April 15th, 2007

Cumulative Advantage Explains Web 2.0, MySpace, The A-List, TechCrunch, Digg, And So Much More

by

Just when you thought you understood Web 2.0, along comes a theory so disruptive it razes everything in its path. The theory of cumulative advantage suggests that every successful Web 2.0 site — and the output of every Web 2.0 platform — is completely arbitrary and random. The head-exploding NYT piece by Columbia professor Duncan Watts is a must read, but here’s the key:

In our artificial market, therefore, social influence played as large a role in determining the market share of successful songs as differences in quality. It’s a simple result to state, but it has a surprisingly deep consequence. Because the long-run success of a song depends so sensitively on the decisions of a few early-arriving individuals, whose choices are subsequently amplified and eventually locked in by the cumulative-advantage process, and because the particular individuals who play this important role are chosen randomly and may make different decisions from one moment to the next, the resulting unpredictably is inherent to the nature of the market.

Oh, how sweet the irony — Web 2.0’s radical openness and transparency, combined with its intensely social nature, are precisely why it brings you the best of nothing.

Here is the most important line in the entire piece, which casts extreme doubt on every notion of democratized collective intelligence — “because the particular individuals who play this important role are chosen randomly and may make different decisions from one moment to the next”

In an open Web 2.0 system, with a randomly chosen group, it’s impossible to generate anything other than arbitrary results. Web 2.0 glorifies the “social,” but in an open system, social behavior becomes “monkey see, monkey do.”

All of a sudden it’s crystal clear what Web 2.0 really is — the greatest platform ever for harnessing randomly imitative social behavior. Before Web 2.0, achieving utterly arbitrary results took time and effort. Now, with platforms like Digg, we can get nowhere in a fraction of the time it used to take.

WOW — I am humbled and awestruck by the power of technology, and the power of randomly socialized human beings to snuff out each others’ critical faculties and personal tastes.

So does that mean Web 2.0 really is DOA? So long as it’s driven by the ideology of egalitarianism and radically transparency, probably so. But I suspect the pendulum is about to start swinging back from that extreme, so maybe there’s still hope.

I realize that I’m painting with very broad brush strokes, and that the fun I’m having with this theory comes unfairly at the expense of some of the myriad notions and sites that fit under the capacious Web 2.0 rubric, but it’s not everyday that you find scientific proof that a concept like Web 2.0 actually deconstructs itself.

As for Web 2.0 winners, is MySpace better than Orkut? Are A-List bloggers smarter or more interesting than Z-List bloggers? Is TechCrunch better than Mashable, ReadWriteWeb, and GigaOm? Are stories on the front page of Digg better than a bunch of random junk?

If you believe the theory of cumulative advantage, the answer is — NO!

Of course, this explains why Google has been the only Web 2.0 company to make a gazillion dollars harnessing human behavior, i.e. linking patterns — PageRank ensures that the link “votes” Google uses to determine search results aren’t from a “randomly chosen group,” and…it doesn’t show you the score.

Comments (36 Responses so far)

  1. Cumulative Advantage Explains Web 2.0, MySpace, The A-List, TechCrunch, Digg, And So Much More » Publishing 2.0

  2. post

  3. wisdom of crowds, collective intelligence, and the network effect that are at the core of the leading approach to succesful web sites and systems. Scott Karp – whose sanity amid the Web2.0 hype I always appreciate has really dropped the hammer: All of a sudden it’s crystal clear what Web 2.0 really is — the greatest platform ever for harnessing randomly imitative social behavior. Before Web 2.0, achieving utterly arbitrary results took time and effort. Now, with platforms like Digg, we can While on one hand, it’s nice to see something we know intuitively – that things already declared popular receive more attention than items without that distinction – proved to be true. On the other, the experiment as performed dramatically

  4. This article about Cumulative Advantage in The New York Times Magazine seems to be sparking quite the fervor on the web. Authors are now convinced that website popularity and the structures of Web 2.0 are not based on quality of the content, but on popularity and randomness. Writers ask, are website such as Digg, (One that I honestly find useless the majority of the time.) MySpace and Blogs just part of a popularity explosion? Is connectivity

  5. Cumulative Advantage Explains Web 2.0, MySpace, The A-List, TechCrunch, Digg, And So Much More  - Apr 16, 2007  - Scott Karp

  6. if someone is popular – for whatever reason, be it real talent or just blind luck – he or she is likely to become even more popular, since people tend to gravitate towards things that are already perceived as being popular”. And as Scott Karp points out, this means that MySpace isn’t the most popular social network because it is in some way inherently the best – it is the most popular because a handful of decisions by the earliest adopters were randomly amplified by subsequent adopters.

  7. and the same overall market tastes would quickly generate different winners: Madonna would have been popular in this world, but in some other version of history, she would be a nobody, and someone we have never heard of would be in her place. Scott Karp is pretty jazzed about this new theory. Oh, how sweet the irony — Web 2.0’s radical openness and transparency, combined with its intensely social nature, are precisely why it brings you the best of nothing. In an open Web 2.0 system, with a

  8. Cumulative Advantage Explains Web 2.0, MySpace, The A-List, TechCrunch, Digg, And So Much More » Publishing 2.0

  9. Publishing 2.0

  10. One other odd example I came across last year is the smart shopping trolley which alerts you to what others are buying in the category – damn, can’t find the link today. Does anyone have one?). Johnnie, meanwhile makes a great point in criticising Scott Karb’s discussion. All too often we see the copying of other people’s behaviour as “dumb” (as opposed to thinking it all through for our selves, like all good individual-decision-making-units should do..???)

  11. A Note of Caution Finally, a note of caution. Over at Publishing 2.0 Scott Karp extrapolates this finding further, suggesting (somewhat tongue-in-cheek) that it explains other phenomena like Web 2.0 and the blogging A-List. He wonders if the A-List is just riding a wave of initial popularity. We should be careful to push this that far, however, because some

  12. (programming, humor, tips, coding) EXCELLENT interpretation of “all I needed to know I learned in kindergarten” through the eyes of a programmer. Must read, hell must print and put beside the coffee machine. [WEB2.0] Cumulative Advantage Explains Web 2.0, MySpace, The A-List, TechCrunch, Digg, And So Much More (publishing2.com, 14 saves) (attention, social, socialsoftware, digg, myspace, web2.0) From the article: ” All of a sudden it’s crystal clear what Web 2.0 really is — the greatest platform ever for harnessing randomly imitative social behavior.

  13. Want to make an Internet TV channel? Tech CEO’s have bigger… umm… salaries. Novices can beat experts. Got a great idea and no cash? Top 10 Venture Capital Lies! Fotowoosh looks pretty damn cool. Cumulative Advantage explains 2.0. Thoughtful post. The Ben Franklin guide to marketing online. Five VC videos from askthevc.com and coloradostartups.com. Intel’s new mobile pc’s will run linux!! Google’s 411 service will train the data center how to interpret

  14. “Oh, how sweet the irony — Web 2.0’s radical openness and transparency, combined with its intensely social nature, are precisely why it brings you the best of nothing.”

  15. and the same overall market tastes would quickly generate different winners: Madonna would have been popular in this world, but in some other version of history, she would be a nobody, and someone we have never heard of would be in her place. Scott Karp is pretty jazzed about this new theory. Oh, how sweet the irony — Web 2.0’s radical openness and transparency, combined with its intensely social nature, are precisely why it brings you the best of nothing.

  16. Hey,

    I wrote about this with regard to Digg, but yes, it does broadly apply across all web media:

    http://mashable.com/2007/04/15/is-digg-the-result-of-cumulative-advantage/

    We used to call this “network effects”, of course. Either way, it’s pretty clear that cumulative advantage would be more noticeable on the web, where people are making decisions based largely on what others think. Top 10 lists, technorati top 100 etc all increase cumulative effects. You’d think we would have realized this by now and figured out that what people want is not popularity, but personalization.

  17. “what people want is not popularity, but personalization.”

    No, I think people WANT to know what’s popular.

    We watch popular TV shows because it gives us something to talk about around the water cooler the next day. If we don’t watch the show, we are out of the conversation.

    We like to know what’s popular on the web so we can talk about it minutes later over IM.

  18. Pete,

    The problem with popularity vs. pesonalization is that, as Watts observes, we are intensely social creatures and thus personalization is actually a function of popularity — our personal tastes are naturally driven by what other people think is popular.

    BUT, that doesn’t mean we don’t have a sense of personal taste — what happens is that our tastes are both driven by AND, in a sense, overrided by what’s popular. We watch popular TV shows to be part of the conversation — to Hashim’s point — BUT we do so even if we actually don’t like those shows.

  19. Um, ok…

    “the long-run success of a song depends so sensitively on the decisions of a few early-arriving individuals, whose choices are subsequently amplified and eventually locked in by the cumulative-advantage process”

    This is more true or less true, depending on the type of network.

    In networks that confer greater advantage to the most advantaged, this is more true. But in networks that limit the advantage of the most advantaged, this is less true.

    By reducing the amplification any given individuals receive, we increase the probability that the message, whatever it is, will be filtered by people who are actually qualified to do so.

    When the rich are allowed to get richer, through no merit of their own, but merely because they are rich, then they will spend money foolishly and wastefully. But when this same money is spread over a larger number of people, it will be spent more carefully and more productively.

  20. So may be Wikipedia works so well because any random person can edit any article (not quite so much any more) they want?

  21. “the long-run success of a song depends so sensitively on the decisions of a few early-arriving individuals, whose choices are subsequently amplified and eventually locked in by the cumulative-advantage process”

    This is often the case with anything that succeeds based on social approval/voting. It is human nature to want to “be part of something,” and all too often I have observed people making foolish choices simply because they didn’t want to be perceived as outside the norm.

    By reducing the amplification any given individuals receive, we increase the probability that the message, whatever it is, will be filtered by people who are actually qualified to do so.

    That does sound good in theory. However, if for example, we reduce the amplification an individual receives simply because he may benefit from the outcome, we risk shooting ourselves in the ‘collectivist foot’ by losing the possible benefit of whatever he may be advocating. All because we rushed to suppress his influence on the group in order to prevent his gaining a financial benefit. The opportunity cost may be more than we think.

    When the rich are allowed to get richer, through no merit of their own, but merely because they are rich, then they will spend money foolishly and wastefully. But when this same money is spread over a larger number of people, it will be spent more carefully and more productively.

    One might be tempted to buy into such a generalized indictment of the wealthy simply because it appeals to a misguided sense of social justice.

    But the social reality tells a different tale. Typical high income/high net worth individuals share common traits like wise spending and investment habits.

    It is typically the less-productive members of society who make foolish spending and investment decisions, as well as lifestyle choices.

    So in the end…are some people more qualified to “vote” on the success of an idea than others? YES

    However, when it comes to pop music/art/fashion and other less important ideas, it just doesn’t matter.

    Tune your radio to the local top 40 station. The fact that unqualified early adopters have an influence on the success of a song has been painfully obvious for decades.

    Thankfully the market has a way of vetting things that truly matter.

  22. [...] a note of caution. Over at Publishing 2.0 Scott Karp extrapolates this finding further, suggesting (somewhat tongue-in-cheek) that it explains other phenomena like Web 2.0 and the [...]

  23. By itself, all this experiment shows me is that a system ordered largely by unbalanced positive feedback will continually amplify that feedback. I learned this in 7th grade with my mock fender and gorilla 12 watt amp. So I’m not so quick to dismiss network feedback as an extremely valuable tool in a decision maker’s toolchest.

    While on one hand, it’s nice to see something we know intuitively – that things already declared popular receive more attention than items without that distinction – on the other, the experiment as performed dramatically over-simplifies the situation for one very big reason: positive feedback loops.

    The most successful emergent systems, those where valuable order arises by individuals acting autonomously, are successful through a balance of positive and negative feedback. Everything from thermostats to a human’s sense of balance to ants’ decisions on the best place to store their waste. Can you tell I just read Stevn Johnson’s Emergence?

    How dramatically would the results change if you simply gave users the ability to vote songs both up and down? Hard to say, but I would have to guess it would make a difference.

    Creating a system that produces quality from simple rules is amazingly complex – netflix is offering a cool $1M for one that improves their movie recommendations. I wouldn’t dismiss the crowd’s wisdom from an experiment that doesn’t allow the wisdom to emerge.

  24. Another thought on your conclusion:

    PageRank ensures that the link “votes” Google uses to determine search results aren’t from a “randomly chosen group,” and…it doesn’t show you the score.

    Your right – Google’s algorythym is very different from the experiment, but not for the reason you state – that it doesn’t show users the score and its seeds aren’t random. And everytime someone searches, the score is right their on the page: results 1 – 10000000. The top search results get more exposure, more links, and longevity at the top – cummulative advantage in effect. And ask someone that has had a site running on the same domain since 96 if their google page-rank hasn’t benefitted from their random early arrival to the page-ranking party.

    Google is successful in many ways because it harnessed the positive feedback of links in, and balanced with all sorts of other network information feedback – a link from a site with a page rank of 1 is of less value than a 2, and links from virtual hosts on the same IP are downgraded to discourage gaming – the algorythym is their very complex secret sauce.

    So my bottom line is that cummulative advantage exists – how else can you explain Sanjaya? – but the makers of effective software and systems know this and the look for rules that amplify the right signals and soften the right signals.

  25. There’s a pretty big aspect to this that you’re missing – “Cumulative Advantage” relies on knowing how other people or voting. “Network Effects” rely on each person acting individually, not knowing how other people are voting.

    I’ve often criticized Digg for this reason – the site does everything it can to get you to vote for things that are already popular. You see the number of Diggs next to every article, and you can even sort by that value.

    But other sites deal with it better. Google’s magic is based on network effects where each inbound link is a vote, as every person who creates a link is doing so pretty much independently and unaware of what anyone else is linking to. Flickr’s interestingness is opaque enough that it doesn’t seem to be subjected to cumulative advantage. The blogosphere is a bit of a gray area where we can see both at work.

    Cumulative advantage, incidentally, is also why publishing political polls ought to be illegal in a campaign season.

  26. Great post! An interesting perspective to consider…

  27. [...] Cumulative Advantage Explains Web 2.0, MySpace, The A-List, TechCrunch, Digg, And So Much More The theory of cumulative advantage suggests that every successful Web 2.0 site — and the output of every Web 2.0 platform — is completely arbitrary and random. (tags: Web2.0) [...]

  28. celeb endorsement and star-power. It explains why Myspace dethroned Friendster –even with its so-called first-mover advantage.

  29. [...] Cumulative Advantage Explains Web 2.0, MySpace, The A-List, TechCrunch, Digg, And So Much More » Pu… (tags: social+media reading) [...]

  30. [...] you can repeatedly roll sixes no matter how carefully you try to throw the die.I was intrigued by Scott Karp’s interpretation of this:All of a sudden it’s crystal clear what Web 2.0 really is — the greatest platform ever [...]

  31. Everything depends on the ratio of songs to things that will keep the random rich from getting richer:

    1. Build in a reputation system (like Slashdot) so that people with good taste have greater influence.

    2. Make sure users have to sample a variety of songs.

    3. Implement a tournament ranking system where users rate 5 random songs against each other at a time. The best 2 would go on to the second round.

    These changes would dramatically increase the chances that the crowd recognizes both genuine hits and lesser-known gems.

  32. (Some of my previous comment was cut off. Here it is in full)

    Everything depends on the ratio of songs to users and users per group. Numbers matter because if there are too many songs per user, they will try only a small sample songs and so less likely find the objectively good songs.

    Fortunately, the right architecture can keep the random rich from getting richer:

    1. Build in a reputation system (like Slashdot) so that people with good taste have greater influence.

    2. Make sure users must sample a variety of songs.

    3. Implement a tournament ranking system where users rate 5 random songs against each other at a time. The best 2 would go on to the second round.

    Such changes would dramatically increase the chances that the crowd recognizes both genuine hits and lesser-known gems.

  33. [...] to another article which examines the Columbia experiment, and comes to a more cycnical conclusion. Scott Karp on Publishing 2.0 writes:All of a sudden it’s crystal clear what Web 2.0 really is — the greatest platform ever [...]

  34. [...] a provocative article in Fast Company about a new disruptive Duncan Watts theory. After last year debunking the “wisdom of the crowds” using the theory of cumulative advantage, Watts is back, this time debunking the idea that there is a class of “influentials” [...]

  35. I don’t think Duncan Watts’s idea really discredits THE TIPPING POINT or THE IDEA VIRUS at all. Here’s why:

    His modeling of the less than dramatic impact of influentials on the spread of a contagion through a population was convincing–BUT–I don’t think his model really applies all that well to the web…

    Because in his model, *every* member of the herd had a *connection* to those around them through which they could infect(influentials simply had a higher probability to infect via these connections). But influentials on the web are different in that their “connections” are to DISCRETE populations–populations whose respective members don’t necessarily have connections to the other population’s members.

    Watts’s model describes the impact of promiscuous sheep on the spread of something within a specific herd. But THE IDEA VIRUS and THE TIPPING POINT describe the impact of a shepherd with access to multiple herds.

  36. [...] a provocative article in Fast Company about a new disruptive Duncan Watts theory. After last year debunking the “wisdom of the crowds” using the theory of cumulative advantage, Watts is back, this time debunking the idea that there is a class of “influentials” who is [...]

Add Your Comment

Subscribe

Receive new posts by email