October 4th, 2007
Disincentives For Gaming The TechMeme Leaderboard
Every ranking system is at risk for gaming, and the TechMeme Leaderboard is no different, as Dave Winer points out regarding today’s pile on to Jason Calacanis Web 3.0 meme. (Ironically, Jason himself anticipated the problem when the Leaderboard debuted.) This was a problem before the Leaderboard, but providing a lasting reward to this behavior certainly runs the risk of exacerbating it.
So here are some suggestions for Gabe to callibrate the TechMeme Leaderboard so that there is less incentive to game it. It’s possible that some or all of these are already included in the algorithm, but I’m throwing them out in public to suggest that the right disincentives can reduce the game of a ranking system like the Leaderboard.
- More weight to the source of the first headline, and less to subsequent subsidiary headlines
- More weight to items published earlier, i.e. less incentive to add redundant contributions late in the game
- Less weight to sources that do nothing but latch on to existing headlines and never produce their own original headlines
- Less weight to all participants in an attention grabbing pile on — objective markers might include a lot of convoluted cross-linking
There’s an opportunity for the TechMeme Leaderboard to be an interesting experiment in using game theory to optimize a dynamic, networked ranking system — Gabe has always been very savvy in revealing some of the “rules” of TechMeme while keeping others hidden.
Eating my own dog food, of course, this post should contribute little or nothing to my position on the TechMeme Leaderboard.
Topics: Aggregation | Blogs
Powered by 
Design by 
I disagree completely. Techmeme and the leaderboard are doing exactly what they’re supposed to do. Jason made a silly post, but tons of people responded. Techmeme doesn’t change people’s behavior, it just mirrors it. If people want Techmeme to change, they should change their own behavior.
Winer is just jealous, it seethes through in every post he wrote. Go back and look at his history and how proud he was to be atop the Technorati list and how annoyed as he started moving down..
I think what a lot of people are missing out on is the massive exposure that TM has been getting from the power of creating a list.
Winer says it’s crap. Well, guess what, he might be right, but I know this… I’m going to bet that the massive amounts of people who now know and visit TM will greatly outweigh the few who disparage the list.
Marketing 101.
gabe listens at least. some good points. dave is a whiner though, because those yutz;s were always gaming it
[…] though Scott Karp tries to offer some serious suggestions to the Techmeme team to help combat this totally expected bastardization of well meaning intentions […]
Why don’t you look at the Leaderboard? I’m ahead of both Calacanis (who isn’t even *on* the list) and Scoble and Karp. I’m supposedly complaining? About what? I’m winning! Geez Louise talk about idiots.
Scott, you have correctly identified the problems in Techmeme. These are the reasons why I thought that Techmeme is going away from the long tail side. On a few occasions, I have written a post first on a particular topic with more personal insights. My article (not on my blog) will then be playing second fiddle to Techcrunch or RWW or some other big player. If they seriously believe that blogosphere is more democratic than traditional media, they need to change the attitude.
Don’t you think he’s tweaking his algorithm this entire time? You know his algorithm is accounting for things more subtle than what you listed there. And no doubt the algorithm is his secret, an advantage of not selling his technology. And because his tool is focused in purpose (unlike Google’s algorithm) he has the power, which no doubt he exercises, to reduce the effects particular abusers may have sensed and exploited. In other words, gaming won’t be a problem.
[…] any link juice. I guess that’s pettiness 2.0. And Scott Karp has some tips for Gabe on how to prevent people from “gaming” Techmeme. But Scott, that would ruin all our fun Tags: Calacanis, […]
Here is my movers and shakers list:
http://www.centernetworks.com/techmeme-leaderboard-movers-and-shakers
The bias on TM is the issue I think will need addressing over time.
On my post I ask an interesting question about blog rankings.
Scott, I don’t think your suggestions would work because the headlines on TechMeme are dynamic. For instance, a story could be the primary headline at one point, but later on the in day, it could be replaced by another headline and subsequently become a child headline of the newer one. So, your suggestions based on whether a story is a primary or secondary headline would not be easy to apply.
The really interesting stuff is off the edge of the leaderboard. Partially that’s because of the gaming, and partially that’s because the leaderboard is all about news aggregators.
More on my blog:
http://smoothspan.wordpress.com/2007/10/01/the-internet-first-breeds-diversity-then-conformity-punctuated-equilibrium/
Less weight to sources that do nothing but latch on to existing headlines and never produce their own original headlines
Unless I’m mistaken many of what you are calling “latch on” blogs *cannot* make a headline regardless of how much better the post is than, for example, Dave Winer’s recent idiot rant. I think upgrading, not downgrading these sites would help bring more balance to the TechMemeMix.
How would weighting the headline stop the gaming of the system? Right now, all someone with any visibility has to do is declare something ‘dead’ and BOOM, it’s on the top of Techmeme.
I’d say weighting posts which add something to the discussion around the original post makes more sense than weighting the headline more than it already is.
The next step will be the creation of a “techmeme awards” event, with nice statuettes. Anytime you try to rank things, it gets messy and ethics become fuzzy. Whether you are calling something the “film of the year” or saying this newspaper is “better” than that one. Amazon’s ranking system has been notoriously bizarre, and a small industry has cropped up to game it for marketing benefit. Bottom line: its a fun list, just don’t take it too seriously.