February 18th, 2006

Idea Filter


Thinking about the attention and information overload problem, I realized that what I want is an idea filter. Richard MacManus seconded the notion (and pointed out that, not surprisingly, he was there long before me.)

I thought it would be helpful to demonstrate what I mean by an idea filter. I read a lot of bloggers, but only a handful consistently have ideas — it’s taken a while for me to figure out who they are. I’ve written before about Umair Haque, who is an idea machine. For now, I’d like to point to three other examples of bloggers with ideas and insight:

Alex Barnett on why real meme-trackers have not yet been invented:

The word ‘memetracking’ or (‘meme tracker’) has been used to describe services such as Memeorandum, Megit, Tailrank and Chuquet.

I can’t call them ‘memetrackers’. I like them, they’re useful sites and all that, but they aren’t ‘meme’ trackers.

A meme, as defined by Richard Dawkins in his 1976 book The Selfish Gene, is

“a unit of cultural transmission, or a unit of imitation.

…Examples of memes are tunes, ideas, catch-phrases, clothes fashions, ways of making pots or of building arches.

… If a scientist hears, or reads about, a good idea, he passed it on to his colleagues and students. He mentions it in his articles and his lectures. If the idea catches on, it can be said to propagate itself, spreading from brain to brain.

(There are other definitions floating around but the originator’s will do to make my point.)

There is imitation going on at these these sites: ‘oh, she’s reporting this news, it’s interesting, so I will’. These sites track the act of passing on units of news. In this sense these sites are tracking the memetic quality of blogs. However, generally speaking, these sites are not tracking the spreading of ‘ideas’ or ‘memes’. They are tracking bits of news being passed on from one blogger or site to another.

A bit of news is not a meme, nor an idea, it is a bit of news.

Noah Brier on how digital technology transformed the economics of attention:

Because for the first time attention is a measurable commodity. Before the internet there was no good way to measure the attention people paid to things. Of course there were some general measures, but beyond paying the entry fee to a movie or the cover price for a magazine, the whole measurement thing was pretty fuzzy. Nielsen tried to do it for years and their numbers have been exposed on multiple occasions. Then all of a sudden digital technology comes along and with it, all the wonderful recordability (my word) that comes along with it. Suddenly there’s a shift from a world where you struggle to measure where attention is being paid to one where you’re buried in data. Cell phone records, email inboxes and internet cookies all contain the pieces that can eventually make up the whole.

When we look back on the early internet, we might very well say that the biggest shift it brought on was forcing the world to rethink advertising. After all, why buy a spot in a magazine where you’ll hope that people will pay attention to your ad on page 63 when you can buy an advertisement on a topic-specific website that guarantees 10,000 people who have made a conscious decision to visit the site will see your ad each day. Or further, why pay for that ad at all if it doesn’t get clicked on? All of a sudden, the happy magazine publishers and television network producers are sweating about the fact that they can’t guarantee people are going to pay attention. It’s almost as if someone pulled away the curtain and revealed the big secret: When you buy an advertisement in traditional media all you can do is hope that people pay attention. When you add in the fact that people are increasingly fragmenting their attention amongst multiple media at once, you’ve got companies like NBC and CBS trying to convince advertisers that people really do care (and this isn’t even to mention the disruptive technologies like Tivo and BitTorrent).

Nick Carr on the Nature article about Wikipedia’s accuracy:

If you were to state the conclusion of the Nature survey accurately, then, the most you could say is something like this: “If you only look at scientific topics, if you ignore the structure and clarity of the writing, and if you treat all inaccuracies as equivalent, then you would still find that Wikipedia has about 32% more errors and omissions than Encyclopedia Britannica.” That’s hardly a ringing endorsement.

The problem with those who would like to use “open source” as a metaphor, stretching it to cover the production of encyclopedias, media, and other sorts of information, is that they tend to focus solely on the “community” aspect of the open-source-software model. They ignore the fact that above the programmer community is a carefully structured hierarchy, a group of talented individuals who play a critical oversight role in filtering the contributions of the community and ensuring the quality of the resulting code. Someone is in charge, and experts do count.

The open source model is not a democratic model. It is the combination of community and hierarchy that makes it work. Community without hierarchy means mediocrity.

Before you respond to my choices, think about this — I chose Alex, Noah, and Nick because they are doing more than echoing — they are doing original thinking with a high degree of analytic discipline. You need not agree with their conclusions to appreciate the quality of their thinking.

I want an idea filter that can direct me to original thinking that will stretch my mind and challenge my own thinking. As I said before, I think such a filter is only possible through the combined power of technology and human intelligence.

More on the idea filter to come — it may be an idea, but it still needs a lot of work.

Comments (16 Responses so far)

  1. The Times discovers sex The cartoon about the cartoons Idea Filter Snobs.com Another Hotel Broadband Trick Don’t read and drive Apple Abusing Copyright Law? Time to blow up blogs How Police States Emerge Class Big Surprise: Cheney Accounts Don’t Add Up Google and

  2. attention.xml and the OPML file….But I’m paying attention.” Indeed Job Trends: sql server, oracle, db2, mysql jobs in sql server, oracle, db2, mysql Idea Filter “I want an idea filter that can direct me to original thinking that will stretch my mind and challenge my own thinking. As I

  3. action of these two tendencies is effectively the dominant hallucination that we call consciousness. I find myself in a kind of hyper mode these days. There is a lot of stimulating writing on filtering and context. Scott Karp does an great job

  4. of the results. … Creativity can endure only so much private control before it careens into a downward spiral of sterile involution.” Idea Filter found: 06:38pm February 19, 2006 Scott Karp: “Thinking about the attention and information overload problem, I realized that what I want is

  5. have a chance of getting this right with our own software!) Alex Barnett blog – The real ‘meme’ trackers are yet to be invented Publishing 2.0 – Idea Filter You’ll notice that I happily post old blog posts – but that’s because I’m not all that interested in New! as much as I am

  6. I want an idea filter that can direct me to original thinking that will stretch my mind and challenge my own thinking.

    I’m going to go out on a limb and call this a contradiction in terms – it seems to me that having your thinking challenged means stepping outside the bounds of your known patterns (the things an attention filter might be sensitive to), colliding with something new, and responding by learning.Serendipity has existed for a long time in online systems (e.g. Wikipedia’s random article feature), but simple unexpectedness is not enough – some interaction with your own prejudices or experiences is required, and therefore the only truly capable filters are going to be human. Your friends, people concerned about your recent behavior, your subconscious mind during periods of crisis.

  7. Michal, I’m not looking for randomness for randomness sake. I’m looking for ideas and insight, which may come from unexpected places, but they are not defined by “randomness.” By subscribing to the feeds of Alex Barnett, Noah Brier, and Nick Carr, I am likely to find ideas, but everything they post does not rise to that level.

    You are right that what I want absolutely requires a human intelligence — I am highly skeptical that technology alone can provide a truly effective filter. That said, I with the immense volume of content now being produced, I think technology is essential for a “first cut.” What I envision is a technology-driven filter that spits out 5,000 (or even 10,000) items a day, and a human editorial team that sends me the 50 items that are worth my time.

    This is the true convergence of media and technology. Old Media still thinks it can be all human, and Web 2.0 advocates think it can be all technology. I’m looking for middle ground.

  8. There’s probably a germ of an interesting idea here. You’re essentially looking for mavens, right? Someone (or something) that knows your tastes & predilections well enough to recommend material you might be interested in, or material you ought to be interested in, for your own good. I’m not just skeptical that technology can do this–I’m rejecting the idea. =o)

    People aren’t that expensive, though I realize it’s disastrously out of fashion to depend on an unscalable resource.

    Amazon has had their Mechanical Turk service going for some time, and I don’t believe anyone has experimented with its use as a taste filter. The other option is to follow in the footsteps of the journalist whose name I can’t remember, and hire a remote executive assistant from a country where the education is good but cost of living is low. A smart, capable south Asian or eastern European assistant who grows to know you without breaking the bank can’t be too hard to find. They might also know enough to not bother you with 50 things a day, and instead concentrate on one or two really beard-pulling ideas.

  9. Michal, here’s what makes it scalable — this is not a recipe for personalized media, so it doesn’t require a personal assistant. I’m sure my taste in “ideas” is not unique — an idea filter could be syndicated — that’s the opportunity for media brands, old and new.

    You are right about one thing — I would be willing to pay for it.

  10. The other option is to follow in the footsteps of the journalist whose name I can’t remember, and hire a remote executive assistant

    That’d be A.J. Jacobs as detailed here: http://www.smartmoney.com/esquire/index.cfm?Story=20050909-outsource&pgnum=1

  11. I would like to start with the question why the original authors of the blogs are not your filters – they know their work, they know when they have something really new so perhaps they should write a blog entry only then?

    Answer no. 1 – they can find readers that like each of their blog entries, even if none of them likes them all, they really need a mechanism to direct them at discreete audiences, I believe the solution here is a PublishSubscribe protocol

    Answer no. 2 – they don’t want to do that – because they need to keep some traffic at their side to keep the readers wake up. This is valid reason – but one that should be destroyed by aggregation (see Aggregation and Transaction Costs in PublishSubscribe).

    Answer no. 3 – the author not always can judge his own writings, so we really need some external filtering as well

    The solution to the above that I propose is http://zby.aster.net.pl/kwiki/index.cgi?SocialRouting – human thinking augmented by automatic Publish Subscribe system. What is important there is that it is not only that the humans choose the good articles and then those individual opinions are aggregated – the individuals define their own aggregation and filtering mechanism. The system only defines a protocol how to connect those individual cells.

    This is the way to have a personalized and P2P filtering system that would harness the processing power of each individual PC.

  12. Zbigniew – on the second page you link to (Social Routing), you say that “every node should have an independent routing algorithm – so that every user could choose his own rules for routing to express his own personal trust.” A working model of this could be less an hour’s work coding a plug-in to Reblog, assuming that the rules are well defined. What kinds of rules do you have in mind?

    The pub/sub infrastructure that’s currently in place with blogs & RSS readers is close to a perfect information filter – it’s run by people, based on a “push” idea, etc. Nothing is really missing, but there are still a few inefficient paths that expand the gap between “this is cool” and “now all my friends know it”. Reblog is trying to short-circuit one of those pathways.

  13. Idea Filter is a great hook, but I like to think of the underlying act as additive, as “making meaning”: your unique talent is in making these new concepts meaningful for others. Meaning has tremendous value. It’s the nucleus of all brands, not just those in media. It’s also the essence of great film, literature, and art. It’s what we naturally seek from life, but it’s really hard find much meaning in the stew of information on the Internet. You’re right, making meaning out of this stuff is the opportunity. I’d love to talk further about this. Keep up the great work. Your site has become a “must read” for me.

  14. Michal – I skimmed the Reblog info pages, it’s cool, interesting how to this time my brain filtered out info about Reblog, because I thought it is a Marc Canter closed source project (also probably at the time when I wrote it Reblog was not yet created, it’s nearly two years old).

    The system I was imaging was something between a semi-automatic router and a filter. I did not think too much about the technical details, but I was imagining it more as a protocol, abstracting the algorithm as a black box and providing the user with a full freedom to choose any one he wants. Plugin architecture is quite close to this, and tags might be a better protocol for passing information from one Rebloger to another than the multiple output channels that I was imaging originaly. So eventually I really need to check this project.

  15. [...] The problem right now is not that there’s too much content, but that the filters are still too primitive and Old Media is still clinging to control of its content, reducing the “liquidity” of the network. [...]

  16. [...] RSS fatigue is already setting in. While multiple posts get you more traffic, they also make it easy to lose loyal readers."The clutter in personal media makes it difficult for anyone to find anything….even with tagging. Scott Karp at Publishing 2.0 rejects the notion that there is too much or that anyone should show restraint. The solution, for him, lies in powerful metadata:"The problem right now is not that there’s too much content, but that the filters are still too primitive and Old Media is still clinging to control of its content, reducing the “liquidity” of the network. [...]

Add Your Comment


Receive new posts by email