January 23rd, 2006
The Technology Intelligence Gap
You can almost hear the techies scrambling back to their drawing boards after an article in today’s New York Times shined a popular spotlight on problems with online recommendation systems. Of course, typical of a Times technology piece, they offer up extreme examples that make it seem like the whole system is broken:
But spewing out recommendations is not entirely without risk. Earlier this month, Walmart.com issued a public apology and took down its entire cross-selling recommendation system when customers who looked at a boxed set of movies that included “Martin Luther King: I Have a Dream” and “Unforgivable Blackness: The Rise and Fall of Jack Johnson” were told they might also appreciate a “Planet of the Apes” DVD collection, as well as “Ace Ventura: Pet Detective” and other irrelevant titles.
Spotlighting the quirks of technology is reminiscent of the “My Tivo Thinks I’m Gay” stories that circulated last year.
To their credit, the Times did point out efforts to close the technology intelligence gap, such as Liveplasma.com (which is experiencing its own technology gap from all the attention — I couldn’t get it to load).
Reactions to this article will probably divide along the usual fault lines, with some arguing that technology will yet find a way to close the gap, and others arguing that there’s no substitute for human intelligence, especially when it comes to summer reading lists.
Sitting on the fence as usual, I’ll ask this question:
Is there a way that humans and technology can better collaborate to help consumers navigate a dizzying world of choice?
Imagine a team of avid reviewers (movies, music, books, etc.) working through an application that scours Amazon, iTunes and the like for interesting cross-recommendations that they might never have considered — these reviewers then apply their own human intelligence to create killer “product bundles” that act as a one-stop shop. These product “remixers” would have every advantage of technology (e.g. the ability to process enormous quantities of data) and every advantage of human intelligence (e.g. the ability to override results that make no sense).
I still trust people more than technology (although some days I’m not sure why), but I appreciate the ability of technology to overcome human limitations.
Why can’t we have the best of both worlds?
Powered by 
Design by 
“Why can’t we have the best of both worlds?”
Scott, I don’t think that’s the question. Of course we can - someday. Sites like Digg and Reedit show some promise, but are far from ideal. People are still learning and tools are still evolving. So I think the question would be better asked - when?
Karl, I don’t see why we can’t pursue the best of both world NOW (the product remix example I gave may not be the best idea, but it’s one we could do right now).
And I disagree that Digg and Reddit are examples of the best of both worlds. They rely almost entirely on human intelligence, with a technology enabler — and they display all the randomness of unfocused human intelligence.
A mashup of Digg and memeorandum — now that would be interesting.
That’s my point
I didn’t mean to imply that Digg and Reedit are the best of both worlds - but they do show a major part of it.
I’m a terrible writer.
To be accurate about a service I would personally like to see - and one that would meet you are looking for (I think): A Memeorandum (algorithmically filtered) + Digg/Newsvine (user filtered) + Slashdot (editor filtered) mashup.
The thing about this is I am positively sure people are already pursuing this now - the fact that we can see the pieces - is what leads me to believe its just a matter of time.
@Karl: A Memeorandum (algorithmically filtered) + Digg/Newsvine (user filtered) + Slashdot (editor filtered) mashup.
But which way would the articles flow? From algo to userbase to editors? The reverse? Userbase at an end point, either end?
David, the simplest form of this, true mashup style, is a page of three lists - “these are the stories the web says are important” (algorithm), “these are the stories your community says are important” (user filtered), and “these are the stories the hosts think are important” (editor filtered).
However, I think the flow you’ve mentioned - algorithm to community to editor has promise - and is in fact - something I want to personally pursue. The simplest form of algorithm can be a hand picked list of trusted feeds that fit a particular subject matter. In fact, that’s what we have already at Philly Future.
Another place to look for where ideas like this are germinating is where Dave Winer is going with his reading list/OPML work. He’s talking about obscure plumbing - sure - but I expect ideas to be bootstrapped on that plumbing that will provide some glimmers soon: http://www.reallysimplesyndication.com/2005/10/13
Recommender systems will evolve into a blending of high-tech and “high-touch”. High-touch recommender systems are social networks.
Bridging these types of systems will create much more effective recommendations.
@Karl: Digg, Slashdot, del.icio.us/popular (via diggdot.us) and tech.memeorandum, all in one RSS feed:
http://feedblendr.com/rss/70
Can’t do anything with Newsvine until it’s out of private beta, I think. But Feedblendr can put those sources together into one feed, sorted by time.
http://feedblendr.com/tips/
An interesting approach to the feed management issue, for those who want a feed of multiple sources organized by time, like email. YMMV
…and naturally as soon as I posted that above it quit working properly. Grrr.
Karl, I think there’s huge promise in a technology-enhanced human editing function, which is where I was trying to go in the original post. I think that’s one way that all of the editorial talent being laid off from Old Media can find a new outlet. And thanks for pointing out Dave Winer’s site — I hope someone hurries up and rolls out these reading lists and other cool apps before too many people tune out RSS. (Also loved his Browse. Search. Subscribe.)
David, thanks for pointing out feedblendr.com — only problem is there’s not enough hours in the day to experiment with all these new apps.
[…] Just came across this NYTimes article, “Like This? You’ll Hate That. (Not All Web Recommendations Are Welcome.),” courtesy of Scott Karp. […]