January 19th, 2008
Developing Algorithms To Prevent Citizen Journalism From Being Gamed: Lessons From Google and Digg
Is there a risk that citizen journalism can be gamed by “PR flacks and unqualified hacks” — Adam Weinstein in Mother Jones thinks so.
Unfortunately, he casts the issue in terms of the risk that economically burdened newsrooms will trade expensive quality journalism for no-cost, untrustworthy content — instead of looking at the very real risk that a more open journalistic process can be more easily gamed — and how that gaming can be prevented.
Which got me thinking about what citizen/networked journalism might learn from the battle that systems like Digg (which accepts anonymous users) and Google (which interprets open linking on the web) have waged against gaming.
By forcing their beleaguered staffs to depend on outsiders for content, then running the content without much editorial oversight, newspapers may be taken in by crackpots and sly marketers who make Jayson Blair look like a grade-school plagiarist. Lobbyists and spin doctors have already taken notice of the new model. Take the South Dakota Politics and Daschle V. Thune blogs, which influenced the Gannett-owned Sioux Falls Argus Leader’s coverage of the state’s 2004 United States Senate race; eventually, the bloggers were found to be on the payroll of just-elected Republican Senator John Thune. “Got a story you can’t convince a mainstream reporter to run?” wrote Chris Suellentrop in Slate. “Leak it anonymously to a blog on your payroll. Then get a local reporter to write a story on the controversial, gossipy, local political blog. Soon everyone in town will be talking about the story you leaked to the blog…. And no one will know that the blog post was a paid placement until after the election.”
The risk that political reporting can be manipulated by political operatives acting as bloggers, for example, is very real. But Weinstein seems more interested in impugning the integrity of every newsroom experimenting with “networked journalism.”
If you could convince me that crowdsourcing and mojos and information centers weren’t about cost cutting or lazy journalism, I’d be all for them. The blogosphere and the 24/7 news cycle are realities, and editors and reporters have a lot of ink-stained baggage to dump if they want to thrive in the new-media world. But that doesn’t mean that bean-counting publishers must recruit mercenary bloggers or convert their cub reporters into untrained, overworked, self-editing news tickers.
I attended Jeff Jarvis’ Networked Journalism Summit last fall, and I can tell you that there was no one advocating “cost cutting or lazy journalism,” nor were there any “bean-counting publishers” seeking to destroy the integrity of journalism — which is not to say this can’t or hasn’t already happened in some instances. But it’s a cynical red herring at best, which distracts from the real issue — the potential for gaming of an open networked/citizen journalism system, and how the problem can be addressed.
Of course, it’s also a canard to argue that the traditional journalistic process was ever immune from gaming, whether by savvy flacks on the outside or Jayson Blairs on the inside. But at least in the command-and-control editorial model, there were processes in place designed to prevent, as much as possible, the manipulation of facts, dissemination of disinformation, and other efforts to advance agendas.
That’s not to say that “objectivity” has ever been easy, particularly when the ideal of balanced coverage conflicts with the imperative to establish “facts” — for example (click through to see the video, it’s an eye-opener):
But as if it weren’t previously hard enough, opening up the journalistic process to a network presents a whole new set of challenges, which Weinstein does legitimately highlight, albeit with the unconstructive polemic.
To address the counter-argument, that there isn’t really a risk of gaming in networked journalism, consider that the web has pretty conclusively demonstrated that any open system that can be gamed, will be gamed. (Blog comment spam is another example)
Fortunately, there are a lot of lessons to be learned outside the sphere of traditional journalism, which can guide the new practice of networked journalism, as it learns how to combat the new potential for manipulation — in particular, newsrooms should study the efforts by open networked systems like Digg and Google to stem the endless tide of gaming.
There are tons of articles out there on the topic (just do an ironic search on Google):
- I Bought Votes on Digg
- “Undetectable” spam — this an example of a post by Matt Cutts, head of Google’s anti-spam team — I won’t go so far as to suggest that his blog be required reading for newsrooms working on networked journalism projects, but they ought to peruse it just to get a feel for what it means to combat the manipulation of open systems.
The basic principle that sites like Google and Digg apply to combat gaming is that there are identifiable strategies for manipulating the system, and if you can identify markers of those strategies, then you can design “algorithms” to detect and block manipulative behavior. These algorithms do not always eliminate gaming, but they can reduce it.
So what would be the equivalent of such anti-gaming algorithms in networked journalism? Journalists have always had the responsibility of vetting the sources for their stories — so now they will have to take on the responsibility of vetting “sources” who are actually writing some of the content.
Take Weinstein’s example of the Tallahassee blogger writing about urban development who previously ran a PR campaign for Wal-Mart — Weinstein complains that the Democrat, which published her blog posts, didn’t disclose this. If it’s because they didn’t know, that seems like a pretty correctable problem.
Algorithm: Does blogger background = PR work for special interests related to blogging topics? If YES, then add disclose and forbid blogging on topics with conflict of interest.
Will this algorithm work perfectly? Of course not. But it’s not like journalists don’t know how to do background checks.
Some algorithms will be points of serious contention — should the “algorithm” block bloggers who contribute to political campaigns? How about those who have worked for a political campaign? What if the blogger discloses all conflicts of interest?
Yeah, well, maybe newsrooms should read Matt Cutts’ blog.
But already there are examples being documented of how the network can manipulate journalists — here’s one from a MediaShift piece on The Benefits and Pitfalls of Using Social Media for Reporting:
Such is the case of Bilawal Bhutto Zardari, son of slain former Pakistani Prime Minister Benazir Bhutto. Back in the old days, after a major event like Bhutto’s assassination, a reporter might jump through hoops trying to score an interview with a member of her family. These days it seems that at least some reporters can’t be bothered, and turned immediately to social networks to see if they could gain insight into his thoughts.
Several big news companies — among them London’s Telegraph and Agence France Presse — lifted quotes about Islam from her son’s Facebook profile. The only problem was that the profile was fake. In this case, traditional media in all its experience didn’t know that social media sources can be a minefield, and it exploded in their faces. Had the “joke” not been discovered sooner or the fake quotes more inflammatory, this could have had serious political implications.
Then there’s the recent President of Facebook incident in France.
But instead of using these instances, as Weinstein does, to support a case for why networked journalism can never work, they should be used to “program” journalism’s manipulation algorithm.
What every newsroom or journalist practicing networked journalism should do is develop a list of all known instances of gaming, and also come up with other possible risks — then they need to develop algorithms to try to protect against these.
So what about the likelihood that some of these algorithms will fail as part of the learning process? The advantage that news organizations have over systems like Google and Digg is that they can actually disclose their algorithms, which Google and Digg can’t do because it would actually give gamers a roadmap.
But news organizations can be transparent. Want to publish content from bloggers who donate to political campaigns? Disclose to readers that you’re doing it, ask the bloggers to disclose the contributions, and leave the rest to readers — they are pretty smart. Given all the right information, they can figure it out for themselves.
Here’s a final piece of advice — don’t look for black and white in this issue, as Weinstein does. Citizen journalism isn’t all good or all bad. The risk of gaming the network can be addressed, although not entirely.
Traditional journalism resisted manipulation, although not always successfully. But in its best moments, it accomplished a lot in the public good. The same can be true of networked journalism.
Powered by 
Design by 
“I attended Jeff Jarvis’ Networked Journalism Summit last fall, and I can tell you that there was no one advocating “cost cutting or lazy journalism,” nor were there any “bean-counting publishers” seeking to destroy the integrity of journalism”
Hardly the kind of place that you’re going to find them, though, is it? But the fact is that a lot of publishers *will* use “citizen journalism” (can’t we call it what it is - amateur journalism?) as an excuse to cut costs.
I’ve never met a publisher who wouldn’t sacrifice 10% quality for a 15% cost saving. The problem is that if you cut the quality of 10% repeatedly, you end up with something with no quality at all.
[...] Karp responding to Adam Weinstein’s claim that newspapers are likely to lose quality because of the [...]
[...] Digg inventou para se proteger dos mal intencionados (o Digg aceita anónimos). É fundamental ler Developing Algorithms To Prevent Citizen Journalism From Being Gamed: Lessons From Google and Digg, de Scott Carp. Fundamental para quem se interessa, claro. Os proprietários e decisores dos nossos [...]
Ian,
“I’ve never met a publisher who wouldn’t sacrifice 10% quality for a 15% cost saving.”
Is that because publishers are evil “bean counters” who don’t care about quality? Or is it because the newspaper business model is collapsing?
The fact is that newspapers do have less money to pay journalists. So we can hand wring and blame it on the bean counters. Or we can figure out new models to enable journalism to evolve through the new economic realities — and maybe find ways to improve the futures of companies that support journalism.
Jeff’s conferences was about figuring out how to make new models work WITHOUT losing quality — doesn’t mean it won’t happen in some instances. But bitterly wishing for the good old days isn’t a very constructive alternative.
“Is that because publishers are evil “bean counters” who don’t care about quality? Or is it because the newspaper business model is collapsing?”
And you have to ask the question: is the newspaper business collapsing because publishers have, since the early 1990’s, been looking for every opportunity to trim the size of their newsrooms?
Does the fact that newspaper reporters have continually been “encourage” to write more and more stories per day not mean that the quality of their output declined - and might that decline in quality not have had something to do with people no longer seeing the value in newspapers?
And this, of course, isn’t just in newspapers: the same is true in magazines. Far too many publishers have seen the switch to digital simply as an opportunity to get more output from less journalists. No matter how you dress it up, more stories per day means less time to work on each story, which encourages “me too” posts and less originality.
Think back to that description of the UK Daily Telegraph news room, where reporters are encouraged to stare at screens all day instead of developing sources and actually talking to people. Does that encourage quality? No - it’s about quantity.
“Jeff’s conferences was about figuring out how to make new models work WITHOUT losing quality — doesn’t mean it won’t happen in some instances.”
Oh, don’t get me wrong. I’m not criticising Jeff’s conferences in any way, and Jeff does some amazing work. Some of it genuinely opens up new ways to do news and disseminate ideas. And he balances things out very well, too.
“But bitterly wishing for the good old days isn’t a very constructive alternative.”
Actually, if quality is being lost then the most constructive thing you can do is point it out. Pretending that the opposite is true, and that you can get the same or better quality of content for less money isn’t being constructive - it’s either naive, or disingenuous.
Ian,
“Pretending that the opposite is true,”
Who is pretending the opposite is true? Painting everything in black and white is why the industry is having so much trouble evolving. This post is all about the risk and trying to look at ways to address it, rather than simply hand-wringing.
“And you have to ask the question: is the newspaper business collapsing because publishers have, since the early 1990’s, been looking for every opportunity to trim the size of their newsrooms?”
It’s notable that you mention the exogenous factor here — that newspaper readership has been declining since the early 1990’s, and that the newspaper’s economic model has been accelerating its decline in the last few years.
Where, exactly, do you believe publishers should find the money to pay journalists to write the same number of stories that they did back in the glory days of the newspaper’s high profit margin?
The risk of declining quality is very real — but so is the rapid decline of newspaper economic fortunes.
Instead of seeing it in black and white, i.e. old model = high quality, and new models = compromised quality, why not focus on how new models can be implemented to maintain quality standards?
It’s easy to argue that the newspaper business is declining strictly because of declining quality. But the reality is that the emergence of digital media and the increasing obsolescence of the newspaper distribution model has do far more to accelerate the decline.
“Where, exactly, do you believe publishers should find the money to pay journalists to write the same number of stories that they did back in the glory days of the newspaper’s high profit margin?”
You do what businessmen everywhere have done during tough times - invest the money and take a risk. The risk, of course, is judging that better quality content will sell more papers/get more views (the medium is actually unimportant) and thus bring you more market share than your competitors. But taking risks is *not* what newspaper or magazine publishers are very good at.
“It’s easy to argue that the newspaper business is declining strictly because of declining quality.”
Actually, I don’t think that *is* an easy argument to make. it’s far easier, in my opinion, to simply say that the underlying economic model has changed and therefore we can’t afford to do quality stuff anymore.
In reality, of course, it’s a mix of the two: declining ad revenues means job cuts means lower quality means less incentive for readers to buy means declining ad revenues… and on and on.
However, to come back to the original point… are you really saying that you can get the same or better quality journalism by reducing permanent staff numbers, increasing the workload of the remaining staff, and increasing reliance on unpaid, amateur journalists?
Scott,
The techniques of gaming Google & Digg (algorithmic systems) have almost nothing in common with gaming newspaper editors (a human system), and thus the techniques of counter-gaming would be entirely different. They have nothing in common: your subject line is misleading.
I sympathize with Weinstein’s argument becaue I’ve argued it before. But it’s wholly incomplete because it fails to recognize the amount of PR-channeling that the traditional press has done. At times like these, I pull out my handy Schudson. “In 1930, political scientist Peter Odegard estimated that 50 perecent of news items originated in public relations work…” (Discovering the News, 1978, p.144, note 66).
The burden remains on us– media obsessives that we are– to provide some solid data either way on quality metrics in journalism per variables like market, staff size, and yes, CitJ. There’s only so many Philip Meyers to go around.
My worry is twofold. One, keeping in mind Philip Meyers, the dean of statistical journalism, is that because of technology the quick appears to be increasingly beating out the deep in journalism. I don’t see CitJ countering that, unless you have CitJ folks putting more effort into stats than into “personal opinion”), and you have editors respecting that (When I finally finished crunching the TimesSelect in November, here’s the response I got from a prominent forwarder of newsroom memos: you’re data’s a month old.)
Second– sorry I can’t link to analysis of BLS numbers that’s still in my draft folders– but we also must consider the occupational pressures of choosing a journalism career. In short, the BLS projects that press jobs are going to stay flat while the PR industry is expected to grow by over thirty percent in the next ten years. And consider the salaries as well. PR salaries are closer to software engineers (and exceed them at the management levels…), while print journalists barely meet the salaries of cable layers.
Ian,
“However, to come back to the original point… are you really saying that you can get the same or better quality journalism by reducing permanent staff numbers, increasing the workload of the remaining staff, and increasing reliance on unpaid, amateur journalists?”
I’m saying that since many newspapers will have no choice but to try, we’d better start focusing on HOW.
More importantly, I’m saying that media has fundamentally changed, and new economic models are still being figured out. Investing more in old models is not a recipe for success.
Allowing people the time and resources to do a good job is not an “old model” - its the fundamental method of achieving quality in any business. If the economic model does not support doing that very simple thing, then we have a duty to say that.
Ian,
No, your duty as a journalist who cares about journalism isn’t just to state the problem — it’s to help figure out the solution. And you know what? It’s going to be HARD — it’s going to require MORE work. Frankly, I think journalists who expect everything to just fix itself without more work are better labeled “lazy”. The sense of entitlement in the midst of difficult, radical change is sometimes striking.
But let me ask you this — do you really think that in digital,network media context most journalists are making best use of their time and available resources? Do you really think the answer is to let journalists continue to do things the same way? Is there no room for improved productivity?
When the word processor was first introduced, journalists kept wailing away on typewriters, much to the detriment of productivity.
Journalists may think they ought to protected from the demands of producivity — but they can’t be any more.
It’s as much the journalist’s responsibility now as the publisher’s to figure out how quality journalism can be economically viable.
Scott, I suggested a solution - put more money into quality journalism. Invest, tough it out, and try and seize market share by doing so. Would it work? Maybe. Would whatever your solution is work? Maybe.
I also hope you’re not aiming the “lazy” jibes at me. You don’t know me from Adam, and I find it offensive and frankly below you to resort to jibes like that. It does nothing for your argument, either.
Improved productivity - yes of course. But at some point, “improved productivity” becomes “less quality”, when you push reduced staffing beyond the point where better technology can compensate. At that point, if you don’t put your hand up and say “hey, this won’t work” you’re not doing your job properly.
“When the word processor was first introduced, journalists kept wailing away on typewriters, much to the detriment of productivity.”
I suspect the journalists I know who bought tandy Model 102’s out of their own money because their papers didn’t see the point would disagree with you.
“Journalists may think they ought to protected from the demands of producivity — but they can’t be any more.”
Well, now you’re just heading into a standard “you old media guys don’t understand!” rant. Which, again, is something you’re better than.
Scott’s more or less right on this matter, except in two ley areas: First, this mater does not require economic or even professional (journlism) models to deal with the citizen journalism issues that arise from technology.
Second, this business about an “algorithm”, if by that Scott means a discrete set of mathematical or procedural(I call it the knitting model, meaning “knit one, pearl two”)steps to obviate what happens when off-the-wall thinking and language emananting from citizen journlaism intersects with technlogy on the one hand and serious news telling on the other.
Scott refuses, so far as I can tell, to entertain something that is really rather obvious if Scott will look at, and THINK about, what (for example) Intel CEO Paul Otellini is talking about in this URL:
http://www.technewsworld.com/story/61112.html
It seems to me that “intellectuals” like Scott always manage to find something problematic in the notion of crises of one kind or another: generally because they invent most of them themselves. In this instance an algorithm?
What is required here is some serious thinking, which Scott is not yet doing, even though he clearly is capable of doing it, about the deep structural impediments manifest in the Web itself, rather than
the knit one, pearl two “problem” of the corrupting influences of the illiterate masses when they invade news telling with web logs or whatever.
I’m not going to tell Scott how to deal with HIS problem, which is that Scott(God bless him!) is not yet thinking hard enough. Read the above-mentioned URL, Scott, and then start thinking and, perhaps, writing about what Otellini is actually saying.
Ian,
Of course I’m not accusing you personally of laziness, and you’re right I would have no basis of do so. But I have seen that attitude first hand in others — my point is that there’s as much “laziness” on the editorial side as the business side, i.e. It cuts both ways.
But what does “put more money into quality journalism” mean? Pay more people to write for print but not the web? Invest more money in investigative journalism that has huge public value but little standalone commericial value when unbundled from the old newspaper economic bundle, without investing in new business models to support the important journalism?
And really, what does the quality of journalism have to do with the classified advertising business that used to subsidize it? Do you think investing in more quality journalism will bring that business back? The reality is the quality journalism was sustained by a monopoly distribution business model, and the web kiled that business, not bean counting publishers.
As for whether jounrnalists are early adopters of new technology, you’re right that some have been and some haven’t been. But in this age, any journalist who is not trying to learn how new technologies can improve their productivity and the quality if their work and the economic viability of their business (YES it’s their businees) is as culpable as the publishers trying to cut their jobs.
And despite much progress in old media types “getting it,” there’s still a ways to go.
That’s the thing Scott- it’s not a question of print OR online. Good journalists can, and should do both (and video, and audio, and any and all other new forms of media).
The problem isn’t that journalists can’t or wont’ do all that. Most - but not, of course, all - welcome the opportunity to enhance their skill set and do new things.
But the problem is that far too many publishers, in all sectors, believe they should more work, rather than different work; that they can, and should, produce ten stories a day instead of three; and are using the shift from print to online as an excuse to make that happen.
Yes, in some cases this is the desperate act of publishers sinking into loss. But in other cases, it’s simply that publishers aren’t making as much profit as they used to, and see the move from print to online as a method of increasing their profit margin.
Those aren’t by and large, the guys going to Jeff’s conferences, but they really do exist.
What I’m not saying here is that print is quality, and online isn’t. I actually don’t do any print work these days, apart from the odd piece of freelance. Falling quality of journalism applies no matter what the medium is.
Of course, I applaud any attempts to do better journalism. But as in any kind of endevour, quality comes down to three things: time, money and talent. If, at the end of the day, the money isn’t there to support quality journalism, let’s just be honest about that - because readers, at least, deserve to know why their papers aren’t as interesting as they used to be.
Mind if I chime in here Scott/Ian?
I organized Jeff’s conference. I also work in citizen journalism. So my potential bias is now laid out before you.
I think the discussion both of you are having is well merited and it could go on forever, but it’s based on a few assumptions (which were in the Mother Jones article as well) which are false.
Assumption:
Citizen Journalism is sub-par: This might be the case for some articles, but it is not a logical truth. I’ve seen some amazing citizen journalism.
Assumption:
Citizen journalism is lazy and cheap:
During Assignment Zero, the first major NewAssignment.net project - I did not sleep. We needed more professional journalists and we needed to pay them for the time commitment we were asking of them.
In the end we hired one extra person at the end of the experiment - but if we had more money, we would have hired more and the end product probably would have been better — and it still would have been “citizen journalism.”
As Scott pointed out - it’s not a black and white world. This is particularly true in citizen journalism.
The complaint is that CJ takes jobs away from journalists and the quality is sub-par. But citizen journalism (or self-publishing on the web) is not going to disappear - And that is NOT an assumption.
Nor do I think it should. If a journalist wants to play the role of the fourth estate - what better way to do that then to MOTIVATE other people to keep a watchful eye as well.
Journalists need to work WITH amateur reporters - and by doing so we will hopefully improve the quality of their content - and finally: If we do, perhaps, as Ian notes: That quality will be recognized and create more jobs for journalists. Granted the job description will be different - but it will be a job - and still a noble one at that (in my opinion).
Dave,
You’re right that this debate is rife with faulty assumptions, and journalism is going to be stuck in the mud until we re-examine all of our assumptions.
I’d also question assumptions about what defines an amateur vs. a professional journalism. Journalism is not defined by a paycheck. There are undoubtedly journalists collecting a paycheck from a newspapers who are terrible at it, and there are also people who don’t work for a news organization with all of the requisite skills and attitude to do good journalism.
And you’re right that instead of pigeon holing citizen journalism as free low quality content, why not look at it as a way for professional journalists to extend their reach — extra sets of hands and eyes and ears for journalists who are overburdened.
Journalists are used to being lone wolves, but in a networked media world, the lone wolf is an island, disconnected and disenfranchised.
A citizen journalist network isn’t going to be up to standard out of the box — it needs to be trained, and journalists need to step up to the plate.
Ian,
Since it’s not the case that all publishers are evil bean counters, I felt it was my duty to call Adam Weinstein on his overly broad brush. I brought up Jeff’s conference because I think the only way to fix the problem of lazy publishers is to bring them new models that have been proven to work.
Here’s one big cause of the overburdened journalist problem — the newspaper business is split is really two businesses, one small and growing, one large and shrinking. If the online business is not to be a sad rehash of the print business, that requires original content — which means net MORE content. Publishers can’t afford to under-resource the growing online business, nor shut down the shrinking print business, which means journalists need to staff two businesses instead of one.
Is it fair? No. Does it lead to unscrupulous practices? Of course.
Can making use of willing contributors other than the overburdened full-time staff possibly help the situation, if done with the aim of achieving the same high standard? I think it can.
Dave wrote: “The complaint is that CJ takes jobs away from journalists and the quality is sub-par.”
I suppose that was Weinstein’s point. I think it’s a fair worry. Of course, the way he phrased it, he needs to prove that publishers are actively pursusing this strategy, and I don’t think anybody can find one to admit such on record.
My sense is– and if I had more solid evidence I would write a full essay on this– that what we will see happen is that we’ll see more people become practioners/bloggers before (or instead of) going into journalism.
Another problem in many discussions over CitJ is the failure to consider the other trends outside of Big Journalism — the trade presses, and hyperlocal media. These have long been realms where more of the copy is supplied by active practitioners.
So, back to the core issue at hand. What methods should editors and readers use to determine what the hidden motivation of various correspondents are? Scott, your premise here is that Google and Digg have something to teach, but you failed to explain how.
[...] Developing Algorithms To Prevent Citizen Journalism From Being Gamed: Lessons From Google and Digg -… “Which got me thinking about what citizen/networked journalism might learn from the battle that systems like Digg (which accepts anonymous users) and Google (which interprets open linking on the web) have waged against gaming.” (tags: internet newspapers participatory journalism citizenmedia crowdsourcing digg google) [...]
Scott:
“I think the only way to fix the problem of lazy publishers is to bring them new models that have been proven to work.”
And that’s why, while I sometimes disagree with the specifics of the ideas that Jeff puts forward, I think his aim is correct - and yours, too.
I think Alan Patrick has actually ended up putting my overall point better than I have in a post over on his blog when he asks “what will you pay for accurate citizen journalism?” The worry I have - and I admit that I’m feeling a bit doom-laden today - is that the answer is “not enough to actually make it work”.
However, I support any and all efforts to make it happen!
If newsrooms lean too heavily on technology and user generated content to save money, then it invariably becomes an arms race between them and automated spam, including for PR purposes. News and social networking technology is a commodity now, and bigger organizations need to devote appropriate personnel levels for sites getting astonishing daily traffic.
Steven Eng
newmediaplanner.com