In recent weeks, Google has indicated that it is revising its search algorithm in order to punish so-called "scrapers" and "content farms"—websites that, respectively, steal articles from other publications and write absurdly banal articles built around common search terms, thereby gobbling up traffic. For anyone who cares about the future of writing and reporting, this was certainly good news. But Google's improvements appeared to be aimed fairly narrowly at the most egregious offenders, sites that no one thinks of as legitimate publications, like Associated Content and eHow. Largely ignored was the question of what Google's incentives are doing to journalism as a whole.
The story of how the chase for search traffic came to define contemporary journalism may not exactly begin with The Huffington Post, but Arianna Huffington's Web magazine probably plays the central role. HuffPo is not a scraper or a content farm; it publishes plenty of original journalism. But it has been successful for the same reason that scrapers and content farms are frequently successful—a penchant for search-engine optimization. Often this means that HuffPo simply does a better job of drawing traffic to its own pieces. It's hard to complain about that. But HuffPo also does a great job of drawing traffic to its excerpts of other publications' pieces. Which is to say, it has become quite adept at reaping traffic—and profits—from other people's work.
This is a milder form of scraping, which goes by the name “aggregation.” And, to be fair to HuffPo, everyone does it now. Aggregation doesn’t involve stealing other people’s pieces. It simply involves curating them—copying and pasting a key passage, providing a link to the original, and then hoping that people searching on that topic will stumble across your excerpt before they stumble on the real thing.
There isn’t anything inherently wrong with aggregation. On the contrary—unless we expect readers to get all their news from one publication or, alternatively, spend all day sifting through numerous websites themselves—the Web needs aggregators. And smart aggregation does, in fact, add something to the world by bringing a certain editorial judgment to bear on the selection of pieces.
So the problem isn’t aggregation. It’s that the entire structure of the media world currently provides publications with huge incentives to aggregate and comparatively small incentives to actually create. If your goal is to increase revenue by increasing Web traffic, it’s simply much cheaper and more efficient to excerpt or summarize other stories, rather than to produce your own. And so, increasingly, we are seeing publications devote more and more attention to aggregating—to the model of success pioneered by HuffPo. The New Republic is no exception: We face the same pressures, the same set of skewed incentives, as everyone else; and we too are trying to figure out ways to garner more search traffic through aggregation. In a different world, this is not where we would choose to put our resources. But the fact is, we don’t have much of a choice.
It’s a cliché, but it’s true: The creation of original information and arguments is tremendously important to both the functioning of democracy and the existence of a decent, reflective culture. So, it’s worth taking a step back and asking if there isn’t something that can be done to remedy this situation. The obvious thing is for Google to think more expansively about how to reward the production of original journalism—that is, not just to penalize scrapers and content farms but also to actively reward publications for writing stories rather than aggregating the material of others.
The other solution is more nebulous. No publication is going to stop aggregating. But journalists might, as a collective whole, pause to think about where our priorities ought to lie. Even as we all play the game by the current set of rules—which means we will all continue to invest in aggregation—we need to remind ourselves that aggregation is not a replacement for journalism. With this in mind, perhaps we can all agree not to let the aggregation arms race escalate into madness. Maybe we can establish a rough norm for ourselves: that we will continue to treat aggregation as a side project to what we really do—and, accordingly, that we will limit it to playing a small role in our institutions relative to the production of real writing and reporting. Otherwise, we are going to wake up one day and discover that we are simply aggregating each other’s aggregation, because no one is any longer bothering to create original material. No doubt The Huffington Post would figure out some way to thrive in that world, but the rest of us should do everything possible to make sure it never comes about.
This article originally ran in the March 24, 2011, issue of the magazine.