This post, along with seventy-eight others, is collected in the book Utopia Is Creepy.
“It’s not information overload. It’s filter failure.” That was the main theme of a thoughtful and influential talk that Clay Shirky gave at a technology conference back in 2008. It’s an idea that’s easy to like both because it feels intuitively correct and because it’s reassuring: better filters will help reduce information overload, and better filters are things we can actually build. Information overload isn’t an inevitable side effect of information abundance. It’s a problem that has a solution. So let’s roll up our sleeves and start coding.
There was one thing that bugged me, though, about Shirky’s idea, and it was this paradox: The quality and speed of our information filters have been improving steadily for a few centuries, and have been improving extraordinarily quickly for the last two decades, and yet our sense of being overloaded with information is stronger than ever. If, as Shirky argues, improved filters will reduce overload, then why haven’t they done so up until now? Why don’t we feel that information overload is subsiding as a problem rather than getting worse? The reason, I’ve come to believe, is that Shirky’s formulation gets it precisely backwards. Better filters don’t mitigate information overload; they intensify it. It would be more accurate to say: “It’s not information overload. It’s filter success.”
But let me back up a little, because it’s actually more complicated than that. One of the traps we fall into when we talk about information overload is that we’re usually talking about two very different things as if they were one thing. Information overload actually takes two forms, which I’ll call situational overload and ambient overload, and they need to be treated separately.
Situational overload is the needle-in-the-haystack problem: You need a particular piece of information – in order to answer a question of one sort or another – and that piece of information is buried in a bunch of other pieces of information. The challenge is to pinpoint the required information, to extract the needle from the haystack, and to do it as quickly as possible. Filters have always been pretty effective at solving the problem of situational overload. The introduction of indexes and concordances – made possible by the earlier invention of alphabetization – helped solve the problem with books. Card catalogues and the Dewey decimal system helped solve the problem with libraries. Train and boat schedules helped solve the problem with transport. The Reader’s Guide to Periodicals helped solve the problem with magazines. And search engines and other computerized navigational and organizational tools have helped solve the problem with online databases.
Whenever a new information medium comes along, we tend to quickly develop good filtering tools that enable us to sort and search the contents of the medium. That’s as true today as it’s ever been. In general, I think you could make a strong case that, even though the amount of information available to us has exploded in recent years, the problem of situational overload has continued to abate. Yes, there are still frustrating moments when our filters give us the hay instead of the needle, but for most questions most of the time, search engines and other digital filters, or software-based, human-powered filters like email or Twitter, are able to serve up good answers in an eyeblink or two.
Situational overload is not the problem. When we complain about information overload, what we’re usually complaining about is ambient overload. This is an altogether different beast. Ambient overload doesn’t involve needles in haystacks. It involves haystack-sized piles of needles. We experience ambient overload when we’re surrounded by so much information that is of immediate interest to us that we feel overwhelmed by the neverending pressure of trying to keep up with it all. We keep clicking links, keep hitting the refresh key, keep opening new tabs, keep checking email in-boxes and RSS feeds, keep scanning Amazon and Netflix recommendations – and yet the pile of interesting information never shrinks.
The cause of situational overload is too much noise. The cause of ambient overload is too much signal.
The great power of modern digital filters lies in their ability to make information that is of inherent interest to us immediately visible to us. The information may take the form of personal messages or updates from friends or colleagues, broadcast messages from experts or celebrities whose opinions or observations we value, headlines and stories from writers or publications we like, alerts about the availability of various other sorts of content on favorite subjects, or suggestions from recommendation engines – but it all shares the quality of being tailored to our particular interests. It’s all needles. And modern filters don’t just organize that information for us; they push the information at us as alerts, updates, streams. We tend to point to spam as an example of information overload. But spam is just an annoyance. The real source of information overload, at least of the ambient sort, is the stuff we like, the stuff we want. And as filters get better, that’s exactly the stuff we get more of.
It’s a mistake, in short, to assume that as filters improve they have the effect of reducing the information we have to look at. As today’s filters improve, they expand the information we feel compelled to take notice of. Yes, they winnow out the uninteresting stuff (imperfectly), but they deliver a vastly greater supply of interesting stuff. And precisely because the information is of interest to us, we feel pressure to attend to it. As a result, our sense of overload increases. This is not an indictment of modern filters. They’re doing precisely what we want them to do: find interesting information and make it visible to us. But it does mean that if we believe that improving the workings of filters will save us from information overload, we’re going to be very disappointed. The technology that creates the problem is not going to make the problem go away. If you really want a respite from information overload, pray for filter failure.
Bottom line: When the amount of information available to be filtered is effectively unlimited, as is the case on the Net, then every improvement in the quality of filters will make information overload worse.
You should look at this from a different perspective. Using the needle in the haystack analogy, we’ve spent too much effort figuring out how to identify the hay and no effort whatsoever on characterizing the needle. We developed intense methods to identify what we don’t want, assuming anything remaining would be interesting, by default. We figure it’s easier to remove all the hay than to search for the needle.
“Better filters don’t mitigate information overload; they intensify it.” Jane Jacobs makes the same point about streets and traffic congestion. Keep building more bigger and better streets, and more cars will come.
“surrounded by so much information that is of immediate interest to us that we feel overwhelmed by the neverending pressure of trying to keep up with it all” If the information is all equally good then there’s no point in looking further. Search is solved. But I suppose we’re never quite sure of that, it is that mistake which keeps us looking, maybe there’s something even better.
The more time passes, the more I am coming to believe that curation is king. It’s funny because I was one of those guys who had always thought that curation would no longer be necessary as technology perfected itself. Maybe technology will reach that point one day, but I am more and more skeptical that other factors (even human nature and gaming systems) would win out.
1) “failure” is conference-clubber coded wording for “the solution to this problem is to throw money at data-mining start-ups, like the ones we promote and have stock options and advisory board positions”. You can have all sorts of failures except for market failure.
2) In the post above, I’d say both Shirky and Carr are following technological determinism, but taking different sides of that coin. But my view is the problem is really more sociological. Why do we think, now, that we NEED to keep track of so much information, if we didn’t before? It’s not volume, objectively, because the amount long, long ago exceeded anything an ordinary person could follow. I suspect this is really about social anxiety where it’s considered safe to blame technology instead of the political trends at work.
Shirky has the problem half right. Attention management is about pushing back and pulling forward. He addresses the pushing back (filtering, attention shielding). You are pointing out the need for pulling forward – that after filtering you still need help noticing the more important stuff left in the pile. This leads to technologies such as keyword subscriptions, social ratings, saved searches, AI approaches, etc.
My response to Shirky is posted at “Clay Shirkey on Information Overload as a Filter Problem” (http://knowledgeforward.wordpress.com/2010/02/19/clay-shirkey-on-information-overload-as-a-filter-problem/).
I dunno — I scroll through a lot of twitter feeds to find signal. That’s because twitter doesn’t have an intelligent, adaptive and tunable recommendation feature. Will it in the future? Certainly (or whatever replaces it will, as will everything else). Will I make use of it? Yes. Will I waste less time? Yes. Will others dial their tuner to let a lot more in than I do? Yes — and they will no doubt continue to bellyache endlessly about information overload . . .
Nick, another interesting pass at this problem assumes that ‘what appears to us as “too much information” could just be the freedom from necessity.’ http://theaporetic.com/?p=228
Loved the article and I agree, with modern tools and exponential growth in connections it is easier than ever to get quality information. But there is so so much more of it.
We can’t expect technology to absolve our personal responsibility to say enough. And turn off.
So what do you do with all the needles? I had an interest in Thailand. Before, I would read some old books I’ve had for years or pick up a book occasionally at used book stores. Now I read blogs, listen to Thai radio, can order books easier than ever from Thailand. Ditto all my other interests. My needle pile is immense. How do you reduce the pile? You can specialize, but how do you maintain the big picture? Do you consolidate the needles by synthesis, by reducing the facts to larger theories that you can hold in memory? Eventually, you just have to recognize limits. You have to get rid of a lot of good stuff. It’s hard.
I agree. Better filters surface more content that is compelling and that makes it more difficult to walk away.
Worse, however, is ‘conversation overload.’ I can walk away from information but walking away from conversations is far more difficult and our technologies make conversation/communication so much easier and that produces more of it.
You are absolutely right, Nick.
The problem is not too much noise, because even after we filter out ALL noise, the problem is we are left with too much signal. Signal — the perfect stuff just for us at this moment — is expanding faster than our attention.
I think this is a psychological problem of dealing with abundance rather than scarcity. What if you lived in a world where everything around you was just what you wanted? How would you choose?
In theory, you would not choose since it does not matter. Leave it to serendipity. If your filters really are working, then anything you accept from them should be satisfying. Is skipping to choose satisfying? Probably not. So I would guess there is a psychological dilemma or paradox here: Ultimate satisfaction will ultimately be unsatisfying. Or rather, no system can be satisfying.
I have enough high quality ambient overload to last me many lifetimes. That is why I take a day or two every week to work OFFLINE with PEN and PAPER and BOOK.
It used to be that the Self was enough to determine importance, value, choice, conversation, interests, refusals, and priorities. Filter just happened. It wasn’t an event in itself but a means to an end we determined. Jane Healy in her now classic book, Endangered Minds (1991) said it well in the chapter, Sesame Street and the Death of Reading.
Mr. Carr, what ever happened to the Carr-Benkler wager? It’s 2011 already!
I came across this essay as I was doing some research for my own writing, and I think you’re absolutely correct about the distinction between a sort of “goal-oriented” situational overload and the broader distractions of an ambient overload. That distinction hadn’t occurred to me, but I think it’s an important and useful one.
This happens to be a specific issue that I’m personally very interested in, and in addition, I think that there are a couple more important dimensions to this discussion:
1) The fact that social context plays an centrak role in how we process and internalize information, so that “ambient needles” that come to us through social mechanisms (such as friends’ Facebook “Likes”) may very well have a different, and deeper, influence than we might initially assume.
2) How “modern digital filters” are expanding our broader social discourse into a realm that is fundamentally different, in some ways, than we’ve seen before. More and more our individual and collective attention is being guided through a new process — one that’s based on statistically emergent results, rather than a conscious human agenda. That’s not inherently good or bad, but when it means that more and more often, we can’t really explain _why_ something is relevant, but can just point to the fact that it apparently _is_, it’s definitely different.
If it’s alright to mention, I’ve actually done a good deal more thinking about these issues — and the insights we can get from some past thinkers, such as Piaget and Foucault — on my site ( http://lairbob.com ). I’d very much like to hear what anyone has to say. As I said before, I think these issues are critical for us to wrestle with, right now.
Great insights Nick. Thanks.
I recently argued that accessing content in my inbox is worse today using a smartphone over 3G than it was back in 1998 when I was one of the few users using a 9600 bauds GSM subscription in New York, my Palm Pilot physically tethered to a bulky Ericsson mobile phone…
Those who have benefited are the great majority of NEW consumers. Yes, they can all discover & consume the Internet, sometimes for the first time in their lives, using a device that fits in their pocket. For some, this may just mean access to information that would otherwise not be available at all (in countries where freedom of expression and information is repressed: no needle because no haystack!).
In my opinion, there is great divide between the early adopters, the elite information workers, the Internet and Mobile industry movers and shakers… and the rest of the World. Clearly, for the great majority of consumers, the filters are probably just hitting a sweet spot, giving them a decent dose of signal (i.e. bayesian spam filters). Indeed, most of them still don’t have a twitter account, and don’t necessarily have over 100 buddies on Facebook (vs. over 500 on both LinkedIn and Facebook). Perhaps the problem still exist on cable TV with a plethora of channels with equally (un)interesting content… but TV is getting old for them.
So yes, I agree with you. We, the information workers, have to suck it up for now. I’m not necessarily good at coping with it. But I’m trying. I think some of the solutions involve offline time (I have yet to manage more than one day!), or delegating, or as suggested in this Newsweek article (http://www.newsweek.com/2011/02/27/i-can-t-think.html), tending to be a “sufficer” rather than a “maximizer”…
The more I read the comments and the more I think of the haystack and the packs of needles, the more I am remembered of an seminal article on a very close subject: the garbage can decision making model… 1972 !!! long before the internet.
A good comment on the article and the original source are linked below:
Quote from “The lid on the garbage can: institutional constraints on decision making in the technical core of college-text publishers”. by Barbara Levitt , Clifford Nas
The garbage can model of organizational choice (Cohen, March, and Olsen, 1972) has been a mainstay of the literature on organizational decision making for over fifteen years. According to this model, many decision processes within organizations do not operate according to rational choice models. Rather, confounding situational elements further limit the cognitive capacities of organizational participants (Cohen, March, and Olsen, 1972; March and Olsen, 1986). Streams of loosely coupled problems, solutions, participants, and choice opportunities flow into the organization at different rates and connect or decouple elements according to a temporal rather than a causal logic. Garbage can processes “depend on a relatively complex intermeshing of elements, [including] the mix of problems that have access to the organization, the mix of solutions looking for problems, and the outside demands on the decision makers” (Cohen, March, and Olsen, 1972: 16). Thus, solutions may seek problems, both problems and solutions may await opportunities for decisions, and participant energy is likely to be distributed according to the overall load and arrival time of the various streams rather than by any “objective” criteria determining the relative importance of a particular issue. In garbage can systems, decisions are often made by flight or oversight rather than by calculation.
End of quote
Full text of this article to be found at http://www.questia.com/googleScholar.qst?docId=5000105617
Original article of Cohen & al can be found at http://www.business.illinois.edu/ghoetker/teaching/ba547papers_files/cohen_march_olsen_1972.pdf
Thanks for having kicked off this discussion
I like this article, it explains a lot, I am in an ever expanding sweet shop of my own making.
I do wonder if this is also an age thing. I am of a generation that still needs to own things, like music I don’t buy CDs but I do buy files files. I notice my son does not have this hang up and just want to know where he can get things. I come from a generation that want to own knowledge, ie: I have to get it into my own head and this is probably what is causing me the stress, maybe I have to develop a different relationship to knowledge.
The old saying “the more I find out the less I seem to know”, so the internet throws up an ever increasing pile of proof that I don’t know much and my inner teacher drives me to keep climb that pile, but I will never get to the top.
I liked the suggestion to go offline. I recently did a drawing project doing a drawing a day for a month and it was great just to get back to the basics.
Feedback Overload (with much credit to Nick Carr!)
Since the invention of writing, the written word has literally piled up. Indeed, from very early on, mankind has been overloaded with information. However, the problem posed by of information overload is not in a metaphorical stack of stuff, but in our relative inability of finding the needle of information we need in the haystack of information we don’t. Things like the Dewey decimal system, book indexes, and a helpful librarian barely addressed the problem until the invention of the internet search engine allows us to find our need, or in this case, needles. As the pundit Nicolas Carr opined , the problem we confront today is not finding a needle in an infinite informational haystack, but finding an infinite stack of needles that all merit consideration. Nowadays, when we electronically search for any topic, we are provided with many similar bits of information that allow us to more precisely fine tune or correct the deficiencies of knowledge. This error correction or feedback function represents a progressive resolution of the discrepancies between what we do and don’t know. Feedback may represent unexpected changes in our progress to a goal and/or unexpected changes in our knowledge of the nature of a goal. Feedback of course is essential to learning, but consequential to that learning is the increased activity of midbrain dopamine neurons, and it is the neuro-modulator dopamine that enables the consolidation of memory as well as heightened alertness and attention on the task at hand. But dopamine also increases positive affect that adds momentary value or ‘incentive salience’ to behavior, but does not intrinsically predict the overall or long term goodness or utility of behavior. Put a bit differently by the neuro-psychologist Kent Berridge, “The brain results suggest that pure decision utility—and not predicted utility—is raised by activating mesolimbic dopamine systems .” What this means is that the importance of the decision in the moment, or its ‘decision utility’ does not influence its long term or ‘predicted utility’. The implications of this are profound, for as the marginal utility of examining each informative ‘needle’ declines, the successive needles of information remain novel, and we continue to dwell on nearly redundant links of information not because they are useful but because they are new. In other words, whereas in the past impoverished feedback environments caused us to waste much time looking for information, the rich feedback environments heralded by improvements in web search lead us to waste much time looking at information! This means that we will be affectively and not rationally inclined to overstay our welcome on sites that not only provide us what we want and need, but infinite variations of the same information that we ‘want’ but don’t need. The problem thus is not information overload, but ‘feedback overload’, as the ever increasing amount and granularity of information feedback provides greater and greater detail that can increase the short term or moment to moment value of behavior to the detriment of our long term interests.
This increase in the momentary incentive salience of behavior can be used to conform with (if not predict) practical ends, but its ultimate value depends upon whose practical ends. For example, the ‘Khan Academy’ (khanacademy.org) is an online math tutorial that uses rich feedback embodied in badges, scores, hints, etc. to increase the decision utility of performing math exercises in service of the predicted utility of long term mastery of say, the mathematical calculus. On the other hand, a Google search also provides rich feedback including social network feeds, instant messaging, videos, helpful links, and now badges in the service of the predicted utility of Google, namely advertising.
Ultimately, the problem is not that we are lost in a haystack, but that we are proverbially resting on a bed of pins and needles with each pin needlessly diverting our attention. The notion of ‘feedback overload’ means we are neurologically inclined to overvalue the short term importance or salience of new information, and when new information scales in amount and availability, we begin to live for a moment that may not conform to our ultimate good. For the rich feedback mechanisms provided by the internet, whether it is social media of just plain search, the solution to this problem is not better filtering of information or better feedback (as this merely acerbates the problem), but less, and can only be accomplished by constraining what information you can see, or when you can see it. The simple solution is keeping your personal library and newspaper, and severely restricting your time with search tools (the internet) that work too well. As internet feedback trends to infinity in ever morphing detail and availability, this will be our only option to spare us a new dark age caused by being blinded by the light.
Berrdge, K. and Aldridge, J. W. Decision utility, the brain, and the pursuit of hedonic Goals, Social Cognition, Vol. 26, No. 5, 2008, pp. 621–646
Nicolas Carr, Roughtype (blog)
above is from my blog
Below is an argument and linked article that attempts to resolve the Carr-Shirky debate. (Hint, Nick is wrong, yet still wins!)
The core assumption behind information overload is that the information we want is the same as the information we need or like. (Both Carr and Shirky evidently subscribe to this) Therefore, we cannot with good reason cut back on the information we want, because it reflects stuff that is important to us. Hence, thanks to the web we are overloaded with needed information that we can’t help wanting. However, from the perspective of contemporary affective neuroscience, wanting and liking are NOT the same thing, and are governed by entirely different neural processes. Thus, what we want is different from what we need because wanting and liking represent distinctive neurological events. Therefore, the key underlying premise of information overload that everything we want is the same as everything we need is based on cognitive principles that have no basis in neural reality, and the concept of information overload must therefore be redefined, or perhaps abandoned.
The linked article questions the concept of information overload by challenging this most elementary assumption. Based on the work of the distinguished neuropsychologist Kent Berridge of the University of Michigan (who also vetted and endorsed it), it is simple, short, and uses the Boston Red Sox title run to make its very radical point. Hope you ‘like’ it or at the very least the Red Sox!