The wisdom of a crowd is often in the eye of the beholder, but most of us understand that, at its most basic level, “crowd wisdom” refers to a fairly simple phenomenon: when you ask a whole bunch of random people a question that can be answered with a number (eg, what’s the population of Swaziland?) and then you add up all the answers and divide the sum by the number of people providing those answers – ie, calculate the average – you’ll frequently get a close approximation of the actual answer. Indeed, it’s often suggested, the crowd’s average answer tends to me more accurate than an estimate from an actual expert. As the science writer Jonah Lehrer put it in a column in the Wall Street Journal on Saturday:

The good news is that the wisdom of crowds exists. When groups of people are asked a difficult question – say, to estimate the number of marbles in a jar, or the murder rate of New York City – their mistakes tend to cancel each other out. As a result, the average answer is often surprisingly accurate.

To back this up, Lehrer points to a new study by a group of Swiss researchers:

The researchers gathered 144 Swiss college students, sat them in isolated cubicles, and then asked them to answer [six] questions, such as the number of new immigrants living in Zurich. In many instances, the crowd proved correct. When asked about those immigrants, for instance, the median guess of the students was 10,000. The answer was 10,067.

Neat, huh?

Except, well, it’s not quite that clear-cut. In fact, it’s not clear-cut at all. If you read the paper, you’ll find that the crowd did not “prove correct” in many instances. The only time the crowd proved even close to correct was in the particular instance cited by Lehrer – and that was only because Lehrer used the median answer rather than the mean. In most cases, the average answer provided by the crowd was wildly wrong.

Peter Freed, a neuroscience researcher at Columbia, let loose on Lehrer in a long, amusing blog post, arguing that he (Lehrer) had misread the evidence in the study. Freed pointed out that if you look at the crowd’s average answers – “average” as in “mean” – to the six questions the researchers posed, you’ll find that they are, as Freed says, “horrrrrrrrrrrrrendous”:

… the crowd was hundreds of percents – yes, hundreds of percents – off the mark. They were less than 100% off in response to only one out of the six questions! At their worst – to take a single value, as Lehrer wrongly did with the 0.7% [median] – the 144 Swiss students, as a true crowd (unlike the 0.7%), guessed that there had been 135,051 assaults in 2006 in Switzerland – in fact there had been 9,272 – an error of 1,356%.

Or, as the researchers themselves report:

In our case, the arithmetic mean performs poorly, as we have validated by comparing its distance to the truth with the individual distances to the truth. In only 21.3% of the cases is the arithmetic mean closer to the truth than the individual first estimates.

So, far from providing evidence that supports the existence of the wisdom-of-crowds effect, the study actually suggests that the effect may not be real at all, or at least may be a much rarer phenomenon than we assume.

But since this is statistics, that’s by no means (no pun intended) the end of the story. As the researchers go on to explain, it’s quite natural for a crowd’s average answer, calculated as the mean, to be way too high – and hence ridiculously unwise. That’s because, while individuals’ underestimates for these kinds of questions are bounded at zero, there’s no upper bound to their overestimates. “In other words,” as the researchers write, “a minority of estimates are scattered in a fat right tail,” which ends up skewing the mean far beyond any semblance of “wisdom.”

Fortunately (or not), the arcane art of statistics allows you to correct for the crowd’s errors. By massaging the results – “tuning” them, as the researchers put it – you can effectively weed out the overestimates and (presto-chango) manufacture a wisdom-of-crowds effect. In this case, the researchers performed this magic by calculating the “geometric mean” of the group’s answers rather than the simple “arithmetic mean”:

As a large number of our subjects had problems choosing the right order of magnitude of their responses, they faced a problem of logarithmic nature. When using logarithms of estimates, the arithmetic mean is closer to the logarithm of the truth than the individuals’ estimates in 77.1% of the cases. This confirms that the geometric mean (i.e., exponential of the mean of the logarithmized data) is an accurate measure of the wisdom of crowds for our data.

Got that?

Well, it further turns out that the median answer – the centermost individual answer – in a big set of answers often replicates, roughly, the geometric mean. Again, that’s no big surprise. The median, like the geometric mean, serves to neutralize wildly wrong guesses. It hides the magnitude of people’s errors. The researchers point this fact out in their paper, but Freed, having criticizing Lehrer for a sloppy reading of the study, seems to have overlooked that point. Which earns Freed a righteous tongue-lashing from another blogger, the physics professor Chad Orzel:

Freed’s proud ignorance of the underlying statistics completely undermines everything else. His core argument is that the “wisdom of crowds” effect is bunk because the arithmetic mean of the guesses is a lousy estimate of the real value. Which is not surprising, given the nature of the distribution – that’s why the authors prefer the geometric mean. He blasts Lehrer for using a median value as his example, without noting that the median values are generally pretty close to the geometric means – all but one are within 20% of the geometric mean – making the median a not-too-bad (and much easier to explain) characterization of the distribution.

You get the sense that this could go on forever. And I sort of hope it does, because I enjoyed Lehrer’s original column (the main point of which, by the way, was that the more a crowd socializes the less “wise” it becomes), and I enjoyed Freed’s vigorous debunking of Lehrer’s reading of (one part of) the study, and I also enjoyed Orzel’s equally vigorous debunking of (one part of) Freed’s debunking.

But beyond the points and counterpoints, there is a big picture here, and it can be described this way: Even in its most basic expression, the wisdom-of-crowds effect seems to be exaggerated. In many cases, including the ones covered by the Swiss researchers, it’s only by using a statistical trick that you can nudge a crowd’s responses toward accuracy. By looking at the geometric mean rather than the simple arithmetic mean, the researchers performed the statistical equivalent of cosmetic surgery on the crowd: they snipped away those responses that didn’t fit the theoretical wisdom-of-crowds effect that they wanted to display. As soon as you start massaging the answers of a crowd in a way that gives more weight to some answers and less weight to other answers, you’re no longer dealing with a true crowd, a real writhing mass of humanity. You’re dealing with a statistical fiction. You’re dealing, in other words, not with the wisdom of crowds, but with the wisdom of statisticians. There’s absolutely nothing wrong with that – from a purely statistical perspective, it’s the right thing to do – but you shouldn’t then pretend that you’re documenting a real-world phenomenon.

Freed gets at this point in a comment he makes on Orzel’s post:

Statistics’ dislike of long right tails is *not a scientific position.* It is an aesthetic position that, at least personally, I find robs us of a great deal of psychological richness … [T]o understand the behavior of a crowd – a real world crowd, not a group of prisoners in segregation – or of society in general, right tails matter, and extreme opinions are over-weighted.

The next time somebody tells you about a wisdom-of-crowds effect, make sure you ask them whether they’re talking about a real crowd or a statistically enhanced crowd.

As part of its Ideas Market speaker series, the Wall Street Journal is hosting a discussion on that venerable topic “The Future of the Book” at the New York Public Library on Tuesday evening. I’ll be one of the panelists, along with tech scribe Steven Levy and Random House e-strategist Liisa Mcloy-Kelley. Moderating will be the Journal’s Alexandra Alter. The event is free and open to the public, but seats are limited and need to be reserved in advance, by sending an email with your name to ReviewSeries@wsj.com. More details here.

The 2011 Pulitzer Prizes were announced today, and I’m thrilled to report that my book The Shallows: What the Internet Is Doing to Our Brains was named a finalist in the General Nonfiction category. The prize winner in the category was Siddhartha Mukherjee’s The Emperor of All Maladies: A Biography of Cancer. The other finalist was S. C. Gwynne’s Empire of the Summer Moon: Quanah Parker and the Rise and Fall of the Comanches, the Most Powerful Indian Tribe in American History.

Are you ashamed that you find Facebook boring? Are you angst-ridden by your weak social-networking skills? Do you look with envy on those whose friend-count dwarfs your own? Buck up, my friend. The traits you consider signs of failure may actually be marks of intellectual vigor, according to a new study appearing in the May issue of Computers in Human Behavior.

The study, by Bu Zhong and Marie Hardin at Penn State and Tao Sun at the University of Vermont, is one of the first to examine the personalities of social networkers. The researchers looked in particular at connections between social-network use and the personality trait that psychologists refer to as “need for cognition,” or NFC. NFC, as Professor Zhong explained in an email to me, “is a recognized indicator for deep or shallow thinking.” People who like to challenge their minds have high NFC, while those who avoid deep thinking have low NFC. Whereas, according to the authors, “high NFC individuals possess an intrinsic motivation to think, having a natural motivation to seek knowledge,” those with low NFC don’t like to grapple with complexity and tend to content themselves with superficial assessments, particularly when faced with difficult intellectual challenges.

The researchers surveyed 436 college students during 2010. Each participant completed a standard psychological assessment measuring NFC as well as a questionnaire measuring social network use. (Given what we know about college students’ social networking in 2010, it can be assumed that the bulk of the activity consisted of Facebook use.) The study revealed a significant negative correlation between social network site (SNS) activity and NFC scores. “The key finding,” the authors write, “is that NFC played an important role in SNS use. Specifically, high NFC individuals tended to use SNS less often than low NFC people, suggesting that effortful thinking may be associated with less social networking among young people.” Moreover, “high NFC participants were significantly less likely to add new friends to their SNS accounts than low or medium NFC individuals.”

To put it in layman’s terms, the study suggests that if you want to be a big success on Facebook, it helps to be a dullard.

Alice Gregory writes:

Shteyngart says the first thing that happened when he bought an iPhone “was that New York fell away . . . It disappeared. Poof.” That’s the first thing I noticed too: the city disappeared, along with any will to experience. New York, so densely populated and supposedly sleepless, must be the most efficient place to hone observational powers. But those powers are now dulled in me. I find myself preferring the blogs of remote strangers to my own observations of present ones. Gone are the tacit alliances with fellow subway riders, the brief evolution of sympathy with pedestrians. That predictable progress of unspoken affinity is now interrupted by an impulse to either refresh a page or to take a website-worthy photo. I have the nervous hand-tics of a junkie. For someone whose interest in other people’s private lives was once endless, I sure do ignore them a lot now.

Via Doc Searls and, with rueful irony, William Blake.

Having recently come off a Red Dead Redemption jag, I decided, as an act of penance, to review the latest studies on the cognitive effects of video games. Because videogaming has become such a popular pastime so quickly, it has, like television before it, become a focus of psychological and neuroscientific experiments. The research has, on balance, tempered fears that video games would turn players into bug-eyed, bloody-minded droogs intent on ultraviolence. The evidence suggests that spending a lot of time playing action games – the ones in which you run around killing things before they kill you (there are lots of variations on that theme) – actually improves certain cognitive functions, such as hand-eye coordination and visual acuity, and can speed up reaction times. In retrospect, these findings shouldn’t have come as a surprise. As anyone who has ever played an action game knows, the more you play it, the better you get at it, and getting better at it requires improvements in hand-eye coordination and visual acuity. If scientists had done the same sort of studies on pinball players 50 years ago, they would have probably seen fairly similar results.

But these studies have also come to be interpreted in broader terms. Some popular-science writers draw on them as evidence that the heavy use of digital media – not just video games, but web-surfing, texting, online multitasking, and so forth – actually makes us “smarter.” The ur-text here is Steven Johnson’s 2005 book Everything Bad Is Good for You. Johnson draws on an important 2003 study, published as a letter to Nature magazine, by University of Rochester researchers Shawn Green and Daphne Bavelier, which demonstrated that “10 days of training on an action game is sufficient to increase the capacity of visual attention, its spatial distribution and its temporal resultion.” In other words, playing an action game can help you keep track of more visual stimuli more quickly and across a broader field, and these gains may persist even after you walk away from the gaming console. Other studies, carried out both before and after the Green and Bavelier research, generally back up these findings. In his book, Johnson concluded, sweepingly, that video games “were literally making [players] perceive the world more clearly,” and he suggested that gaming research “showed no evidence of reduced attention spans compared to non-gamers.”

More recently, the New York Times blogger Nick Bilton, in his 2010 book I Live in the Future, also suggested that videogaming improves attentiveness as well as visual acuity and concluded that “the findings argue for more game playing.” The science writer Jonah Lehrer last year argued that videogaming leads to “significant improvements in performance on various cognitive tasks,” including not only “visual perception” but also “sustained attention” and even “memory.” In her forthcoming book Now You See It, Cathy N. Davidson, an English professor at Duke, devotes a chapter to video game research, celebrating a wide array of apparent cognitive benefits, particularly in the area of attentiveness. Quoting Green and Bavelier, Davidson notes, for example, that “game playing greatly increases ‘the efficiency with which attention is divided.'”

The message is clear and, for those of us with a fondness for games, reassuring: Fire up the Xbox, grab the controller, and give the old gray matter a workout. The more you play, the smarter you’ll get.

If only it were so. The fact is, such broad claims about the cognitive benefits of video games, and by extension other digital media, have always been dubious. They stretch the truth. The mental faculties of attention and memory have many different facets – neuroscientists are still a long way from hashing them out – and to the extent that past gaming studies demonstrate improvements in these areas, they relate to gains in the kinds of attention and memory used in the fast-paced processing of a welter of visual stimuli. If you improve your ability to keep track of lots of images flying across a screen, for instance, that improvement can be described as an improvement in a type of attentiveness. And if you get better at remembering where you are in a complex fantasy world, that improvement can be described as an improvement in a sort of memory. The improvements may well be real – and that’s good news – but they’re narrow, and they come with costs. The fact that video games seem to make us more efficient at dividing our attention is great, as long as you’re doing a task that requires divided attention (like playing a video game). But if you’re actually trying to do something that demands undivided attention, you may find yourself impaired. As UCLA developmental psychologist Patricia Greenfield, one of the earliest researchers on video games, has pointed out, using media that train your brain to be good at dividing your attention appears to make you less able to carry out the kinds of deep thinking that require a calm, focused mind. Optimizing for divided attention means suboptimizing for concentrated attention.

Recent studies back up this point. They paint a darker picture of the consequences of heavy video-gaming, particularly when it comes to attentiveness. Far from making us smarter, heavy gaming seems to be associated with attention disorders in the young and, more generally, with a greater tendency toward distractedness and a reduced aptitude for maintaining one’s focus and concentration. Playing lots of video games, these studies suggest, does not improve a player’s capacity for “sustained attention,” as Lehrer and others argue. It weakens it.

In a 2010 paper published in the journal Pediatrics, Edward L. Swing and a team of Iowa State University psychologists reported on a 13-month study of the media habits of some 1,500 kids and young adults. It found that “[the] amount of time spent playing video games is associated with greater attention problems in childhood and on into adulthood.” The findings indicate that the correlation between videogaming and attention disorders is at least equal to and probably greater than the correlation between TV-viewing and those disorders. Importantly, the design of the study “rules out the possibility that the association between screen media use and attention problems is merely the result of children with attention problems being especially attracted to screen media.”

A 2009 study by a different group of Iowa State researchers, published in Psychophysiology, investigated the effects of videogaming on cognitive control, through experiments with 51 young men, both heavy gamers and light gamers. The study indicated that videogaming has little effect on “reactive” cognitive control – the ability to respond to some event after it happens. But when it comes to “proactive” cognitive control – the ability to plan and adjust one’s behavior in advance of an event or stimulus – videogaming has a significant negative effect. “The negative association between video game experience and proactive cognitive control,” the researchers write, “is interesting in the context of recent evidence demonstrating a similar correlation between video game experience and self-reported measures of attention deficits and hyperactivity. Together, these data may indicate that the video game experience is associated with a decrease in the efficiency of proactive cognitive control that supports one’s ability to maintain goal-directed action when the environment is not intrinsically engaging.” Videogamers, in other words, seem to have a difficult time staying focused on a task that doesn’t involve constant incoming stimuli. Their attention wavers.

These findings are consistent with more general studies of media multitasking. In a much-cited 2009 paper in Proceedings of the National Academy of Sciences, for example, Stanford’s Eyal Ophir, Clifford Nass, and Anthony D. Wagner show that heavy media multitaskers demonstrate significantly less cognitive control than light multitaskers. The heavy multitaskers “have greater difficulty filtering out irrelevant stimuli from their environment” and are also less able to suppress irrelevant memories from intruding on their work. The heavy multitaskers were actually less efficient at switching between tasks – in other words, they were worse at multitasking.

So should people be prevented from playing video games? Not at all (though parents should monitor and restrict young kids’ use of the games). Moderate game-playing probably isn’t going to have any significant long-term cognitive consequences, either good or bad. Video-gaming is fun and relaxing, and those are good things. Besides, people engage in all sorts of pleasant, diverting pursuits that carry risks, from rock-climbing to beer-drinking (don’t mix those two), and if we banned all of them, we’d die of boredom.

What the evidence does show is that while videogaming might make you a little better at certain jobs that demand visual acuity under stress, like piloting a jet fighter or being a surgeon, it’s not going to make you generally smarter. And if you do a whole lot of it, it may well make you more distracted and less able to sustain your attention on a single task, particularly a difficult one. More broadly, we should be highly skeptical of anyone who draws on video game studies to argue that spending a lot time in front of a computer screen strengthens our attentiveness or our memory or even our ability to multitask. Taken as a whole, the evidence, including the videogaming evidence, suggests it has the opposite effect.

Q: How do people cook these days?

A: They cook with Google.

When you’re looking for a good recipe today, you probably don’t reach for Joy of Cooking or Fannie Farmer or some other trusty, soup-stained volume on your cookbook shelf. You probably grab your laptop or tablet and enter the name of a dish or an ingredient or two into the search box. And that makes Google very important in the world of eating. Very, very important. I’ll let Amanda Hesser, noted food-writer, cookbook-author, and web-entrepreneur, explain:

The entity with the greatest influence on what Americans cook is not Costco or Trader Joe’s. It’s not the Food Network or The New York Times. It’s Google. Every month about a billion of its searches are for recipes. The dishes that its search engine turns up, particularly those on the first page of results, have a huge impact on what Americans cook.

Once upon a time, Google didn’t distinguish recipe search results from any other sort of search result. You typed in, say, “cassoulet,” and that keyword ran like any other keyword through the old Google link-counting algorithm. Recipes that had earned a lot of links from a lot of good sites appeared at the top of the list of results. But then, about a month ago – on February 24, 2011, to be precise – Google rolled out a special algorithm for finding recipes. And it added a “Recipe” button to the list of specialized search options that run down the left side of its search results pages. And it allowed searchers to refine results by ingredient, calories, or cooking time.

On the surface, all these changes seemed to be good news for cooks. What’s not to like about a specialized recipe search engine? Beneath the surface, though, some funny things were going on, and not all of them were salubrious. In fact, the changes illustrate how, as search engines refine their algorithms, their results become more biased. In particular, the changes reveal how a powerful search engine like Google has come to reward professional sites that are able to spend a lot on search engine optimization, or SEO, and penalize amateurs who are simply looking to share their thoughts with the world. Originally celebrated for leveling the media playing field, the Web has come to re-tilt that field to the benefit of deep-pocketed corporations.

Let’s look at the actual effects that Google’s changes have had on the kind of sites that show up in recipe search results. I’ll let Meathead Goldwyn, proprietor of a barbecue website and self-described “hedonism evangelist,” take up the story:

When one enters “ribs” in Google, my website AmazingRibs.com is #1. [But] if you search for “ribs” and then click on the new “Recipes” option in the column on the left on most browsers, the results are limited to only those that Google is sure are recipes and not articles about some football player with broken ribs. My ribs recipes are nowhere in sight. How does Google know a recipe when it sees one? The authors have included code that tells Google “this is a recipe.” … Handy for consumers, but a pain for food bloggers like me. I’m getting smashed because I did not get around to installing the new recipe codes when Google announced them in April 2010 because the instructions were too confusing. Now the top slots are all occupied by the big-time corporate food sites, Foodnetwork.com, Epicurious.com, About.com, AllRecipes.com, etc.

If you’re publishing recipes online and you want them to rank highly in Google’s recipe results, it’s no longer enough simply to publish really good dishes and get lots of people to link to them. Now, you have to be adept at (or hire someone who’s adept at) SEO in order to code your pages in ways suited to Google’s increasingly complex algorithm. If you want to get a sense of how complicated this is, you can check out this page at Google’s Webmaster Central, which describes how the publisher of a food site needs not only to tag a page as a recipe but to put various “microdata,” “microformats,” and “RDFa” tags into the source code of their pages. As Meathead notes, the page “was obviously written by engineers for engineers.” Here’s an eye-boggling sample that Google provides for a recipe called Grandma’s Holiday Apple Pie:

It may be Grandma’s apple pie, but I don’t think Grandma is going to be able to crank out that kind of coding. And I don’t think Google’s explanation of how the coding works is going to be much help to the old gal:

• On the first line, <itemscope itemtype="http://www.data-vocabulary.org/Recipe"> indicates that the HTML enclosed in the <div> represents a Recipe. itemscope indicates that the content of the <div> describes an item, and itemtype="http://www.data-vocabulary.org/Recipe" indicates that the item is a Recipe.
• The sample describes properties of the recipe, such as its author, ingredients, and preparation time. To label recipe properties, each element containing one of these properties (such as <div> or <span> is assigned an itemprop attribute indicating a property. For example, <span itemprop="author">.
• A property can consist of another item (in other words, an item can include other items). For example, the recipe above includes an Review-aggregate item (itemtype="http://www.data-vocabulary.org/Review-aggregate") with the properties rating and count, and a Recipe-ingredient item (ingredient), which in turn has the properties amount and name.

No, Grandma is out of luck.

And that’s the point. As Google’s army of codesmiths – with the best of intentions, I’m sure – make the company’s search algorithms ever more complex, ever more “refined,” the art of creating pages that will rank highly becomes ever more a job for professionals, for SEOers who spend all their time analyzing Google’s arcane instructions and mastering the esoteric codes those instructions demand. Amateurs and small-timers, like Grandma and Meathead, have little chance to compete with the big corporate sites, which can afford to spend big bucks on SEO. Once antagonists, Google and the SEO industry have developed a tightly symbiotic relationship that seems to be mutually beneficial. The folks who lose out are the little guys.

Here’s Amanda Hesser again:

Google has, in effect, taken sides in the food war. Unfortunately, it’s taken the wrong one … Imagine the blogger who has excellent recipes but has to compete against companies with staff devoted entirely to S.E.O. And who now must go back and figure out the calorie counts of all of his recipes, and then add those numbers, along with other metadata. That’s not going to happen. So the chance that that blogger’s recipes will appear anywhere near the first page of results is vanishingly small. What this means is that Google’s search engine gives vast advantage to the largest recipe websites with the resources to input all this metadata.

But that’s not all. Other biases – these having to do with Google’s idea of what people should be cooking and eating – are also at work. In setting up parameters for refining results based on cooking time and calories, Google explicitly, if subtly, gives privilege to low-calorie recipes that can be cooked quickly, as shown in the options it allows for refining a recipe search:

Those choices may seem innocuous, but they have important consequences, as Hesser describes:

Google unwittingly – but damagingly – promotes a cooking culture focused on speed and diets.

Take, for instance, a recent search for “cassoulet.” The top search result is a recipe from Epicurious, one of the larger and better sites. But if you refine by time, your choices are “less than 15 min,” “less than 30 min,” or “less than 60 min.” There is no option for more than 60 minutes. In truth, a classic cassoulet takes at least 4 hours to make, if not several days (the Epicurious recipe takes 4 hours and 30 minutes; yet there in the results are recipes under each of these three time classes. One from Tablespoon goes so far as to claim to take just 1 minute. (It’s made with kidney beans, canned mushrooms, and beef, so it’s not long on authenticity.) … Refining recipe search by time doesn’t result in better recipes rising to the top; rather, the new winners are recipes packaged for the American eating and cooking disorder.

The proof is no longer in the pudding. It’s in the search results. And baked into those results are the biases, ideologies, and business interests of the people running the search engines. The code is not neutral.