The quality of peer production

In an important essay at First Monday, Paul Duguid takes a hard and rigorous look at whether, and to what extent, web-based peer production can produce quality work.

“Two ideas are often invoked,” he writes, “to defend the quality of peer production.” The first of these “laws of quality,” borrowed from open-source software development, is Linus’s Law: “Given enough eyeballs, all bugs are shallow.” Duguid points out, as others have before him, that applying this law outside the sphere of software production is problematic. The “stern gatekeeper” of software quality – the program has to run – is absent in the production of most cultural works. The second law of quality is what Duguid dubs “Graham’s Law,” after Paul Graham, who “claims that ‘The method of ensuring quality’ in peer production is ‘Darwinian … People just produce whatever they want; the good stuff spreads, and the bad gets ignored.'” This law, too, is problematic, argues Duguid. It reflects

an optimistic faith that the “truth will conquer.” While this optimism has roots in Milton’s Areopagitica, it is perhaps a particularly American, democratic belief, enshrined in the First Amendment. Such optimism no doubt makes good political principle, but it does not dictate political process. Freedom of speech is not the same as the freedom to replace other’s versions of the truth with your own. The authors of the U.S. Constitution and the Bill of Rights may have believed that open debate leads to political truth, [but] they did not believe that the Constitution would improve were it changed at the whim of each citizen’s changing view of truth. Consequently, the U.S. Constitution has significant built–in inertia … As this example may suggest, Graham’s implication that continuous tinkering only makes things better is highly suspect. It is hard to see why entropy would be indefinitely suspended by peer production. In areas of “cultural production,” in particular, progress is not necessarily linear, and neither the latest (nor the earliest) version of a work [is] always the best …

Rather than taking the laws on faith, we need to ask in which cases the laws work, in which they do not, and if they do not, why not. So we need to look at cases where the laws have failed to work and then to ask — in general, systemic rather than individual, particularistic terms — why.

Duguid goes on to examine the products of two well-established and seemingly fairly simple peer-production processes on the Internet: Gracenote (for compiling information about the contents of compact disks) and Project Gutenberg (for publishing online versions of out-of-copyright texts). While finding the products of both these projects “immensely useful,” he also documents, painstakingly, that they have deep and persistent flaws: “both suffer from problems of quality that are not addressed by what I have called the laws of quality – the general faith that popular sites that are open to improvement [will] iron out problems and continuously improve.”

He then examines Wikipedia, a more complex project in peer production. He documents the many flaws that bedevil the online encyclopedia, from plagiarism to the use of unreliable sources to sloppy writing, and shows how Wikipedia, “despite its creed of continuous improvement, can defy Graham’s Law” and evolve toward lower rather than higher quality as edits pile up. Duguid focuses his analysis on two entries, for the early English novelists Laurence Sterne and Daniel Defoe, and he admits that these entries “do appear to fall into a backwater of Wikipedia. Thus it may seem unfair to choose these as examples to illustrate aspects of the whole.” But he then makes a crucial point about assessing an encyclopedia’s quality:

I suggested earlier, however, that judging overall quality from the less– rather than the more–frequented parts, the weak rather than the strong links, is not a bad idea. After all, how is the ordinary user to know when he or she has landed in a backwater? With Linus’s Law in mind, we should acknowledge that the eyeballs that consult encyclopedia entries are, in the default case, quite unlike those beta testing or developing code and quite unsuited to recognizing or characterizing any but the most obvious errors. To use an Open Source program is in itself often an acknowledgment of a certain level of skill. To turn to the encyclopedia is, by contrast, more likely a confession of ignorance. If I want or need to find out about Defoe, then I’m not likely to be in a position to critique an entry on him.

“Editing,” Duguid writes, “is a hard task and needs to attract people prepared to think through the salient issues. Wikipedia is very sensitive to malice. It needs to be as sensitive to ineptitude.”

In the end, Duguid concludes that the two “laws of quality” underpinning today’s peer-production projects are insufficient. “If we are to rely on peer production in multiple different spheres of information production,” he says, “we need to look for other ways to assure quality.” But one comes away from this excellent paper wondering whether, once these “other ways” of quality assurance are imposed on a process, it would still qualify as “peer production.” As Duguid eloquently demonstrates, quality doesn’t just happen; it’s not an emergent phenomenon. It’s imposed on a work by people who know what they’re doing. Quality – true quality – may thus be incompatible with the democratic ideal that lies at the heart of what we call peer production.

4 thoughts on “The quality of peer production

  1. Brad

    It is fitting that eyeballs are mentioned, because the problem for the “emergent quality” crowd is exactly the difference between seeing, which is automatic, and thinking, which takes two things to succeed: the individual’s volitional effort, and his or her cognitive context — which is just the sum of prior effort.

    The idea that democratic debate leads to truth probably has more to do with Dewey than the Founding Fathers, who consciously defined a republic instead of a democracy.

  2. mazedesigner

    Duguid makes the point, “… in trying to estimate the qualities of peer production systems, we should not infer the quality of its quieter backwaters from the gross numbers of users.” I wonder if there could be a more complex way to infer the quality of an article based on the behavior of the contributors. What research has been done to identify the patterns that are commonly associated with articles that are deemed to have high quality? Perhaps Wikipedia could dynamically flag articles with seemingly suspect contributor patterns. Clearly this would never be perfect, but maybe it could help users know when they have “landed in a backwater.”

  3. Graham Hill


    The history of the quality movement shows that quality is in the eye of the beholder. Quality is inextricably linked with being fit for purpose.

    If you need a simple overview of something new to you – let’s say, “Social Networks” – Wikipedia will probably serve your purpose. It’s quality is good enough as defined by most users.

    If you need much more detail, Wikipedia will not serve your purpose, but it will provide you with references – in this case to Scott and Wassermann & Faust’s standard textbooks – that contain all the material you are likely to need. Although not of high quality per se, the reader is shown where to go for the quality they desire.

    The prudent reader will look at several sources for information, not just rely upon a single source, no matter how expert the source claims to be. This applies to searches in wikipedia, traditional encyclopedias and of course, to expert opinion of all types.

    Graham Hil

  4. jesse ezell

    The statement:

    “Hence, while Open Source software has relied heavily on peer production and to a lesser extent on peer review, for quality, it relies as heavily though perhaps less obviously, on the chip and the compiler as ultimate arbiters.”

    Shows a complete lack of understanding of how software is developed. Bugs are not found by the compiler. Compiler errors result in the build breaking and the program not compiling. However, these are generally just syntax errors and take seconds to fix. In any case, these aren’t bugs because you cannot possibly ship the code until it compiles, and it won’t compile unless you fix them. People still end up being the ultimate arbiter, deciding what is and is not a bug. Compilers are extremely limited and don’t really help at all.

Comments are closed.