Britannica’s indictment

In February, I took a look at the journal Nature’s much-publicized comparison of the accuracy of Wikipedia and Encyclopedia Britannica. I concluded that there was much less to the study than originally met the eye. Now, as the Register reports, the editors of the Britannica have issued their own critique of the Nature analysis. They don’t mince words:

Nature’s research was invalid. As we demonstrate below, almost everything about the journal’s investigation, from the criteria for identifying inaccuracies to the discrepancy between the article text and its headline, was wrong and misleading. Dozens of inaccuracies attributed to the Britannica were not inaccuracies at all, and a number of the articles Nature examined were not even in the Encyclopædia Britannica. The study was so poorly carried out and its findings so error-laden that it was completely without merit. We have produced this document to set the record straight, to reassure Britannica’s readers about the quality of our content, and to urge that Nature issue a full and public retraction of the article.

Even allowing for the bias of the source, the Britannica’s assessment shows that the Nature study was, at best, sloppy. In a number of cases, the Britannica entries used by Nature weren’t even actual Britannica entries. They were lifted from other publications, such as the Britannica yearbook or the student version of the encyclopedia, or they were short excerpts of much longer entries, or they were “patchworks of text taken from two or more articles and pieced together in a way that made a mockery of the original entries.”

What’s most troubling, though, is that Nature appears to be withholding the full documentation of its analysis. “We contacted Nature,” write the Britannica editors, “asking for the original data, calling their attention to several of their errors, and offering to meet with them to review our findings in full, but they declined.” Given the broad attention paid to Nature’s conclusion that “Wikipedia comes close to Britannica in terms of the accuracy of its science entries,” Nature needs to set the record straight.

UPDATE: Nature has posted a response to Britannica’s criticisms. It says, in part:

The [Britannica] company claims that our article gave a misleading impression of Encyclopaedia Britannica’s accuracy. Specifically, the company … objects to the fact that in some cases we took material from Britannica’s Book of the Year and its Student Encyclopedia. This was done in a few cases when the Britannica website provided articles from these sources when queried on the pre-determined topics; as we said, the survey compared the content of the websites. In a small number of cases, to ensure comparable lengths, we provided reviewers with chosen excerpts, not full articles; this was done with entries from both Encyclopaedia Britannica and Wikipedia. In one instance Britannica alleges that we provided a reviewer with material that was not from the Britannica website. We have checked and are confident that this was not the case.

It concludes: “We do not intend to retract our article.”

10 thoughts on “Britannica’s indictment

  1. Gianni

    Sorry, this discussion is – at best – irrelevant; the bottom line is Britannica is static, Wikipedia is dynamic.

    Photos versus Cinema – two different animals

  2. Dr P Miller

    The scariest aspect of Sam and Gianni’s comments is that they don’t care about data integrity. Or the consequences of using unreliable information.

    In my professional experience, people like this are dangerous and to be avoided all costs.

  3. RET

    The comment that I took away from the article on El Reg was this:

    What do these seemingly disparate projects have in common? The idea that you can vote for the truth.

    A more succinct encapsulation of the problem with Wikipedia I am yet to read.

    cheers, RET

  4. Gianni

    Dr. Miller,

    I could not agree more on the importance of data integrity.

    I am saying that IMHO, given an adequate timespan, a dynamic model will ALWAYS deliver better quality results.

    The errors found on Wikipedia by Nature are long gone, the ones in Britannica re still there.

  5. Topper

    Yes, the errors found on Wikipedia by Nature are all gone, replaced by thousands of new dynamically created mistakes…

  6. Johan Sundström

    It would seem (whether explicitly stated so or not in the Nature article) that the original review was comparing the freely available online content of both works, rather than being a Wikipedia versus Britannica (bound version), which the Britannica critique addresses.

  7. H.A. Page

    The comments by Carr’s readers in this space are the most insightful and relevant comments to be found regarding the debate. We tried to have Jimmy Wales participate in our graduate class discussion on Wikipedia but he was too busy…

    We’ve looked at this issue and I’ll be adding more beyond this.

    Appreciate your (collaborative) thinking here.

    Cheers – Hattie

    new media and motherhood at motherpie.com

Comments are closed.