In February, I took a look at the journal Nature’s much-publicized comparison of the accuracy of Wikipedia and Encyclopedia Britannica. I concluded that there was much less to the study than originally met the eye. Now, as the Register reports, the editors of the Britannica have issued their own critique of the Nature analysis. They don’t mince words:
Nature’s research was invalid. As we demonstrate below, almost everything about the journal’s investigation, from the criteria for identifying inaccuracies to the discrepancy between the article text and its headline, was wrong and misleading. Dozens of inaccuracies attributed to the Britannica were not inaccuracies at all, and a number of the articles Nature examined were not even in the Encyclopædia Britannica. The study was so poorly carried out and its findings so error-laden that it was completely without merit. We have produced this document to set the record straight, to reassure Britannica’s readers about the quality of our content, and to urge that Nature issue a full and public retraction of the article.
Even allowing for the bias of the source, the Britannica’s assessment shows that the Nature study was, at best, sloppy. In a number of cases, the Britannica entries used by Nature weren’t even actual Britannica entries. They were lifted from other publications, such as the Britannica yearbook or the student version of the encyclopedia, or they were short excerpts of much longer entries, or they were “patchworks of text taken from two or more articles and pieced together in a way that made a mockery of the original entries.”
What’s most troubling, though, is that Nature appears to be withholding the full documentation of its analysis. “We contacted Nature,” write the Britannica editors, “asking for the original data, calling their attention to several of their errors, and offering to meet with them to review our findings in full, but they declined.” Given the broad attention paid to Nature’s conclusion that “Wikipedia comes close to Britannica in terms of the accuracy of its science entries,” Nature needs to set the record straight.
UPDATE: Nature has posted a response to Britannica’s criticisms. It says, in part:
The [Britannica] company claims that our article gave a misleading impression of Encyclopaedia Britannica’s accuracy. Specifically, the company … objects to the fact that in some cases we took material from Britannica’s Book of the Year and its Student Encyclopedia. This was done in a few cases when the Britannica website provided articles from these sources when queried on the pre-determined topics; as we said, the survey compared the content of the websites. In a small number of cases, to ensure comparable lengths, we provided reviewers with chosen excerpts, not full articles; this was done with entries from both Encyclopaedia Britannica and Wikipedia. In one instance Britannica alleges that we provided a reviewer with material that was not from the Britannica website. We have checked and are confident that this was not the case.
It concludes: “We do not intend to retract our article.”