« Alien territory | Main | The hitmaking machine »
Britannica's indictment
March 23, 2006
In February, I took a look at the journal Nature's much-publicized comparison of the accuracy of Wikipedia and Encyclopedia Britannica. I concluded that there was much less to the study than originally met the eye. Now, as the Register reports, the editors of the Britannica have issued their own critique of the Nature analysis. They don't mince words:
Nature’s research was invalid. As we demonstrate below, almost everything about the journal’s investigation, from the criteria for identifying inaccuracies to the discrepancy between the article text and its headline, was wrong and misleading. Dozens of inaccuracies attributed to the Britannica were not inaccuracies at all, and a number of the articles Nature examined were not even in the Encyclopædia Britannica. The study was so poorly carried out and its findings so error-laden that it was completely without merit. We have produced this document to set the record straight, to reassure Britannica’s readers about the quality of our content, and to urge that Nature issue a full and public retraction of the article.
Even allowing for the bias of the source, the Britannica's assessment shows that the Nature study was, at best, sloppy. In a number of cases, the Britannica entries used by Nature weren't even actual Britannica entries. They were lifted from other publications, such as the Britannica yearbook or the student version of the encyclopedia, or they were short excerpts of much longer entries, or they were "patchworks of text taken from two or more articles and pieced together in a way that made a mockery of the original entries."
What's most troubling, though, is that Nature appears to be withholding the full documentation of its analysis. "We contacted Nature," write the Britannica editors, "asking for the original data, calling their attention to several of their errors, and offering to meet with them to review our findings in full, but they declined." Given the broad attention paid to Nature's conclusion that "Wikipedia comes close to Britannica in terms of the accuracy of its science entries," Nature needs to set the record straight.
UPDATE: Nature has posted a response to Britannica's criticisms. It says, in part:
The [Britannica] company claims that our article gave a misleading impression of Encyclopaedia Britannica’s accuracy. Specifically, the company ... objects to the fact that in some cases we took material from Britannica’s Book of the Year and its Student Encyclopedia. This was done in a few cases when the Britannica website provided articles from these sources when queried on the pre-determined topics; as we said, the survey compared the content of the websites. In a small number of cases, to ensure comparable lengths, we provided reviewers with chosen excerpts, not full articles; this was done with entries from both Encyclopaedia Britannica and Wikipedia. In one instance Britannica alleges that we provided a reviewer with material that was not from the Britannica website. We have checked and are confident that this was not the case.
It concludes: "We do not intend to retract our article."
Advertisement: Are you ready for "The Big Switch"? Fast Company calls Nicholas Carr's new book "compulsively readable - for nontechies, too." Salon says it's "magisterial." Order now from Amazon.com.
Comments
Tempest in a teapot. Why? Wikipedia's good enough for me.
Posted by: Sam Hiser at March 23, 2006 12:42 PM
Sorry, this discussion is - at best - irrelevant; the bottom line is Britannica is static, Wikipedia is dynamic.
Photos versus Cinema - two different animals
Posted by: Gianni at March 23, 2006 01:09 PM
The scariest aspect of Sam and Gianni's comments is that they don't care about data integrity. Or the consequences of using unreliable information.
In my professional experience, people like this are dangerous and to be avoided all costs.
Posted by: Dr P Miller at March 23, 2006 07:53 PM
The comment that I took away from the article on El Reg was this:
What do these seemingly disparate projects have in common? The idea that you can vote for the truth.
A more succinct encapsulation of the problem with Wikipedia I am yet to read.
cheers, RET
Posted by: RET at March 23, 2006 09:53 PM
Dr. Miller,
I could not agree more on the importance of data integrity.
I am saying that IMHO, given an adequate timespan, a dynamic model will ALWAYS deliver better quality results.
The errors found on Wikipedia by Nature are long gone, the ones in Britannica re still there.
Posted by: Gianni at March 24, 2006 05:41 AM
Yes, the errors found on Wikipedia by Nature are all gone, replaced by thousands of new dynamically created mistakes...
Posted by: Topper at March 25, 2006 02:33 AM
It would seem (whether explicitly stated so or not in the Nature article) that the original review was comparing the freely available online content of both works, rather than being a Wikipedia versus Britannica (bound version), which the Britannica critique addresses.
Posted by: Johan Sundström at March 26, 2006 07:29 PM
That's like comparing a fresh bunch of grapes & raisins
Posted by: Anonymous at March 27, 2006 06:43 PM
The comments by Carr's readers in this space are the most insightful and relevant comments to be found regarding the debate. We tried to have Jimmy Wales participate in our graduate class discussion on Wikipedia but he was too busy...
We've looked at this issue and I'll be adding more beyond this.
Appreciate your (collaborative) thinking here.
Cheers - Hattie
new media and motherhood at motherpie.com
Posted by: H.A. Page at March 29, 2006 02:33 PM
The Wikipedia article on Puppets Who Kill was more complete and balanced than that of Britannica
http://en.wikipedia.org/wiki/Puppets_Who_Kill
Posted by: Eric Stratton at March 29, 2006 10:51 PM
Post a comment
Thanks for signing in, . Now you can comment. (sign out)
(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)
Nick's new book:
"Future Shock for the web-apps era" -Fast Company
"Ominously prescient" -Kirkus Reviews
"Riveting stuff" -New York Post
Greatest hits
Avatars consume as much electricity as Brazilians
The love song of J. Alfred Prufrock's avatar
Other writing
The end of corporate computing
Nick's last book:
Order from Amazon
Visit book site