This text was originally provided by the Encyclopedia Britannica in PDF format (here). PDFs aren't web-friendly, so I've translated it into HTML and added a table of contents (and put the appendices, which are really long, on a separate page).
Britannica holds the copyright, so they can ask me to take it down (though I hope they don't).
– Ben Yates, March 2006
Fatally Flawed
Encyclopaedia Britannica, Inc., March 2006
Contents
- Main Section
- Appendix A: The Original Data
- Appendix B: Nature's Errors
Refuting the recent study on encyclopedic accuracy by the journal Nature
In its December 15, 2005, issue, the science journal Nature published an article that claimed to compare the accuracy of the online Encyclopaedia Britannica with Wikipedia, the Internet database that allows anyone, regardless of knowledge or qualifications, to write and edit articles on any subject.1 Wikipedia had recently received attention for its alleged inaccuracies,2 but Nature's article claimed to have found that "such high-profile examples [of major errors in Wikipedia] are the exception rather than the rule" and that "the difference in accuracy [between Britannica and Wikipedia] was not particularly great."
Arriving amid the revelations of vandalism and errors in Wikipedia, such a finding was, not surprisingly, big news. Within hours of the article's appearance on Nature's Web site, media organizations worldwide proclaimed that Wikipedia was almost as accurate as the oldest continuously published reference work in the English language.3
That conclusion was false, however, because Nature's research was invalid. As we demonstrate below, almost everything about the journal's investigation, from the criteria for identifying inaccuracies to the discrepancy between the article text and its headline, was wrong and misleading. Dozens of inaccuracies attributed to the Britannica were not inaccuracies at all, and a number of the articles Nature examined were not even in the Encyclopaedia Britannica. The study was so poorly carried out and its findings so error-laden that it was completely without merit. We have produced this document to set the record straight, to reassure Britannica's readers about the quality of our content, and to urge that Nature issue a full and public retraction of the article.
In rebutting Nature's work, we in no way mean to imply that Britannica is error-free; we have never made such a claim. We have a reputation not for unattainable perfection but for strong scholarship, sound judgment, and disciplined editorial review. These practices are the foundation of any reliable reference work, and Nature's careless analysis demeaned them.
Britannica undergoes continuous revision and fact checking. Our editors work unceasingly to revise and improve the encyclopedia and to publish the results in a timely way. We work with thousands of contributors and advisers around the world--scholars and experts all--and maintain a brisk correspondence with our readers as well.4 We investigate any claims of error that come to our attention, and when one is valid, we fix the error. Where Nature's reviewers found genuine inaccuracies or important omissions in the Britannica, we have corrected them, but as a work of research from which conclusions may be drawn, Nature's study was without value. The purpose of this document is to enumerate the scores of serious errors and misjudgments that undermine Nature's study so that its lack of validity can be understood.
Nature accused Britannica of "omissions" on the basis of reviews of article excerpts, not the articles themselves. In a number of cases only parts of the applicable Britannica articles were reviewed.
One Nature reviewer was sent only the 350-word introduction to Encyclopaedia Britannica's 6,000-word article on lipids. For Nature to have represented Britannica's extensive coverage of the subject with this short squib was absurd, and it invalidated the findings of omissions alleged by the reviewer, since those matters were covered in sections of the article he or she never saw.
Other reviewers were sent only sections taken from longer articles. For example, what the Nature editors referred to as Britannica's "articles" on "kin selection" and "punctuated equilibrium" are actually separate sections of our article on the theory of evolution, written by one of the foremost experts on evolution in the world. What they claimed to be an "article" on field-effect transistors was actually only one section of our article on integrated circuits. For Nature to have excerpted our articles in this way was irresponsible.
Nature rearranged and re-edited Britannica articles. In some cases reviewers were sent patchworks of text taken from two or more articles and pieced together in a way that made a mockery of the original entries. The "article" on "aldol reaction" that the journal sent its reviewer consisted of passages taken selectively from two different Encyclopaedia Britannica articles and joined together with text evidently written by Nature's editors. This was dishonest, and it completely misrepresented Britannica's published coverage of the subject.
Nature failed to check the factual assertions of its reviewers. Everyone makes mistakes, even experts, which is why all factual assertions should be verified. Nature, however, decided not to do this; they did not even require reviewers to provide sources for their assertions because "it would have been simply too time consuming."6 Instead, Nature assumed that what its reviewers said was true and, when it contradicted something in the Britannica, that the reviewer was right and Britannica was wrong. The result was predictable.7 Examples:
Nature's review of the Britannica article "Pythagorean theorem" claimed the Britannica misspelled an Italian town that the reviewer said should be spelled "Crotona." However, according to the U.S. Board of Geographic Names, the preferred spelling is in fact "Crotone," as Britannica spelled it. Other reliable sources also give "Crotone" as the right spelling. For Nature's editors to have ruled this an error on one reviewer's say-so, without confirming the spelling, was inexcusable.
For the article on Subrahmanyan Chandrasekhar, Nature's reviewer quibbled with the dates Britannica gave for one of Professor Chandrasekhar's academic appointments and for the publication of his book Principles of Stellar Dynamics. Once again, on the basis of these unverified objections, Nature ruled Britannica to be in error, but we have solid sources for both dates, and we stand by them. See Appendix B for details.
Nature failed to distinguish minor inaccuracies from major errors. By counting up the alleged inaccuracies for both encyclopedias, Nature treated all mistakes equally and failed to observe that Wikipedia had many more shortcomings of a fundamental kind. Reviewers told the journal that many of the Wikipedia articles were "poorly structured and confusing"--a fact that made those arti
cles resistant to basic fact checking and thus not suitable for comparison with Britannica--yet the fact was buried deep in the article and its methodological implications ignored.
Nature counted "errors" and "critical omissions" that did not exist. As we have said, where reviewers found genuine inaccuracies in the Britannica, we corrected them, but dozens of the so-called inaccuracies they attributed to us were nothing of the kind; they were the result of reviewers expressing opinions that differed from ours about what should be included in an encyclopedia article. In these cases Britannica's coverage was actually sound, most often reflecting the considered judgment of editors who have extensive experience in publishing an encyclopedia, in deciding what information should be included in it for the general reader, and in how it should be presented. Nature's reviewers--who were scientists, not encyclopedists--were certainly entitled to their opinions on these matters, but for Nature to have ascribed inaccuracies to Britannica simply on the basis of their opinions was invalid. For example:
According to the reviewer of the Britannica article "Nobel Prize," the fact that the 1935 Peace Prize was awarded to Carl von Ossietzky in 1936 should have been stated. We disagree. To raise this minor fact and then be required to explain it would, at that point in the Nobel Prize article, have distracted the reader from the main discussion. This was not an omission; it was an editorial judgment. (N.B.: The fact that the award was conferred in 1936 is reported in Ossietzky's biographical article, where it belongs.)
The reviewer of the article on Paul Dirac objected that certain areas of Dirac's work were not covered. Our coverage, however, was appropriate for a general-reference encyclopedia. By design, the 825-word article explained, for the lay reader, Dirac's most significant contributions, not all of them.
The reviewer of the article on the Haber-Bosch process suggested that the article should have shown the chemical equation for the reaction, and Nature therefore called its absence from the article an omission. Not so. The article's verbal description of the process was clear and sufficient for the general reader.
Conclusion
"No test is perfect and we acknowledge that any of our reviewers could themselves have made occasional errors," said the editors of Nature about their study. "But by choosing reviewers who were highly qualified in the specific area described by each entry, we aimed to subject the encyclopaedia entries to the fairest and most stringent test that we could."8
Indeed, perfection is unattainable, and a few mistakes in Nature's analysis could perhaps have been overlooked. But alas, as we have shown above, the study's shortcomings went well beyond "occasional errors." The entire undertaking--from the study's methodology to the misleading way Nature "spun" the story--was misconceived. Among other things, while it is important to engage "reviewers who were highly qualified," that alone is not enough: as every editor at Britannica knows, even the assertions of experts must be confirmed and their observations considered in the proper context for a general-reference work like Britannica. Nature failed to take these and so many of the other steps that would have been required to make its research valid. The results were the errors we describe above and in Appendix B.
We now call on Nature to fulfill its commitment to good scholarship and send us the unabridged reviewer reports on which the study was based. And as we have shown here, the facts call for a complete retraction of the study and the article in which it was reported. We call on Nature to make the retraction and make it promptly.
Notes
1Jim Giles, "Internet encyclopaedias go head to head," Nature, December 15, 2005: 900-01.
2John Seigenthaler, "A false Wikipedia 'biography,'" USA Today, November 29, 2005 (Link). "Free Encyclopedia Wikipedia.de Has Copyright Issues," DW-World/Deutsche Welle, November 29, 2005 (Link). "Wikipedia Caught in Podfather Turf War," Podcasting News, December 5, 2005 (Link). All accessed March 7, 2006.
3Dan Goodin, "Wikipedia Science Topics As Accurate As Britannica-Report," Associated Press, December 14, 2005. Gregory M. Lamb, "Online Wikipedia is not Britannica - but it's close," Christian Science Monitor, January 5, 2006 (Link). "Wikipedia Gets Things Right," Red Herring, December 14, 2005 (Link). "Assessing Wikipedia's Accuracy," All Things Considered, December 15, 2005 (Link). Julian Dibbell, "Factually Speaking," Village Voice, December 22, 2005 (Link). "Fact or fiction? Online encyclopedias put to the test," The Age, December 15, 2005 (Link). All accessed March 7, 2006.
4See Michael J. McCarthy, "It's Not True About Caligula's Horse; Britannica Checked --- Dogged Researchers Answer Some Remarkable Queries," Wall Street Journal, April 22, 1999: A1.
5"Supplementary information to accompany Nature news article 'Internet encyclopaedias go head to head'" (Link). Accessed February 23, 2006.
6"Supplementary information," 2.
7Nature's editors appear to have thought that since their reviewers were scientists, those reviewers' judgments trumped the encyclopedia where there was a disagreement. This was presumptuous, since Britannica engages expert advisers and contributors who are in every way equal to Nature's reviewers. Our distinguished contributors have included more than 110 Nobel laureates.
8"Supplementary information," 1.