This text was originally provided by the Encyclopedia Britannica in PDF format (here). PDFs aren't web-friendly, so I've translated it into HTML and added a table of contents (and put the appendices, which are really long, on a separate page).

Britannica holds the copyright, so they can ask me to take it down (though I hope they don't).

Ben Yates, March 2006

Fatally Flawed

Encyclopaedia Britannica, Inc., March 2006

Contents

Refuting the recent study on encyclopedic accuracy by the journal Nature

In its December 15, 2005, issue, the science journal Nature published an article that claimed to compare the accuracy of the online Encyclopaedia Britannica with Wikipedia, the Internet database that allows anyone, regardless of knowledge or qualifications, to write and edit articles on any subject.1 Wikipedia had recently received attention for its alleged inaccuracies,2 but Nature's article claimed to have found that "such high-profile examples [of major errors in Wikipedia] are the exception rather than the rule" and that "the difference in accuracy [between Britannica and Wikipedia] was not particularly great."

Arriving amid the revelations of vandalism and errors in Wikipedia, such a finding was, not surprisingly, big news. Within hours of the article's appearance on Nature's Web site, media organizations worldwide proclaimed that Wikipedia was almost as accurate as the oldest continuously published reference work in the English language.3

That conclusion was false, however, because Nature's research was invalid. As we demonstrate below, almost everything about the journal's investigation, from the criteria for identifying inaccuracies to the discrepancy between the article text and its headline, was wrong and misleading. Dozens of inaccuracies attributed to the Britannica were not inaccuracies at all, and a number of the articles Nature examined were not even in the Encyclopaedia Britannica. The study was so poorly carried out and its findings so error-laden that it was completely without merit. We have produced this document to set the record straight, to reassure Britannica's readers about the quality of our content, and to urge that Nature issue a full and public retraction of the article.

In rebutting Nature's work, we in no way mean to imply that Britannica is error-free; we have never made such a claim. We have a reputation not for unattainable perfection but for strong scholarship, sound judgment, and disciplined editorial review. These practices are the foundation of any reliable reference work, and Nature's careless analysis demeaned them.

Britannica undergoes continuous revision and fact checking. Our editors work unceasingly to revise and improve the encyclopedia and to publish the results in a timely way. We work with thousands of contributors and advisers around the world--scholars and experts all--and maintain a brisk correspondence with our readers as well.4 We investigate any claims of error that come to our attention, and when one is valid, we fix the error. Where Nature's reviewers found genuine inaccuracies or important omissions in the Britannica, we have corrected them, but as a work of research from which conclusions may be drawn, Nature's study was without value. The purpose of this document is to enumerate the scores of serious errors and misjudgments that undermine Nature's study so that its lack of validity can be understood.

Misleading Headline

In the pages below we describe the errors Nature made in its study, but first a word about the misleading way in which it was presented.

Anyone who read the article with even a modicum of care would have noticed a discrepancy between the headline and the data themselves. While the heading proclaimed that "Wikipedia comes close to Britannica in terms of the accuracy of its science entries," the numbers buried deep in the body of the article said precisely the opposite: Wikipedia in fact had a third more inaccuracies than Britannica. (As we demonstrate below, Nature's research grossly exaggerated Britannica's inaccuracies, so we cite this figure only to point out the slanted way in which the numbers were presented.) Even if Wikipedia were "only" a third more inaccurate than Britannica, this would be a large difference, especially in a study that focused exclusively on factual accuracy, disregarding other important properties of encyclopedias, such as the organization of information, the quality of writing, and the readability of the articles. Why Nature tried to minimize this considerable difference in accuracy is unclear, but the fact is that Britannica was far more accurate than Wikipedia according to the figures; the journal simply misrepresented its own results.

As we would soon learn, however, this was only the beginning of the investigation's errors and misrepresentations. In the days after the article was published, Britannica's science editors, with the help of our outside advisers and contributors, began reviewing the list of inaccuracies the journal claimed to have found, with the aim of addressing every claim that had validity. We discovered in Nature's work a pattern of sloppiness, indifference to basic scholarly standards, and flagrant errors so numerous they completely invalidated the results. We contacted Nature, asking for the original data, calling their attention to several of their errors, and offering to meet with them to review our findings in full, but they declined.

The Study and the Data

According to Nature's description of its study, 42 pairs of articles on scientific subjects, from the Britannica and Wikipedia respectively, were reviewed by outside experts, mainly academic scientists, who were offered anonymity. (Most of them chose to remain anonymous.) According to a document posted on Nature's Web site, reviewers "were asked to look for three types of inaccuracy: factual errors, critical omissions and misleading statements. . . . The reviews were then examined by Nature's news team and the total number of errors estimated for each article."5

The reviewers produced reports for the articles assigned to them, and those reviews were excerpted in the document on the Web. However, contrary to the usual practice of making all data freely available in order to facilitate a study's replication by others, Nature declined our repeated requests to make the full reports available. For more on this, see Appendix A.

Even without the original data, however, Britannica editors analyzed Nature's findings, using the truncated versions of the reviewer reports posted on the Web and copies of the articles reviewed that we obtained from Nature. We identified a multitude of serious flaws in their procedures and conclusions and found that in dozens of cases Britannica information that Nature claimed to be inaccurate was not inaccurate at all. We review some of Nature's most significant errors below. A more complete list can be found in Appendix B.

Nature reviewed text that was not even from the Encyclopaedia Britannica. Several of the articles Nature sent its reviewers were not from our core encyclopedia, and in one case it was not from any Britannica publication at all.

Articles on Dolly the Sheep and Steven Wolfram reviewed by Nature were taken not from the Encyclopaedia Britannica but from previous editions of the Britannica Book of the Year, which are archived on our site and clearly dated and identified. Yearbook authors are often given greater latitude to express personal views than writers of encyclopedia articles. In the Wolfram article, the Nature reviewer disagreed with Britannica's author on the phrasing of two sentences in which point of view figured significantly, and on the basis of those disagreements Nature's editors counted the two points as "inaccuracies" in Britannica. In addition to the fact that reviewing yearbook articles in a study of encyclopedias is inappropriate, these particular judgments were simply unfounded. The reviewer was entitled to his or her opinion about how a point might best be presented, but that opinion did not make our author's presentation "inaccurate."

Nature's comments on the article "ethanol" were based on text not from the Encyclopaedia Britannica but from Britannica Student Encyclopedia, a more basic work for younger readers. One of the reviewer's comments referred to text that does not appear in any Britannica publication.

Nature accused Britannica of "omissions" on the basis of reviews of article excerpts, not the articles themselves. In a number of cases only parts of the applicable Britannica articles were reviewed.

One Nature reviewer was sent only the 350-word introduction to Encyclopaedia Britannica's 6,000-word article on lipids. For Nature to have represented Britannica's extensive coverage of the subject with this short squib was absurd, and it invalidated the findings of omissions alleged by the reviewer, since those matters were covered in sections of the article he or she never saw.

Other reviewers were sent only sections taken from longer articles. For example, what the Nature editors referred to as Britannica's "articles" on "kin selection" and "punctuated equilibrium" are actually separate sections of our article on the theory of evolution, written by one of the foremost experts on evolution in the world. What they claimed to be an "article" on field-effect transistors was actually only one section of our article on integrated circuits. For Nature to have excerpted our articles in this way was irresponsible.

Nature rearranged and re-edited Britannica articles. In some cases reviewers were sent patchworks of text taken from two or more articles and pieced together in a way that made a mockery of the original entries. The "article" on "aldol reaction" that the journal sent its reviewer consisted of passages taken selectively from two different Encyclopaedia Britannica articles and joined together with text evidently written by Nature's editors. This was dishonest, and it completely misrepresented Britannica's published coverage of the subject.

Nature failed to check the factual assertions of its reviewers. Everyone makes mistakes, even experts, which is why all factual assertions should be verified. Nature, however, decided not to do this; they did not even require reviewers to provide sources for their assertions because "it would have been simply too time consuming."6 Instead, Nature assumed that what its reviewers said was true and, when it contradicted something in the Britannica, that the reviewer was right and Britannica was wrong. The result was predictable.7 Examples:

Nature's review of the Britannica article "Pythagorean theorem" claimed the Britannica misspelled an Italian town that the reviewer said should be spelled "Crotona." However, according to the U.S. Board of Geographic Names, the preferred spelling is in fact "Crotone," as Britannica spelled it. Other reliable sources also give "Crotone" as the right spelling. For Nature's editors to have ruled this an error on one reviewer's say-so, without confirming the spelling, was inexcusable.

For the article on Subrahmanyan Chandrasekhar, Nature's reviewer quibbled with the dates Britannica gave for one of Professor Chandrasekhar's academic appointments and for the publication of his book Principles of Stellar Dynamics. Once again, on the basis of these unverified objections, Nature ruled Britannica to be in error, but we have solid sources for both dates, and we stand by them. See Appendix B for details.

Nature failed to distinguish minor inaccuracies from major errors. By counting up the alleged inaccuracies for both encyclopedias, Nature treated all mistakes equally and failed to observe that Wikipedia had many more shortcomings of a fundamental kind. Reviewers told the journal that many of the Wikipedia articles were "poorly structured and confusing"--a fact that made those arti

cles resistant to basic fact checking and thus not suitable for comparison with Britannica--yet the fact was buried deep in the article and its methodological implications ignored.

Nature counted "errors" and "critical omissions" that did not exist. As we have said, where reviewers found genuine inaccuracies in the Britannica, we corrected them, but dozens of the so-called inaccuracies they attributed to us were nothing of the kind; they were the result of reviewers expressing opinions that differed from ours about what should be included in an encyclopedia article. In these cases Britannica's coverage was actually sound, most often reflecting the considered judgment of editors who have extensive experience in publishing an encyclopedia, in deciding what information should be included in it for the general reader, and in how it should be presented. Nature's reviewers--who were scientists, not encyclopedists--were certainly entitled to their opinions on these matters, but for Nature to have ascribed inaccuracies to Britannica simply on the basis of their opinions was invalid. For example:

According to the reviewer of the Britannica article "Nobel Prize," the fact that the 1935 Peace Prize was awarded to Carl von Ossietzky in 1936 should have been stated. We disagree. To raise this minor fact and then be required to explain it would, at that point in the Nobel Prize article, have distracted the reader from the main discussion. This was not an omission; it was an editorial judgment. (N.B.: The fact that the award was conferred in 1936 is reported in Ossietzky's biographical article, where it belongs.)

The reviewer of the article on Paul Dirac objected that certain areas of Dirac's work were not covered. Our coverage, however, was appropriate for a general-reference encyclopedia. By design, the 825-word article explained, for the lay reader, Dirac's most significant contributions, not all of them.

The reviewer of the article on the Haber-Bosch process suggested that the article should have shown the chemical equation for the reaction, and Nature therefore called its absence from the article an omission. Not so. The article's verbal description of the process was clear and sufficient for the general reader.

Conclusion

"No test is perfect and we acknowledge that any of our reviewers could themselves have made occasional errors," said the editors of Nature about their study. "But by choosing reviewers who were highly qualified in the specific area described by each entry, we aimed to subject the encyclopaedia entries to the fairest and most stringent test that we could."8

Indeed, perfection is unattainable, and a few mistakes in Nature's analysis could perhaps have been overlooked. But alas, as we have shown above, the study's shortcomings went well beyond "occasional errors." The entire undertaking--from the study's methodology to the misleading way Nature "spun" the story--was misconceived. Among other things, while it is important to engage "reviewers who were highly qualified," that alone is not enough: as every editor at Britannica knows, even the assertions of experts must be confirmed and their observations considered in the proper context for a general-reference work like Britannica. Nature failed to take these and so many of the other steps that would have been required to make its research valid. The results were the errors we describe above and in Appendix B.

We now call on Nature to fulfill its commitment to good scholarship and send us the unabridged reviewer reports on which the study was based. And as we have shown here, the facts call for a complete retraction of the study and the article in which it was reported. We call on Nature to make the retraction and make it promptly.

Notes

1Jim Giles, "Internet encyclopaedias go head to head," Nature, December 15, 2005: 900-01.

2John Seigenthaler, "A false Wikipedia 'biography,'" USA Today, November 29, 2005 (Link). "Free Encyclopedia Wikipedia.de Has Copyright Issues," DW-World/Deutsche Welle, November 29, 2005 (Link). "Wikipedia Caught in Podfather Turf War," Podcasting News, December 5, 2005 (Link). All accessed March 7, 2006.

3Dan Goodin, "Wikipedia Science Topics As Accurate As Britannica-Report," Associated Press, December 14, 2005. Gregory M. Lamb, "Online Wikipedia is not Britannica - but it's close," Christian Science Monitor, January 5, 2006 (Link). "Wikipedia Gets Things Right," Red Herring, December 14, 2005 (Link). "Assessing Wikipedia's Accuracy," All Things Considered, December 15, 2005 (Link). Julian Dibbell, "Factually Speaking," Village Voice, December 22, 2005 (Link). "Fact or fiction? Online encyclopedias put to the test," The Age, December 15, 2005 (Link). All accessed March 7, 2006.

4See Michael J. McCarthy, "It's Not True About Caligula's Horse; Britannica Checked --- Dogged Researchers Answer Some Remarkable Queries," Wall Street Journal, April 22, 1999: A1.

5"Supplementary information to accompany Nature news article 'Internet encyclopaedias go head to head'" (Link). Accessed February 23, 2006.

6"Supplementary information," 2.

7Nature's editors appear to have thought that since their reviewers were scientists, those reviewers' judgments trumped the encyclopedia where there was a disagreement. This was presumptuous, since Britannica engages expert advisers and contributors who are in every way equal to Nature's reviewers. Our distinguished contributors have included more than 110 Nobel laureates.

8"Supplementary information," 1.

Appendix A: The Original Data

Appendix B: Nature's Errors