Back to Articles

Making the Most of Getting It Wrong

What do we do when science makes a mistake?
December 30, 2018 | Comments

Most scientific projects these days involve teams of scientists, often working at different institutions. But the job of writing the manuscript about the project and its results that is submitted to a scientific journal for publication consideration is usually given to a single person, most often the first author of the paper. After he or she has written the first draft of the paper, all the co-authors must review it, make comments, and check the data. Once they agree on a final draft, the paper is submitted to the journal, where it is assigned to independent peer reviewers who either recommend further modifications and re-review or that the paper be rejected (they rarely recommend it be accepted after first submission).

        Thus, many eyes fall on a scientific paper before it is published and in that process errors in data analysis and interpretation are found and fixed. On occasion, an error in a paper is only discovered after it has already been published. Then, embarrassed authors must publish a correction (or if the errors are deemed to make the paper fatally flawed, a retraction) in the journal that published the paper. This makes error discovery transparent. The original authors may be disappointed that the description of their results contained an error, but acknowledging it clears the record and allows the overall scientific field to move on to further experiments without being influenced by an erroneous data set.

        Errors in science are inevitable and as long as no fraud or cover-ups are involved, scientists’ reputations are not usually affected when they have to issue the occasional published correction. But a recent high-profile instance of a reported mistake in a scientific paper captured the attention of mass media and raised the question of how the scientific community and the general public do and should respond when science makes a mistake. And because many people do not fully understand the scientific process, the incident raised fears that such errors can be exploited by science deniers to manipulate public opinion.

The Oceans are Heating Up

        The present case involves a paper in the journal Nature, one of the two most prestigious scientific journals in the world. The paper, authored by scientists from four countries, found that the earth’s oceans are warming much faster than had been previously reported by climate scientists.  The paper’s implications for climate change are clear—if the oceans are warming faster than previously estimated, then climate change is advancing at an even more reckless pace than we feared and attempts to prevent disastrous consequences will be even more challenging. The paper’s findings were picked up by many major newspapers and broadcast media. A headline in the Washington Post on October 31, 2018, for example, warned that “Startling new research finds large buildup of heat in the oceans, suggesting a faster rate of global warming”.

An Error is Found

        Then, however, Nicholas Lewis, a British researcher, found some errors in the calculations the team of researchers had made and published his findings on the blog of Judith Curry. Lewis provided an in depth explanation of the errors he found in the Nature paper, and the paper’s authors agree that he is right. The errors made in the paper are highly technical, but we will try to give a brief explanation and see if we (who are not climate scientists) can get it right.

        The paper, by Laure Resplandy of Princeton University, Ralph L. Keeling of the Scripps Institute, and eight other co-authors, introduces a new method for calculating the amount of ocean warming. This turns out to be a technically very difficult endeavor and existing methods are highly prone to errors. So Resplandy and colleagues hoped the new method they devised and describe in their paper would be more accurate. The new method involves measuring concentrations of oxygen and carbon dioxide in the atmosphere. Because less atmospheric oxygen and carbon dioxide can be dissolved in the oceans as ocean temperatures rise, by calculating how much of these gases are lost from the atmosphere, one can infer the amount of ocean temperature increase. The result of their measurements and calculations was that the oceans have gained 1.33 ± 0.20 X 1022 joules of energy per year between 1991 and 2016.

        That number may not mean much to most of us, but it turns out to be much higher than previous methods calculated and translates to a much faster rate of global warming than previously believed to be the case. It would make the agreed upon limit of a 2.0o C rise in the earth’s temperature made in the Paris Climate Agreement more difficult to achieve and demand that countries take even more aggressive steps to limit the burning of fossil fuels and the release of carbon dioxide into the atmosphere.

        But the catch here is in the term ± 0.20. That is what statisticians refer to as the “error term,” usually the standard deviation of the mean. Whenever scientists measure something, they usually do so in sample of the total population. So if, for instance, we want to know how many Americans like chocolate ice cream, we don’t have to ask every single American but rather take a randomly selected subgroup of Americans that are picked to represent the whole population. This means, however, that the percentage of chocolate lovers in the sample of people actually studied may be a little less or little more than the percentage in the total population of Americans. Statistics allows us to figure out how much wiggle room there is in the sample percentage compared to the total population. In the case of ocean warming, the Resplendy team calculated an error term of plus or minus 0.20, meaning that the real value is somewhere between 1.33 minus 0.20 (i.e. 1.13) and 1.33 plus 0.20 (i.e. 1.53, with all the other numbers and units after it, of course). Because the 0.20 number is much smaller than the 1.33 number, scientists can conclude that there is not much error in this calculation.

        But Nicholas Lewis looked at the calculations in the paper and realized that the actual error term should be considerably higher than 0.20, meaning that the amount of energy being absorbed by the oceans might be much lower than the 1.33 number. Noting that the media had broadcast the finding to the general public, Lewis concluded his blog by stating “Of course, it is also very important that the media outlets that unquestioningly trumpeted the paper’s findings now correct the record too…But perhaps that is too much to hope for.”

The Correction is Made

Lewis needn’t have worried, however, because media outlets indeed picked right up on the error as soon as Resplendy and colleagues acknowledged it in a correction published in Nature. The Washington Post, for example, carried a story on November 13, 2018 with the headlineScientists acknowledge key errors in study of how fast the oceans are warming.

The concern is that climate change deniers will leap at this acknowledged error and attempt to call all of climate science into question. In fact, what the error means was nicely summarized by science journalist Christa Marshall, who quotes one of the paper’s authors, Keeling, as follows: “The overall conclusion that oceans are trapping more and more heat mirrors other studies and is not inaccurate, but the margin of error in the study is larger than originally thought”. In other words, there is absolutely no question that the oceans and the entire planet are warming; all this error means is that the amount may not be greater than scientists already know it is. That we are in trouble because of global warming is not at all called into question by the correction.

The concern, however, that the correction will be misused is not misplaced. The blog in which Nicholas Lewis chose to present his findings belongs to one Judith Curry, a former professor at the Georgia Institute of Technology’s School of Earth and Atmospheric Sciences. Curry has taken positions on her blog and elsewhere that disputes the climate change scientific consensus. In a testimony before a U.S. Senate subcommittee in 2015, for example, she stated that, “The growing discrepancy between climate model predictions and the observations has raised serious questions about the climate models that are being used as the basis for national and international energy and climate policies”. Curry seems to be among the 3% of climate scientists who do not think we are increasing the earth’s temperature by burning fossil fuels. Note, however, that typical of science deniers in general, she does not state that the climate science is wrong but rather that there is “doubt.” In fact, the consensus is so overwhelming among climate scientists that we are warming the earth to the point of catastrophe that we can safely say there is no doubt about the issue.

Now, Nicholas Lewis could have shared his findings directly with the editors of Nature, who would then have alerted the study’s authors. Instead, he chose to post it on a venue that devoted to climate change denial. This seems like a deliberate attempt to use the error to provoke other climate change deniers and to spread the myth that climate change is a myth.

Christa Marshall heard this worry that climate deniers may be salivating over the correction. “In the past, scientific debates about climate science have prompted skeptics to attack mainstream climate science generally,” she wrote. “Some climate scientists said they are concerned that could happen again in this case and the outcome wildly misinterpreted.”

An Immediate Response is Needed

How then should scientists and people who are devoted to promoting the truth about climate change respond? Marshall quotes one climate scientist as saying, “We can’t worry about that…We have to just call it as we see it, do good science, put it out there, defend it and, when necessary, correct.” Scientists often take this position when science denial rears its head, and insist that it is not their responsibility to take any action to ensure the public understands the facts.

We think, however, that this is not the correct response. Climate change is an urgent problem. As Bill McKibben writes in The New York Review of Books, if we held warming to 1.5oC, we would still lose 70 to 90 percent of coral reefs. At 2.0oC warming, we will lose 90 percent. Currently, he says, we are on track to warm the earth by 3.5oC. This will result in the forced migration of millions of people as parts of the earth become uninhabitable and rising sea levels wash away coastal cities and some islands. The article correction in Nature changes none of these facts.

We believe that a first-responder program needs to be established to cope immediately with the onset of any anti-science communications that spring up in mainstream media, social media, blogs, or any other mode of communication seen by a significant number of people. The program would be sophisticated in monitoring all of these communication outlets and in using evidence-based methods for communicating science to the public. Scientists themselves can no longer refuse to worry about science denial and must instead be part of an effort that does not take days or weeks to respond to anti-science challenges but rather anticipates them and acts immediately.

We also must do a much better job educating the public about the scientific process. Errors in a published paper do not mean there is widespread fraud or that an entire field of research is automatically suspect. That is what anti-science types would have us believe. In the case of the now corrected Nature paper on ocean warming, the scientific process worked well. An error was reported, the scientists immediately acknowledged it, the problem was made known widely and transparently, and the scientific community can now move on to improve on new methods of measuring things like ocean warming. It will be nothing short of a disaster if climate deniers successfully use this incident to convince people that climate change is not real. Scientists do their best to learn from their errors; the rest of us need to develop that capacity as well.

More Like This