Food for Thought Hampers Trust in Science
The Perils of Nutritional Science Reporting
There are few areas of science that attract as much attention by media and its consumers than nutrition. It seems that almost every time a study is published that touts a particular food or diet as having special health benefits, it garners headlines.
One of the few things most of us believe we control is what we eat. That makes it highly attractive to think we can prevent seemingly uncontrollable events like heart disease, cancer, and dementia by choosing the right foods. It is even better when we are told we can do this without ever feeling hungry.
It is not surprising, therefore, that science and the public intersect so frequently around diet and nutrition. Given that we want people to trust science and incorporate it into decisions about personal health and safety, it would seem of critical importance that the nutritional science imparted to the public be trustworthy and reported accurately. We have been following stories about nutritional studies closely because of their potential impact on how non-scientists judge the value of what they hear and read about scientists’ work and findings. Sadly, there are few areas of science as prone to mistakes, misunderstandings, and hype than nutrition.
The Perils of Observational Studies
Why are nutritional studies so prone to these problems? First of all, it is because most of them are what we call “observational” studies. In this type of study, scientists observe what people are already doing and see if it affects a chosen outcome. In the case of observational nutritional studies, scientists choose a particular food or diet and then collect data on whether people in a selected population who eat more of it have a different health outcomes—like fewer heart attacks or better memory—than those who eat less of the same food or foods. But this means that the scientists have to take the people in the study’s word for what they eat, and the nutritional questionnaires typically used to collect such data are notoriously prone to error because people often don’t remember—or care to report—what they really eat.
A further problem with observational studies is that active and control interventions aren’t randomly assigned to two groups of similar people. Perhaps people who choose to eat more of one particular food than other people also have other idiosyncratic qualities, any one of which could be responsible for a difference in health outcome. It is very hard to isolate the effect of a specific food or diet without random assignment of groups to an active and to a control intervention.
We can see how this can affect conclusions we may draw from a study in a recent contretemps over a press release about a Chinese study that looked at the effect of egg consumption on developing heart disease published in the medical journal BMJ. In this study, half a million Chinese adults were asked how many eggs they eat daily and then followed for the development of heart disease and stroke over several years. There was a small decrease in these health outcomes in people who ate an egg every day compared to those who did not eat eggs at all. BMJ issued a press release about the study titled “Daily egg consumption may reduce cardiovascular disease: Having an egg a day could reduce risk of stroke by 26 percent”. Some media outlets immediately picked this up as indicating that eating eggs will prevent you from dying of a heart attack or stroke.
But observational studies cannot ever establish such a cause and effect relationship. And as Health News Review’s Kevin Lomangino reported, there are numerous problems reaching such a conclusion from this kind of study including that “food frequency questionnaires…are known to be unreliable”. In fact, the study cannot possibly tell us that eating eggs was the sole reason that the study subjects avoided heart attacks and strokes. Who knows what other things people who eat an egg a day also eat that are different from people who shun eggs altogether, for example? Scientists call extraneous things they cannot account for that might affect their study results “confounding variables.”To their credit, the BMJ editors responded to Lomangino’s criticism by promising to do a better job in the future monitoring what is written in its press releases. An observational study can only suggest something might be going on about eating eggs and heart disease, not prove that one is linked to the other causally.
Even Controlled Studies Can Suffer
One would hope that a more sophisticated type of study could address some of the technical problems with observational studies and permit us to reach firmer conclusions about what foods and diets are best for us. But even when scientists are able to do a seemingly sophisticated and prospectively designed, randomly assigned study, problems can arise that shake our confidence in its findings and conclusions.
In 2013, a group of Spanish scientists published a paper in the prestigious New England Journal of Medicine that concluded “we observed that an energy-unrestricted Mediterranean diet, supplemented with extra-virgin olive oil or nuts, resulted in a substantial reduction in the risk of major cardiovascular events among high-risk persons. The results support the benefits of the Mediterranean diet for the primary prevention of cardiovascular disease”. Now this was a randomly assigned, controlled study in which subjects drawn from the same populations in Spain were randomly assigned to one of three diets. This kind of study design should eliminate the problem of confounding variables because the only thing that is different among groups in a randomized study is the actual study intervention(s). And the conclusion that the two groups that received different forms of modified Mediterranean diets had fewer cardiovascular events than those assigned to receive a low-fat diet was not surprising given all that has been written lately about the Mediterranean diet’s supposed advantages.
But earlier this year the New England Journal retracted the article because numerous problems were found in the way the study was conducted. It turned out that further review of the study methods revealed that about 1600 of the 7500 people in the study hadn’t been properly randomized to one of the groups. Randomization is essentially choosing by chance—you flip a coin and heads you go to one group and tails to the other (it is actually done by sophisticated computer programs that can handle randomization to more than two groups). But in the Mediterranean diet study, sometimes if one family member was assigned to a group, his or her family members were also put in the same group for convenience. That isn’t random assignment, of course, and adversely affects the study’s validity. A reanalysis of the data eliminating the subjects whose group assignment was not done randomly still yielded the finding that the Mediterranean diet led to fewer instances of heart disease and stroke, but confidence in the study is obviously significantly weakened.
While scientists debate the study’s usefulness, it would not be surprising to find that the public is once again soured on believing what they say about nutrition in general. It just happens too frequently that nutritional studies get big headlines, only to be proven wrong later on. Either the headlines, based on press releases, exaggerated the meaning of the study or the findings are not replicated by a new study. It’s confusing and contributes to science denial.
Where Angels Fear to Tread?
A further problem is that when science fails to convince the public that its recommendations are correct, non-scientists rush in with recommendations that may or may not have any basis at all in science. One such example is the recent revival of the supposed benefits of the ketogenic diet. In a recent Reader’s Digest article “12 Things That Happen to Your Body on a Keto Diet” these benefits are once again rehearsed, but the science behind them is shaky at best.
The “keto” or more technically “ketogenic” diet is an ultra-low carbohydrate diet that rapidly depletes the body of its usual source of energy, carbohydrates like glucose. This forces the body to break down fat stores to produce an alternative energy source, called ketone bodies or ketones. This takes only a few days for most people and can result in rapid weight loss. The Reader’s Digest article does a reasonable job explaining the physiology involved, and even lists some of the adverse side effects that are possible from following a ketogenic diet, like “irritability, insomnia, fatigue, headache, and mood swings.”
But as is characteristic of this kind of article, it fails to explain what science there is—and mostly isn’t—to justify recommending the keto diet. And there are serious misstatements. One is the claim that “the brain loves fat, and the keto diet is full of fat for it to use. This may result in better brain function.” In fact, the brain normally burns only glucose and like the rest of the body only switches to ketones when no carbohydrates are available. There is actually zero evidence that the keto diet improves cognitive or memory function. There are data, on the other hand, that link heavy consumption of saturated fats, which are part of the keto diet, to impaired cognition. The article also implies that the ketogenic diet might be beneficial for people with a condition associated with obesity called non-alcoholic fatty liver disease, although the data for this are once again seriously lacking.
Perhaps most important, the Reader’s Digest fails to explain that there are almost no studies that look at the long-term effects of the keto diet, mostly because it is so hard to stay on it that few people could make it through such a study. That means that the weight loss someone might experience with a keto diet is often transient. And we know that roller-coaster ups and downs in weight are also harmful to health.
But how can we ask people not to trust Reader’s Digest when respected journals like the BMJ and the New England Journal of Medicine do not always present accurate information on diet and nutrition? We believe the first order of business is to help people understand that mistakes in science, especially nutritional science, are part of the scientific process. That means that even when a study in a top-notch, peer-reviewed scientific journal seems to indicate some food or other will make us live longer, we need to be skeptical. We now know for sure that eating high-carbohydrate, processed foods is harmful. We have some reason to believe that diets high in plants, oils, and nuts are beneficial. We have to be humble with everything else we recommend.