Things We Just Don’t Know
Why Is Nutritional Research So Inconclusive?
Editor’s Note: We thank our frequent reader and colleague, Dr. Richard Plotzker, for calling to our attention the articles in the Annals of Internal Medicine about coffee consumption that prompted this post. We also disclose that one of us (Jack Gorman) has a patent pending for a new use for caffeine in tablet form.
Have you had your coffee yet today? We guess you have, because a survey recently conducted by the National Coffee Association found that 64% of Americans aged 18 and older drank coffee on the previous day. And we are not even the most avid coffee consumers in the world, only 26th. The top five countries for coffee consumption are Finland, Norway, Iceland, Denmark, and the Netherlands.
If one wanted to conduct a study to see if drinking coffee is good or bad for our health it would surely seem we would at least have available a very large sample of people from whom to acquire data. Millions of people drink coffee and they do so in various amounts and for prolonged periods of time. Epidemiologists should therefore have relatively little trouble matching the amount of coffee individuals consume with their health outcomes and longevity and figure out if all that caffeine is helpful or harmful. After all, that is essentially what they did to find out that, beyond question, cigarette smoking is disastrous for health and clearly cuts down on expected lifespan.
Surprisingly, however, it is not at all clear whether coffee drinking is particularly good for us. An article last year in the Annals of Internal Medicine reviewed the results of two large epidemiological studies on coffee drinking and concluded that “Although drinking coffee cannot be recommended as being good for your health on the basis of these kinds of studies, the studies do suggest that for many people, no-long-term harm will result from drinking coffee”. How’s that for an entirely lukewarm and unhelpful conclusion?
This lack of certainty afflicts almost all research that tries to assess whether what we eat and drink is healthy. We are justified in our skepticism about the results of nutritional research studies because they always seem to be contradicted by the next study that comes out. Some of the triumphs in medical research have come from epidemiology. The cigarette smoking finding mentioned above is one example. Another is the amazing speed with which scientists figured out how the virus that causes AIDS—HIV—is transmitted, thus saving millions of lives around the world. Why is it so hard to get definitive answers when epidemiology turns its attention to what we eat?
Confounds and Errors
There are many answers to this question. Let’s start with the scientific issues.
Most epidemiological studies are not randomized, controlled trials. Once scientists suspect that something, often called the “exposure,” is the cause of an adverse outcome, like getting a bad disease or dying, they usually cannot design a trial in which a group of people are randomly assigned to exposure versus no exposure. For instance, we cannot ethically randomly assign people to smoke or not smoke cigarettes in order to find out if smoking causes cancer. That means we have to follow large groups of people who choose whether or not to smoke over long periods of time and see if the rates of cancer are higher in the former group compared to the latter.
The problem with studies that lack random assignment to an exposure like cigarette smoking is that the people in the two groups are likely to have many other differences between them besides whether they smoke. Cigarette smokers might also drink more alcohol, exercise less, and more likely be male. Epidemiologists have developed powerful statistical tools to control for these so-called confounding variables, but they have to know what the variables are in order to control for them. If there are differences between groups about which we are unaware, it is not possible to control for them and this can distort the signal from the exposure of interest.
In the case of cigarette smoking, the signal is overwhelming. Smokers have such a greater chance of getting lung cancer and heart disease compared to non-smokers, that no extraneous, confounding variables can make the association go away. In statistical terms, we say that the effect size for the association between cigarette smoking and cancer is large. In other words, we now know for certain that cigarette smoking causes cancer.
But the signal from drinking coffee is not nearly as strong. People who drink coffee, especially a lot of coffee, may be different in other ways that affect health from people who drink less or no coffee. Coffee drinkers may have different personalities and genetics compared to non-drinkers, for example, and these are very hard to control for in epidemiological studies. There are likely many other differences between the groups, making it extremely difficult to be sure that it is coffee that is responsible for any particular good or bad outcome.
The same is true for almost everything we eat. Whether red meat causes cancer or drinking alcohol decreases the risk for heart disease are both controversial claims, but what can be said about each is that confounding variables and small effect sizes make them difficult to state with great confidence. We just eat too many different things and we eat them in such varying quantities that it is very difficult to pick out which specific part of our diets is really responsible for any particular outcome.
And that variability also helps explain another major problem with figuring out whether coffee or anything else is good or bad for us. Almost all nutritional studies rely on self-report—people enrolled in the study are given questionnaires to fill out that detail everything they eat and how much of each thing listed was actually consumed. Quantities of food and drink consumed are extremely important because if coffee is good for you, it is possible that one cup a day has a very different effect than either five cups a day on the one hand and a cup a month on the other. But how accurate are people at recording all of this? It turns out that food survey questionnaires are prone to significant amounts of error, especially when people are expected to complete them regularly over long stretches of time. People forget what they ate for breakfast, miscalculate amounts, and sometimes even, dare we say, lie. Many people don’t like to admit that they ate a candy bar or drank two cans of soda. And further complicating the matter is the finding that the very act of filling out food intake questionnaires can induce people to change their eating habits (g). While it might be nice to see that people who keep a food diary adopt a healthier diet, that is going to make the results of a study on food intake as a cause of some disease lose its ability to tell us what happens to the great majority of us who are not enrolled in nutritional studies and don’t fill out food and drink diaries.
Conflicts of Interest in Nutritional Research
There are also some important non-scientific reasons that nutritional studies are suspect. Many nutritional studies are funded by food manufacturers. Corporations that manufacture food often do everything possible to make sure we only see studies that support their products. Marion Nestle (no relationship with the eponymous food manufacturing company), has studied and written persuasively on this issue. “When food companies funded research,” she says, “their research questions were phrased in such a way that the results could be used for marketing purposes. It became clear to me that this wasn’t about basic science and nutrition, but more about how the benefits of a product could be established according to the funder”. It is thus always critical for journalists and the public in general to question who funded a nutritional study as part of our evaluation of its merits.
When scientists who are not influenced by industry funding do decide that something is dangerous or safe for us to eat or drink, it is not necessarily the case that people will believe them. For example, most scientists involved in nutrition and agriculture believe that genetically-modified foods, often referred to as GMOs, are safe for us to eat. Nevertheless, a large segment of the population, perhaps supported by the very lucrative organic food industry, insist they are dangerous. One study recently showed that a lack of science literacy predicts fear of GMOs. So lack of fundamental knowledge about science can also lead to mistrust of the results of nutritional studies.
We have seen scientific reports lately claiming that caffeine reduces the risk of death from chronic kidney disease and liver disease; the incidence of multiple sclerosis, anxiety-related behaviors, depression, dementia, and heart disease; and mortality in general (references available upon request). All of that might make one think that that cup of coffee we are dependent on to get us going in the morning isn’t so bad after all.
But take into account all the scientific and conflict of interest problems that arise in reaching that conclusion and it is no wonder that people are skeptical. We are left with the not very satisfying conclusion that “no-long-term harm will result from drinking coffee.” Most people seem to drink coffee and for them, that will have to be sufficient reassurance for now and probably for the foreseeable future. Nutritional research has given us some tantalizing things to think about, but when it comes to definitive conclusions about what is beneficial and what is harmful, that best advice still seems to be to just not eat too much of anything.
Now go figure out what “too much” is exactly.