Why are all nutrition studies untrustworthy? The problem is you

How far off are the data? Archer and a pair of collaborators published a paper in 2013 that attempted to put a number on it. They looked at almost four decades of results of the National Health and Nutrition Examination Survey (NHANES)—the big U.S. government database on diet—and compared what people said they ate with how much they’d need to eat simply to stay alive. Their findings: “Across the 39-year history of the NHANES, IE data [that is, energy intake data—how many calories were consumed] on the majority of respondents (67.3% of women and 58.7% of men) were not physiologically plausible.” That is, if people ate what they said they ate, they would have starved. In other words, more than half the data were demonstrably wrong. And that’s not even counting people who were over-reporting.

Now, there’s wrong and there’s wrong. If you have an otherwise functional clock that just happens to be set ten minutes slow, it’s no problem to adjust the incorrect data it provides you at any given moment. The question is whether faulty food data can be similarly manipulated to make it useful.

And there, opinions seem to vary. Some researchers (these for instance) argue that survey techniques have gotten better, that the amount of error is less than Archer and others have argued, and that with appropriate statistics techniques self-reported data can be useful. Others, like the Energy Balance Measurement Working Group, take a harder line: “For research, clinical practice, and policy determination,” they argue, “a great need exists for accurately determining EI and expenditure. It is time to recognize that even if methods are cheap and convenient, inaccurate scientific methods will lead to inaccurate conclusions. Interpretations based on inadequate measurement tools may cause misguided national and local health care policies, funding for and by major organizations, and health care advice to individuals. Like other antiquated methods, self-reported PAEE [physical activity energy expenditure] and EI need to be recognized as inaccurate measurements by the scientific and funding communities.”

Pay attention to that mention of the funding community. I mentioned that nutrition research suffers from conflicts of interest. Sometimes that means corporate sponsorship, of course. But almost always it involves preserving nutrition research itself. What happens if survey-based data suddenly become unacceptable? A lot of nutrition researchers will quickly discover it’s a lot harder to find funding, conduct studies, and publish the kinds of articles that provide tenure, job security, and prestige. And these endangered scholars are the peers who pass judgment on the articles that appear in peer-reviewed journals. They have a powerful incentive not to rock the boat.

You could look at food’s data problem and say that it means we should be a lot more circumspect about believing the latest bit of hype coming out of universities (especially if the university in question is Cornell and the researcher is working in the lab of Brian Wansink). But Archer goes considerably further. He doesn’t just say that most nutritional studies are questionable. He says they’re wrong.

Dietary cholesterol? There was never evidence that it caused harm—something that even the scientific committee of the Dietary Guidelines for Americans conceded in 2015.

Does consuming sugar cause diabetes? No, he says. (Though if you have diabetes, diet remains important.) Archer believes that many of the problems we associate with diet are actually the product of insufficient physical activity. But in the case of type 2 diabetes—or adult-onset—he thinks it’s actually passed on from generation to generation. (I can’t do justice here to the theory—“non-genetic evolution.” But you can read about it for yourself here.)

The Mediterranean diet? Reducing sodium? The evidence is actually against the Mediterranean diet, Archer says. And for many people, reducing sodium might actually do more harm than good.

(How did we travel down this path in the first place? That’s a question for another day. But if you’re impatient, Archer maps it out in great detail here.)

Let’s pause and take a deep breath.

OK, there are cranks out there. Could Archer be one? I suppose. He’s got that whole voice-of-one-crying-in-the-wilderness thing going on, which is worrisome. He was recently let go by the University of Alabama, where he worked as a nutrition researcher. (Don’t feel too sorry for him. He’s now chief science officer at a startup focused on using data to improve health.) But his articles are showing up in the right places, and even the people who disagree with him seem to treat him with respect. And he’s not the only scientist who thinks the way he does. For those of us who haven’t got the scientific expertise to sort things out for ourselves, it’s always hard to know what to think when faced with this sort of controversy. Here’s how I approach it:

First, I already knew I wasn’t satisfied with the quality of what I’ve been reading about food. The expert opinions swing about too wildly, and there’s too much out there that’s just stupid. (Did you read that the “most nutritious” food of all is the almond? In second place is the cherimoya, a tropical fruit. Pig fat is eighth. I’m sure that someone has a methodology that justifies the list. But in what world does it mean anything?) I’m getting to the point where I believe very, very little.

Second, the question about data is truly bothersome. I ask, “Would I lie on a food questionnaire?” and the answer is “almost certainly.” I’m pretty sure I have lied to my doctor about my diet, even though I think it’s a terrible idea. Fortunately, I’m not sure he actually listens to me. If there’s no data, I don’t want to hear the conclusions.

So I’m inclined to think that Archer may well be right. At least he’s given me a tool for chipping away at the quasi-scientific garbage that I encounter on a daily basis. And if there’s no evidence of connections between specific foods and diets and diseases, does that mean that we just haven’t proven them yet, or that there aren’t any to be discovered?

At this point, you may have written me off as being a crank myself. Fair enough, I suppose. But stick around for one last question: If it is in fact true, as it might be, that diet has little or no impact on diseases—if that becomes the accepted wisdom—what happens to the “good food” movement? Would it be a disaster, or maybe the beginning of a truer, kinder approach to the vexing problems we all face.

That’s what we’ll talk about next time. See you then.