Searching for nutrition research on gluten-free, fat, or sugar? It’s all wrong.
You may have missed it (I certainly did), but over the holidays the Journal of the American Medical Association published an enigmatic little article about nutrition research: “Disclosures in Nutrition Research: Why It Is Different.” The title may suggest a real tub-thumper, an anti-industry screed full of dark accusations (at least, it did to me), but it’s not. Instead, it’s a modest, temperate little suggestion that nutrition researchers might do well to be a bit more open about their conflicts of interest.
Well, duh, huh?
“Nutrition research is among the most contentious fields of science,” the authors write. “Although the totality of an individual’s diet has important effects on health, most nutrients and foods individually have ambiguously tiny (or nonexistent) effects. Substantial reliance on observational data for which causal inference is notoriously difficult also limits the clarifying ability of nutrition science. When the data are not clear, opinions and conflicts of interest both financial and nonfinancial may influence research articles, editorials, guidelines, and laws.”
Is that worth another duh? Probably.
Who could disagree? Which, of course, raises the question of why we’re hearing it at all—and why nutrition research is being singled out.
To answer that, it helps to know that the lead author of the piece is a guy named John Ioannidis. If you don’t know the name, you should. He’s a mathematician and physician, a professor at Stanford, and one of the 20 most-cited researchers in the world. (“This probably only proves that citation metrics are highly unreliable,” he writes in his Stanford bio.)
Ioannidis’s claim to fame is a pair of extraordinary articles about medical research. The first, published in Journal of the American Medical Association in July 2005, examined all the highly cited original clinical research studies published in three top medical journals in 1990–2003, comparing them with later studies that used bigger samples. There were 45 studies that claimed that a specific medical intervention worked. Seven were disproven, and another seven were shown to have overstated the effectiveness of the intervention. Twenty were replicated, and 11 were unchallenged.
Think of what that means: Fewer than half of the studies were confirmed by subsequent research, and as for the ones researchers did try to confirm, roughly 40 percent were overturned.
When is research likelier to be wrong? Ioannidis lists a few situations: Smaller studies yield less reliable results. Studies that look at smaller effects (a hangnail as opposed to instant death) are more likely to be wrong. You should also be wary of fields that are testing large numbers of potential relationships (the way that genetic researchers are screening thousands of genes to see what effects they might have on health), fields with lots of flexibility in experimental design, fields where there are lots of potential financial conflicts, and “hot” fields with lots of researchers jumping on board.
The way the math seems to work out (though I’m no expert), the more factors you have lowering the likelihood of accurate results, the more impact bias has. Which means that nutrition research needs to pay more attention than many other fields.
I wanted to ask Ioannidis what he had in mind when he published the JAMA piece, but he never got back to me. But here’s what I take away from it: One of the world’s foremost experts on scientific error thinks that it’s possible or even likely that most scientific findings are wrong. And he thinks that nutrition research is in a worse state than that.
And that, I’d say, gets more than a duh. It gets an uh-oh, or maybe a “holy shit.”
The story doesn’t stop there. Come around next time, and I’ll introduce you to a researcher who’s taken the obvious next step: He doesn’t just find nutrition research problematic. He thinks it’s almost all completely wrong, and that today’s obsession with food as the key to health is just bunk. He’ll explain why he thinks so (he’s pretty persuasive), why the committee preparing for the next round of the United States dietary standards doesn’t want to hear about it—and, of course, what he’s doing now that his university has dumped him.
See you then.