I recently read The China Study, a book on nutrition. Or, rather, a bit over half. I stopped after a few hours because he was starting to repeat the narrative over and over, and I wanted to look at rebuttals before I put in the time to finish the book. I should clarify, at this point the book sounded pretty convincing.
The go-to rebuttal was written by Denise Minger, and took her a month to write and almost that long for me to read (over an hour). After reading it, it too sounded pretty convincing.
I aired my thoughts to a family friend, and his defense of The China Study sounded pretty convincing too.
This is not a healthy pattern. Luckily, at each point, I realized what was going on. I had gone into this endeavor with a highly, almost radically skeptical mindset, and it paid off. But many times, I and others aren’t so lucky.
One of my housemates sometimes complains that he believes too many things he reads. I completely empathize; humans evolved such big brains largely to make very compelling arguments. Nearly everyone disseminating information to us has an incentive to exaggerate as much as they can get away with, be it reporters, peddlers, researchers, or Donald Trump. Even researchers, whom we usually think of as trustworthy in cases other than climate change, are still motivated to exaggerate: if they can sway their field or the public to think their work more important than it is, it means more citations or media attention for them. Much of this is because good hypotheses are hard to come by; if a compelling one presents itself it may be your only chance for fame, and you certainly don’t want to pass that up because the data is unlucky. From HPMOR, ch. 78:
[He] is new to the business of having ideas, and so when he has one, he becomes proud of himself for having it. He has not yet had enough ideas to unflinchingly discard those that are beautiful in some aspects and impractical in others; he has not yet acquired confidence in his own ability to think of better ideas as he requires them. What we are seeing here is not his best idea, I fear, but rather [his] only idea.
This trend is especially, especially prevalent in nutrition. Having such wide interaction with the public, with so many people willing to early adopt the latest supposed panacea, leads to potent incentives to rush to press with the first tenable idea rather than double-check. How many one-shot diet solutions have there been over the years? For every big one, there are also numerous small ones: reduce sodium, take ginseng, etc. The FDA doesn’t regulate supplements, so that domain is of course a cesspool, but it hasn’t even done very well on the food and drugs it has been created for. The recommended daily allowances (RDAs) it releases have been revised in very different directions over the years, and some have said that the failure of their low fat recommendations in past decades condemned hundreds of thousands to cardiac- and obesity-related deaths.
The diet pushed in the China Study is plant-based, with no meat protein. But it is not enough to judge it based on the health evidence put forth—there are other directly conflicting diets that claim similar results, including low-carb and low-fat, each with their own apparently overwhelming evidence. You might protest that many low-fat diets have already been debunked, but this makes me more worried rather than less. After decades of government and scientific support for fat reduction in our diets (not only support, but fantastic promises of weight loss, elixir of life, and virgins) turned out to be faulty, it seems proper to be very skeptical of further diets claiming to Get Rid of This One Nutrient and Have All Your Problems Solved, even if, like the China Study, they look like they are firmly backed by evidence.
Why is “evidence” not the cut and dry standard of truth that the 21 million people in IFLS would have you believe? This is an important ground rule for science, and nutrition above all: it’s surprisingly easy to get results in favor of whatever you want. The latest viral post about it is from our stats heroes at 538, showing how you, yes you, can manipulate data to prove with statistical significance that either democrats or republicans have a positive effect on the national economy. Similarly, Scott Alexander shows how even parapsychologists researching telepathy can reliably find a needle of signal in a haystack of noise, and one of them produced a p-value of 10^-10, effectively damning all of science. This comes a decade after Ioannidis first published his pair of findings about how many studies in medicine, etc, are non-replicable or overstated (somewhere between 10 and 90%, depending on how strict you are).
So. Not only do we have many competing hypotheses, but many of them have good evidence. It is perhaps not surprising why no one knows what is going on.
While certainly annoying for those with practical concerns about nutrition, one man’s trash is another man’s epistemological goldmine. And mine it we must, with the state of the data.
I am highly intrigued at the possibilities nutrition offers to train rationality. If you missed out on the chance to test yourself on the many-worlds interpretation of quantum mechanics vs the Copenhagen interpretation because Yudkowsky did it for you, there are still many others in evolutionary theory, catastrophe theory, Greek history, etymology, Pinker vs Taleb on the Long Peace, the AGI safety problem, futurology, and more. Usually these are fairly two-sided, and some are more complex than others. On one side of the spectrum, simple case studies in rationality with limited options abound, as you notice when reading the irrational things other people write; but a truly godawful mess like nutrition research only comes around once in a blue moon.
Yudkowsky talked every once in a while about the rationalist community, musing on how to build a formal course of study or a formal order of members. In a few short stories, he called them the Bayesian Conspiracy. To be initiated, one must pass a short test against biases; later, their master had them practice rationality by working on large open problems in science. Unfortunately, most genuine problems in science have enormous barriers to entry, and require you to devote substantial temporal resources to even having a shot at them. His storyline seems a little unrealistic (though don’t let me stop you from shooting for the stars).
Whereas solving the hardest theoretical problems from a classroom might be too difficult for reality, sifting through the empirical data in nutrition science sounds, while still painful and hard, much more tenable. Some of the better known rationalists have already been here: Scott Alexander notes that himself, Luke Muehlhauser, and Romeo Stevens are all unimpressed by low-carb diets, and asks this position to be inducted as Rationalist Consensus. Luke’s is a beautiful leviathan about low-carb diets on behalf of the Open Philanthropy project. Alexander has written about wheat, Taubes, and a paleo-ish diet, but he’s cheating because he’s a doctor. These are fantastic analyses. They make me trust the rationality of the person more. These are the kind of Rationalist Theses I would expect from one trying to prove themselves.
But before you run off and consume these write-ups (hopefully I am not already too late), I again want to encourage you to try the exercises yourself. This is like a coloring book: there are a lot of pages, so you aren’t likely to run out fast, but you can’t re-use them and all the good ones tend to end up finished first by you or your friends.
The reason they’re hard to re-use is because being blind to the expected result is one of the foremost qualities of nutrition (for rational testing; this is almost exactly the opposite of its intended). Nutrition has a long history of being wrong on almost every level many times in a row, so this is great to train an open mind (or non-overconfident priors). And because it’s been wrong about almost everything, nutrition has an almost infinite number of sub-problems. Is salt good? Are megadoses of vitamin C good? What about other vitamins taken separately? What about animal protein? What about gluten? Tryptophan does weird things, how do we feel about that? Etc etc. The last great characteristic recommending nutrition for this role is that the barriers to entry are low: I can upperbound the scholastic prerequisites at AP Bio, and a single high school biology course (with some wikipedia supplementation) might certainly be enough too. Anything else, you can learn as you go from wikipedia.
To put my time where my mouth is, I’m writing up a review of The China Study and Whole now. If any others of you see nutrition’s shitshow for the golden opportunity that it is and decide to test yourself, for training purposes or to signal to others, let me know. I’m interested in compiling both questions and verdicts (partly to test for overconfidence, so be warned). If you are the 99% and are somewhere on the spectrum of not actually going to spend your time on that, I hold no grudges—but if nothing else from this post, remember how easy it is to be swayed by promises of solutions to your problems, how easy it is to build a convincing argument for any nutritional verdict, and how likely it is that any opinion you hold now is overconfident and will be overthrown in the next few decades.
ETA: even medical science has been shown to later recant about 40% of their taught material. Imagine how high this number would be for a subfield as hard to quantify as nutrition.