Gary Taubes is an investigative science and health journalist who has written three books (; ; and ) that have challenged conventional ideas and beliefs about nutrition. This is a version of a post originally on that has been edited for brevity.
Four hundred years ago, give or take a couple of years, Francis Bacon pioneered thinking about the scientific method by noting that humans are programmed to pay more attention to evidence that agrees with their preconceptions and to reject evidence that doesn't, and that this thinking leads to very effective religious dogma but not to reliable knowledge of the universe. Hence what was needed was a new technology of reasoning that would minimize these tendencies, although he recognized that getting rid of them entirely was not an option. Humans didn't function that way:
"The human understanding, once it has adopted opinions, either because they were already accepted and believed, or because it likes them, draws everything else to support and agree with them. And though it may meet a greater number and weight of contrary instances, it will, with great and harmful prejudice, ignore or condemn or exclude them by introducing some distinction, in order that the authority of those earlier assumptions may remain intact and unharmed."
In good science, this kind of cognitive bias is addressed, among other methodologies, by deciding in advance of looking at the evidence (or doing a trial) what criteria will be used to judge the worth of the evidence (the results) without knowledge of whether that evidence supports our hypotheses. This is one reason why clinical trials are done double blind, and why the data are ideally analyzed by researchers who are blinded to whether the subjects were interventions or controls, such that the biases of the investigators (or even the subjects) don't bias the interpretation of the results.
For whatever reason, when it comes to heart disease and dietary fat, the investigators whom the American Heart Association (AHA) chooses to determine what we should or should not eat have never been believers in this kind of, well, scientific methodology. This was the general conclusion of my first investigation into the . I'd like to say the situation has improved, but clearly it hasn't. The latest on saturated fat is the AHA's expert authorities – what Inspector Renaud in Casablanca would have called "the usual suspects" – reiterating that they were right 50 years ago, and they were right 20 years ago, and they're still right.
A Scottish cardiologist/epidemiologist described this pseudoscientific methodology to me as "Bing Crosby epidemiology" – i.e., "accentuate the positive and eliminate the negative." In short, it's cherry picking, and it's how a lawyer builds an argument but not how a scientist works to establish reliable knowledge, which is the goal of the enterprise. While all good science requires making judgments about what evidence is reliable and what isn't, scientists have to do this keeping in mind that the first principle of good science, now quoting Feynman, "is that you must not fool yourself and you're the easiest person to fool." The history of science is littered with failed hypotheses based on selective interpretation of the evidence.
The new Presidential Advisory, written by a dozen esteemed experts led by Harvard's Frank Sacks, may be the most egregious example of Bing Crosby epidemiology that I've ever seen. Whether consciously or unconsciously, they assume that what they think is true surely is, and then they methodically eliminate the negative and accentuate the positive until they can make the case that they are surely, clearly and unequivocally right.
The AHA concludes that only four clinical trials have ever been done with sufficiently reliable methodology to allow them to assess the value of replacing saturated fats, SFAs, with polyunsaturated fats, PUFAs (in practice, replacing animal fats by vegetable oils), and concludes that this replacement will reduce heart attacks by 30%. In the history of this debate, this is a huge, if not unprecedented number. These four trials are the ones that remain after the AHA experts have systematically picked through the others and found reasons to reject all that didn't find such a large positive effect, including a significant number that happened to suggest the opposite. For these trials that didn't make the cut, the AHA authorities carefully identify why these trials were critically if not fatally flawed, and so why their results cannot and should not be used in any reasonable assessment.
The trials rejected include among the largest ever done: the Minnesota Coronary Survey, the Sydney Heart Study, and, most notably, the Women's Health Initiative, which was the single largest and most expensive clinical trial in history. All of these resulted in evidence that refuted the hypothesis. All are rejected from the analysis. And the AHA experts have good reasons for all of these decisions, but when other organizations – most notably the Cochrane Collaboration – did this exercise correctly, deciding on a strict methodology in advance that would determine which studies to use and which not, without knowing the results, these trials were typically included.
What the AHA experts don't appear to do is make the same effort with the trials that do support their hypothesis and assumptions. Of the four studies that support the 30% reduction, all are ancient by the standards of nutrition science. All date to the 1960s. One of them, for instance, is the Oslo Diet-Heart Study. This trial reported a significant reduction in CVD events, in line with the beliefs of the AHA authors, and so it's included among the four trials considered worthy. The Oslo trial was indeed typical for the era, which means very primitive by today's standards. It has a single investigator, Paul Leren. Patients from the Oslo City Hospitals are then recommended to him as subjects if they are at high risk of heart disease or have already had heart attacks. He randomizes half of these patients, now subjects, to eat a low-SFA, high-PUFA diet and then gives them intensive counseling for years ("continuous instruction and supervision," as Leren puts it), and he compares them to a control group that gets no counseling and eats the standard Norwegian diet.
So one group gets a "healthy" diet and intensive counseling for years; the other group gets nothing. This is technically called performance bias, and it's the equivalent of doing an unblinded drug trial without a placebo. In this case, as Leren explains, all the physicians involved also knew whether their patients were assigned to the intervention group or the control, which makes investigator bias all that much more likely. We would never accept such a trial as a valid test of a drug. Why do it for diet?
Why do I know this about Oslo? My curiosity could not be satisfied by reading the published literature because Leren didn't give the necessary details in the published studies. He probably didn't have the space. He did in . I bought a copy a few weeks ago. That's how curious I was. It's in this monograph that Leren assesses the state of the science and describes in pretty good detail what he actually did in the trial. He also discusses the dietary changes achieved in his intervention group, and here's where the performance bias, rather than the PUFA/SFA shift, may have determined the study outcome.
Leren mentions in passing that sugar consumption in his intervention group was very low, about 50 grams a day, which is about 40 pounds a year and is . (I'm extrapolating back from this data -- i.e., guessing.) So this is a critical problem with performance bias in a diet study, any diet study. As we're taught in eighth grade science classes, good scientific experiments change a single variable with an intervention such that we can see the effect of that single change. In this trial, the variable that's supposed to be different is the SFA/PUFA ratio, but the performance bias introduces another one. One group gets continuous counseling to eat healthy; one group doesn't. Now is it possible that continuous counseling influenced health status independent of the fat content of the diet? One way is that apparently the group that got it decided to eat a hell of lot less sugar. This unintended consequence now gives another possible explanation for why these folks had so many fewer heart attacks. I don't know if this is true. The point is neither did Leren. And neither do our AHA authorities. So why didn't they reject the Oslo study with its clear performance bias? I'm guessing they didn't want to look for performance bias because the study confirmed their preconceptions.
All of the four studies used to support the 30% number had significant flaws, often this very same performance bias, and reason to reject them.
The is another good example. PrediMed may be the single most influential clinical trial of the last decade, and it's used by the AHA experts to argue for the benefits of a supposedly PUFA-rich Mediterranean diet. But it, too, was fatally flawed. You had to read the supplemental data in this case to find out. The researchers randomized subjects to three arms, one of which got nuts (Mediterranean) and regular counseling; one got olive oil (Mediterranean) and counseling; one (non-Mediterranean) got bupkis and no counseling. Hence, significant performance bias. Midway through the trial, the researchers actually realize that this was a problem and decide to address it. Here's how they describe this revelation on page 10 of the supplemental material:
"The initial dietary protocol for the Control group started with the delivery of a leaflet summarizing the recommendations to follow a low-fat diet (Table S2-S3) and scheduled one yearly visit. In October 2006, 3 years into the trial, we realized that such a low-grade intervention might potentially represent a weakness of the trial and amended the protocol to include quarterly individual and group sessions with delivery of food descriptions [my italics] shopping lists, meal plans and recipes (adapted to the low-fat diet) in such a way that the intensity of the intervention was similar to that of the Mediterranean diet groups, except for the provision of supplemental foods for free."
Sound of throat clearing ... Imagine a drug trial, in which "three years into the trial" the investigators realize that it might be a problem that they neglected to give the control group a placebo. Oops. Would editors of a prestigious journal buy the idea that "such a low-grade intervention might [might!!] potentially represent a weakness of the trial"? Would such a trial get published in any respectable journal? In nutrition, it's published in the New England Journal of Medicine and makes it to And the admission of this potential weakness is only made in the supplemental material. Not in the paper itself. Imagine had the study found that the Mediterranean diet was actually harmful. That giving nuts and olive oil increased the risk of death. Do you think the AHA's experts would have included it in this assessment, or would they have found this performance bias problem and rejected it on that basis? I'm voting for the latter, but we'll never know.
Ultimately this AHA document is a recapitulation of what the AHA experts have been arguing for decades. The only reason to publish it is because it's been taken heat lately from folks like me and Nina Teicholz and a host of others who point out that we're dealing with a pseudoscience here and the public deserves far better. Those of us who have become critics may indeed be biased about what we believe now – I certainly am -- but ultimately we're arguing for better science. This kind of post-hoc analysis of clinical trials, whether subgroup analysis or otherwise, can only be hypothesis generating. What the AHA experts are doing here is saying that their assessment of the data leads to what they consider a compelling hypothesis: replacing SFA with PUFA should reduce heart disease by 30%. But that's all they can say. By deciding what data to include and what not based on their preconceptions of what's true and what's not, they cannot say this is a fact, as they claim, only that it's still a reasonable hypothesis and has yet to be refuted.
The AHA experts do acknowledge that they're discussing the same decades-old trials that still cannot resolve this controversy, and then they state explicitly what would be necessary to do so:
"Readers may wonder why at least 1 definitive clinical trial has not been completed since then. Reasons include the high cost of a trial having upward of 20,000 to 30,000 participants needed to achieve satisfactory statistical power, the feasibility of delivering the dietary intervention to such a large study population, technical difficulties in establishing food distribution centers necessary to maintain high adherence for at least 5 years, and declining CVD incidence rates caused by improved lifestyle and better medical treatment."
So a rigorous test probably can't be done. And, more importantly, if this is what it takes to rigorously test the hypothesis -- "20,000 to 30,000 participants needed to achieve satisfactory statistical power" -- then why are we even discussing these other trials with nothing like that number? (And, of course, that's why they had to dismiss the WHI as meaningful because that trial does have this kind of statistical power.) They've effectively eviscerated their own case. And this leads to another point.
Back in 1981, Geoffrey Rose, a pioneer thinker in the field of preventive medicine, wrote an article , and he pointed out the same problem about vegetable oils that confronts us today. As Rose observed, it's one thing to tell people not to eat something because we evolved to eat very little of it and there's good evidence that eating less of it will reduce chronic disease risk.
But telling people to eat something new to the environment -- an unnatural factor, à la virtually any vegetable oil (other than olive oil if your ancestor happen to come from the Mediterranean or mid-East), which was what concerned Rose and concerns us today -- is an entirely different proposition. Now you're assuming that this unnatural factor is protective, just like we assume a drug can be protective say by lowering our blood pressure or cholesterol. And so the situation is little different than it would be if these AHA authorities were concluding that we should all take statins prophylactically or beta blockers. The point is that no one would ever accept such a proposal for a drug without large-scale clinical trials demonstrating that the benefits far outweigh the risks.
So even if the AHA hypothesis is as reasonable and compelling as the AHA authors clearly believe it is, it has to be tested. They are literally saying (not figuratively, literally) that vegetable oils -- soy, canola, etc -- are as beneficial as statins and so we should all consume them. Maybe so, but before we do (or at least before I do), they have a moral and ethical obligation to rigorously test that hypothesis, just as they would if they were advising us all to take a drug. And then, well, they should probably do it twice, since a fundamental tenet of good science is also independent replication. And what we need here is good science.