Over the last year, one message has been clearly emphasized: trust science. Evidence, and only high-quality evidence, will form the basis for policy. How has this influenced the coronavirus vaccine campaign? On the one hand, there has been strict adherence to scientific rigor when it fits the desired narrative. On the other hand, scientists may differ in how they interpret the science. For example, some insist treatments are only "evidence-based" when they were evaluated in a double-blind randomized clinical trial and applied using protocols that exactly mirror the research studies. Others are willing to accept evidence from modeling exercises based on questionable assumptions. Yet, when advocating for greater acceptance of vaccines, the scientific standard is far from uniform. When communicating with the public, the same scientists may apply different standards depending on whether study conclusions fit the desired message.
Why would the world's greatest scientific thinkers apply the "good science" label so inconsistently? Perhaps the best explanation is what psychologists call , which is the tendency to interpret observations or data in a manner consistent with previously established beliefs and values. Thousands of studies conducted over the last 50 years show how confirmation bias clouds conclusions in the sciences, the arts, politics, judicial decisions, finance, and medicine. Nobel laureate Daniel Kahneman, PhD, has contributed numerous showing how, even among the most experienced scientists, preconceived notions color the interpretation of data and events. And worse, prior beliefs influence how we scrutinize information. have consistently shown that people uncritically accept evidence that confirms their beliefs, while subjecting disconfirming information to rigorous skeptical evaluation.
Before we continue, we want to be clear: we support the widespread deployment of coronavirus vaccines. We both were vaccinated as soon as we were eligible. But, we worry that bias can shade interpretation of evidence where scientific uncertainty remains.
Let's examine how the confirmation bias tendency has played out in the evaluation of studies in support of vaccines. For example, within days of a suggestion of increased myocarditis following vaccination among young Israeli men, CDC Director Rochelle Walensky, MD, MPH, ignored evidence in the CDC's own surveillance systems and leaped to the that vaccines posed no threat. But now, FDA has added a warning about the risk for myocarditis after vaccination with the mRNA shots, and CDC agreed to update their fact sheet.
NIH Director Francis Collins, MD, PhD, who by anyone's standards is a model of personal and scientific integrity, published a with the title, "Studies Confirm COVID-19 mRNA Vaccines Safe, Effective for Pregnant Women." The evidence was based on two studies. included only 30 pregnant women and did not measure outcomes in terms of maternal or child health. The small sample size is an issue. Imagine concluding that maternal age is unrelated to trisomy 21 based on 30 women ages 35 to 40. Down syndrome, which occurs in about eight per 1,000 live births for 40-year-old moms, would most likely be overlooked. The included just 84 vaccinated women who had given birth. Examinations showed the placentas from these births were comparable to those from a group of women who had not been vaccinated. These two studies, comprising a total of 114 pregnancies, were then generalized to all women and to birth outcomes rather than surrogate measures of immunity or placental pathology.
Evidence used to reassure men may be even weaker. In June, JAMA that was designed to determine whether mRNA vaccines diminish fertility. The investigation included a grand total of 45 young (median age 28) volunteers. Semen was collected pre- and post-vaccination. There was a modest increase in sperm concentration, motility, and semen volume following the vaccine. No data on pregnancies, live births, or neonatal complications were available. app reported the results under the heading, "Hopeful Dads Can Relax About COVID Vax: No Link to Infertility." A quote from the senior author diminished the methodological limitations: "...even though the 45 number is small, we're confident that we can generalize this to the rest of the population." He went further to express confidence that the Johnson & Johnson and Novavax vaccines, which were not evaluated in the study, would similarly not affect sperm counts. Urology Times reported, "Study shows COVID-19 vaccines do not affect male fertility" without raising a single question about methodological limitations. CNN, "Sperm count not harmed by Covid-19 vaccine, study says," quoted several experts who reassured men that the study removes any concern about vaccine effects on fertility. Yet, these small studies exert outsize influence because JAMA publications often get extensive media attention.
Now, let's do a thought experiment. Suppose the study showed a decrease in sperm concentration or motility after the vaccine. Would JAMA have accepted the paper? Or would reviewers have said: 1) there were only 45 subjects, 2) it used a convenience sample that is unrepresentative of the U.S. population of men, 3) there was no control group, 4) the outcomes were surrogate markers, not actual measures of reproductive success, and 5) follow-up was limited to 70 days after the second dose.
The list goes on. The concern, of course, is that confirmation bias is at work. JAMA upholds very high methodological standards for papers that challenge the dominant narrative. But for studies that reinforce the prevailing wisdom ... not so much. To be fair, we are not aware of any evidence that vaccines adversely affect fertility. But we need more time and evidence to affirm the vaccines have no effect on birth outcomes. That is why Pfizer, Moderna, Johnson & Johnson, and CDC have remained cautious -- pregnant women can get vaccinated and should discuss any questions with a healthcare provider; NIH also just to learn more about the vaccine in pregnant women.
This leads us back to the complicated morass of vaccine advocacy, science, and confirmation bias. No doubt, leading physicians and scientists, eager to encourage vaccination, sincerely believe vaccines do not affect fertility or birth outcomes. Does the evidence justify the enthusiasm? The vaccine makers have clearly stated that data are not sufficient to determine whether the vaccines are safe and effective for pregnant women. They are only now beginning studies that will provide sufficient evidence on safety. Policy should be based on unimpeachable science. We need to recognize our fallibility in interpreting evidence: When using science to warn or reassure the public, even the best scientists must recognize that we are all potential victims of confirmation bias.
is a faculty member at Stanford University's Clinical Excellence Research Center and a Distinguished Research Professor at the UCLA Fielding School of Public Health. , is the David and Marianna Fisher University Professor of International Relations at Brown University.