Why Experts Disagree: The Nature of Scientific Controversy

In 1987, Philip Tetlock began one of the most ambitious studies ever conducted on expert judgment.

He recruited 284 political scientists, economists, and policy analysts - people who made their living thinking about politics and giving advice to governments and institutions - and asked them to make probability forecasts about future political and economic events.

They would predict election outcomes, economic turning points, international crises, the stability of governments.

The study ran for twenty years. By the time Tetlock analyzed his results, he had collected 82,361 forecasts.^[1] His conclusion, published as Expert Political Judgment in 2005, was damning: the experts barely beat chance. They did worse than simple statistical extrapolations.

More prominent experts with larger media platforms performed worse than less prominent ones. The more confident an expert, the less reliable their predictions tended to be.

This was not a finding about a fringe of incompetents. These were credentialed professionals at respected institutions, paid for their expertise, cited in policy documents, quoted in newspapers.

The finding that they could not reliably predict events in their own domain raises an urgent and practical question: if expert judgment is often unreliable, how should we interpret the persistent, visible, and often acrimonious disagreements among experts that shape public policy on climate, nutrition, economics, and medicine?

Is all expert disagreement equally troubling? Are there types that warrant deference and types that warrant skepticism? The stakes are real: how citizens and policymakers interpret expert disagreement determines responses to pandemics, climate legislation, drug regulation, and economic reform.

The answer, it turns out, requires a more sophisticated taxonomy than most people use. Expert disagreement is not a single phenomenon.

It encompasses genuine frontier uncertainty, values embedded in empirical questions, motivated reasoning by researchers, and in some cases the deliberate manufacture of the appearance of scientific controversy.

Understanding which type you are looking at is the first and most important analytical step. Treating all expert disagreement as equivalent - either dismissing it all as corruption or treating it all as legitimate uncertainty - produces errors in both directions.

"The most important thing I can tell you about being a good forecaster is this: start with a blank slate. Don't just consult the people who agree with you." - Philip Tetlock, Superforecasting (2015)

Key Definitions

Manufactured doubt - A deliberate strategy of creating the appearance of scientific uncertainty to delay regulatory or policy action, even when a strong scientific consensus exists.

Distinguished from genuine scientific uncertainty by its external commercial funding, political motivation, and focus on amplifying doubt rather than producing evidence. The strategy was documented in detail by Oreskes and Conway in Merchants of Doubt (2010).

Cultural cognition - Dan Kahan's term for the tendency of people to process risk information in ways that reinforce their cultural identity and group memberships, rather than updating beliefs proportionally to the evidence.

Higher scientific literacy is associated with greater polarization, not less, on culturally contested questions - because educated people are more effective rationalizers.

Replication crisis - The discovery, following the Open Science Collaboration's 2015 Reproducibility Project, that a large proportion of published scientific findings cannot be reproduced using the original methodology.

Only 36% of 100 psychology studies selected for replication produced results consistent with the original at the same significance level.

p-hacking - The practice of running multiple statistical tests or analytical variations and reporting only those that achieve conventional significance thresholds (p < 0.05), inflating the false-positive rate of the published literature without any individual act of fraud.

HARKing - Hypothesizing After Results are Known: presenting post-hoc explanations as if they were pre-registered hypotheses, a form of scientific misconduct that is often entirely unconscious and operates through motivated memory.

Hedgehogs vs foxes - Philip Tetlock's adaptation of Isaiah Berlin's distinction: hedgehog experts who know one big thing and apply it confidently everywhere generate compelling media narratives but forecast less accurately than fox experts who draw on many ideas, maintain calibrated uncertainty, and acknowledge the limits of their models.

Pre-registration - A reform practice in which researchers publicly register their hypotheses and planned analytical methods before data collection, preventing post-hoc adjustment and making HARKing detectable.

Superforecasting - The finding, from Tetlock and Gardner's Good Judgment Project, that a small subset of non-expert forecasters consistently outperforms expert analysts through disciplined application of probabilistic reasoning, active evidence-seeking, and willingness to update on new information.^[2]

Publication bias - The tendency of journals to accept positive results (confirming a hypothesis) over null or negative results, creating a published literature that overstates the prevalence of real effects.

Identity-protective cognition - Kahan's term for the process by which people engage their cognitive capacities to protect culturally important beliefs rather than to update them in response to evidence, explaining why information campaigns on culturally charged scientific topics often backfire.

Types of Expert Disagreement

Not all expert disagreement is alike, and conflating its different varieties is the most common error in public reasoning about science. There are at least four distinct types, each with different implications for how much weight a non-expert should give to the disagreement.

Genuine Frontier Uncertainty

The first and most legitimate form is genuine scientific uncertainty at the frontier of knowledge. When data are sparse, methodologies contested, or causal mechanisms poorly understood, reasonable scientists working in good faith can reach different conclusions from the same evidence.

This is not a failure of science - it is science operating normally. The history of science is a history of frontier uncertainty gradually being resolved as evidence accumulates.

The debate about minimum wage employment effects is largely of this type. When David Card and Alan Krueger published their 1994 natural experiment comparing fast-food employment in New Jersey and Pennsylvania after New Jersey raised its minimum wage, they found no employment decrease.^[9]

Critics attacked the methodology; defenders replicated and extended the findings. David Card received the Nobel Prize in Economics in 2021 partly for this work.

The debate continues because the evidence genuinely supports different interpretations under different model assumptions about labor market structure, the magnitude and timing of wage increases, and geographic variation.

Recognizing frontier uncertainty matters because it sets the appropriate epistemic posture: calibrated confidence with acknowledged error bars, not false certainty.

The Intergovernmental Panel on Climate Change has developed formal likelihood language for this purpose - "virtually certain" means greater than 99% probability; "likely" means greater than 66% - and when applied rigorously, this precision is a model of honest scientific communication.

Values Embedded in Scientific Questions

The second type of disagreement is often misdiagnosed as factual dispute when it actually reflects disagreement about values. Science can establish facts about probabilities and magnitudes.

It cannot, by itself, determine what level of risk is acceptable, how to weigh competing harms, or whose interests should be prioritized when they conflict.

The mammography screening debate illustrates this clearly. The United States Preventive Services Task Force and the American Cancer Society have issued contradicting guidelines on screening frequency and starting age. They share largely the same scientific information.

They weight the evidence differently because screening involves a genuine tradeoff: earlier detection of real cancers against the costs of false positives - unnecessary biopsies, overdiagnosis of indolent cancers that would never have caused harm, and radiation exposure.

How to weigh the benefit of catching a true cancer against the harm of treating a cancer that would never have progressed involves value judgments, not probability estimates alone. Experts who weight these differently will recommend differently, and both can be scientifically rigorous.

Roger Pielke Jr. argued in The Honest Broker (2007) that this conflation of fact and value is a pervasive problem at the science-policy interface.^[8] Scientists who present value-laden recommendations as if they were purely factual conclusions misrepresent both the science and their own role.

Motivated Reasoning Among Researchers

The third form is less comfortable to acknowledge: experts are not immune to the motivated reasoning that affects all human cognition.

Scientists have careers, funding dependencies, theoretical commitments, and cultural worldviews that influence which hypotheses they test, which results they publish, and how they interpret ambiguous findings.

The evidence for systematic motivated reasoning among researchers is now extensive. John Ioannidis and colleagues have conducted multiple meta-analyses showing that industry-funded studies of pharmaceuticals, nutrition, and chemicals produce results favorable to the funder at rates far above what chance would predict.

A 2003 analysis in the Journal of the American Medical Association found that industry-sponsored clinical trials were significantly more likely to report favorable outcomes than non-industry-sponsored trials.

The effect operates not primarily through overt fraud but through subtler mechanisms: choice of comparators, analysis endpoints, publication decisions, and the quiet abandonment of unfavorable trials.

Career incentives produce similar distortions. Uri Simonsohn, Leif Nelson, and Joseph Simmons documented what they called "false-positive psychology" - researchers engage in exploratory analyses, select the most interesting findings to report, and frame them as confirmatory while being entirely sincere about their scientific intentions.

The aggregate effect is a published literature with much higher error rates than the nominal 5% significance threshold implies.^[7]

Manufactured Controversy

The fourth type is the most troubling: the deliberate manufacture of the appearance of scientific uncertainty to serve commercial or political interests.

Naomi Oreskes and Erik Conway documented this in forensic detail in Merchants of Doubt (2010).^[3]

Their central finding was that the same small group of scientists - many with genuine credentials - and the same network of public relations firms appeared across multiple major scientific controversies over four decades: tobacco and cancer, acid rain, the ozone hole, secondhand smoke, and climate change.

The tobacco industry's playbook, developed in the 1950s and documented in internal corporate records obtained through litigation, was explicit: the goal was not to disprove the evidence that smoking caused cancer but to manufacture sufficient public uncertainty to prevent regulatory action.

Internal documents put it directly: "Doubt is our product."

The same firms and many of the same individuals subsequently worked on campaigns to manufacture doubt about ozone depletion, acid rain, and climate change.

Oreskes and Conway identified signature patterns that distinguish manufactured from genuine controversy: the disagreement exists primarily in media and policy forums rather than in peer-reviewed literature; a small group of contrarian scientists appear across multiple unrelated fields; the contrarians consistently align with regulated industries.

These patterns do not reflect scientific debate. They reflect a replicable strategy applied by the same network across different industries facing similar regulatory threats.

Tetlock's Forecasting Research: Foxes and Hedgehogs

Tetlock's twenty-year study was not simply a negative finding about expert failure. It was also a positive finding about what distinguishes better from worse forecasters.

The most accurate forecasters in his dataset - what he called foxes, following Isaiah Berlin's distinction - shared a cluster of characteristics. They drew on many different analytical frameworks rather than one overarching theory. They expressed calibrated uncertainty, using probabilistic language rather than projecting confidence.

They were willing to update their forecasts incrementally as new information arrived. They held their own views with intellectual humility, actively seeking disconfirming evidence rather than avoiding it.

The least accurate forecasters - hedgehogs - had one big idea that they applied everywhere with confidence. Hedgehog experts generated compelling media narratives and achieved high public prominence. They also forecast less accurately than foxes, and more prominent hedgehogs performed worse than less prominent ones.

The implication is unsettling: the qualities that make an expert quotable in media - confident, articulate, committed to a coherent framework - are negatively correlated with forecasting accuracy.

Tetlock's subsequent work, conducted with co-author Dan Gardner and reported in Superforecasting (2015), went further. In the Good Judgment Project, run as part of a U.S.

government-sponsored forecasting tournament, Tetlock identified a subset of non-expert forecasters - educated civilians with no special access to classified information - who consistently outperformed professional intelligence analysts with access to classified data.

These "superforecasters" used the same cognitive strategies Tetlock's foxes had used: probabilistic thinking, active evidence-seeking, willingness to update, decomposition of complex questions into components.

The finding suggests that forecasting accuracy is substantially a matter of cognitive style and discipline, not just domain knowledge.

Cultural Cognition: Why More Information Can Deepen Disagreement

Dan Kahan's research at Yale has documented one of the most counterintuitive findings in science communication. On culturally contested topics - climate change, nuclear waste disposal, gun control - cultural worldview is a stronger predictor of risk perception than scientific literacy or numeracy.

More troublingly, higher scientific literacy is associated with greater polarization on these topics, not with convergence.

Kahan and colleagues demonstrated this in a series of studies, including a 2012 paper in Nature Climate Change (doi: 10.1038/nclimate1547) that showed that people with higher science comprehension scores showed wider gaps between cultural groups in climate risk perception, not narrower ones.^[6]

People with high science literacy are better equipped to find, evaluate, and deploy evidence that confirms their cultural group's position. They are more sophisticated rationalizers, not more rational processors of evidence.

Kahan calls this "identity-protective cognition": because cultural identity is bound up with certain positions on contested empirical questions, receiving scientific information that challenges those positions functions as an identity threat.

The response is to engage sophisticated cognitive machinery in defending the position rather than updating it.

The result is that information campaigns aimed at correcting scientific misconceptions on culturally charged topics have limited effects and can even backfire, increasing polarization by giving people better tools for motivated reasoning.

This finding does not mean that knowledge is irrelevant or that nothing can be done. Kahan's research suggests that how information is framed, who delivers it, and whether it affirms or threatens cultural identity significantly changes its reception.

The naive information-deficit model - that public disagreement about science stems from ignorance and that better science education will fix it - is not supported by the evidence.

The Replication Crisis and What It Means for Consensus

The replication crisis has further complicated the relationship between published literature and reliable knowledge. The Open Science Collaboration's 2015 Reproducibility Project (doi: 10.1126/science.aac4716) attempted to replicate 100 published psychology studies using the original methodology.^[5]

Only 36% produced results consistent with the original at the same significance level. Subsequent replication efforts in medicine, economics, and nutrition research found similar patterns.

The mechanisms are now well understood. Publication bias means that journals preferentially accept positive results, so the published literature systematically overstates the prevalence of real effects - a null result finding no difference between an intervention and control condition is substantially less likely to be published than a positive finding, even if both were equally well-conducted.

P-hacking inflates false-positive rates. HARKing - presenting post-hoc hypotheses as pre-registered - makes exploratory analyses look like confirmatory ones.

This matters for expert consensus because consensus is often built on published literature. The case of dietary fat illustrates the problem. Ancel Keys's work linking dietary saturated fat to heart disease shaped nutritional science and public health guidelines for decades.

Critics like John Yudkin, who argued sugar was the more important culprit, were marginalized partly because of Keys's institutional dominance and partly because of undisclosed sugar industry funding of research designed to deflect attention from sucrose.

More recent evidence has produced a more nuanced picture, and the strong anti-fat consensus is now recognized as substantially overstated. The consensus was built on a literature with systematic problems.

Pre-registration and registered reports - in which journals agree to publish a study regardless of results, contingent on pre-specified methodology - are the primary structural reform proposed and implemented in response to the replication crisis.

Evidence from journals that adopted registered reports shows that the proportion of null results rises substantially, confirming that publication bias had been systematically inflating the published literature's positive-result rate.

When Expert Intuition Can Be Trusted

Against this backdrop, Kahneman and Klein's 2009 collaborative paper in Psychological Review (doi: 10.1037/a0016755) asked a pointed question: when should expert intuition be trusted?^[4]

Coming from intellectual traditions that had reached opposite conclusions about expert judgment - Kahneman emphasizing its failures, Klein emphasizing its successes in naturalistic settings - they converged on a surprisingly clear answer.

Expert intuition is reliable when two conditions are met: the environment must provide regular, clear, and timely feedback on the expert's judgments, and the expert must have had adequate opportunity to accumulate experience with a large number of comparable cases.

A chess grandmaster satisfies both conditions - the rules are stable, feedback is immediate and unambiguous, and relevant patterns recur reliably. A firefighter reading a burning building satisfies both as well.

A clinical psychologist predicting a patient's long-term outcome satisfies neither: feedback is delayed, confounded by other factors, and often never received. A financial analyst predicting market movements fails even more clearly - markets are too noisy and too adaptive to allow reliable pattern learning.

The domain, not the credential, determines the reliability of expert intuition. This has an important implication: in low-validity environments like geopolitical forecasting, macroeconomic prediction, and long-term clinical prognosis, the confident intuitions of experienced experts are not more reliable than statistical models and may be less reliable.

Years of practice in a domain without reliable feedback builds confidence, not accuracy.

How Non-Experts Should Navigate Expert Disagreement

The practical question that follows from all of this is how a non-expert should evaluate expert disagreement. Several principles emerge from the research.

Look at where the disagreement lives. If disagreement exists primarily in peer-reviewed literature, with competing factions producing and responding to evidence, it is likely genuine.

If it exists primarily in media, think tanks, and policy debates while the peer-reviewed literature shows substantial consensus, manufactured doubt is more probable.

Consensus among major scientific institutions - not unanimity, but the weight of the evidence as expressed in systematic reviews and meta-analyses - carries more epistemic weight than disagreement among media commentators.

Distinguish facts from values. When experts with access to the same evidence recommend different policies, the disagreement may be about values embedded in the scientific question, not about the evidence itself. Identifying which components of a controversy are empirical and which are normative clarifies the nature of the dispute.

Examine incentives and funding. Industry-funded studies in multiple domains show systematic bias toward sponsor-favorable outcomes. Calibrate confidence in individual studies accordingly, and look for independent replication.

Prefer calibrated uncertainty. Experts who express graded confidence - who use probabilistic language and acknowledge what they do not know - are generally more reliable than those who project certainty. The appeal of the confident hedgehog is a cognitive bias, not a guide to truth.

Distinguish domain expertise from adjacent claims. A climate physicist's expertise on the greenhouse effect does not automatically extend to optimal carbon pricing mechanisms. An immunologist's expertise in vaccine efficacy does not automatically extend to vaccine mandate policy. Domain boundaries matter.

For a deeper look at the psychological mechanisms that create and sustain false certainty, see why most published research is wrong. For the social and cultural dynamics that amplify expert disagreement in public discourse, see why political polarization increases.

For the cognitive biases that affect non-expert evaluation of evidence, see how to think clearly.

Sources & Further Reading

1. Tetlock, P. E. (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton University Press.
2. Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The Art and Science of Prediction. Crown.
3. Oreskes, N., & Conway, E. M. (2010). Merchants of Doubt. Bloomsbury Press.
4. Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. Psychological Review, 116(4), 515-526. DOI: 10.1037/a0016755
5. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. DOI: 10.1126/science.aac4716
6. Kahan, D. M., Peters, E., Wittlin, M., Slovic, P., Ouellette, L. L., Braman, D., & Mandel, G. (2012). The polarizing impact of science literacy and numeracy on perceived climate change risks. Nature Climate Change, 2(10), 732-735. DOI: 10.1038/nclimate1547
7. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLOS Medicine, 2(8), e124. DOI: 10.1371/journal.pmed.0020124
8. Pielke, R. A. (2007). The Honest Broker: Making Sense of Science in Policy and Politics. Cambridge University Press.
9. Card, D., & Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772-793.