“Adam Kucharski, mathematical modeller and Professor of Infectious Disease Epidemiology at the London School of Hygiene & Tropical Medicine in London, UK, was one of the most reliable expert sources for many reporters wrestling with the scientific debates and dilemmas [around COVID-19]. He has now distilled his experience from working on both the pandemic and the epidemiology of other disease outbreaks, such as Zika virus disease and Ebola virus disease, into Proof: The Uncertain Science of Certainty, an exceptionally clear and engaging account of how scientists demonstrate truth and falsity. By showing that the matter often requires us to accept uncertainty without having to resort to mere guesswork or opinion, this book ought to be on the reading list of policy makers everywhere, and should help any reader make more informed judgements in their lives. [..]
Most questions in science cannot be adjudicated on a rigorous basis, however. They demand that we proceed in a more legalistic manner, collecting evidence to guide judgement. Kucharski rightly remarks that many scientists have an “overly simplistic” idea of how this “scientific method” is supposed to work, based on Karl Popper’s notion of falsification: a theory can never be definitively proved, but only disproved if it conflicts with experiment or observation. In reality, hypothesis-testing is much more complex—for example, does a conflict between theory and observation signal a real problem with the theory or a flaw in the experiment? And we need to decide what counts as evidence in the first place. When scientists were debating whether the alpha variant of SARS-CoV-2, recognised in late 2020, was more transmissible than the original virus, Kucharski was struck by how specialists in different fields, such as epidemiologists and virologists, disagreed about where the burden of proof lay. “For certain biologists”, he writes, “it seemed Alpha wouldn’t be more transmissible unless the evidence came from the methods they had expertise in.”
In situations with any degree of causative complexity, especially in medicine, the issue is more often not about developing theories to explain the observations but of deciding what makes a difference to outcomes. Spotting a correlation between two variables, Kucharski explains, is just the first step on the “ladder of causation” proposed by computer scientist Judea Pearl. The next step is to ask whether a particular intervention will have the desired effect, which does not depend on having an explanation for the association itself. As epidemiologist Bradford Hill, a leader of the team that established the causal role of smoking in cancer, pointed out, we did not need a biological explanation of how smoking has that effect before making interventions that reduce cancer. Plenty of drugs today, not to mention general anaesthetics, are known beyond reasonable doubt to work even though it is not always known why.
Above the intervention rung of Pearl’s ladder is that of counterfactuals: being able to say what would have happened had the intervention not been made. Counterfactuals are arguably the most fraught aspect of proving a hypothesis, because they are by definition never observed, leaving uncertainty about what they imply for causation. On the one hand, an ineffectual intervention might be awarded spurious causal power. On the other hand, an effective intervention can blind us to what might have been. Even now, some people regard the predictions by epidemiological modellers at Imperial College London in the UK in March, 2020, of massive numbers of hospitalisations and deaths from COVID-19 without lockdown-type intervention as alarmist, because that outcome “never happened”. Even some knowledgeable statisticians have argued that infections would have fallen in any case through self-imposed physical distancing. Modelling of climate change faces the same challenge: if dire hypothetical scenarios prompt interventions, the predictions will never be tested and thus become vulnerable to accusations of alarmism.
In medicine such hypotheticals may be rendered immune to testing for another reason: ethics. Kucharski explains why the randomised controlled trial (RCT) cannot be the “gold standard” by which all treatments are assessed. Quite aside from all the usual (but sometimes neglected) complications of adequate sample size and hidden biases in sample selection, the RCT of a life-saving treatment that quickly shows high efficacy cannot ethically be continued by withholding the treatment from all. (Fortunately, if tragically, the efficacy of the COVID-19 vaccines was underscored by the experiences of those who did not receive them either because it was unavailable or refused.) [..]
Kucharski makes some pointed and timely comments about such standards in medicine and psychology. He argues that the statistical criteria adduced by early pioneers such as Karl Pearson and Ronald Fisher have become enshrined as almost talismanic emblems of proof today, most notoriously the p values used to assess the “significance” of a finding. For one thing, researchers are often confronting problems for which rigorous, high-quality discriminating data just do not exist. Kucharski explains how, in such cases, researchers can wring conclusions from sparse or indirect data. [..] In the end, however, we may have to evaluate the merits of making statements that include uncertainty and probability or of accepting that there are some things we just do not yet know. [..]
Proof is lifted from the good to the exceptional by its discussion of what happens to evidence and proof in the public and political arenas—the matter not of generating knowledge but of what people do with it. Here again the COVID-19 pandemic was revealing. Kucharski was as surprised and disturbed as many scientists by the vehemence with which apparently clear evidence about fatality risk, vaccine efficacy, and much else was rejected by many people. His response is nuanced and thoughtful. It is not enough, for example, to call conspiracy theorists lazy; Kucharski quotes one researcher who says “They spend a lot of time misinforming themselves”, and such misinformation has become easy to generate and propagate. Kucharski recognises that scientists can be too complacent in expecting to be believed, not least because of the distrust engendered by scientific fraud and irreproducibility. “A simplistic view of scientific evidence risks a path of too much faith followed by too much disillusionment”, he writes. “Societies must get better at communicating how the process works, monsters and all.” This book is an important part of that process.”
Full editorial, P Ball, The Lancet, 2025.3.29