Scientists must take responsibility for their own credibility
You need not look beyond the coronavirus pandemic to see that science can counter a deadly health threat. But that progress was hindered by another ongoing epidemic: A lack of faith in science.
This unhealthy suspicion hindered widespread acceptance of effective therapies and vaccines, leading to needless suffering. As a physician and scientist, I know that mistrust of science jeopardizes its health benefits. And it is worse when scientists themselves provoke suspicion.
The threat from within
Consider a recent commentary about scientific integrity by editor-in-chief of Science magazine, H. Holden Thorp. Thorp cited Stanford University undergraduate Theo Baker, who led investigative reporting at the student newspaper and questioned the integrity of scientific findings published by Stanford’s president, Marc Tessier-Lavigne.
Baker’s journalism won him the coveted George Polk Award and led to Tessier-Lavigne’s resignation. This was more than a role reversal where a student taught his teacher. It was a harbinger of criticism to come.
That case likely inspired an article in Nature about untrustworthy clinical trials. It spotlighted John Carlisle, an editor of the journal Anaesthesia, who examined data from randomized clinical trials published in the journal over a three-year period. He found that 26% of them were difficult to trust because the data
were faked or flawed. He called them “zombie trials.”
One contributing factor was an inability to access the data from those trials. When the raw data collected by researchers is hidden, it casts doubt over the findings they derive from that data. If similar deficiencies are widespread in the scientific literature, it would be intolerable. For the sake of our own credibility, and the ability of our work to truly improve people’s lives, physicians and scientists must take this problem seriously.
Changing incentives
Irreproducible research — that is, studies that seem authoritative but whose results can’t be replicated by independent scientists — is a complex and multifaceted problem. It can generally be traced to flaws in study design, experimental techniques, protocols, data analysis or reporting. The good news is that something can be done about it. Putting in place consensus-based standards, carefully designed protocols and data analyses would increase reproducibility and the returns on funding investments. These standards can be required by the most respected journals, and will in turn boost trust in science.
Many journals have these standards, but there is no agreement on how best to implement them. At the same time some institutions launched programs to reduce irreproducible research results. Among them are the Center for Open Science at the University of Virginia and The Scientific Standards Hub at Frederick National Laboratory, where I work. Still more needs to be done. One way forward is to change scientific incentives. The current measures of success are about gaining grants, publications, promotions, tenure and prizes. Not surprisingly, this reward system drives how science is done. It explains why some scientific publications do not stand the test of time. Scientists are motivated to report their findings before others scoop them. But our pursuit should be to uncover durable truths about nature. We must increase the cost of publishing bad research, without undermining the process of trial-and-error on which science is based.
Stress-testing science
Science can benefit from another strategy to increase confidence in any discovery. The influential twentieth century scientific thinker Karl Popper felt that the prevailing experimental method was fraught since it sought to prove rather than to disprove a discovery. Like anyone, scientists are enamored by their own theories. But at times they are misled by them. This could affect how they design or interpret their studies.
To counteract this, Popper offered a provocative proposal called empirical falsification. He encouraged scientists to challenge rather than verify their ideas. If this were done rigorously and the theory stood up, they might be on to something.
Popper’s idea seems straightforward, but it is difficult to put into practice. That is because none of us works in isolation: We have competitors who encourage us to publish our work as soon as we can. If we were asked to show that experiments were done to question a hypothesis, it might slow the pace or increase the cost of our work.
Consider, however, the cost both to individual scientists and to humanity of the alternative: the time and expense used to vainly advance a flawed theory. It is said that science corrects its mistakes. While true, this does not diminish the fact that any correction will waste precious time. Rather than sort things out after publication, it is best to avoid errors from the start.
Popper’s approach could bolster faith in scientific discoveries and reduce the need for retracting publications. Moreover, it would hasten benefits to victims of frightening maladies. Could those sufferers ask any less of us?