Calling Out Junk Science
If you hear about some important finding from a new scientific study, be skeptical: There’s a good chance it’s not true.
That’s a key point in the National Association of Scholars’ new report, “The Irreproducibility Crisis of Modern Science,” by David Randall and Christopher Welser. And it should raise eyebrows.
“Half the results published in peer-reviewed scientific journals are probably wrong,” write Randall and NAS President Peter Wood in a recent op-ed column, confirming the claim by Stanford professor John Ioannidis that made headlines in 2005.
Since then, they note, “researchers have confirmed [Ioannidis’] skepticism by trying — and often failing — to reproduce” the results of “influential” research. One 2012 study was unable to replicate the findings of fully 47 out of 53 “landmark” reports in hematology and oncology. Scary.
And while the best way to see if a finding is valid is to try to reproduce it, the scientific community doesn’t do that remotely enough.
The NAS paper cites political “threats” to the validity of scientific studies but focuses more on “the use and abuse of statistics.” That is, many findings can’t be reproduced because they’re wrong.
Yet shoddy research can become the basis of new medical recommendations, social or political movements or government laws and regulations.
The biggest flawed “newsmakers” have been in psychology. Wood and Randall cite studies on “negative stereotypes” and “implicit bias,” which have been widely embraced, especially on the left. Yet researchers have been unable to reproduce the findings on those subjects. In 2015, they were able to validate only 39 percent of the conclusions in 100 key psychology studies.
Also badly troubled: climate-change science, “a farrago of unreliable statistics, arbitrary research techniques and politicized groupthink,” as Randall and Wood put it.
The biggest danger, of course, is when bogus findings lead to new laws, policies or ways of living. That has happened, for example, at the Environmental Protection Agency, where, says Wood, “opaque and often doubtful claims” are used as the basis of new regulations.
The EPA, he argues, is known for pushing “scientists to produce the results [it] wants in order to justify policies it wishes to implement.” Good for EPA boss Scott Pruitt for now seeking to limit studies his agency relies on to only those that make their data and methodology public — so other researchers can attempt to reproduce them.
NAS’s paper offers several reforms, including more resources for studies that seek to reproduce findings. It also urges the government to avoid basing new regulations on studies that don’t meet “strict reproducibility standards.”
Anyone interested in honesty in research should rally behind the paper’s recommendations. Meanwhile, accept findings of scientific studies at your own risk.