Rush to publish studies touchy

Analysis by Stanford trio shows risk of quick release of info

In the race to combat the COVID-19 pandemic, the world’s scientists have embraced a radically new method of disseminating information about their research, offering it quickly and without filters in the effort to understand and control the deadly disease. But their new communication model is striking at the heart of scientific integrity, publicizing research that has been corrupted by speed, sloppiness and opacity. And now the academic world is being roiled by a question for which millions of lives hang in the balance: Is the public being wellserved by the fast and free flow of research, or dangerously misled? Nowhere is the question over scientific conduct louder than at Stanford University, where a trio of researchers are accused of promoting faulty analysis and “tipping the scale” on antibody studies that they say proves the virus is more widespread and less lethal than we feared, and that public health restrictions are too strict. Now the university, which has also come under fire, is investigating the veteran professors’ research, a significant step in a world that cherishes credibility and reputation. COVID-19 has changed, perhaps forever, the way scientists do their work. Faced with an overwhelming sense of medical urgency, the practice of using “preprint” servers — online platforms that allow scientists to share preliminary findings quickly, before formal and protracted evaluation — has soared in popularity, with bold and unvetted claims going straight to the public. Peer review? It’s raucous and transparent, crowdsourced — via email and Twitter — by scores of commentators. At no other time in his

tory have so many critical insights been learned so quickly, upending the slow and insular traditions of science. According to an analysis by Aleszu Bajak and Jeff Howe of Northeastern University, more than 10,000 Covid-related papers have been published since January. By comparison, only 29 studies were published during the 2003 SARS pandemic. But a lot of what researchers are producing is rubbish, experts say. How many of us will die on ventilators? A study in the Journal of the American Medical Association reported a terrifying 88% fatality rate, and then, in a little-noticed correction two days later, dropped the number to 24.5%. Does hydroxychloroquine, endorsed by President Donald Trump, help? At least 18 clinical trials enrolling more than 75,000 patients are testing nearidentical hypotheses. So far, none of them have shown any promise. When launching tests of the new Moderna vaccine, researchers didn’t wait to see how well it prevents infection in animals before trying it in people, according to STAT News. And in this week’s media blitz, based on tests involving just eight people, the company revealed very little data about the vaccine’s success. Fastidious research standards may seem a luxury during a global pandemic that has claimed more than 335,000 lives, Mcgill University biomedical ethicist Dr. Jonathan Kimmelman said. But now, more than ever, it’s critical to do good work, he said. “Any time you present research findings where the ink is not yet dry,” he said, “you need to be crystal clear that your findings are preliminary, provisional and unvetted.” When the Stanford team — Drs. Jayanta Bhattacharya, John Ioannidis and Eran Bendavid — released the first draft of their Santa Clara County-based preprint, the news was stunning. The nation’s first study of its type, it found that the virus was astoundingly 50 to 85 times more prevalent than presumed. But that meant the death rate was far lower. Yet the project raised eyebrows from the start. Even before they started collecting data, the researchers openly questioned stay-at-home orders. Ioannidis wrote a provocative article arguing that if COVID-19 is less deadly, widespread restrictions “may be totally irrational.” A Wall Street Journal editorial by Bhattacharya and Bendavid was entitled “Is the Coronavirus as Deadly as They Say?” Bhattacharya revisited that theme in the Hoover Institution and Fox Nation program “Questioning Conventional Wisdom.” When their preprint was published, its conclusions backed the trio’s policy arguments, and it was saddled with statistical problems. It failed to describe key calculations and made at least five material mistakes, according to statistician Will Fithian of UC Berkeley. The population-weighted intervals in a table were miscalculated. The authors plugged the wrong interval into a formula. They made two math errors in executing that formula. And, misreading their test kit’s manufacturer insert, they used the wrong numbers for the antibody test’s specificity. “They’re the kind of screw-ups that happen if you want to leap out with an exciting finding and you don’t look too carefully at what you might have done wrong,” said Andrew Gelman, a professor of statistics and applied science at Columbia University. Even as Fithian and other outside experts alerted the team to errors within two days of publication, the researchers promoted their findings in national media, from NPR to Fox News. Without initially disclosing his role in the study, coauthor Andrew Bogan published an op-ed in The Wall Street Journal asking, “If policy makers were aware from the outset that the COVID-19 death toll would be closer to that of seasonal flu … would they have risked tens of millions of jobs and livelihoods?” Knowingly stepping into the partisan fray “injects their science blatantly into a political fight,” said Stanford’s Hank Greely, a prominent medical and scientific ethics expert. “It’s letting their research be weaponized for political ends, which guarantees it will be distorted.” Their paper was revised and republished on April 30, addressing many problems and estimating a new infection rate on the low end of the original range, but outside experts say it’s still flawed. Meanwhile, the team calculated startling new infection estimates for Los Angeles County — using the same methodology and no explanation of how the numbers were calculated. Since then, questions have been raised about the project’s funding. Who is Bogan, a Palo Alto-based manager of global equity funds who left science two decades ago? There also are suspicions about the role of David Neeleman, the JetBlue founder who is outspoken about lifting restrictions and said he “consulted” with the team. He also contributed $5,000 to Stanford for the researchers, according to Buzzfeed News. Even more incriminating is an anonymous whistleblower complaint to Stanford, obtained by Buzzfeed, which asserts that microbiologist Taia Wang, who performed the team’s assays, refused to be an author and wrote that she didn’t trust their test. Respected Stanford pathologist Scott Boyd also tested their samples and then distanced himself from the work. “In all studies — and especially those that could impact public health — rigorous design, statistical correctness and responsible reporting of findings are essential,” said Robert Tibshirani, a professor in Stanford’s Departments of Statistics and Biomedical Data Science. “And transparency is a key component throughout: Authors should provide full details of what they did, their computer code, and the source data, except when protected by privacy rules.” Stanford’s reputation will pay a price for publicizing this work, “because people will remember that ‘the Stanford study’ was hyped but it had issues,” Columbia’s Gelman predicted. “The next study out of Stanford will have a little less of that credibility bank to borrow from.” Stanford said it is reviewing the team’s work, insisting “the integrity of Stanford Medicine’s research is core to our mission. When we receive concerns such as this, they are taken extremely seriously,” Stanford Medicine spokesperson Julie Greicius said. The team staunchly defends its research, saying the study was done in accordance with good research practice and university policies and approval. They revised their preprint with alacrity, addressing the criticisms, they add. “I understand that the university is investigating some of the allegations that have been made in the media and expect that the study will be fully vindicated,” said Bhattacharya, senior author of the study. Co-author Ioannidis expressed gratitude for the public critiques, saying “this is exactly how science should work. … Preprints are not immutable final versions.” Political bias played no role in the analysis, he said. “I have repeatedly stated that science should not mix with ideology and I find the allegation of conservative ideology for myself to be very weird given my entire career,” he said. “We are facing a very serious problem with many human lives being at stake. My only goal is to do the best possible science and to save lives.” Mistakes and wrong turns are endemic to the scientific process, even when scientists try their best to be careful, UC Berkeley’s Fithian said. “We’re all racing against time.” “But I think science can and must do better. If you learn you have made mistakes, acknowledge them publicly, before your next media appearance.”