CMU works to predict cases of COVID-19
Predicting real-time COVID-19 infections in the United States — with Carnegie Mellon University’s first predictions expected as early as March 22 — will involve dozens of volunteers making educated guesses about disease trends.
And anyone 19 and older can join in.
In successfully predicting realtime influenza cases nationwide in recent years, CMU’s Delphi Research Group keeps standings of its volunteers’ accuracy and declares end-of-season champions, as determined once the Centers for Disease Control and Prevention releases case totals long after the flu season ends.
The group will continue making flu predictions, while also turning its attention to the more challenging task of predicting real-time COVID19 cases.
The problem is, no previous COVID-19 epidemic has occurred, so information about how the disease spreads and how long it lasts is scarce, said Roni Rosenfeld, a CMU professor of machine learning who also leads Delphi.
“With COVID-19, I’m anticipating that aggregate human judgment will do better than computerized machine-learning analytics,” he said. “We will use the system we have now, and CDC wants us to
start focusing on COVID-19 on March 15. It’s going to be wobbly in the beginning, but we hope to get better every week.”
Ryan Tibshirani, a CMU associate professor of statistics and machine learning, is the group’s co-founder.
This “wisdom-of-crowds” protocol, and perhaps surprisingly so, represents an important modern-day method that CMU has employed successfully in predicting real-time flu cases, along with those likely to occur in the following weeks and months. The question is, will it work with COVID-19?
CMU’s weekly real-time estimates will be posted on its delphi.cmu.edu/nowcast website. CDC did not respond immediately to requests for comment but has made previous announcements about CMU’s analysis of flu activity, including being the world’s most accurate in 2017-2018 among 30 different forecasting systems.
In keeping with surveillance standards, real-time estimates will be expressed as a percentage of doctors’ office visits that are due to influenza-like illness. Future trends will be presented as probabilities of different outcomes, similar to the percentage likelihood of precipitation in weather forecasts.
In October, the CDC named CMU as a Forecasting Center of Excellence for providing highly accurate realtime totals of influenza cases over a period of years. That five-year “Center of Excellence” designation includes $3 million in research funding. The University of Massachusetts Amherst also has an Influenza Forecasting Center of Excellence.
Such real-time analysis goes far beyond adding totals of diagnosed cases in each state.
Instead, the mission involves estimating how many people are infected right now, including those with the virus but not yet diagnosed or showing symptoms, and where those cases are occurring nationwide. Accurate real-time totals, Mr. Rosenfeld said, are integral to determining future trends.
The CDC and other health agencies nationwide will use the estimates in strategies to mitigate the spread of the SARS-2-CoV virus, which causes respiratory problems, fever and diarrhea symptomatic of COVID-19. Other goals are preventing patient overload at key hospitals and having an available tool to determine when best to declare quarantines, cancel public events, and close schools, government offices and businesses.
In predicting flu cases, the group uses two systems it developed.
The Delphi-Epicast system uses “wisdom-of-crowds,” with volunteers making decisions predicated on historic flu data that Delphi provides them.
CMU’s Delphi-Stat system uses artificial intelligence — machine-learning technology — “to make predictions based on past patterns and input from the CDC’s domestic influenza surveillance system,” CMU said previously.
Both systems will be employed in its COVID-19 analysis, said Mr. Rosenfeld, who holds a doctoral degree in computer science.
The Delphi Group also was working on forecasting systems for dengue fever and various other infectious diseases, including HIV, drug resistance, and such epidemic viral infections as Ebola, Zika and chikungunya.
Wisdom-of-crowds
When historical data is available, as with the flu, computational and machinelearning analytics typically produce more accurate results. When data is lacking, as is the case with the new coronavirus pandemic, the wisdom-of-crowds protocol can be more accurate, Mr. Rosenfeld said.
An example of how wisdom-of-crowds works is having people guess the number of pennies filling a large jar. Throw out the extremes, and the median often comes close to the actual number, with accuracy generally increasing as the number of guesses increases.
Delphi will track respiratory illnesses in general, with an understanding that for now, most likely are flu-related. But as the flu season wanes, chances will rise that the majority of respiratory illnesses are COVID-19, he said.
In predicting real-time flu cases in recent years, Delphi Group estimates — compared with the CDC’s seasonend totals — were nearly spot-on.
Mr. Rosenfeld said the volunteers who spend more time studying the information that Delphi provides them, and pay attention to details while making necessary adjustments, typically have better results and place higher in the standings. Of the 70 current volunteers, 40 are regular participants.
Anyone 19 and older can participate in the process through the website, delphi.cmu.edu/crowdcast. No prior expertise is needed.
“The more participants, the more locations we can provide forecasts for,” Mr. Rosenfeld said.
While it might seem unscientific, the wisdom-of-crowds tool combines available information and human reasoning. It’s more reliable than one might expect.
“We look for volunteers to use their sensibilities and general reasoning rather than hunches,” he said, noting that people, as opposed to computers, “are good at integrating everything they know and making reasonable estimates, even in novel circumstances.”