Why are the models so off?

Experts mention their lack of knowledge about the virus and how fast people changed their behavior

The researchers trying to model how many Americans might be hospitalized or killed by COVID19 have been criticized from multiple sides, with allegations they needlessly scared people or gave leaders false reassurances it was OK to reopen businesses.

In Colorado, recent projections that the state could run out of hospital beds as early as September appear unlikely to come true, as new coronavirus infections slowed down after a rapid increase in mid-july.

Other projections far underestimated the outbreak’s toll, predicting about 300 would die of the virus by early August. As of Friday, 1,736 Coloradans had been killed by the virus, and an additional 121 people had died with it in their systems.

Experts point to at least three factors to explain why projections haven’t hit the mark: the inherent limitations of models, holes in our knowledge of the new virus and people changing their behavior faster than anticipated.

Jimi Adams, an associate professor of health and behavioral sciences at the University of Colorado Denver, said there’s often a misunderstanding of what models do. Some, like new two-week predictions from Google, try to forecast what will happen. Others try to lay out what could happen under different possible scenarios, including if no policies change, which isn’t likely during a pandemic, he said.

Think of the difference like Ebenezer Scrooge’s question to the Ghost of Christmas Future: “Are these the shadows of the things that will be, or are they shadows of the things that may be only?” Basically, Scrooge is asking if he has any power to avoid a future where he dies alone and Tiny Tim dies young. While the ghost doesn’t answer, Scrooge changes his behavior to try to avert the worst-care scenario, and the future that comes is less bleak.

We saw something like that happen as states and cities required people to stay home or wear masks, and as people reacted to a resurgence in cases by avoiding high-risk settings, said Beth Carlton, an associate profes

sor of environmental and occupational health at the Colorado School of Public Health.

It’s unusual for people to change their behavior as quickly and dramatically as they have in response to COVID-19, she said. That’s good for public health, but makes it more difficult to project what might happen.

“People do have the power to change the direction of these forecasts,” she said. “Some of the power is individual, and some is policy.”

Up-and-down swings

Cellphone location data used to estimate social distancing and survey responses about mask wearing show people get strict about distancing when media report cases are going up and relax when the picture starts to improve, which wasn’t what scientists initially expected, said Ali Mokdad, a professor at the University of Washington’s Institute for Health Metrics and Evaluation. While it’s good that people are taking precautions when they feel at-risk, relaxing as cases fall just sets up the next spike, he said.

“Unfortunately, we’re going to see these up-anddown swings unless people adapt to the virus,” he said.

The IHME’S latest model was close on deaths in Colorado by early August, projecting 1,869 fatalities by Wednesday — 18 more than the total number of people who died after contracting the virus. Looking forward, it estimates between 2,149 and 5,006 people could die by Dec. 1 if almost all Coloradans wear masks in public, and the toll could range from 2,744 to 18,244 if they don’t.

Initially, models had to rely on data from China and Europe, which didn’t always apply to the American situation, Mokdad said. The picture was also limited in the early months of the virus’ spread in the U.S. because a lack of testing meant no one really knew how far the virus had spread, he said.

“Our model evolved as we have more data,” he said.

Katie Colborn, research director for surgical outcomes and applied research on CU’S Anschutz Medical Campus, had worked on forecasting malaria outbreaks. With malaria, there’s about a decade’s worth of high-quality data about weather and cases, so it’s possible to predict that an unusually rainy season will likely result in more cases, at least in the short term, she said. For COVID19, the early warning signs aren’t nearly as clear.

Researchers are still trying to work out how the disease transmits, including how long it could stay in the air after an infected person speaks, Colborn said. Other questions, like why it spreads widely in nursing homes, but apparently not in child care centers, also need answers, she said.

Refining approaches

The Policylab at Children’s Hospital of Philadelphia also is refining its approach as knowledge evolves. In May, they had predicted there was little chance of a renewed outbreak in Colorado Springs this summer, but that Denver could be in danger. Both cities have seen an uptick in coronavirus infections, but the growth is steeper in El Paso County, where residents seems to be practicing less social distancing.

In more recent projections, they’ve shifted to focus on the next one to four weeks, said Brian Fisher, an associate professor of pediatrics and epidemiology at the University of Pennsylvania. The farther out you look, the more likely you are to be off, because behavior or policies may have changed, he said.

They also have learned that the effects of temperature and humidity are more complicated than originally believed, Fisher said. This spring, they had thought hot days would work against the virus, but that hasn’t happened, perhaps because people started congregating in air-conditioned spaces, he said.

The Policylab model uses cellphone data about visits to non-essential businesses, like stores and restaurants, to estimate social distancing, Fisher said. That doesn’t account for other precautions people might have taken, like wearing masks or getting takeout instead of sitting in a restaurant, so it’s possible some counties won’t see the expected rebound in cases as people go out more, he said.

“It may be that they’ve reduced social distancing, but they’re doing it in an intelligent way,” he said.

Models will differ based on the quality of their data, as well as their makers’ mathematical choices and assumptions about human behavior, said Aditya Khanna, director of network modeling at the University of Chicago. While specific numbers may not reflect what happens, most have been accurate in predicting that if people began living like they did before the pandemic, there would be a spike, he said.

“There are some things that are very clear,” he said. “In the broad picture, there’s a lot that we have learned already.”