National Post (National Edition)
COVID fight has relied too much on stats, some say
`Models are not a panacea or magic ... They are a tool'
THE KIND OF LOCKDOWNS THAT WE HAVE SEEN HERE IN THE WEST ARE JUST AN IMPOSSIBILITY IN MANY GEOGRAPHIES … THEY WOULD LEAD TO MASS STARVATION. THESE MODELS HAVE REAL CONSEQUENCES IF THEY ARE APPLIED HAPHAZARDLY. — DR. ZULFIQAR BHUTTA
The first several pages of the Public Health Agency's epidemiological update last week struck an optimistic note.
The number of COVID-19 cases, hospitalizations and deaths were all falling steadily after a major early winter surge.
Then came slide 13 in the presentation, an alarming new model of what could happen next.
Based on more transmissible variants of the virus spreading widely, it showed an almost vertical spike in the case count should public health restrictions be eased. An only slightly less mountainous climb would occur into March and April, even if current lockdown measures were kept in place.
The only way to avoid a feared third wave of COVID-19 illness, the report suggested, was by imposing new restrictions.
It was a sobering message, but hardly unfamiliar by now. Almost since the pandemic began, scientists have been using high-level mathematics and powerful computers to churn out models of how the virus might spread and affect Canadian society, and what mitigation measures would do to slow it down.
Few experts deny that modelling has been of at least some value, while measures taken in response seem to have staved off nightmare scenarios of sickness and death.
But a year into the pandemic, skeptics worry that the battle against COVID-19 has relied too heavily on mathematical projections that can be undermined by sparse data and an unpredictable virus — and are often released with scant mention of their limitations.
The inadvertent side effect may be public distrust of government officials fighting the coronavirus, critics argue.
“When you put out a model that is not credible and it turns out to be ridiculed, you've threatened the integrity of the whole health-care system,” argues Ed Mills, a Vancouver-based epidemiologist and part-time McMaster University professor. “So don't be surprised if people start not believing what the public health agencies say.”
In fact, there's already skepticism about those federal predictions of a third wave in March fuelled chiefly by the B.1.1.7 mutation of the SARS-CoV-2 virus — the so-called U.K. variant.
Ontario released a similar model on Feb. 11, and a recent decline in cases seems to be flattening off.
But Mills, an adviser to the Gates Foundation, and others argue there simply is not enough evidence to conclude the trend is all but inevitable.
Projections for Canada are based on what happened in the U.K. in late December, when the more-transmissible variant contributed to a troubling surge in cases, notes Dr. Zain Chagla, an infectious disease specialist at McMaster University.
But it's unclear, he says, how much of that spike can be blamed on the variant, and how much on the somewhat eased restrictions in place in England at the time, plus the inevitable rise in contacts around Christmas.
In Denmark, meanwhile, the U.K. variant is prevalent, yet the case count has fallen for weeks, albeit in the wake of a strict lockdown, Chagla notes.
“In terms of predictions for a third wave, I just don't know,” says epidemiologist Prabhat Jha, director of the Toronto-based Centre for Global Health Research. “We don't have enough data.”
Despite such reservations, though, modelling has many boosters in the scientific community.
While no one should view the science as some kind of crystal ball to peer into the future, it has played a “huge role” in pandemic planning, says one prominent expert in the field.
“I think it's been remarkably successful,” says Kumar Murty, a University of Toronto mathematics professor and chair of Ontario's science modelling table. “(But) models are not a panacea or magic or anything. They are a tool. They are one tool to help understand a complex phenomenon.”
It's easy to complain about a model being wrong when the actual outcome differs from the projection, he says.
But such criticism ignores the changes in policy and personal behaviour that followed the model, often in reaction to it, Murty says.
“We never wanted to be left with the scenario that it did happen and we weren't prepared,” Chagla says.
To try to foresee what might happen — and guide preparations — modelers crunch real-world data through mathematical equations to approximate the future course of a disease or the impact of interventions.
The field has been around literally for centuries, the first recorded model predicting in 1760 that universal smallpox inoculation would boost life expectancy by more than three years.
The math application has boomed in recent months as scientists worldwide have put aside their regular pursuits to address the COVID-19 crisis.
The medrxiv.org preprint site and the PubMed registry list between them about 1,300 studies that mention modelling and COVID-19 in the title or abstract. But there have been questions since the beginning about some of the results.
U.K.'s Imperial College jolted the world with modelling that indicated the U.S. would see 2.2 million deaths if it did not take action to limit COVID-19's spread.
In Canada, early models foresaw an almost apocalyptic impact on health care.
The University of Toronto-led COVID-19 Modelling Collaborative suggested last March that under a “conservative” scenario as many as 5,000 coronavirus patients at a time would need to be treated in an intensive-care unit by midMay, many requiring ventilators that were in limited supply.
To make room for that kind of deluge, hundreds of thousands of elective surgeries were cancelled across Canada, as governments desperately sought out new supplies of breathing machines.
COVID-related hospitalizations didn't peak in Ontario until last month during the pandemic's second wave, when ICUs cared for up to about 400 people daily.
It was a serious strain on the system and weary health professionals, but less than a tenth of what that initial model had predicted.
Modelling early in the pandemic also suggested schools could be a major vector for the virus, leading to widespread closures in the spring.
More recent projections have reached a different conclusion, though the role of schools as a possible accelerator of the pandemic continues to be hotly debated.
Beyond this continent, models by Imperial College predicted a huge death rate in South Asia, more than 2.5 million fatalities in Pakistan alone by last November, notes Dr. Zulfiqar Bhutta, co-director of the Centre for Global Child Health at Toronto's SickKids Hospital.
Others projected a smaller, but also hugely inaccurate toll, he noted.
Pakistan has had about 12,000 deaths and, as it turns out, politicians largely ignored the foreign modelling, Bhutta says.
Perhaps in response to such prognostications, though, India imposed lockdowns last spring that caused widespread hunger among migrant workers. Its COVID death rate has also remained low, one of the mysteries of the pandemic.
“The kind of lockdowns that we have seen here in the West are just an impossibility in many geographies … they would lead to mass starvation,” Bhutta says. “These models have real consequences if they are applied haphazardly.”
Mills says models produce by Seattle's Institutes of Health Metrics have become the world's most reliable. But the University of Washington institute generated controversy itself when it predicted in April that the U.S. would see only about 60,000 total deaths, a model then-president Donald Trump seized on as he tried to downplay the pandemic's gravity. What, if anything, went wrong? Especially early on, models likely erred by treating populations as homogeneous, though some people were more or less resistant to the SARS-CoV-2 virus because of pre-existing immunity or other reasons, Jha says.
Mills argues that mathematicians often have failed to validate their models by running past data through them to see if they could accurately predict what has already happened. Or such data was simply unavailable.
And much is unusual or unknown about the virus, he says.
Murty says modellers have, in fact, worked hard to incorporate new evidence about the disease into their work.
The U of T's Fields Institute for Research in Mathematical Sciences, which he directs, even conducted a survey to determine what percentage of people followed public health edicts on the coronavirus. The finding was about 70 per cent, information that can now be input into models on the potential impact of anti-COVID measures, Murty says.
Still, Jha worries that governments have placed too much emphasis on modelling and not enough on empirical study of the pandemic as it unfolds.
He says he and academic colleagues, for instance, had to take the initiative to conduct a large “sero-prevalence” study to examine what percentage of Canadians have antibodies to the new virus — hard evidence of how widely it has spread.
“Every past public health pandemic that has been successfully dealt with has also created new knowledge,” says Jha. “Aside from the extraordinary success in vaccines, this pandemic has not produced extensive new knowledge.”
If there is one thing that critics and champions of modelling seem to agree on, it is the need to communicate the field's limitations, something health officials, politicians and the media sometimes ignore.
Failing to explain fully the basis for projections and the fact they could well be wrong, risks fuelling conspiracy theories and COVID denialism, undermining the whole effort, Chagla says.
“We're a society that has been so paralyzed by COVID … everybody is looking at this stuff and saying `Where did this come from?' ” he notes. “You can't go `Here it is, see you guys later' … It's really important to be transparent.”