THE MODELS MUDDLE
Can scientific models be trusted? The gap between expectation and outcome has raised questions about the projections that have shaped lockdowns and disrupted economies around the globe
The early figures read like a dystopian, end-of-the-world movie script. As the Covid-19 pandemic was taking hold, epidemiological models predicted 350,000 deaths in SA without a lockdown, 96,000 in Sweden (26,000 under a best-case lockdown scenario), 2-million in the US this year (peaking in July), and 500,000 in the UK.
Thankfully, these models are wrong. SA has so far had about 500 deaths from Covid19 — a number that is, however, expected to rise exponentially for months. There have been about 4,029 deaths in Sweden, where death rates have been in decline since April 22; about 100,000 in the US, where daily figures have dropped since April 15; and about 37,000 in the UK, where daily numbers have declined since April 9.
This difference between expectation and outcome has raised questions about the scientific models that have shaped lockdowns around the world and left economies reeling.
In the UK, the Imperial College London model that influenced the government decision to lock down the economy could best be described as the song of Covid-19 models: it stuck like an earworm, made the world sit up listen and is now, if UK headlines are anything to go by, somewhat despised.
The model, which predicted 500,000 UK infections, was released on March 16 and, according to media reports, informed Prime Minister Boris Johnson’s decision to announce a lockdown a week later.
After weeks of pressure to make the coding behind its model public, Imperial College study lead Prof Neil Ferguson relented. It’s a decision he may come to regret.
The coding the model relied on has been pulled apart by health experts and global coders, derided as the “worst coding I have ever seen”; as a 13-year-old, 15,000-line code not up to industry coding standards; and as “riddled with bugs”. One senior industry coder reportedly said he would have fired anyone who produced that code for him.
When tested by epidemiologists at the University of Edinburgh, the model gave different answers when it was run on different computers — and different ones again when rerun on the same computer.
In SA, similarly dire predictions have been made about the course of the pandemic.
Stellenbosch University’s SA Centre for Epidemiological Modelling & Analysis (Sacema), for example, released a model in early March suggesting a death rate of 350,000 if no lockdown were implemented.
Health minister Zweli Mkhize has been at pains to deny that the government’s lockdown was based on that figure. But he admitted in a virtual meeting last week that the first models the government had sight of showed far, far higher fatality figures than the figure of 40,000 some local models have converged on.
What do the models in SA say now? And can we trust them?
The Actuarial Society of SA (Assa) predicts about 48,300 deaths, with a high end of 88,000. The SA Covid-19 Modelling Consortium model, designed by a range of academics and used by the government for planning purposes, predicts 40,000 to 48,000 deaths.
The FM tried to contact epidemiologist Dr Harry Moultrie and other members of the consortium after Moultrie promised “transparency in [the] interests of democracy”, but they did not answer, or even acknowledge, the questions. Assa answered the FM’S questions in detail.
In its public documents, the SA consortium warns models are based on assumptions which can be incorrect. The model write-up states, in distinctive bold font: “These projections are subject to considerable uncertainty and variability. Estimates will change and improve as the epidemic progresses and new data become available.”
To illustrate just one of the difficulties facing modellers: many people who contract Covid-19 remain asymptomatic or have such mild symptoms that they don’t seek medical attention and so aren’t tested, making disease prevalence difficult to measure.
Sacema’s Prof Juliet Pulliam said at the virtual launch of the consortium’s modelling results that as deaths increase in SA, this data will be used to improve the consortium model.
Deaths can be a better indicator of infections. But even this is tricky. European countries, US states and cities abroad are now measuring “excess deaths” — a measure of unexpectedly high deaths compared with previous years. These figures suggest there are more Covid-19 deaths than have been officially counted.
So if even death figures are uncertain, one can only have sympathy for those modelling data for a six-month-old disease that contains so many unknowns.
Models on new diseases have been wrong before.
Barry Childs, who is in charge of the Assa model, says initial models on HIV, for example, “were based on uncertain assumptions and were therefore often wrong. Today we have a pretty solid idea of how HIV works and models are therefore very reliable.”
So how much store are we to put in such models if the input data is likely incorrect?
them asymptomatic and not detected.
Remove children from the equation — they seem less susceptible to infection — and that means about one in three adults in SA will have been infected within six months.
The model doesn’t explain why this is likely in SA, when no country in Europe has come close to such a number. There, studies of antibodies to the coronavirus in people’s blood show a prevalence of 4%-7% in the population.
Studies in New York, in the US, however, put the number much higher, at an infection rate closer to 20%.
When it comes to predictions of fatalities, these are calculated as a percentage of the number of people who get infected — so if modellers get the infection numbers wrong, the projected death rate could be inflated.
Economist Alex van den Heever, who put together a completely different model for the National Treasury in March, argues the point of the lockdown is to “suppress the epidemic”. To assume it will “run its course” and infect almost everyone before subsiding is, he says, to plan for failure.
This is exactly what the Assa and consortium models assume.
Childs admits this isn’t what has been seen in Europe, but it could be because the lockdowns worked.
Another explanation for the high rates is the type of model used: the susceptible-exposedinfectious-recovered (SEIR) model. This assumes everyone who has not had a new disease is susceptible to it, and if exposed could become infected.
“The one big assumption on the SEIR model for a novel virus is that everyone is susceptible, which is why the projections of total infected numbers are so high,” says Childs.
He says the Assa team is looking for other explanations “in the emerging data” which shows not everyone gets the disease.
“The current Assa Covid-19 model predicts high prevalence. Given the novelty of the virus, without effectively curbing its spread most of the population would be exposed and infected, the vast majority being asymptomatic,” he says.
As Assa tries to refine its model, it is “considering allowing for variation in susceptibility”. This means trying to calculate that some people will get infected, while others exposed to the virus just won’t catch it.
In the end, what everyone agrees on is that all models are highly limited by their assumptions. The consortium even warns the government to use its model “with caution”. Yet the power of models to shape drastic government decisions will be long debated, even after Covid-19 infections have waned.