Data collecting method called ‘among worst’ in country
Province’s reporting changes make it harder to identify COVID-19 trends, experts say
As countless epidemiologists, doctors, professors, economists and politicians have stated during the COVID-19 pandemic, good data is key to responding quickly and effectively to the virus.
Timely and accurate data has helped provinces such as British Columbia and Alberta begin to jump-start their economies after riding out the initial wave of infections.
In Ontario, the primary public sources of information on the spread of the virus have been the province’s 34 local public health units and Public Health Ontario, a government agency created to protect and promote health.
Since the beginning of the pandemic, Public Health Ontario has reported daily on the number of COVID-19 cases and deaths across the province. Over time, it has added other related metrics on hospitalizations, outbreaks in long-termcare settings and types of exposure.
But as the Star has found over the past two months, time lags, changes in how the data is reported and the use of unusual definitions of key metrics have raised questions about whether Ontarians are getting a true sense of the virus’s burden on society.
One of the main problems, the Star found, is that the province’s integrated Public Health Information System (iPHIS) is seriously under-reporting deaths due to COVID-19, particularly in seniors’ homes.
“There is massive room for improvement in how Ontario collects and reports data,” said Dionne Aleman, a professor of industrial engineering at the University of Toronto who models infectious disease outbreaks and pandemics.
“I assume PHO is operating with better data than they show to the public, but the data presented to the public in Ontario is among the worst of any province or country I have looked at.”
Here are four changes the province has made to the way it reports on COVID-19 and why they have complicated both the government’s and the public’s ability to understand the spread of the virus.
Patients tested vs. samples tested
On April15, the province quietly changed the way it had been reporting COVID-19 testing data. Until that day, it had published the total number of patients tested for the virus, but that metric disappeared only to be replaced by the number of samples tested each day. Because multiple samples are taken from some patients, it suddenly became impossible to know how many people the province was testing.
Ontario Health, the province’s new health super agency, which took over the public reporting of lab tests from Public Health Ontario on that date, told the Star it was determined that tracking the number of tests performed is a “more meaningful measure” of provincial testing capacity “because the difference between the number of people tested and number of tests performed can vary.”
Ashleigh Tuite, an infectious disease epidemiologist at the Dalla Lana School of Public Health, said knowing the number of people tested is important for calculating the positivity rate — the percentage of people who test positive on any given day.
Any day could include a large number of followup tests on people we already know are positive, she said. “You don’t necessarily want to be counting these extra tests that you’re doing on somebody who has already tested positive because that’s not providing you with additional information.”
Aleman said simply presenting the number of samples tested may give the impression the number of people tested has gone up, when in fact that may not be the case.
“It may still be the same. It may even be less,” she said, adding if the province wants to emphasize that the rate of testing is in fact going up, it should provide both the number of samples and the number of patients tested.
When asked about this change by reporters at the time, Health Minister Christine Elliott said the number of people who had been tested more than once is “not significant.”
No more negatives
At the end of March, the Ministry of Health quietly stopped reporting the number of patients who had tested negative for COVID-19. Up until then, this metric had been included with the number of patients who had tested positive, and those still “under investigation,” meaning their test hadn’t yet been processed by a lab.
Fortunately, it was still possible to calculate the number of people testing negative by subtracting positives from the total number of patients tested. But this became impossible once the province switched to reporting only the number of samples tested.
“In this case, it’s even more critical for us to know of the people who are being tested, what per cent of them are actually coming back positive, what per cent are coming back negative, because that can help at least give us some idea of how to extrapolate those numbers to the wider community,” said Aleman, noting the caveat that such an extrapolation would be imperfect because of a biased selection scenario where the province has only been testing people strongly suspected of having the virus.
Other provinces, including Quebec, continue to report a daily number of negative cases.
Numbers hard to compare
On May 19, the Ministry of Health announced it would no longer include in its daily reporting any COVID-19 cases and deaths in long-term-care homes that had not been confirmed by a lab test.
Before that date, Ontario’s data included some cases of COVID-19 that had been confirmed by the fact a patient had symptoms and was living in a facility that was experiencing an outbreak, but not a test — the logic being that if a resident got sick where the virus was present, there was no need to conduct a test to confirm the obvious.
The ministry said that removing those cases would result in a measure for long-term-care COVID-19 cases that could be comparable with the rest of the population, where only labconfirmed cases are reported.
But because the province never did release the total separately, it is not clear how many non-lab-confirmed cases and deaths suddenly disappeared from the province’s data.
Aleman noted that when metrics like this are changed suddenly, it becomes hard to identify trends over time.
“We can’t really look back and know what changed if things start to look different now than what we expect,” she said, noting that separating or combining certain metrics is not a bad thing in and of itself. “But you have to do that retrospectively through all of your data as well.”
An inaccurate picture On May 22, Public Health Ontario quietly reclassified more than 5,000 cases of the virus previously thought to have no source of transmission.
Up until that date, the agency had been telling the public that between 35 to 40 per cent of overall reported cases of the virus in the province had no known origin. On May 21, there were 8,471cases with this classification. The next day, the agency moved 5,309 cases out of this category because it turned out that these cases were outbreak-related (i.e. there was, in fact, a known source of transmission).
As a result, the percentage of cases thought to have no known source — the key measure of “community spread” — dropped from an average of about 36 per cent for the month before to about 13 per cent in one day.
“If it’s just — all of a sudden out of the blue — all these cases are updated to say, ‘oh, you know, we’re just going to say that as of today we know how all these people became infected,’ well, now we don’t really know what’s been happening,” Aleman said, noting that knowing how people became infected is a good thing.
“Again it comes back to that issue of needing to understand trends in order to make a decision for the future and here we just have no trends because suddenly the data is totally different.”
She suggested that there should be a retroactive update to indicate on what day we learned how they became infected.
University of Waterloo professor of economics Mikal Skuterud, whose research includes linking return-to-work rates within regions to changes in COVID-19 case numbers, said generally that updates and changes to the data can be a good thing: “You want to get it right.”
But he notes that there is a cost.
“It makes the research more difficult for sure,” he said, noting that if there is overcounting or undercounting occurring, and it’s related to the extent to which regions are sending people back to work, “it’s going to throw off all your estimates.”
“You’re going to say sending people back is a really bad thing. Or the opposite, even scarier, sending people back is not a bad thing, when in fact it is. Then you’re completely the wrong policy prescription,” Skuterud said. “We want to get it right.”
Abetter way
In contrast to Public Health Ontario’s daily reporting, several regional public health units have started publishing comprehensive and easy-to-understand data on local cases.
For example, York Region’s interactive dashboard has been updated again and again to provide a variety of figures, charts and models. Crucially, users can also download the underlying data.
Similarly, Waterloo Region provides a variety of data visualizations, including epi-curves by symptom-onset date and case-reported date.
Back in April, frustrated with the province’s “outdated and clunky” system, Toronto Public Health built its own COVID-19 tracking system, known as Coronavirus Rapid Entry System or CORES. That tool now underlies the city’s COVID-19 information portal.
None of these existed before the pandemic, but similarly detailed reports are now being published daily across the province in Durham and Peel Regions, Ottawa, Kingston, Simcoe Muskoka and more. Even tiny Hastings Prince Edward publishes a variety of easy-toread charts and maps with case counts.
“If Ontario wants more public compliance with physical distancing and PPE guidelines, they need to clearly illustrate on their webpage the trends that are happening, including trends on testing so that we in the public can understand if lower infection numbers are likely to be driven by lower testing rates,” said Aleman.