How to glean useful info from mess of public COVID-19 data

With situation always in flux, clarity is crucial

The COVID-19 pandemic in Canada has ushered in a golden age of amateur charts on epidemiological curves. Everywhere you turn, someone’s got another chart.

The problem is that much of it’s based on public data that’s limited, incomplete and often outdated, meaning it can lead to wrong conclusions about what’s really happening.

It’s not all bad news. Public health agencies are slowly improving at providing detailed and up-to-date information. Testing backlogs are starting to clear and lab capacity is rising. But it remains very difficult to find detailed sources of data and accurately compare jurisdictions, and most charts should still be viewed with a skeptical eye.

Here’s a guide to the known problems with our data right now, and how to sort through the mess to find what’s truly useful.

PAY MORE ATTENTION TO HOSPITALIZATIONS

Most of the charts you see are based on “confirmed case” counts, which are tests for COVID-19 completed by a qualified lab. But these case counts present a very limited picture of a country’s situation — and sometimes a highly distorted one.

Case counts are always about two weeks behind due to the lag in people developing symptoms and getting tested. They also substantially underestimate the real case count due to limited testing capacity.

But Canada’s case count is particularly tough to analyze because of differences in how provinces test. British Columbia and Alberta, for example, started out testing widely and then tightened their criteria to high-priority cases. Ontario and Quebec are the opposite: they started out slowly and are now testing more quickly and widely (though Quebec ramped up much faster).

Furthermore, beware of misleading spikes in daily case counts. They tend to be due to backlogs being cleared or a change in how tests are processed, not a real daily increase in cases.

For all these reasons, case counts must be treated as just one indicator of a country’s situation. Hospitalizations and deaths are more reliable metrics, though both of these also come with caveats.

People hospitalized due to respiratory problems are a high priority for testing, so these numbers are less affected by varying testing standards and backlogs. Hospitalizations — particularly intensive care stats — are also crucial to track because COVID-19 becomes especially deadly when hospitals run out of capacity.

Provincial health agencies have been slow to provide hospitalization info on their websites, but most are now doing it. For the best example, Quebec’s health data agency posts a chart tracking hospitalizations and intensive care cases over time. Ontario published hospitalizations for the first time on Thursday.

However, hospitalization indicators lag and are more than two weeks old.

Deaths due to COVID-19 are also a more accurate picture of the spread of the disease, but this indicator lags even further behind hospitalizations. Deaths can also be disproportionately affected by outbreaks in long-term care homes.

LOOK FOR LOCAL DATA

The Public Health Agency of Canada recently started posting a daily epidemiological summary of COVID-19, including age group breakdowns and common symptoms.

But the report comes with a warning: they only have detailed information on barely half the cases, as they’re reliant on provincial data. Of the 9,017 confirmed cases in their April 1 report, the feds only had age and gender info on 5,590, and only had hospitalization status for 3,177.

A general rule to follow is that the higher up a government body is, the less updated their info probably is.

This has been starkly illustrated in Ontario.

For example, on Wednesday reporters asked Ontario’s associate chief medical officer why local health units were collectively reporting 65 COVID-19 deaths, while the provincial count was still 37. “I am kind of surprised at that discrepancy, I have to say,” Barbara Yaffe responded. She said the province’s numbers depend on health units inputting the data.

If you’re collecting data, first check the most local levels of health authority. Many are now reporting their own COVID-19 data, and it may take days for that data to trickle up to higher levels.

FOLLOW EXPERTS

Many people have the basic coding knowledge or software to make their own charts. That does not mean they know a single thing about epidemiology.

But there are many people who are subject matter experts and are commenting publicly on the data, often on their own social media accounts.

If you want to know what’s going on, find the infectious disease experts, critical care physicians, statisticians and public health authorities, and pay close attention to what they’re saying.

As a recent example, Ontario is now (finally) posting an open data set for its individual COVID-19 cases. On first glance, putting it into a graph could show the province is bending the curve.

But David Fisman, an epidemiologist at the University of Toronto’s Dalla Lana School of Public Health, ran that data through his own modelling with adjustments for known data flaws. His conclusion? COVID-19 is still on the march in Ontario.

“So sadly we must now retract our high-fives,” Fisman tweeted. “But it’s important to actually know where we’re at ... We are not out of the soup yet.”