Efficacy debacle shows why vaccine league tables are very unhelpful

If there is one key rule for staying on a medical regulator’s good side, it is “don’t revise published data”. Yet AstraZeneca did exactly that, revising the efficacy of its Covid-19 jab from its US trial last Thursday.

The company, having announced a 79 per cent efficacy, within three days re-issued a lower finding of 76 per cent, following a rebuke from independent scientists overseeing the trial that it was using out-of-date data.

It was hardly a huge cut to the number, and the reason was relatively benign, but the debacle speaks to the fierce amount of scrutiny vaccine trial data is now facing. More than anything, the revision exposes an obsession with a single vaccination metric – efficacy.

The world faces a growing menu of vaccines – 23 in the final stage of testing and 80 in clinical trials – and many use efficacy to help pick out their preferred dish from this global buffet.

At the top comes vaccines like Pfizer and Biontech’s, which boasted a 95 per cent efficacy from its trial. At the bottom comes those like the single dose Johnson & Johnson jab on 66 per cent.

One widespread misconception is that efficacy is a measure of absolute protection against Covid-19. Many take ‘95 per cent efficacy’ to mean that 95 per cent of people with a vaccine were protected in the trial, while 5 per cent still got the virus anyway.

This is not what efficacy measures. Instead the statistic captures the relative chance of getting Covid-19 between the placebo and vaccinated groups in a trial.

A 95 per cent efficacy means that the number of vaccinated participants who got Covid-19 after the jab was 5 per cent of the number who only got a placebo. In other words, the vaccinated had a 95 per cent less chance of catching the virus – not that 5 per cent of the vaccinated group got the disease. This makes ranking jabs on efficacy hard because this kind of relative, individual risk is dependent on the conditions of the trial.

Both the AstraZeneca and Pfizer jabs saw part of their trials run in the US but over widely different time periods, with the number of cases circulating there far higher by the time of AstraZeneca’s analysis. Yet the company also had other parts of its trial running in different places, further muddying the waters.

Indeed Johnson & Johnson’s tests were conducted in Brazil and South Africa, just as new variants of Covid-19 were emerging. Who participates also counts and differences are inevitable.

This does not all mean efficacy is a useless measure. Yet it does severely limit the conclusions one can come to – it is not possible to say a jab with a 20 per cent higher efficacy over another is 20 per cent better.

On what counts most – stopping severe cases of Covid-19 – comparing jabs is splitting hairs. Though AstraZeneca revised the overall efficacy, left intact was the share of the vaccinated people avoiding a serious case of Covid-19: 100 per cent.

Nobody getting a dose in the trial ended up in hospital or dying after a jab, even after the data update.

Choosing one jab for marginal benefits over another is at most a luxury afforded to only the few – New Zealand, Australia, South Korea – who have stamped out Covid-19 completely.

Efficacy is just one of the first steps in assessing vaccines, and most jabs now are being examined on their effectiveness. This looks beyond performance in perfect trial conditions to real world performance.

This stage of assessment, however, is also where controversies like the blood clot scandal in Europe are bound to happen. Though all vaccine trials feature sizable numbers of participants – typically tens of thousands – they pale in comparison to the millions of jabs actually being rolled out globally.

Unlikely events thrive at this scale. Everyone knows their lottery chances are abysmally low – but that buying a million tickets would tip impossible odds in your favour.

Yet despite conclusions from the European Medicines Agency that any tiny risk of blood clots from the AstraZeneca vaccine is outweighed by the benefits, several nations have yet to resume giving doses of the jab.

Unfortunately, misconceptions around the stats are far too common – and the stakes are not academic rigour but actual lives.