Trial by spin and bad science
Ben Goldacre is on the phone from London and he’s getting exercised about statins.
“Statins are the canary in the cage for problems in modern medicine,” he says.
They are also the subjects of his next book – yet to be completed – and were addressed at some length in his last one, Bad Pharma (2012).
His continued focus is not surprising. For a host of reasons, research into statins and the prescribing patterns that research catalyses pretty much epitomise his concerns about medical academia.
They are concerns that, since he qualified as a medical doctor in 2000 after studies in Oxford, Milan and Los Angeles, have seen him move from GP, to author, to researcher, to, now, medical activist.
Goldacre, Uk-born to Australian parents, fronts two long-term campaigns designed to bring about root and branch reform of the ways clinical trials are conducted and reported.
The first, Alltrials, has been running since 2013. It seeks to ensure that every trial is both registered with an appropriate authority and then published, regardless of result. At present, perhaps as much as 50% of trial write-ups never see the light of day.
The second campaign, Comparetrials (CT), kicked off late last year and aims to enforce procedural transparency in trials. At issue is the matter of switched priorities: trials that start by researching one outcome, but then change to another before the exercise is concluded.
Earlier this year, the CT team checked every trial published between October 2015 and January 2016 in five major publications, including the British
Medical Journal and the Annals of Internal Medicine. The results reported were compared to the outcomes specified when the trials were initially registered, or the founding protocols published. Out of 67, only nine did what they set out to do. In the rest, 354 stated outcomes were not completed, and 357 new ones were added.
Goldacre happily admits that CT is a “preposterously nerdy venture”. That said, however, it doesn’t diminish its importance.
“There’s a real problem in the way that clinical trials report their results. You can measure the outcome of your trial in hundreds of different ways,” he said.
He poses the hypothetical example a drug aimed at improving cardiovascular health. How baseline and improved health are assessed are matters of great complexity. Blood tests can measure perhaps 20 applicable lines of evidence, each set against potentially hundreds of cut-off points. Symptom questionnaires can be measured against a plethora of ratings scales. Hospital admissions can be recorded by treatment, code, doctors’ notes or length of stay. Patients can be monitored over days, weeks, years, decades.
“So you potentially have thousands and thousands of ways of measuring something like cardiovascular health,” he says, “And because there are so many ways of measuring it, that means the results are really vulnerable to cherry-picking.
“That’s why traditionally we ask people at the beginning of a clinical trial to specify exactly what they are going to measure as the success criteria, and exactly how they are going to measure it.”
Tradition, obligation, and principle, however, are not unbreakable bonds (as any subscriber to Retraction Watch can testify). Much of Goldacre’s
Bad Pharma comprises a detailed and depressing catalogue of the many ways in which researchers – sometimes at the behest of the pharmaceutical companies sponsoring the work – recalibrate their
WE HAVE FAILED, SO BADLY, TO COMMUNICATE THE MODEST BENEFITS OF THESE TREATMENTS TO THE PUBLIC.
initial intentions and generally fiddle with the data.
Sometimes this is merely a matter of spin – such as expressing a benefit as a relative rather than absolute risk reduction – but sometimes it is much more organic.
Outcome priorities are changed; negative results are omitted; trials are foreshortened or extended to better massage the data. In the book, Goldacre terms these tactics “a quiet and diffuse scandal”.
Perhaps this would not matter quite so much if it weren’t for the fact that published trial results feed into pharmaceutical marketing and doctor prescribing choices.
In Bad Pharma, Goldacre relates a trial featuring a new painkiller, celecoxib, that was tested against two other pills to assess side-effect gastrointestinal complications. The published study showed clearly that over a six-month period the new drug was way better than the old ones, leading many GPS to preference it in prescribing.
But it eventually came to light that the original intention of the trial was to test the three pills for a 12-month period – over which celecoxib performed no better than its rivals.
At the other end of scale, in the late 1990s pharmaceutical company GSK investigated anecdotal reports of deaths associated with its asthma inhaler drug Salmeterol. It set up a large clinical trial, with participants monitored intensively for 28 weeks. It then asked participating doctors to keep an eye out for adverse events for another six months – but did not actively search for cases.
Not surprisingly, the period of intensive monitoring revealed a higher number of negative outcomes (measured against a placebo) than the follow-up period when no one was looking too hard. Changing its initial trial protocol, GSK reported the figure for the two periods combined, thus reducing the apparent severity of the problem.
Which brings us back to statins, the most commonly prescribed medication in the developed world, and Goldacre’s canaries.
In Bad Pharma, he points out that the two most popular prescribed statins, atorvastatin and simvastatin, both work well, but no one has ever tested them against each other to determine which one works better. This is an important point, because if one works only slightly better than the other it’s still a result that could convert into the prevention of thousands of strokes and heart attacks every year.
In the absence of this data, prescribing, Goldacre points out, is effectively a random act. But attempts to formally constitute that randomness as a countrywide trial in England met with a farcical level of bureaucratic complication. That makes statins “one of the most fascinating problems in medicine right now”.
“Over 100 million people take a statin every morning and yet there are huge gaps, firstly in our knowledge about which is best, and gaps in our knowledge about side-effects,” he says.
“But also, we have failed, so badly, to communicate the modest benefits of these treatments to the public that there is huge widespread panic and anxiety among not just patients but also doctors, in many cases, about what the benefits of these treatments are.
“If we can’t get this stuff right for [statins] the single most commonly prescribed class of drug in the whole of the developed world – a tablet that is taken every day by 100 million people – then that’s a real window into our failures to do appropriate clinical trials throughout the whole of medicine.”
It’s a subject to which he will no doubt return, not only in his forthcoming book, but also in his upcoming speaking tour of Australian capital cities, kicking off in Brisbane on 22 September. ANDREW MASTERSON is an author and journalist based in Melbourne, Australia.
IMAGES 01 Ben Gilbert 02 Science Photo Library / Getty Images 03 Harper Collins
Dates and tickets can be found at www.thinkinc.org.au/events/ben- goldacre