Exploring the ‘credibility chasm’

Think-tanks are an ever-present, yet somehow under-examined feature of the public policy landscape. Think-tanks get a lot of press, at least partly because they are adept at issuing press releases advertising their work to the media, complete with pullquotes and readily available experts for radio and TV hits. Academic studies — the sort of work written by professors for professors — pass almost unnoticed, mainly because most of it is not relevant to current policy debates, and because peer-reviewed publications are not so readily accessible. But is visibility the same thing as credibility?

It would seem not. Carey Doberstein, a political scientist at the University of British Columbia, recently published a study in Canadian Public Policy on the credibility gap — he calls it a “credibility chasm” — between academic research and research published by think-tanks and advocacy organizations. Interestingly, his study is not carried out among the general population, but among policy analysts in the provincial governments of British Columbia, Saskatchewan, Ontario and Newfoundland and Labrador.

Participants in the study were asked to read and evaluate the credibility of different studies in two areas of provincial competence — minimum wages and incomesplitting. The analysts were asked to evaluate a set of five or six studies produced by academics, think-tanks and advocacy groups. Doberstein very sensibly does not draw inferences about credibility from these evaluations: one study is hardly enough to evaluate the credibility of one group, or even of one researcher. He focuses instead on how the source of a study affects policy analysts’ perceptions of its credibility.

Instead of sending the studies out to the analysts under their proper affiliations, Doberstein randomly altered them. For example, a study on the effects of an increase in the minimum wage written by researchers at the University of Toronto and published in a peer-reviewed journal was sent out with the correct affiliation to one group of analysts, under the name of the Canadian Centre for Policy Alternatives (CCPA) to another group, and under the name of the Fraser Institute to yet another group. Similarly, in addition to being sent out under its own name to one group, a CCPA study would be sent out as a University of Toronto study to a different group, and represented as a Fraser Institute study to yet another set of analysts, and so on. Two advocacy groups, the Wellesley Institute and the Canadian Federation of Independent Businesses, rounded out the minimum wage exercise, and a similar mix of academic, think-tank and advocacy groups was used for the income splitting case.

This randomization strategy allows Doberstein to identify the reputation effects of the various sets of researchers: How is a study’s credibility affected by its affiliation? The answer is: pretty much in the way you’d expect. Adding a university affiliation to a think-tank or advocacy group study increases analysts’ perceptions of its credibility, while adding a think-tank or advocacy group’s name to an academic study makes it less credible. Generally, credibility among policy analysts declines as you move from university affiliations to think-tanks to advocacy groups.

These results aren’t hard to explain. Policy analysts know full well that advocacy groups cannot be expected to publish anything that does not fit their stated agendas, so a study showing (once again!) that the data supports their previously-held position is not a particularly strong signal. Doberstein finds a similar effect among thinktanks: Think-tanks with a more stridently ideological focus (CCPA, the Fraser Institute) are viewed as being less credible than the relatively neutral C.D Howe Institute.

Is this good news or bad? On the positive side, it shows that policy analysts are well aware of the incentives facing various sets of researchers, and know enough to put their work in context. On the downside, one might have hoped that analysts could set all that aside and evaluate the research on its own merits. Of course, that’s an ideal that almost no one can match: this is why so many academic journals use double-blind peer review, in which neither authors nor reviewers are identified to each other.

Perhaps the more interesting question is why advocacy groups and ideologicallydriven think-tanks even bother to produce reports that are discounted so heavily by policy analysts. One answer might simply be that their reports aren’t written for the benefit of analysts; they’re written for the benefit of their donors. People like to have their beliefs confirmed, and they’re willing to pay to have someone tell them that they were (once again!) right.

This discussion also provides some insight into the challenges facing the media, particularly as it concerns the markets for news and opinion. Asking people to pay someone to tell them what they want to hear is a viable business model, and many digital outlets — from The Rebel through Canadaland to Rabble — are in the process of filling out that landscape. (It also raises the question of why the CBC would want to cut into this action with its CBC Opinion site. There’s no obvious market failure here that needs a public-sector fix.)

News, on the other hand, has the elements of a pure public good: everyone benefits from knowing the basic facts of what is going on, and technology has made it almost impossible to control access to news once it’s been published. Profits from advertising revenues can no longer finance news gathering to the same extent that they used to, but academic researchers can still fall back on teaching to cross-subsidize their research work. If you really want to make an academic researcher sweat, ask her to imagine trying to make a living from her research alone.