CONSIDER THIS A WARNING

‘CORRELATION IS NOT CAUSATION’ IS NOT A MAGIC PHRASE THAT WINS EVERY ARGUMENT

If you’ve taken a course in statistics, you’ve probably been warned against mistaking correlation for causality, and it’s good advice. But this is one of those cases where a little knowledge can be a dangerous thing: “correlation is not causality” is not an incantation that gives you the power to think of yourself as having beliefs based on data while dismissing out of hand any evidence that challenges your previously-held opinions.

To be sure, correlation and causality are different things, and data can never prove causality. Even in a (relatively) simple context such as a billiards table, causal relations cannot be observed directly. Causality is part of the theoretical framework that underlies our understanding of how billiard balls move, not something we can measure. As the 18th-century philosopher David Hume observed in An Enquiry Concerning Human Understanding, we don’t actually observe the transfer of momentum from one billiard ball to the next.

It’s easy to come up with a causal explanation for what we see. Too easy, in fact: there are an infinite number of theories that can potentially explain a finite set of facts. Isaac Newton’s explanation for what happens after one billiard ball hits another is not the only one: there’s always the possibility that everything is being manipulated by the Flying Spaghetti Monster’s invisible tentacles. Or that everything we perceive is part of the Matrix or that we’re just a dream someone is having. The possibilities are endless, and not all of them have been made into films yet. The way to avoid going further down that rabbit-hole is to limit ourselves to theories that make predictions that might be wrong, or falsifiable.

Of course, no theory perfectly forecasts every outcome: measurement errors and confounding factors (friction, weather, politics, etc.) impose limits on how precise a prediction can be. This is where probability and statistics come in: theories that produce smaller errors are given more credence. The more information there is — more data, or data that clearly favour one side over the other — then the easier it is to decide which theory is in line with the data.

The real problem occurs when the same correlation is consistent with more than one theory: much of the time spent on empirical projects consists of trying to make sure that the effect we’re measuring is the effect we’re interested in. If prices and quantities are determined simultaneously by the interactions of supply and demand, then there’s no way of telling if a correlation between prices and quantities reflects a supply effect, a demand effect, or a combination of both. This “identification problem” is pervasive in economics.

This is why people prefer working with data generated by controlled experiments when they can. A properly designed experiment will ensure that the variation in the (hypothesized) causal factor is purely random, and not subject to feedback from the effect you’re trying to identify. Unfortunately, experimental data are hard to come by in economics. Some economists have taken to performing their own experiments — using laboratory techniques similar to those used in psychology — but we’re not yet at the point where lessons learned in the lab are routinely applied to policy problems. Randomized controlled trials (RCT) — in which people are chosen at random to receive a “treatment” — are still rare, and are usually not feasible.

“Natural experiments” occur from time to time, when what amounts to a random change in the economic environment — a policy change, a natural disaster — affects only part of the population. In this case, the differences in outcomes in the two subgroups can be potentially explained by the fact that one received the treatment, and the other did not.

But for the most part, economic data are non-experimental (or observational) — just as they are for the other social sciences and for some natural sciences such as fields like astronomy and climatology. In this case, you have to make explicit assumptions about how the world works — that is, set out a model specifying the theory of causality — and compare its predictions to the data. This modelling exercise is at the heart of all empirical studies that try to extract causality from correlation, and you can’t cheat by saying that everything depends on everything else. If you don’t have a controlled experiment to close off some causal relations, you have to close them off by assumption. Without these “identifying” restrictions, you can’t make the connection between correlation and causality.

These restrictions are definitely up for debate: you can’t appeal to the data to test if an identification strategy is valid. These assumptions have to stand on their own, on the basis of their intuitive plausibility and/or other empirical work. For example, a popular identification restriction that says that technical change and trade liberalization affects labour demand, but not labour supply. This may not be literally true — it’s easy enough to come up with some sort of multi-step counterfactual — but it’s plausible enough for empirical work.

It’s fair game to question identification assumptions, but the real challenge is to come up with an even more plausible set of restrictions. Unless you have a more plausible identification strategy for interpreting the results, saying “correlation is not causation” to deflect a conclusion you don’t like is the equivalent of putting your fingers in your ears and yelling “La la la, I can’t hear you.”

This happens all the time. My favourite recent example was a recent study linking the introduction of universal daycare in Quebec to a decline in children’s non-cognitive skills. This resulted in a wave of indignant denunciations of its “flawed” methodology, but of course the study’s real sin was in producing a conclusion that displeased many people. Meanwhile, studies that use the same identification assumptions and find that universal daycare increased female labour force participation rates in Quebec were met with nods of approval. When the same methodology produces two sets of results and you reject only one, that’s confirmation bias at work, not analysis.