Jobs data a piece of work

Recall about a year ago the mystery of how widespread business claims of labour shortages, and their calls to preserve the Temporary Foreign Workers program, could co-exist with critics who pointed to a job vacancy rate of 1.5% in a Statistics Canada survey. Some of the mystery was cleared up last week, when Statistics Canada quietly put out a technical paper called 2011 Workplace Survey Summary and Lessons Learned. It holds lessons not just for understanding vacancy rates but also for anyone relying on data alone to provide the answer to all policy questions and conundrums.

Statcan found that the 1.5% vacancy rate in its business payroll survey nearly doubled to 2.7% in its Workplace Survey. The increase was most pronounced for the industries that were complaining the most about shortages in 2011, such as mining, transportation and construction, and for the Western provinces. In other words, these data make a lot more sense and better fit the narrative coming from the business community at the time.

So what was the difference between the two vacancy surveys? Statcan pointed to two factors. One was a subtle change in the questionnaire that redrew the boundary between whether searches were only outside the firm or included some internal searches. But mostly the change reflected who in the organization was targeted to respond as “having a knowledgeable respondent answering the questionnaire was an issue in data collection.” No kidding. It makes a big difference whether human resource personnel answered instead of the usual “Statcan guy” in the payroll department, who knows little about the organization’s human resource needs. (And how far down the totem pole do you have to be as the designated “Statcan guy” who responds to its surveys? If it was me, I would definitely be updating my CV and LinkedIn contacts.)

In a December 2014 paper for a School of Public Policy Conference in Calgary, I noted these and other problems with surveys of vacancy rates. The real conundrum is that, unlike straightforward variables like retail sales, the concept of vacancies is open to interpretation by the respondent, and this varies significantly with firm size. It is always problematic to ask people to interpret economic concepts themselves; ask 10 people what is unemployment, and you will get 10 different answers. So I have lingering doubts about the vacancy rate estimate of 2.7%. Another survey could find a totally different answer. Pouring money into more surveys is unlikely to

It makes a big difference whether

human resource personnel answered instead of the usual

‘Statcan guy’

produce a definitive answer, apart from further discrediting the unlikely result of the first survey that found vacancies were almost non-existent.

The real problem is not the challenge of measuring nebulous concepts like vacancies, but our society’s exclusive reliance on data, of whatever quality, as an evidence-based guide to policy. Knowledge based on hard to measure variables like vacancies is “fragile” to borrow a concept from Nassim Taleb; that is, conclusions based on such data can change significantly in response to seemingly small modifications to its environment, such as how a question is phrased, the