Why statistics should always be treated with some caution

Debates on everything from Brexit to education are dominated by numbers. Gareth Rees, Research Professor at Cardiff University looks at the power of statistics in public debate in relation to Wales’ international Pisa results

NOWADAYS, political debates are often dominated by numbers. We only have to look back at last year’s Brexit referendum, for example, to see how important statistics – numbers of immigrants, patterns of trade with other countries, the size of payments to the European Union and so on – were in the campaign.

However, as the use of statistical information about the economy and society in which we live has become more significant, so people have become increasingly cynical about its truthfulness. Paradoxically, then, statistics lie at the heart of what some people have dubbed “post-truth politics”.

Here in Wales, the inherently controversial nature of statistics is most frequently reflected in the neverending debates about the effectiveness of our public services. We have become all too familiar, for instance, with the wrangles over the National Health Service in Wales, as reflected in data about the length of waiting times or average expenditure on treatments.

In education, however, there seems to be a much greater degree of consensus. Policy-makers and the commentariat more generally share a concern over what is viewed as the relatively poor performance of the Welsh education system.

Views differ markedly, of course, about the most effective policies to pursue, but there is nevertheless widespread agreement that Welsh schools, in particular, are less successful than those in the other countries of the UK, as well as internationally, in achieving high levels of attainment amongst their pupils.

Here too, the basis of this evaluation is almost entirely statistical. More specifically, tremendous weight is now attached to the results of the Programme for International Student Assessment (Pisa).

This is an international survey, conducted every three years by the Organisation for Economic Co-operation and Development (OECD), which tests the attainment of a sample of 15-year-olds in mathematics, science and reading.

It claims to provide a robust basis for assessing the performance of national education systems, by enabling comparisons to be made between countries.

The latest results, published just before Christmas, confirmed what previous Pisa surveys have shown: Wales’s scores compare rather badly with those of the other countries of the UK, as well as internationally.

However, one would never know from the debates about these issues in Wales that numerous researchers have contested the statistical procedures by which the Pisa scores are produced.

In part, this is a problem of the simplistic way in which the results are reported by politicians and the media. They focus on single average scores for each country and the “league tables” that can thereby be constructed.

More fundamentally, however, as Pisa produces successive “snapshots”, rather than truly longitudinal data, tracing the progress of individual pupils over time, it does not provide a robust basis for identifying the effects of educational systems on pupils’ performance.

And it is the latter that we need in order to make informed judgements about the quality of educational provision in Wales.

In short, therefore, we should treat the Pisa results cautiously. They are best considered as a starting point for further research, rather than providing a definitive measure of educational performance in Wales. Certainly, even a brief examination of the latest round of results raises a number of questions that warrant further analysis.

For example, why are the scores achieved by the pupils who perform best on the Pisa tests significantly lower in Wales than in the majority of other countries? Relatedly, why is the performance of pupils from the most advantaged socio-economic backgrounds also comparatively weak? Why did pupils who took the Pisa tests in Welsh score significantly lower than those who took the test in English (even where they are in Welsh-medium schools)? Why have Pisa science scores in Wales declined significantly since 2006?

It is also important to understand that Pisa is not the only way in which the quality of educational provision in Wales can be measured. Hence, for instance, the National Survey for Wales for 2012-2013 revealed that parents in Wales were overwhelmingly satisfied with the school that their child attended.

There is also evidence (from the Millennium Cohort Survey) that indicates that pupils in Wales express relatively high levels of wellbeing compared with those in other parts of the UK.

However, these “softer” measures of how the education system is performing do not generally feature in the public debates.

Public concerns over the performance of the Welsh education system are wholly legitimate.

However, an informed debate requires a more rounded examination of the statistical evidence than the current preoccupation with “headline” results from Pisa implies.

Gareth Rees is a Research Professor at Cardiff University, based in the Wales Institute of Social and Economic Research, Data and Methods (Wiserd). He is a Fellow of the Learned Society of Wales