Prof proves SA kids know their stuff
A Stellenbosch University professor has caused red faces at a leading international education research body by showing its assessment of how local children are faring in reading ability is wrong.
Education economist Martin Gustafsson found that contrary to the dismal verdict of the Netherlands-based International Association for the Evaluation of Educational Achievement (IEA), reading competency in grade 4 improved between 2011 and 2016.
“The analysis points to the gains being among the largest of all 2011-2016 gains among 43 countries reflected in the Pirls [Progress in International Reading Literacy Study] 2016 report,” said Gustafsson.
The Pirls test, administered every five years by the IEA, provides comparative data on how well children in a certain grade can read, assessing whether this standard improves, remains stagnant or gets worse.
As a result of the blunder, the IEA was forced to declare on its website earlier this year that the original South African reading trend “can no longer be considered reliable”.
It removed all references to the trend results for SA from its website and wrote to basic education director-general Mathanzima Mweli in March apologising for “the confusion caused”.
Gustafsson said that when the 2016 Pirls results were published in December 2017 he found it “a bit odd” that there was no improvement in grade 4 pupils’ reading performance as it was inconsistent with “considerable progress seen in a couple of other testing programmes”.
“The deputy minister [Enver Surty] at the time was hopping crazy, saying it [the results] didn’t make sense.”
Although analysing the Pirls data was on his “to-do list” he only got round to it last year. After poring over the raw data, he discovered that the 2011 Pirls figures had been calculated incorrectly because of a mistake in converting results from one scale to another.
The error resulted in an inflated score of 323 points for Pirls 2011 — the correct score was 295. This meant that there was an improvement of 25 points between 2011 and 2016 that researchers had missed.
In a recent report, the department said
that “such an improvement would generally be considered good, and would make SA one of the fastest Pirls improvers for the period 2011-2016. This is good news as improving reading in the early grades is a clear priority of the president.”
Gustafsson said he was “really astounded” when he discovered the error. “I didn’t think this kind of mistake was possible … The IEA prides itself in never making mistakes and this was a really serious mistake. This is really bad for their reputation.”
He said the Centre for Evaluation and Assessment (CEA) at the University of Pretoria, which ran Pirls 2016 in SA, released its own report in 2017. “Both reports pointed to there having been no improvement in reading in SA between 2011 and 2016.”
He said the CEA did not conduct its own analysis of the raw data it used.
“I think the centre could have examined the data in more detail but I think the reason they didn’t has to do with data analysis capacity.”
Professor Sarah Howie, the CEA’s national research co-ordinator at the time, said the centre’s report was based on data sent by Boston College, which analysed the data for the IEA.
In an e-mailed response to the Sunday Times, the IEA’s executive director, Dirk Hastedt, said that following Gustafsson’s “valuable work” in drawing the association’s attention to the issue earlier this year, “SA’s trends in Pirls between 2011 and 2016 were considered less reliable than originally expected”.
Hastedt said SA will be included in the Pirls 2021 assessment. Participation in the 2021 “paper-only version” of Pirls costs more than R4m.
Basic education department spokesperson Elijah Mhlanga said Pirls provided a valuable opportunity to assess whether the overall levels of reading comprehension in the country are improving.