Surveys — much ado about nothing
World rankings don’t you just love them! All sorts of outfits produce these great big lists of comparative international performance. But even when the survey’s authors say their data is not really sufficient to make definitive international comparisons there is an enormous flurry of self congratulation from those at the top of the list and great wails of despair from those at the bottom.
We saw a classic example of this at the beginning of the month when the Organization for Economic Cooperation and Development published the results of its Program for International Assessment — or PISA test for 2012, which compares global educational systems by testing 15-year old students.
Even cynics seem to have missed the wonderful pun in the acronym for this survey because the Italian city of Pisa is most famous for its leaning tower that looks as if it is about to topple at any moment yet remains in place.
The same can be said of this survey that was greeted with wild enthusiasm in East Asia where schools scored highly, especially in Shanghai with students topping the mathematics, reading and science categories. Even here in Hong Kong there were celebrations over making it to third place in the mathematics rankings.
It would be churlish not to congratulate Shanghai’s schools and in Hong Kong, where criticism of the educational system is rife; the good news has been grasped with some alacrity by the government
In fact, PISA is one of the better global surveys because it has the intelligent objective of assessing not just what students know but also what knowledge and skills they can use for participation in daily life. However, the survey’s organizers have themselves pointed out that the findings are not sufficiently comprehensive to draw sweeping conclusions.
This, of course, has not stopped sweeping conclusions being drawn. Thus, we have seen much celebration in Asian countries following the survey’s publication while in countries such as the United States and those in the Nordic world, where educational systems are generally highly regarded, there has been a flurry of self abasement.
The problem is that focusing on international comparisons often obscures the reality at ground level. I have a lot of contact with educators in Hong Kong who regularly complain that their students lack curiosity and initiative and focus almost entirely on examination results.
Yet Hong Kong does well in international surveys of examination results. However, look closely at the overcrowded schools in poorer areas and the droopy eyes of better-off students subject to far too many hours of extra exam-oriented tuition and you will see another picture. This one suggests the education system has profound problems.
International surveys don’t pick up on this, but that does not mean international comparison surveys have lost their popularity. One of the surveys I like best, for its risible quality, is the much publicized annual global sexual activity survey conducted by the condom maker Durex. This purports to tell us how many times people in various countries have sex.
Hong Kong regularly ranks very low in this survey, generating much idiotic comment about how the pressures of local society mitigate against people having sex. Another way of looking at this is that when someone with a clipboard comes around asking how many times you have sex; an honest answer tends not to be the first response that comes to mind. In some countries, people like to boast of their sexual prowess, in others, Hong Kong would be an example here, people are more modest and more reticent in discussing this intimate subject with strangers.
Therefore, I would conclude that the value of the Durex survey is around nil, but it is only one in a group of other surveys measuring global happiness, contentment and goodness knows what else. It really is nonsense because the subject matter is highly subjective and there is no tangible way of measuring these things.
The same cannot be said of the PISA test but valid questions need to be asked about the test’s methods, about its cultural biases and about who participated in the survey because, generally speaking, educational surveys show big differences between the performances of students from different social classes.
Most worrying is what happens after these international comparisons are published. This can lead to complacency among policy makers who have no right to be complacent; alarm, among those who really have no need to be alarmed; and, most damagingly, concern about things that do not really matter.
However, international surveys are much loved by the media because they make for good copy. Countries also have an obsessive tendency to pour over international performance comparisons.
I recall being sent as a young reporter to a British city that was ranked in some survey as being the most miserable place in Britain. My job, I presumed, was to find lots of miserable people and make something out of their misery. The city looked rather depressing but somehow the people refused to be as depressed as they were supposed to be. I interviewed one older man who could not understand what all the fuss was about, but pointed to a house across the road and said, ‘mind you Albert over there is a miserable bugger’. That comparison made quite a lot of sense.