ON DECEMBER 3RD the OECD, a group of mainly rich countries, published the latest round of its Programme for International Student Assessment (PISA), a study of the academic performance of half a million 15-year-olds in 65 countries in reading, maths and science. Parents, teachers and politicians set great store by the results. Countries such as Finland and Sweden, which dropped down the rankings this year as Asia consolidated its lead, are engaged in a bout of national soul-searching. But just how accurate are the tests, and which country's children are really the cleverest?
It is hard to fault PISA on its detail and scope. Its sample is vast and the OECD publishes an accompanying 500-page tome comparing everything from outcomes by gender and wealth to rates of improvement, alongside earnest inquiries into how much children in different countries enjoy learning maths (or don't). Because such large surveys are difficult and expensive to carry out, there are few alternative rankings with which to compare PISA. The best of the bunch for maths (PISA has focused most closely on that subject this year) is the Trends in International Mathematics and Science Study (TIMSS), which holds four-yearly assessments of progress among pupils aged 9-10 and 13-14 years. Taking the older children as the nearest equivalent to PISA's sample of 15-year-olds, the top of the table is strikingly similar. Clever (or at least well-schooled) east Asians lead the rankings. Shanghai, which tops PISA, is not assessed in TIMSS. Otherwise the leaders are the usual suspects: South Korea heads the TIMSS table, followed by Singapore, Chinese Tapei and Hong Kong. Allowing for statistical noise, there is no great difference in results.
But a few notches down, things are intriguingly different. TIMSS places America and England at places nine and ten, behind Russia and Israel. On the latest PISA maths rankings, by contrast, England is placed 21st, America 29th, Russia 38th and Israel 42nd. TIMSS has also been historically much kinder to Belgium, ranking it well above the position given to it by PISA. Subtle differences in the questions asked of the children, as well as technical sampling differences, account for some of the variation. But there are also distinctions in what is being tested. PISA emphasises the contexts in which mathematics is applied, so word problems feature highly. TIMSS, meanwhile, is designed to assess whether curriculums show progression, hence its testing at two ages, four years apart. It is thus more geared towards testing a learned body of knowledge, whereas PISA busies itself more closely with measuring the “yield” at the end of a decade or so of schooling. This suggests that countries that do well in TIMSS but poorly in PISA are succeeding in imparting knowledge about a subject, but failing to turn it into conceptual skills that can be applied when pupils leave school.
