Wrong numbers

MEASURING things is hard. Counting mentions of them is easy. Google's new search tool Ngram Viewer lets users find out how often a word occurs in the millions of books the firm has scanned. It also helps analyse the salience of abstract ideas—corruption, say—in different languages and places (see chart).

Lots of new numbers will be grist to the mills of those who compile international rankings. The hunger for crunchy comparisons of everything from venality to brainpower is huge and growing, not least among media such as this newspaper. It is tempting to try and pin ideas down by turning them into figures. Such measures can be handy. But they have serious flaws.

The quality of economic statistics is often dubious in developing countries, for example. Figures are often only rough estimates and revised often and extensively. Inferences are risky. The latest version of the Penn World Tables, the best source for such numbers, has Equatorial Guinea's GDP growing by 4% annually between 1975 and 1999. The 2002 version put the rate at minus 2.7%. The country may thus have been sub-Saharan Africa's slowest-growing economy—or the second-fastest.

At least teachers have drilled generations of economics students in the limitations of GDP data, even if the public and politicians give them undue weight. But experts too can be fooled by “synthetic” indices combining several related measures into a single number which are often used to back broad claims. For instance, an index may rely on data on a few rather narrow items, such as the number of convictions for certain crimes, but claim to measure something broader, such as the quality of a country's legal system.

Rankings based on perceptions are particularly problematic. One example is the World Bank's survey on the quality of developing countries' infrastructure. In places where potholes and dirt tracks are the norm, the people interviewed may be so used to them that they do not see poor roads as much of a problem. Those accustomed to good roads may grumble more loudly about much smaller shortcomings.

Combined indices can also be oversensitive to updates. A recent study of the United Nations Development Project's widely cited Human Development Index (HDI) by the economists Hendrik Wolff, Maximilian Aufhammer and Howard Chong finds that small revisions to the data that go into the index can substantially alter a country's position in the rankings.

Part of the problem is that few people pay attention to margins of error. As a result, even differences that are too small to be statistically significant are used to rank countries. Unsurprisingly, such rankings change with every minor revision. A detailed internal evaluation of the World Bank's influential Doing Business index in 2008 found that because countries were given different ranks despite fairly similar index values, small data revisions could send them shooting up or down the indices of business-friendliness.

This would matter less if the rankings were used as broad guides. But phoney precision (among users if not producers of these data) is endemic. The UNDP puts countries into three categories based on their HDI number: low, medium, and high human development. Mr Wolff and his co-authors find that the probability that any country is in the wrong category is as high as 45%. Yet Merck, a pharmaceutical company, bases local prices for some medicines on a country's HDI category. Some even want a country's HDI number to determine the effort it has to put into reducing carbon emissions.

Rankings based on large global databases such as Google's will also have their share of problems. In the case of the Ngram Viewer, the body of books in different languages may not be comparable, for instance.

Yet few will grumble as the number and scope of international rankings mushroom. Many agree with Lord Kelvin, the 19th-century physicist after whom the unit of absolute temperature is named: he reckoned that measuring something provides additional knowledge. And so it does, in the physical sciences. But where humans are involved, more data sometimes yield less truth.

The Economist received the following letter from the United Nations Development Programme in response to this article

SIR - You used the Human Development Index (HDI) to illustrate misclassifications of countries by aggregate indices and relied on recent research by Hendrik Wolff et al that criticises the HDI because it did not update country classifications to take account of formula or data revisions. However, what you did not mention is that this criticism applies to the old version of the HDI rankings and not to that published in 2010 (which is available at http://hdr.undp.org/en/). Our new method groups countries by quartiles in the HDI distribution and thus automatically updates these classifications when there are data or formula revisions. The possibility of country group misclassification is at most 5%, well below the 45% cited in your article.

Jeni Klugman
Director
Human Development Report Office
United Nations Development Programme
New York