Science & technology | National security

Intelligent intelligence

Just how good are government analysts?

PREDICTION is difficult, as Niels Bohr once observed, especially about the future. But that has not stopped a huge industry that promises to do so from springing up. Commuters pore over newspaper columns; politicians spend fortunes on deciphering polling data; bosses sit in thrall to the pronouncements of coiffed young flip-chart merchants from management-consulting firms.

Governments are as keen as anyone to know the future, and one of the main jobs of intelligence agencies is to arm them with forecasts about how the world will look months or years from now. How good these are is an open question: spies often complain the public hears about their failures (to predict the September 11th attacks, for instance) but never of their successes.

In a paper in the Proceedings of the National Academy of Sciences, David Mandel, of Defence Research and Development Canada, and Alan Barnes, a former intelligence analyst for the same country, take a stab at an answer. They analysed more than 1,500 intelligence forecasts produced by a nameless (but presumably Canadian) agency, covering the period from March 2005 to December 2011.

Their results suggest that the old joke about “military intelligence” being an oxymoron is unfair. When they compared what the analysts had said with what actually came to pass, they found that the predictions were right about three-quarters of the time. Cynics might wonder if the analysts merely restricted themselves to easy cases, but Dr Mandel and Dr Barnes also found they were good at calibrating their judgments. Events they deemed unlikely did not happen often, whereas those they thought likely occurred frequently. Indeed, if anything they were underselling themselves, tending to err more than necessary on the side of uncertainty. And there was evidence that their skills could be learnt—for more-experienced analysts tended to do better than their junior counterparts.

The result is even more striking because of its contrast with a famous earlier finding. In 2005 Philip Tetlock, a management theorist at the University of Pennsylvania, announced the results of a 20-year study in which 284 experts—professors, journalists, civil servants and so forth—were invited to make more than 28,000 predictions. Their performance was abysmal: barely better than chance, and inferior even to simple computer algorithms.

Why the difference? Drs Mandel and Barnes do not know. But their favourite hypothesis rests on cultural differences—that professional analysts are more cautious than your average pundit, and that this diffidence serves them well. That fits with some of Dr Tetlock’s later findings, in which he identified a small group of “superforecasters” who actually seemed competent at their jobs. One characteristic of these superforecasters was caution about their own abilities.

And working as an analyst offers plenty of incentives to be cautious. Unlike pundits, who can pontificate from the safety of their armchairs, analysts know that their advice is likely to have consequences in the real world. Drs Mandel and Barnes found that analysts’ inherent underconfidence became even more pronounced when confronting particularly important or difficult questions.

Analysts must also defend their claims to managers, who are trained to be sceptical, and to their political masters. Other studies have shown such accountability encourages careful thinking and reduces self-serving cognitive biases. Journalists, media dons and other pundits do not face such pressures. Today’s newsprint is, famously, tomorrow’s fish-and-chip wrapping, which means that columnists—despite their big audiences—are rarely grilled about their predictions after the fact. Indeed, Dr Tetlock found that the more famous his pundits were, the worse they did.

This article appeared in the Science & technology section of the print edition under the headline "Intelligent intelligence"

America’s lost oomph

From the July 19th 2014 edition

Discover stories from this section and more in the list of contents

Explore the edition

Discover more

Antarctica, Earth’s largest refrigerator, is defrosting

The world must pay more attention to its southern pole

Killer whales deploy brutal, co-ordinated attacks when hunting

Their techniques are passed down through the generations


A new generation of music-making algorithms is here

Their most useful application may lie in helping human composers