Graphic detail | Daily chart

Mainstream election-forecasting could be improved by a popular academic approach

It correctly predicted Donald Trump’s victory but is no crystal ball

By THE DATA TEAM

UNPRECEDENTED, unbelievable and earthquake: three words used in book titles to describe the outcome of America’s most recent presidential election. Pundits are fond of blaming quantitative forecasters, trafficking in polls and probabilities, for their failure to foresee Donald Trump’s ascension to the presidency. The forecasters, for their part, have argued that the median voter (and the median pundit) misinterpret their methods, and that a few bad apples should not spoil the bunch. Should forecasters adopt a new statistical method, such unpleasant post-mortems could become rarer.

Forecasting the American presidential election usually relies on a clear-cut approach: first, modellers construct polling averages for each state. For those states with less available polling information, they supplement these with data from national polls. From this, they create complex computer programs to simulate several thousand contests and report the results. All popular forecasts—such as those from FiveThirtyEight and the New York Times—are built upon this basic structure.

This method is not perfect. Even though American pollsters predicted the national popular vote almost perfectly, some state-level polls had deep methodological flaws. When these were funnelled into statistical forecasts, they produced an inflated sense of certainty in Hillary Clinton’s victory, by incorrectly suggesting victory in five states that Mr Trump would actually win: Wisconsin, Michigan, North Carolina, Florida and Pennsylvania. Some modellers emerged with more egg on their face than others. FiveThirtyEight had given Mr Trump a 29% chance of winning; the Princeton Election Consortium argued it was 1% or less.

In the world of imperfect state-level polling, forecasters can turn to an academic statistical technique to sidestep this problem, called multilevel regression with post-stratification. Because that is a mouthful, researchers usually refer to it as Mr P. The approach would allow forecasters to produce state-level estimates by applying sophisticated demographic modelling to a single, large national poll. For example, the national poll would gather data about a large number of college-educated white women—including both their likely voting intention and chances of turning out on election day—and then estimate their differing impact in each state. Because demographic factors like age, race, education and income determine voting much more strongly than geography, this method produces accurate estimates. So long as researchers have good demographic estimates—helpfully provided by the Census Bureau—they can produce a good poll.

A recent one, published by the researchers Chad Kiewiet de Jonge, an affiliated professor at the Centro de Investigación y Docencia Económicas in Mexico, and Gary Langer and Sofi Sinozich, president and research associate at Langer Research, showcases the power of these methods. According to their calculations, an election model based on Mr P would have called the winner in the recent presidential election correctly in 50 of 51 American states (including Washington, DC). Also, the margins of error in those states they predicted Hillary Clinton would win are much smaller than those from mainstream outlets.

But Mr P is no Mr Perfect, and the method is no crystal ball. The predictions that the authors make for the 2004 to 2012 elections contain a hefty dose of error. In 2000, for example, the method would have given Al Gore a 67% chance of winning the election. (It is unclear what an alternative method would have predicted.) Moreover, in 2008 and 2012, the model did worse than the polling-aggregation-based forecasts published by Nate Silver, a prominent psephologist. Maybe Mr Silver luckily drew from good polls in 2008 and 2012, but Mr P nevertheless performed worse.

The paper also benefits from prediction being easier after the event. It took into account that the American electorate was deeply divided by education level, which public-opinion researchers began paying serious attention to after the 2016 election. Some key state-level polls did not take account of this fissure before the election, producing bad estimates that contributed to the misfiring of poll-based forecasting models, like those at FiveThirtyEight and the New York Times.

The advantage of using Mr P in 2016 was its ability to better predict turnout patterns than state-level public polls, which often had to rely on outdated results from 2012 to produce their estimates. Since the authors of the paper were able to use more updated data on the propensities of black and white, educated and uneducated, and wealthy and middle-class voters to cast ballots, they foresaw an electorate that was whiter and less educated than others were expecting.

However, Mr P does not provide a solution for some larger challenges in today’s public-opinion polling. Bias can be introduced into polls if too few voters belonging to certain demographic groups answer their phones, for example. And even when pollsters try to correct for these biases by weighting their samples, if response rates dip too low, errors can arise in previously unknown directions, such as when one young black man from Illinois biased an entire national survey conducted by the Los Angeles Times in 2016. Still, even if half of the population of a state answered their phones (an impossible number in modern survey research, to be sure), researchers face the problem of correctly identifying the voting population of a state. Even as this method has been around for quite some time—the earliest papers to use multiple regression and poststratification were published at the turn of the century—weaknesses in survey research that pose a threat to polling aggregation also jeopardise forecasts made with Mr P. As some researchers would say about trying to model your way out of bad data, “garbage in, garbage out.”

Mr P promises to be a useful tool for forecasting future elections, and an application of the method correctly predicted that Donald Trump would win the presidency. It’s a pity that the authors predicted that only after the fact. Future success will depend on consistently getting elections right—before they happen.

Discover more

Three reasons why oil prices are remarkably stable

Can it last?

Why America is a “flawed democracy”

EIU’s index plots the country’s democratic decline since 2006


Five charts compare Democrats and Republicans on job creation

Our analysis of the past eight American presidents