Free exchange | Big data

Separating tweet from chaff

Social networks may be able to shed light on important economic trends

By S.P. | LONDON

WHAT Twitter tells you about the world largely depends on whom you choose to follow. Personal experience of hours wasted on the microblogging service suggests that few of the 15 billion “tweets” posted every month are of any interest at all. But taken as a whole, many believe the aggregated musings of 241m people tapping away on their phones might form an interesting data set which can provide real-time information on the state of the economy.

The latest attempt to extrapolate a signal from the noise focuses on the American labour market. Researchers at the University of Michigan (Correction: and Stanford University) have created indexes of job losses, job searches and postings. Counting phrases such as “lost my job” or “help wanted”, the researchers think they can gauge what’s going on in the labour market weeks before official data is compiled. Anyone who has seen “Trading Places” knows how valuable that can be.

Does it work? Sort of. The researchers don’t claim their new-fangled index can predict unemployment, for example, merely that it foresees the direction in which forecasters are likely to err. And even that only happens haphazardly and after a pretty intense massaging of the data. Plenty of tweets have to be ignored, for example if they comment about unemployment statistics (“looks like plenty of people lost their jobs, the official data suggest”) rather than personal circumstances. Nor do phrases that might have been correlated to job losses, such as “sacked” or “let go”, make the cut, for example. That suggests only the terms that are known to correlate with joblessness in the period concerned were included. Annoyingly, such things are rarely constant. Google Flu Trends, which aggregates web searches for phrases like “flu remedy” to paint a picture of influenza outbreaks, was long held as a breakthrough for “Big Data” enthusiasts. But then its famed correlations stopped working, as Tim Harford wrote in the Financial Times this weekend.

More to the point, is the Twitter-mining useful? As the researchers point out, official labour-market data in America is published often and without sampling error. So there is little that a social-media based sample can offer, beyond perhaps greater granularity as to where, specifically, people are losing their jobs, for example. Their aim is to prove that areas which aren’t so well covered by official statistics can be usefully tracked through Twitter.

Plenty of hedge funds claim, sometimes cryptically, to use Twitter as a data source. Some merely use it as a newswire, tracking developments from people they trust, for example when a Wall Street Journal reporter tweets that a planned merger between two companies is off. But a few claim that they can gauge the sentiment around a company’s shares using proprietary algorithms. This latest research suggests that, for now, mining social media is a useful add-on to the old-fashioned way of doing things.

Discover more

Religious competition was to blame for Europe’s witch hunts

Many children are still persecuted as alleged witches in Africa for similar reasons

Has BRICS lived up to expectations?

The bloc of big emerging economies is surprisingly good at keeping its promises


How to interpret a market plunge

Whether a sudden sharp decline in asset prices amounts to a meaningless blip or something more depends on mass psychology