Turks of the world, unite!

By G.F. | SEATTLE

AMAZON'S Mechanical Turk service was designed to turn humans into automata for tasks that could be broken down into suitably trivial and repetitive bits. It has been used to great effect by Amazon and others, and typically has tens of thousands of Human Intelligence Tasks (HITs)—like categorising images, transcribing segments of audio, or writing short bits of text—available to anyone willing to grapple with them for a pittance a pop.

Requesters do not know the identity of workers, who are assigned unique IDs, but who may not be requested by name. Jobs may be restricted to workers who pass certain qualifying tests, however, which allows a prospective employer to screen for appropriate candidates. Workers only see the name of the requester without verification or identifying details.

This relative anonymity means that Turk may be used for less savoury purposes like posting spam comments, ad listings, reviews, or other material on sites that require registration or verification of humanity through a CAPTCHA. It is regularly used to recruit (often unwitting) workers as human pawns in games of chess with site owners and search engines. For example, a typical task available as Babbage was writing this piece asked a worker to post a Craigslist ad for a "Like new Blackberry Torch and Iphone 4 Price: 85". The description contained links that redirect and appear to generate affiliate revenue for the requester for sales resulting from heedless Craigslist users following the path.

Sometimes, spammers fail to pay workers, either claiming the results are too poor to merit payment (a situation which Turk allows to limit botched jobs), or through invalid means of payment, such as stolen credit cards. Such miscreants foul the pool of workers, who are less likely to trust new companies, says Panos Ipeirotis, a professor at New York University's Stern business school, who studies and tests such matters. This forces companies to pay higher rates to attract leery workers, turning many potential clients off task-based crowdsourcing.

To see how big a problem this represented, Mr Ipeirotis and his colleagues used Turk itself. First, they selected thousands of HITs from recently registered requesters. Then, they created their own HITs, asking workers to identify whether other requesters' HITs involved spamming, and if so, what sort (creating a Twitter account and posting ready-made tweets, clicking on ads, writing a positive review, or pressing the Facebook Like button).

First, though, Mr Ipeirotis had to eliminate dubious results. Many of these, it turns out, arise from scams involving software which imitates human workers only to return a random answer—or one coordinated with other robots. One way to screen for robots is to see whether many different workers return the same response to the same query. Alas, enough robot workers programmed to give the same coordinated response may afford an air of consensus, so other tests are needed. He turned to Crowdflower, a company that helps to filter out robots and other poor performers thanks to more rigorous qualifying tests and evaluating responses to queries to which an answer is known. He also analysed surveys by workers (each of whom has a unique ID) to check whether responses to questions about gender, age, etc, matched over time.

The results from those workers who successfully jumped through all the hoops were disturbing. Mr Ipeirotis's group found 41% of the nearly 6,000 HITs tested from relatively new requesters were judged to be spam. They also discovered that spammers request almost entirely spam tasks, while honest turkers nearly always provide only legitimate jobs. This sounds obvious, but it reveals that Amazon should have been capable of fingering the spammers with ease.

Mr Ipeirotis posted his blog entry with details and a spreadsheet on December 16th, 2010. Amazon must have taken it to heart. A few days later all the spam tasks had disappeared. However, the problem has not gone away. In early May, Mr Ipeirotis added a chart to his Mechanical Turk Tracker site showing the number of spam HITs that are currently listed on the service. The site displays current and historical data, including the number of projects, each of which may involve anything from one HIT to thousands of them. It also archives and allows searches of task descriptions. On May 11th, for example, the chart showed a sudden spike of several hundred spammy jobs that disappeared an hour or so later. More typical is the current average of roughly 200 dubious projects out of 1,500 available.

Mr Ipeirotishopes analyses like his will keep Amazon on its toes. Having done his bit to curb worker exploitation, Mr Ipeirotis's next goal is to improve their lot further, by raising remuneration standards. His motives are far from socialist, though. He believes that better pay will lead to sounder results, and make it more difficult for spammers to despoil Mechanical Turk's ingenious ecosystem.