Science & technology | The scientific method

Let’s just try that again

Reproducibility should be at science’s heart. It isn’t. But that may soon change

THE Journal of Irreproducible Results is a long-running satirical magazine, designed for the amusement of scientists. If the title were not already taken, though, it would be a good one for another, more serious publication that is being launched on February 4th. The Preclinical Reproducibility and Robustness Channel, an electronic rather than a paper journal, is dedicated to the task, found tedious by most academic researchers, of replicating and testing the experiments of others. Professional egos, the exigencies of career-building and the restricted sizes of grants and budgets all conspire against the rerunning, in universities, of old studies instead of the conducting of new ones.

Commercial researchers cannot afford to be so choosy. If they pick an idea up from academia, they have to be sure that it works. Often, it doesn’t. For example, when staff at Amgen, a Californian drug company, attempted to reproduce the results of 53 high-profile cancer-research papers they found that only six lived up to their original claims.

This figure is probably typical. What was not typical was that Amgen actually offered it for publication, and it was, indeed, published (in 2012, by Nature). Mostly, journal editors, like academic scientists, are more interested in new work than in the refutation of old stuff. Amgen’s submission was spectacular, because so many papers were involved at one go. This may have been what got it over the bar. If each refutation had been written up and submitted separately, the outcome would probably have been different. As for the six successful replications it included, why would anyone bother to publish work confirming what was already known?

Yet knowing which previous research is and is not correct is crucial to the progress of science, and a repository of such information would be useful. Hence the Preclinical Reproducibility and Robustness Channel. Its publishers, Faculty of 1000, based in London, hope to provide an outlet for the accumulated replications gathering dust in commercial laboratories (Amgen has promised its trove to the venture), and also to stimulate academic scientists to follow suit and provide more.

The problem, though, is not restricted to medicine. An analysis of 98 psychology papers, published in 2015 by 90 teams of researchers co-ordinated by Brian Nosek of the University of Virginia, managed to replicate satisfactorily the results of only 39% of the studies investigated. Again, it was the very size of this project that got it into print, as smaller studies languished.

Things may now be changing. Both reproducibility and the whole openness of the scientific process are discussed much more than once they were. An entire institute dedicated to the matter, the Meta-Research Innovation Centre at Stanford, in California, opened for business in 2014.

If this institute flourishes—and even more so if it is emulated—it may even become possible to make a career out of being a buster of others’ questionable efforts: a forensic scientist of science, as it were. That is by no means certain, and there will probably be few Nobel prizes in it. But mopping up messes is an honourable activity, and this week’s launch of a new outlet for the publication of duplication is part of the clean-up.

This article appeared in the Science & technology section of the print edition under the headline "Let’s just try that again"

How to manage the migrant crisis

From the February 6th 2016 edition

Discover stories from this section and more in the list of contents

Explore the edition

Discover more

Antarctica, Earth’s largest refrigerator, is defrosting

The world must pay more attention to its southern pole

Killer whales deploy brutal, co-ordinated attacks when hunting

Their techniques are passed down through the generations


A new generation of music-making algorithms is here

Their most useful application may lie in helping human composers