Two Things Papers Don’t Show, but Should

edenNegative results and reproducibility, although central in science, are neglected by scientific publications. We urgently need specialized journals to deal with them.

1. Negative Results

A good colleague and friend of mine—for privacy let’s fictionally call him Oliver—has a problem: his simulations don’t match the experiments.

For months, he and his team worked on the computational simulations of a certain molecule. He did, as usual, a high-level and serious research. By the end, however, Oliver found out that the results of his simulations were exactly the opposite of the experimental results published by another group not long before.

He checked and rechecked everything. He masters all those methods and procedures. There was no mistake: the simulations given by the theoretical model that he adopted really insisted on contradicting the lab measurements.

And that was a dead-end for the project. Those data were simply unpublishable in any respectable scientific journal, unless he could also find out why simulations and experiments were fighting. However, to find this underlying reason would be another project, which he considered not worth pursuing when allocating his resources.

Those thousands of hours of computation and hundreds of hours of highly specialized human work will never hatch and make their way to the electronic pages of a journal.

This is a pity. Oliver’s negative results should be made public, not only due to the costs that they involved, but mainly because they are of interest to the community of specialists in his field. Why do theory and experiment not match? Are the theoretical approximations not valid in that case? Are the experimental results wrong?

Negative results is a niche completely neglected by scientific publications; and they are so common. Every scientist often faces negative results during their research: simulations that didn’t match experiments; hypotheses that weren’t confirmed; derivations that didn’t result in the expected output.

We should be able to make them public, first, to let other specialists evaluate and eventually find out the source of the problem; second, to make people aware that they may be wasting resources on something that won’t work.

2. Reproduced data

Negative results isn’t the only editorially neglected niche. Reproducibility—the heart of the scientific enterprise—is also strangely abandoned.

Everyone knows that science requires reproducibility. Reproducibility, however, naturally doesn’t mean that every piece of scientific information must be double-checked by an independent part. It only means that every piece of information should be companied by enough technical information to be eventually repeated and verified.

Nevertheless, this isn’t always the case: C. Glenn Begley and Lee M. Ellis, for instance, revealed that a biotech company could confirm the scientific findings of only 6 out of 53 landmark papers on cancer research. And my own experience tells me that this is a very common problem in my field as well.

When we read a paper, we usually just assume that the authors did a good job, specially if the results are sound. We can’t aford to test everything we read. But should we blindly rely on published data? Maybe the authors did a technical mistake and the results are wrong. Or maybe they’re crooks and cooked up data. Or maybe the publisher made a copy mistake when preparing the manuscript and it went unnoticed in the proofs.

You may remember the Schön scandal. In 2002, the highly awarded German physicist Jan Hendrik Schön was found to be publishing fake data on semiconductor physics. Probably he was caught only because he called too much attention, peacoking with his many papers published on Science and Nature. Were he publishing on low-profile journals, maybe no one would ever care to try to reproduce his data, and he would be comfortably nesting in some minor university now.

It’s not that scientists are lazy and don’t want to check what the giants have under their shoulders. In fact, we do a lot of reproduction as a routine work. But this reproduced data, which consume time and resources, don’t have a proper way to be publicly shared. Like the negative results, they’re fated to remain buried in our notebooks.

We certainly could improve the overall quality of the published science, if we had not only the potential of reproducing data, but also the practice of doing that and the habit of publishing them.

Researchers would probably be more careful, precise, and comprehensive in their method description and data presentation, if they knew that they may be shown wrong or inconsistent at anytime. We would all work on safer grounds, if we could count on published compilations of independently verified results.

Moreover, to have reproducibility as a legitimate publishing area would lay excellent landmarks for people migrating into a new research field. They could safely fly over already well-known grounds, before exploring their own lands.

We can even bet that a new class of specialists dedicated to the niche of data reproduction would evolve. If you don’t like the idea, just remember that vultures play a quite relevant ecological role.

Editors, opportunities are waiting for you

In principle, we don’t need journals to make negative and reproduced data public. We could add them to our group websites or send them to public archives.

But I don’t think this is the way to go. We need to make this kind of research public within the formal peer-reviewed publication system. To be reliable, accessible, and worth investing in, negative and reproduced data should appear in properly reviewed papers, contributing to the production and evaluation indexes of their authors.

Dear publishers and editors, negative and reproduced data are a great opportunity. Don’t miss it!

As for negative data, in particular, I may even leave a suggestion for a journal name: “Proceedings on Negative Analysis in Sciences,” or PNAS to shorten.


More about scientific publishing:

The illustration is a detail of Franz Stuck’s “The Lost Paradise“, from 1897.

Categories: Science Policy

Tags: , , , , ,

3 replies

  1. Proceedings on Negative Analysis in Sciences,” or PNAS superb abbreviation! 😀

  2. Maybe:
    1) attached in the PhD thesis. I know one experimental Professor ask his student to do it like this way. So it’s very thick for the thesis,,,
    2) wikipedia like Surface Hopping, CASSCF,,,: technical details or experience, failed system,,,

A penny for your thoughts...

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: