A Journal Editor’s Perspective on Publishing Negative Results

In which a journal editor weighs in on getting negative results into the hands of people who can use them. (Photo credit: North Carolina State University)

Earlier this week I wrote about two questions regarding negative results. First, should researchers publish their negative results? Second, why is it so hard to publish negative results?

Some of the responses I got on Twitter and Facebook drove home how divisive the issue can be. Many researchers thought publishing negative results would be incredibly helpful. But others were decidedly less enthusiastic. As Twitter user Peter Dudek put it, “If I chronicled all my negative results during my studies, the thesis would have been 20,000 pages instead of 200.”

To get some input on publishing negative results, I thought it would make sense to reach out to the publishing community. So I contacted Damian Pattinson, editorial director of PLOS ONE, with my questions. Damian, in turn, asked for input from Eric Martens, a senior editor at PLOS ONE. [Note: I contacted other journals as well, and will share their responses if/when I receive them.]

Here’s my Q&A with Martens on whether negative results matter, why there are so few publications on negative results, and what the publishing community can do about it.

Communication Breakdown: It seems to me that publishing negative results would be a service to the research community, as it would help prevent people from duplicating the same efforts — and expedite the process of finding positive results. But I may be wrong. What do you think? Why should (or shouldn’t) negative results be published?

Eric Martens: The publication of negative results is important for exactly the reasons you mention. There are millions of potentially valuable negative results sitting unpublished in labs around the world. There is no doubt that many researchers come up with similar hypotheses to test and have thus been duplicating each other’s efforts without knowing, wasting valuable time and research resources in the process.

In addition, there are some fields where publishing negative results are not commonplace, particularly when a positive effect has been demonstrated previously. (Social psychology is a prominent example and some information about that field and a recent PLOS ONE paper are here: http://www.nature.com/news/disputed-results-a-fresh-blow-for-social-psychology-1.12902). Thus, publication of negative results can be an important element of validating and correcting the scientific literature.

Lastly, publication of negative results is important for meta-analyses, which draw conclusions from a series of related studies and are often used as an important validation of clinical outcomes. Hesitance to publish negative results may distort meta-analyses because the studies included in the analyses will be biased toward demonstrating the effect in question.

CB: Are you aware of any numbers on the percentage of published papers that have “negative result” findings? Do you have an estimate?

Martens: I haven’t heard any specific numbers, but there is some evidence that the fraction of studies reporting negative results is decreasing. At PLOS ONE, we probably publish more than the rest of the literature does, as a whole, given that our publication criteria state that we will consider such studies. However, the proportion of purely negative studies is still very small. We hope that this number will increase as publication of negative results becomes more mainstream, and PLOS ONE is increasingly recognized as a venue for this work.

CB: I’m curious as to why there seem to be so few publications on negative results. I.e., whether it stems from a lack of submitted papers, or a lack of accepted papers. Do researchers submit papers on negative results? If so, how often are they accepted for publication?

Martens: There are a number of reasons why there are so few publications on negative results.

Why don’t we share negative results with others? (Photo credit: North Carolina State University)

On the submissions side, researchers may observe a negative result, but don’t feel it’s worth the effort to flesh out and publish the results. For example, a pilot experiment may suggest that a particular avenue is not worth the investment of additional resources, yet the pilot data does not meet the standard of proof for publication in the field. The growth and acceptance of venues that publish individual experiments or figures, e.g., PLOS Currents and FigShare, may start to change this behavior.

Researchers may also be disinclined to publish negative results because they feel it doesn’t improve their chances of promotion or being funded, or generally advance their careers. Further, though I don’t have any direct evidence for this, I could also imagine that some researchers would prefer not to make their negative results available to competitors when they feel there is no major benefit for themselves.

On the publishing side, many journals would reject studies reporting negative results because they feel the work is not impactful enough. PLOS ONE does not do this; our publication criteria state that we will consider all work that makes a contribution to the field, independent of impact. It can be difficult to assess what constitutes a contribution, so for replication studies, we require that “authors must provide a sound scientific rationale for the submitted work and clearly reference and discuss the existing literature,” and we apply the same principle to submissions reporting negative results. That is, negative results are valuable to the community in cases where the result is illuminating in the context of previous work. Thus, we will not consider work where a negative result is obviously expected, for example a hypothetical study that concludes “Precipitation levels in California have no effect on cancer rates in Indonesia.”

CB: Is there anything that could be done to encourage the publication of negative results?

Martens: There are some journals that peer review negative results, which is good for ensuring the reliability of the data. However, reducing barriers to posting these results may make it more prevalent. If researchers could post negative results quickly and easily – i.e., without formatting requirements or concerns about making revisions – to a database or server, it might happen more often.

CB: I’m also curious as to whether anyone actually reads negative result papers, or cites them. Are citation rates relatively high, low or “normal” for negative results papers published by PLOS ONE? Do those papers get an average number of views?

Martens: This is highly variable depending on the type of negative result. Some negative results are of great interest because they contradict previously published results (e.g., “Three Unsuccessful Attempts to Replicate Bem’s ‘Retroactive Facilitation of Recall’ Effect”, which has been viewed over 21,000 times to date). Others may be of high interest because they contradict expectations of the community (e.g., “Lack of an Antibacterial Response Defect in Drosophila Toll-9 Mutant”). I would imagine that negative results that are not unexpected would garner fewer readers and citations. I don’t have any aggregate data for PLOS ONE, but recent research has shown that negative results are less-cited than positive.

CB: Has PLOS ONE spent any time thinking about the issue of publishing negative results? If so, how might you address the issue?

Martens: Yes, PLOS ONE in particular has thought about publishing negative results and we state in our publication criteria that we will consider these for publication. Also, this fact is often accentuated when staff editors give presentations about the journal. Partly, publication of negative results suffers from a PR deficit, so we seek to promote the benefits of making these available.

CB: Do you know whether this is an issue being considered by other prominent journals?

Martens: The most overt efforts at addressing this issue [are via the] journals that are dedicated to publishing negative results, e.g. Journal of Negative Results in Biomedicine from BMC and Journal of Negative Results [which covers research on ecology and evolutionary biology]. Other journals that work in a similar way to PLOS ONE are also exploring the idea. Scientific Reports from Nature Publishing Group, for example, states [on its website] that “Papers describing negative results will also be considered if their insight is useful.” There hasn’t been much movement from the more prominent journals, though, beyond the occasional statement that negative studies are important. Because these papers rarely garner high numbers of citations, they are less attractive to publishers that compete for high impact factors.

[Random Note: I read this week in the DrugMonkey blog that the open access journal F1000Research is waiving its publication fee for manuscripts on negative results. The offer appears to be good until the end of August. I should mention that I had not previously heard of F1000Research, but that likely says more about me than about the journal. And Ivan Oransky wrote about F1000Research here. Hat tip to Peter Edmonds for bringing this to my attention.]