While publishers and researchers have mixed feelings about whether scientists should publish negative results in peer-reviewed journals, with some arguing that negative results are essentially a waste of time, federal funding agencies appear to be largely in favor of publication.
I approached a number of federal funding agencies in the U.S. about their positions on publishing negative results, and the results ranged from relative indifference to vocal support in favor of publishing them.
[Note: this is the third in a series of posts about publishing negative results. The first two posts were a general overview of the subject and an interview with a senior editor at the journal PLOS ONE.]
The National Science Foundation (NSF) does not appear to have strong feelings about publishing negative results, but is certainly not opposed to the idea. In response to a query, an NSF spokesperson said: “NSF funds a broad range of disciplines and there is substantial variation in the scholarly communication practices among these disciplines and consequently in the types of products resulting from NSF funding. These kinds of value judgments – for example, the merit of publishing negative findings – are left up to the research communities.”
But the National Institutes of Health (NIH) has staked out a much firmer position on the issue. In a statement from an NIH spokesperson, the agency said: “The dissemination of research findings, both positive and negative, is a critical step in the scientific process, and NIH promotes data sharing and encourages the transparency of research results through publications and other mechanisms, such as the posting of data in publicly accessible databases. Additionally, NIH encourages the publication of results of all of the research it funds, regardless of outcome.”
That support was echoed by research administrators in the institutes themselves.
Jeffery Schloss, director of the Division of Genome Sciences at the National Human Genome Research Institute (NHGRI), said in an email: “NHGRI is a leader in promoting the rapid and wide dissemination of research results because we think results are of most use when they’re available to everyone who might wish to use them (within legal constraints, such as those that result from consent provided by human subjects for use of their information). Consistent with this position, we are unaware of any practice among our staff that would give either lower ‘credit’ for publication of negative results than of positive results. That being said, we also have not explicitly discussed encouraging investigators to publish negative results.”
Dr. Michael Lauer, director of the National Heart, Lung and Blood Institute’s (NHLBI) Division of Cardiovascular Sciences, was also a vocal supporter of publishing negative results – and answered my questions at greater length.
Communication Breakdown: Does NHLBI track the journal articles that result from NHLBI-funded research? If so, why?
Michael Lauer: Yes we do, as does all of NIH. This is just one way in which we can measure the impact of our work. Anyone can see those publications by using the NIH Reporter portal.
For example, go [to this] description of an NHLBI-funded grant. If you click on the “Results” tab, you will see the publications – there are many in this case – as well as data on citations, to which we also pay attention.
CB: Does NHLBI give journal articles about negative results the same weight as articles about positive results? If not, why not?
Lauer: Yes, we do. We don’t differentiate articles by their results (“positive” versus “negative”).
CB: Does NHLBI consider the impact factor of journals where these articles are published? I ask because a journal that focuses primarily (or exclusively) on negative results would likely have a low impact factor (because there would likely be fewer citations).
Lauer: No, but we do pay attention to citation data. [Lauer then refers to a 2011 commentary in Circulation Research]. See table 2 [in the 2011 commentary], in which we note the serious problems associated with looking at Impact Factor as opposed to other measures. I would note that nearly every issue of NEJM and JAMA – two of the most highly cited journals – contains at least one article with a “negative” result. For example, this just-published trial on tonsil surgery for sleep apnea was negative for its primary endpoint.
CB: Does NHLBI think it is important to publish negative results?
Lauer: Yes. In fact some of our most highly cited papers (WHI, ACCORD, AIM-HIGH) were “negative,” producing results contrary to the study hypothesis.
I’d still like to get input from others in the publishing community. What do big-name publications like Nature and Science make of negative results?
I’m also curious as to whether this post, or my other recent posts on negative results, have been useful or interesting to anyone in the research community. Did I simply rehash old arguments, or did these posts offer additional insight? I would love your feedback.