When in doubts, prefer unimpressive negative results

When you first hear about peer review, you are lead to believe that the purpose of peer review is detect mistakes and improve papers. Then, one day, you submit a beautiful paper presenting a correct result. Surprise! It gets rejected! What happened? Someone decided your work was not impressive enough.

In Why Current Publication Practices May Distort Science, Young et al. explains why the artificial scarcity of publication venues in science may systematically favor wrong results. The short story is this:

  • journals and conferences pick papers that are most likely to get cited;
  • papers with impressive positive results are more likely to get cited;
  • hence, papers with impressive positive results are favoured against unimpressive negative results;
  • however, unimpressive negative results are actually more likely to be correct!

Hence, the modern version of peer review may actually favour misguided, but impressive results over carefully crafted, but boring results.

One of their conclusions is particularly interesting:

To exorcise the winner’s curse, the quality of experiments rather than the seemingly dramatic results in a minority of them would be the focus of review, but is this feasible in the current reality?

Maybe my advice to submit your papers where they are likely to be accepted might actually lead to better science?

Related posts: Productivity measures are counterproductive?, Are we destroying research by evaluating it?, and On the upcoming collapse of peer review.

Source: Guillaume Guité

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

Leave a Reply

Your email address will not be published.

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax

You may subscribe to this blog by email.