Peer review is an honor-based system

It would take too long to expose all of the flaws of peer review, here are some:

  • some work is just flat wrong because the reviewers cannot analyze all of the mathematical results, and because they cannot redo the experiments;
  • numerous researchers cheat, sometimes in small ways (“2 out of 3 experiments agree with by theory, let us drop the third one”), sometimes in big ways (“I don’t have time to run these experiments, so let me make up some data”);
  • peer review may perpetuate some biases and prevent researchers from putting into question some fundamental questions (“we decided that this is the right way, if you question it, you are a loony”).

However, for all its faults, peer review remains essential in science. I want other researchers to read and criticize my work. I enjoy it very much when people try to find flaws in my work. I think that my work is serious enough that when people point out flaws, I am usually aware of them at some level and I can respond easily (and enjoy the process).

The type of peer review I do not enjoy is the country-club approach: 1) does the paper agrees with the goals and views of the reviewers 2) is the paper written by someone we can respect? Fortunately, you can navigate the system and stay away (mostly) from country-club peer review.

But why do I still like peer review despite its obvious flaws? Because I see it as an honor-based system. In such a system, you have to accept that there will be cheaters. A lot of them. And there will lots of mistakes. All we have to do is be open about it. That is, you cannot say “but my work was peer reviewed so you cannot question it!” or “I am very good, look at these prestigious publications!”. The peer review is there to help the authors. It is not, however, an insurance against fraud or mistakes. I like peer review because it helps me become better, but I do not use the system to determine how good someone else is.

So, what do we do if we want to know how good someone is? You read his work. You reproduce his experiments. You redo his math. Of course, this scales poorly. If you have to hire someone, you cannot read the work of 50 or 500 candidates. So? I think we have to be realistic. It is hard to know how good someone is. You can get to know 10 or 20 researchers in your life. That is about all.

Hiring processes are flawed. You will hire cheaters. Get over it.

Published by

Daniel Lemire

A computer science professor at the Université du Québec (TELUQ).

2 thoughts on “Peer review is an honor-based system”

  1. Daniel, I am not so optimistic. I think you can cheat, cheat a lot, and never get caught.

    First of all, there are ways to make it improbable to get caught:

    1) Never release your data, if you ever bothered to collect some.
    2) Never release your software.
    3) Never flush out fully the proof of a theorem (after all, if people believe your sketches…).
    4) Make it very difficult to reimplement your work.
    5) Cheating by omission is not really cheating, is it? How can you prove that the authors had other data than the one they presented?

    Then, even when you do get caught, there are lots of ways around it:

    1) Peer-review will ensure that it is difficult to publish a paper whose main contribution is to prove that another paper was wrong. We rarely see these papers for good reasons: people hate negative results.
    2) Accuse whoever can’t reproduce your results of having made a mistake or having misunderstood your work.
    3) Say that you have “moved on” and that this problem is no longer interesting.
    4) Say that it was the work of your collaborators and you had very little to do with it.

    In short, cheaters can get by just fine. I am quite certain.

    As long as we take the peer review system for anything else than an honor system, cheaters have a huge edge over honest folks, and this ensures that cheating gets built into the system.

    But the minute you realize that reviewers can’t possibly be able to detect well-crafted fraud, then you start rethinking your metric.

    I am not saying we should not pay tribute to those who publish lots of papers in high places. Just… be careful in how you interpret it. Maybe they careful crafted their papers for this very purpose, you know?

  2. I would think the other aspect of peer review is that peers read the papers you publish. If you succeed in selling your work as consequential (i.e., claiming to solve an important problem and publishing in a prestigious venue), then a lot of your peers will read your work and try to build on it. That makes it pretty likely that you’ll be exposed.

    Sure, that won’t happen with less ambitious publications. But it’s a lot like theft. You can probably steal a little and not get caught most of the time. But I suspect that the expected value of violating the honor system is negative, at least if you’re aiming for any kind of status in the research community–which is presumably the primary motive for cheating. After all, the pay isn’t that good!

Leave a Reply

Your email address will not be published. Required fields are marked *

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax