Many scientific journals use double-blind peer review. That is, the authors submit their work in a way that cannot be traced back to them. Meanwhile, the authors do not know who the reviewers are. In this way, the reviewers are free to speak their mind. It feels fair because the reviewers cannot be influenced (in theory) by the declared affiliation of the authors or their relative fame.

How well does it work in practice? You would expect double-blind reviewing to favor people from outside academia. Yet Blank (1991) reported that the opposite is true: authors from outside academia have a lower acceptance rate under double-blind peer review. Moreover, Blank indicates that double-blind peer review is overall harsher. This is not a surprise: It is easier to pull the trigger when the enemy wears a mask.

Meanwhile, there is at best a slight increase in the quality of the papers due to double-blind peer review (De Vries et al., 2009)—everything else being equal. However, not everything is equal under double-blind peer review. What is the subtext? That somehow, the research paper is a standalone artefact, an anonymous, standardized piece of LEGO. That it should not be viewed as part of a stream of papers produced by an author. It sends a signal that an original research program is a bad idea. Researchers should be interchangeable. And to assess them, we might as well count the number of their papers—since these papers are standard artifacts anyhow.

But that is counter-productive! Research papers are often only interesting when put in a greater context. It is only when you align a series of papers, often from the same authors, that you start seeing a story develop. Or not. Sometimes you only realize how poor someone’s work is by collecting their papers and noticing that nothing much is happening: just more of the same.

Researchers must make verifiable statements, but they should also try to be original and interesting. They should also be going somewhere. Research papers are not collection of facts, they represent a particular (hopefully correct) point of view. A researcher’s point of view should evolve, and how it does is interesting. Yet it is a lot easier to understand a point of view when you are allowed to know openly who the authors are.

Are there cliques and biases in science? Absolutely. But the best way to limit the biases is transparency, not more secrecy. Let the world know who rejected which paper and for what reasons.

Source: This blog post came about through an online exchange with Philippe Beaudoin.

References:

  • Blank, R.M., The effects of double-blind versus single-blind reviewing: Experimental evidence from the American Economic Review, The American Economic Review 81 (5), 1991.
  • De Vries, D.R. and Marschall, E.A. and Stein, R.A., Exploring the Peer Review Process: What is it, Does it Work, and Can it Be Improved? Fisheries 34 (6), 2009.

Update: Mark Wilson has another argument against double-blind peer review. What if you pick up good ideas from double-blind papers that are later rejected and remain unpublished? How do you acknowledge the contribution of the authors of the unpublished work?

Update 2: Patrick Lam points out that the programming languages community now uses a variant of double-blind review for some conferences (like PLDI or POPL, the top PL conferences) where the authors are asked to submit blinded papers, but the identities are revealed to the reviewers after they submit their first-draft reviews.

15 Comments

  1. Daniel,

    While the case put forward against double blind review looks good, one case where your argument may fail is submission by a new author OR a publication which may be a part of a new story. I am aware of researchers changing areas and working on different projects, problems and hence, putting it into context isn’t possible. I do concur on the “transparency” aspect, however, I would vouch for half transparency viz. make the comments public and if any discrepancy found let the program committee decide on the fate of the reviewer. This may mitigate various risk scenarios arising out of the current review system.

    Comment by Denzil — 28/4/2011 @ 17:08

  2. @Denzil

    As a member of program committees, I often find it useful to look at who the authors are and what they have done. It seems almost essential.

    Comment by Daniel Lemire — 28/4/2011 @ 17:51

  3. “Research papers are often only interesting when put in a greater context.”

    That is true but what that’s why papers should cite prior work and how the particular paper relates to it. It shouldn’t matter if it was your own prior work or someone else’s to make it worth publishing.

    Comment by Sebastian H. — 29/4/2011 @ 3:00

  4. @Sebastian

    I disagree. See my following blog post:

    http://lemire.me/blog/archives/2011/04/29/is-science-more-art-or-industry/

    Comment by Daniel Lemire — 29/4/2011 @ 8:14

  5. I reviewed papers for ICDAR recently, and I found its single-blind system to be very helpful. For one, knowing authors tells me how carefully I need to check their derivations. In ideal world I’d rederive every formula in every paper I review, but who has this kind of time?

    In another example, I googled authors publications and found essentially identical paper published in earlier year. Author failed to mention this in their submission.

    Recently there’s an increasing amount of research that is happening outside of traditional publishing system. What would you do if the submission is almost identical to a work you’ve seen on someone’s research blog/mathoverflow/etc? With double blind you have to guess if it’s the author finally packaging up their work for publication, or an impostor with copy/paste.

    Comment by Yaroslav Bulatov — 1/5/2011 @ 18:59

  6. I used to question double-blind review like you, but the best argument in its favour I have heard is that there is apparently a verifiable bias against women and minorities in “transparent” reviewing. I am not aware of references, but this point was made by prominent female researchers in my field (they are few and far between!), which I would tend to trust.

    Technically this doesn’t mean we could not use a reverse single-blind system where authors are hidden and reviewers names are published. They could be listed (without their reviews) with a paper without jeopardizing the confidentiality, if there are enough of them, which there should be anyway.

    Point is: it’s easy to come up with examples where keeping names hidden causes problems, but the trade-off is real and arguably worse.

    Comment by Cyril — 3/5/2011 @ 10:24

  7. @Cyril

    We recognise that women remain poorly represented at senior levels across the sciences, including ecology and evolution. Given the lack of evidence that double-blind review favours female authors, we suggest that efforts to address this imbalance should be directed into supporting innovative schemes aiming to change working conditions, including initiatives such as increased flexibility of working hours, support for scientists returning to research after career breaks and mentoring schemes. We see no reason to direct time and resources into overcoming the acknowledged resistance of editors and the scepticism of many in the field regarding the alleged benefits of double-blind review.

    Reference:
    Webb, T.J. and O’Hara, B. and Freckleton, R.P., Does double-blind review benefit female authors?, Heredity 77, 2008.

    Comment by Daniel Lemire — 3/5/2011 @ 10:30

  8. Daniel,

    Note sure what the point of that excerpt is, especially in light of the fact that the paper right after that one (last page of the pdf you link to), as well as the paper they are replying to, seem to put things very clearly:

    “[...] the study is a compelling indication that changes in review policy [to double blind] increase female representation through editor, reviewer or author behaviour.”

    As it’s the nature of such contentious issues to be argued either way, I’m actually more interested in the actual perception of female/minority researchers in my field. Yours and others may be different!

    This is a point that should at least be addressed as it seems to be more serious than potential recycling/plagiarism.

    Comment by Cyril — 3/5/2011 @ 11:22

  9. I’m interested in your opinions on this one, In practice, how far does double blind review got in retaining author anonymity?

    Given that authors tend to cite their own previous works, as Daniel indicates, and that experts in a field tend to know each others work very well, how often are you able to guess who is the author of a paper that you are reviewing?

    Comment by Kris Jack — 3/5/2011 @ 11:43

  10. @Cyril

    I think we agree that a more female-friendly science would be better for everyone.

    I’m actually more interested in the actual perception of female/minority researchers in my field.

    Fair enough. Let us accept as a fact that women feel that they are treated more fairly with double-blind peer review… as opposed to, say, open peer review.

    Still. We have to watch out for unintended consequences.

    Do consider however that double-blind peer reviews are reportedly harsher (and this makes intuitive sense to me). What impact do these strongly worded reviews have on women over time? Do men and women cope equally well with “macho” reviewing?

    Comment by Daniel Lemire — 3/5/2011 @ 12:59

  11. @Kris Jack:

    I’d say that out of all double-blind papers I have reviewed over the last 15 or so years, I’ve suspected who the author was many times, but mostly in a non relevant way (ie from references to past papers and work, but I did not know these people anyway). I have been dead certain only once or twice because I knew the authors, how they write and the field well enough.

    Comment by Cyril — 5/5/2011 @ 9:25

  12. I am emphatically FOR double-blind reviews, and reading the above comments re-inforce my conviction. Reviewers “suspect” who the author is, “google” his/her prior work, fail to check derivations based on the authors’ reputation, and spend their otherwise precious time on inconsequential aspects – focus on the paper given to you for review, and do not concern yourself with the author and his affiliation. Look, if you don’t have time to do a proper review, THEN DON’T DO IT, and say so from the start – there are probably other people in the whole wide world ready to step up and do a proper job (but you wouldn’t want that, would you, you want to be part of the game, don’t you, you want to be able to make or break other people, right?). The absence of a double-blind review perpetuates the rigging of the system and is deeply unfair to the authors – if you don’t want to do that, fairer would be to disclose the identity/affiliation of the reviewers to the author, so that he/she could have a better idea about how competent they really are to judge his/her work. How comfortable would reviewers feel in that situation?

    Comment by Epizdochie — 22/5/2012 @ 14:31

  13. @Epizdochie

    Reviewers “suspect” who the author is, “google” his/her prior work, fail to check derivations based on the authors’ reputation, and spend their otherwise precious time on inconsequential aspects – focus on the paper given to you for review, and do not concern yourself with the author and his affiliation. Look, if you don’t have time to do a proper review, THEN DON’T DO IT

    The person taking the decision to accept or reject the paper, the editor, knows your reputation and your affiliation. So, sadly, there is no way to take this out of the equation. In fact, in a double-blind system, the editor has even more power.

    To properly review a paper, I need to compare it with the state-of-the-art and previous work. Typically, this means reviewing (even unknowingly) what the author worked on previously. So, you cannot just focus on the paper as a separate entity.

    (…) fairer would be to disclose the identity/affiliation of the reviewers to the author, so that he/she could have a better idea about how competent they really are to judge his/her work. How comfortable would reviewers feel in that situation?

    I have been advocating precisely this scheme for years. If you are going to criticize someone, you should be able to do it openly, in front of witnesses, not under the cover of anonymity.

    But it works both ways… if you are going to ask me to review your work, through a journal or otherwise, please let me know who you are.

    Once everything is public, the focus will naturally be on the science. If people unfairly review your work (because you lack a prestigious affiliation) then you’ll be able to point to the evidence, as it will all be out there. It also takes power away from the editors, and it puts it back in the community at large.

    Comment by Daniel Lemire — 22/5/2012 @ 16:03

  14. You’re very prompt, Daniel, I commend you for that (I also have an appreciation for the situation at hand, I am anonymous in this conversation, but you’re not, sorry).
    I feel that the editor would have no ammo once he receives the reviews from the reviewers, so a double-blind would hurt no one, Also, the rejection of a paper based on a perceived lack of prestigious affiliation would be hard to prove even if the reviewers are known (after all, the author is a nobody to start with, AND, there is no published work to speak of, so who would pay any attention to him/her?) – a double-blind probably seems a better choice there, at least no one can cry “foul”.

    But anyway, I’m glad that turning the tables on the reviewers (i.e., making them non-anonymous) seems like a good idea to you, too, so let’s work on that, it looks like a good start,

    Again, thanks for replying.

    Comment by Epizdochie — 22/5/2012 @ 20:42

  15. @Epizdochie

    I feel that the editor would have no ammo once he receives the reviews from the reviewers

    The objectivity of the editor is important especially because reviewers are rarely unanimous. Editors effectively act as super-reviewers. An editor can stop a paper from going to peer review. He can pull the paper out of the process after receiving even a hint of criticism, or he can ignore much of the criticism and push the paper forward.

    The decision of the editor is typically final and without appeal. The recommendation of a reviewer is always only a recommendation, it can always be ignored by the editor.

    So, really, if you feel that it is important that your identity be hidden, it should be hidden from the editor.

    Comment by Daniel Lemire — 22/5/2012 @ 21:45

Sorry, the comment form is closed at this time.

« Blog's main page

Powered by WordPress