Peer review without journals or conferences

Almost all scientists ask their peers to review their work outside of the formal process offered by journals and conferences. Young scientists are routinely told to take their manuscripts to a senior scientist and ask for a review.

With some success, I have even used my blog as a peer review device: e.g., we received many comments on our manuscript on fast integer decoding through a blog post I wrote. If you read some of my recent research articles, you will find that I acknowledge people who commented on this blog. (I have even ended up writing papers with some of them, but that is a different story.) In effect, my blog has helped me get extra reviews for some of my work! I couldn’t be more grateful!

I am not really interested in making journals and conferences go away, but I am interested in going beyond them. I fear that they are often limiting us:

  • In fields such as computer science, limiting the review to journals and conferences effectively cuts most non-academics out. They are also limiting the review to a narrow band of experts. If you are trying to solve problems that matter, this might be entirely wrong.
  • Traditional peer review is anonymous. In principle, this makes it fair and transparent. In practice, it can be needlessly alienating. For example, I would much rather have an open exchange with some authors that I criticize rather than just send an anonymous note. We might both benefit from the exchange. One pattern I have noticed is that some of my (well meaning) comments get ignored. Even when they require little work and can only benefit the authors. We have put walls in the peer review process, and there are good reasons for these walls, but we could do without them if we reinvented the process.

Hence, I have launched an open invitation to the world: send me your drafts, and if I find them interesting, I will review them and then tell the world about them (through social networks) if you revise them to my satisfaction.

My goal is not to replace journals and conferences all by myself, but I do see a growing trend whereas people point to papers that have not yet been peer reviewed and say: I read this, it looks good to me! I’d like more people to participate in this new emerging model.

Anyhow, so far, only one courageous fellow agreed with my terms: Nathanaël Schaeffer sent me his paper Efficient Spherical Harmonic Transforms aimed at pseudo-spectral numerical simulations. I wrote a review, the same way I would for a journal, and I sent it to him. He produced a revised version that took into account my criticism. I am now telling you that if you care for spherical harmonic transforms, you should definitively check out his paper. (Update: Schaeffer’s paper has now appeared in a good journal.)

So where does that leave us?

  • If you are a researcher, and you would be willing to review manuscripts openly the way I did, please let the world know!
  • If you think your work could interest me and you want to try a different type of peer review, please send me your paper!
  • If one of my papers interests you and you want to write a review and share it with me, please do! I also have software that needs reviewing.

I stress that you do not need to be affiliated with a college or have researcher as your title for this model to work. Anyone can write or review a research paper. (Admittedly, few people can write good papers or produce deep reviews but that is another story.)

I am not sure exactly how far we can go with such open peer review processes. But I think we can improve the current system significantly. To make my point stronger, I plan to write a blog post describing how I benefited from reviews and criticisms I have received through my blog and social networks. For now, please believe me: I received insights that I would never had received through the traditional peer review process.

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

6 thoughts on “Peer review without journals or conferences”

  1. I completely agree with your idea of engaging reviewers in a dialog. There’s a lot of mis-understandings that happen with blind reviews and I would often have preferred transparency both as a submitter and as a reviewer.

    But I’m a little worried by your proposal – if it takes off. Suppose everyone did this, including Peter Norvig. Now, say I have this cool idea for fast integer decoding – who am I going to send it first: Norvig or Lemire? A simple game theoretical analysis says Norvig, even though Lemire is clearly the better choice. Why? Because Norvig might get me a high-paying job with a cool company and has thousands more Twitter followers than Lemire.

    I think that keeping the function that journal editors play as “switchboards” for peer-matchups might still be better than “send me your draft, and I’ll advertise your paper if you revise it to my satisfaction.” We just need to change how journals work.

  2. I see a business opportunity: an “E-harmony” for matching authors and reviewers. A technology development opportunity for a better kind of recommender too – reviewers could be “rated” and recommended across different dimensions (subject matter knowledge / detail of reviews / helpfulness). Authors could be rated according to how finished their submissions are (I hate it when I get a poorly written submission with mistakes everywhere in order to get free proof-reading)….

  3. @Andre

    In the future I imagine, you’d seek a reviewer for your latest manuscript, maybe using some kind of computer-assisted process. You’d obviously want your reviewers to be as well known and as influential as possible. But this is actually a good thing!

    I don’t imagine this would replace journals. It is more about going beyond journals.

    That’s like saying “YouTube won’t replace TV channels”. No, it won’t.

  4. @Andre

    By the way, though I can only guess, I suspect that Norvig might be open to something like I describe. Google has not focused its research on journals and conferences.

  5. Could it be built upon ?

    It’s already there and it’s open source.

    In my opinion we would (at the very least) need to (1) differentiate (i.e. givign different weigths) between pseudonym and “real name” users and (2) differentiate between pure up/downvote and actual reviews/comments.

  6. Yes I am very much interested in this.
    Open reviews. Reviews of reviewers. Some closed reviews if really needed. Repositories/versioning for papers.
    At the end only citations really count. Reviewers can help a paper get better IF there is no competition. Ideally… but we all know this is not true.
    So better leave it to the judgement of the masses. Crowdsource reviews and comments.
    At the end only the number of citations and impact of the paper really count.

    I wrote about this also here:

    You might also be interested in:
    Some of this is coming!!!

Leave a Reply to Andre Vellino Cancel reply

Your email address will not be published. The comment form expects plain text. If you need to format your text, you can use HTML elements such strong, blockquote, cite, code and em. For formatting code as HTML automatically, I recommend

You may subscribe to this blog by email.