The art of paper review
I do not claim to be an expert at reviewing academic papers, but I have done my share of work. Here is my recipe:
- Reproducibility, (self-)plagiarism and presentation are easy to evaluate and I usually spend quite a bit of time on these issues. Science should be reproducible. (Panos Ipeirotis seems to agree with me.) Plagiarism can be surprisingly hard to detect, but it is also amazingly frequent, so I usually search for a few word cooccurrences in Google. Presentation is, on average, quite poor. Figures are often ugly. Poor English is frequent.
- The relevance and strength of the paper is something I usually have an opinion about. Alas, it is easy to be wrong about the importance of a paper, so I usually do not have much to say unless I have directly worked on the same problems for a couple of years.
- Correctness is hard to check especially if I am not a domain expert. I usually pick up on secondary details. Are the results credible? Do the authors mention some special cases that should have arisen in their analysis or experiments? I must unfortunately admit that I usually cannot be sure that the papers I have reviewed are correct. At best, I can voice an opinion about their credibility.
Montreal, Canada 
Follow on
Peer review is great, but it can be a rather inefficient evaluation mechanism:
- Should good reviewers receive more weight than bad reviewers? We can operationally define good based on how well a reviewer approximates the mean behavior across reviews.
- Shouldn’t authors–particularly, established ones–be allowed / expected to put their credibility on the line when they publish? Again, that could take some of the onus off of reviewers, since authors will have an incentive to filter their own work more harshly as their reputation increases.
I suppose the current system works, but it does seem wasteful.
p.s. this issue has also come up on Panos’s blog.
Comment by Daniel Tunkelang — 13/5/2008 @ 14:18