In my previous post, I argued that people who pursue double-blind peer review have an idealized “LEGO block” view of scientific research. Research papers are “pure” units of knowledge and who wrote them is irrelevant.
Let us take this LEGO block view to its ultimate conclusion.
If science is pure industry, producing standardized elements—called research papers, why should papers be signed as if they were pieces of art? The signature is obviously irrelevant. Nobody cares who made a given LEGO block. Thus, I propose we omit names from research papers. It should not change anything, and it will be fairer.
Indeed, why not have anonymous papers all the way? Journals could publish articles without ever telling us who they are from. We would ignore, for example, which papers were written by Einstein or Turing. How is that relevant? How does it help us to appreciate a given paper to know it was written by Turing?
What would we do for conferences? Because papers are standard units, people could attend conferences and be assigned a paper, any paper, to present. Presenting your own work is a bit too egotistical anyhow.
Of course, for recruiting or promotion purposes, we would need to be able to map research papers to individuals. But, because papers are standard units, all you care about is the number of papers and related statistics. Thus, an academic c.v. would not list research papers, but instead provide a key that could be used to retrieve productivity statistics.
Of course, this is not, even at a first approximation, how science works. Science is more art than industry. That is why we put our names on research papers. It does matter that it is Turing that wrote a given paper. It helps us understand the paper better to know its author, its date and its context. When I receive a paper to review, I try to see how the authors work, what their biases are.
Research papers present a view of the world. But like Plato’s cave, this view is fundamentally incomplete. If a paper report the results from some experiments they conducted, the paper is not these experiments: it is only a view on these experiments. It is necessarily a biased view. Do you know what the biases of these particular authors are?
Let us be candid here. When reviewing research papers, there is no such thing as objectivity. Some papers are interesting to the reviewer, some aren’t. What makes it interesting has to do with whether the world view presented is compatible with the reviewer’s world view. And because different individuals have (or should have) different world views, it does matter who wrote the paper even if we omit names. It helps me to find your paper interesting if I can put myself in your shoes, get to know who you are. An anonymous paper is far more likely to be boring to me, because it is hard to have empathy for the authors.
Some days, we all wish it did not matter who we are. Can’t people just look at our work on its own? You can get your wish by becoming a bureaucrat or a factory worker. Science is for people who want to see their name in print, people who want to build their reputation and cater to their inflated ego. In short, good science is interesting.
If the reviewers’ point of view is merely tainted (in an “honest” way), we don’t need double-blind reviews. We already require 3-4 different people to evaluate a paper in order to average out any reasonable bias related to a paper’s *content*.
In my view, double-blind reviews are a tacit acknowledgment that some reviewers are not just biased, but downright hostile and dishonest when evaluating other people’s work when they know who they are. We should be concerned that we let such people review papers in the first place.
A conference which I won’t name even had to let each member of the PC select one paper to be accepted without argument. I presume the reason of this measure is that paper reviews regularly degenerate into cockfights, to the point of being dysfunctional.
Bottom line: starting to patch your review process is probably a sign of a deeper problem in your community that you are avoiding to address.
@Hallé
To some extend, people react how we expect them to react. We keep the reviewers anonymous, so that they can be nasty to the authors without fear of reprisal. And guess what happens?
The assumption, it seems, is that people would not be openly critical in the open. But we know that to be false. For example, most of us got Ph.D.s under an open process. You know who reads your thesis, and they know who you are. [After all, we do put our names on our Ph.D. thesis, don’t we?] Well… Nobody that I know has ever hinted that getting a Ph.D. was a joke. It is hard work and much rigor is needed. The system works rather well. Getting a Ph.D. does not mean that you are a genius, but generally speaking, most people who have a Ph.D. have done a sizeable chunk of scientific work.
It works because external reviewers will come down hard on a Ph.D. student when needed. Why? Because nobody wants to have been the external on a “garbage Ph.D. thesis”. It looks bad on a c.v.
Overwhelmingly, external reviewers, when they are not anonymous, try to be fair… because they know that it would tarnish their reputation if they were unfair. You don’t want to sink a good student just for spite because it can come back to haunt you later.
As I wrote, we need more transparency, not less. This will make communities healthier.
@Daniel: I agree with your view of openness. I once stumbled upon this text:
http://www.cs.rutgers.edu/~muthu/ccmfun.pdf
Go look at p. 9, section “E”. The author suggests that the reviewing process be public, so that paper reviews of both accepted and rejected papers be freely available on some form of centralized repository. I would totally go for such an open model. We’ve all once received reviews that were plain nasty, and I bet the reviewers would be more nuanced if their responses were published.
Anonymity != Interchangeability. Even on the assembly line there are varying levels of skill. One worker might thread a screw perfectly every time, another might leave it loose now and again. That mistake could cause my GPS to die after 6 months. I have no knowledge of who put my technology together, but it can certainly matter. Thus anonymous authors by no means implies you could evaluate a researcher by article count. You could, however, read anonymized articles by all candidates, and rank them that way.
What if Einstein hadn’t signed his papers? From a human drama angle, sure, our scientific story cache would be poorer without the Annus Mirabilis. But if nobody knew the stochastic explanation of brownian motion and special relativity had come from the same person, would science have been held back?
Which isn’t to say that we should write anonymous articles. From a PR standpoint, from a practicality standpoint, from a transparency standpoint it all makes sense to sign away. But I, personally, don’t see the signature as at all central to the process of science. Maybe it’s because exist outside academia, but when I need to learn about the cutting edge on some topic I find the relevant papers and read them, not particularly noticing who wrote what.
@Daniel nice arguments. You probably have seen this, but for the readers of the blog. Workshop on organizing conferences (http://www.usenix.org/events/wowcs08/index.html), good collection of experiences in organizing.
@Pradeep
A workshop about organizing workshop?
What next, we are going to have papers about writing papers?
;-0
“A workshop about organizing workshop?”
I actually wish there were more workshops like these to bring more transparency. This is especially useful for new grad students and some research communities that don’t have a good ecosystem.
“What next, we are going to have papers about writing papers?”
🙂 There are plenty of papers like those already.
Daniel,
“Let us be candid here. When reviewing research papers, there is no such thing as objectivity. ”
If objectivity isn’t maintained I don’t see any type of review process to help. I would rather review work in the context of the conference rather than the context of author’s previous work. As I mentioned in your earlier blog post, the proposed review system is heavily loaded towards people working in an area for some time and hence, may act as an extra barrier to new entrants.
What next, we are going to have papers about writing papers?
Actually, not a bad idea. If I had to write such a paper, I would start it with: Never, ever, ever use Microsoft Word!
I support this idea of open reviews. It certainly would not stop people being critical, but it would make them polite and constructive. If a reviewer claims some bullshit she/he must be responsible for it.
@Pradeep I agree it is a good idea. I was joking.
Since 2009, the Agile conference has been using a completely open, collaborative process for selecting most of its content. I am not sure that they have applied this process to ALL tracks (I think the more academic track still follows a traditional process).
http://agilepainrelief.com/notesfromatooluser/2011/04/how-we-reviewedagile-2011-coaching-stage.html
I participated in the 2009 edition (both as a reviewer and a submitter), and I have to say that this process did have the advantage of offering early feedback to authors. Not sure that it’s necessarily more fair, or that it leads to the best papers being selected in the end. It’s too complex a question to be answerable through a single experiment like that.