Are we destroying research by evaluating it?

This morning, I read a fascinating paper, Evaluations: Hidden Costs, Questionable Benefits, and Superior Alternatives by Bruno S. Frey and Margit Osterloh (October 2006). This paper is concerned with the undesirable effects of the focus on bibliometric indicators (“publish or perish”). In many context, it is very difficult for a researcher to land a job, or to keep the said job, or the necessary funding, unless he publishes regularly in prestigious venues. Intuitively, these measures would insure that research is of higher quality. Is that really so?

Their main point is that such (rigid) evaluations distort incentives for researchers.

The measurement exerts not only pressure to produce predictable but unexciting research outcomes that can be published quickly. More importantly, path-breaking contributions are exactly those at variance with accepted criteria. Indeed innovative research creates novel criteria which before were unknown or disregarded. The referee process, by necessity based on the opinions of average peers finds it difficult to appreciate creative and unorthodox contributions.

They argue that we see a homogenization of research endeavors. All laboratories and departments end up looking the same. Fads are followed religiously.

They argue that this disconnects researchers from the real world:

Research departments give no credit to faculties who write books and magazine articles designed to intermediate between the research community and the general public because they don’t contribute to the citation record. As a consequence, the gap between rigor and relevance of research is deepened and the dialogue between science and practice is undermined.

I often complain about this very fact on this blog. But I especially like this bit:

The tendency to measure research performance by the size of grants received creates an incentive to undertake more expensive, rather than relevant research.

I find the paper really fascinating. They go on to say that researchers who act as reviewers have a incentive to rate poorly competitors or potential competitors. If everyone gets to review in an open way, I guess this incentive is small, but in the current setup where few people get to kill or allocated most grants and paper acceptance, there is a real worry that, without ever realizing it, they may seriously hurt potential competitors only to protect their own interests.

Ah! But the people getting reviewed can play games as well. For example, one can always create new metrics against which one performs well. That’s ok, but then, it can get uglier:

Authors raise their number of publications by dividing their research results to a “least publishable unit”, slicing them up as thin as salami and submitting them to different journals. Authors may also offer to include another scholar among the authors in exchange for being put as co-authors on his or her paper. Time and energy is wasted by trying to influence editors by courting them e.g. by unnecessarily citing them. More serious are manipulations of data and results.

There is the nice concept of academic prostitution (!!!) which relates to the way in which authors are inclined to change their papers to favor the reviewers, by citing them, for example. This is exemplified by this quote:

According to the study by Simkin and Roychowdhury (2003) on average only twenty percent of cited papers were ever read by the citing authors.

They explain that the system is self-sustained because anyone who questions it is then suspected of not meeting the required quality standards. The system locks people in.

What do they propose as an alternative? In short that you should choose the right people, coach them adequately, and then leave them alone. They also correctly point out that the Web makes it less important to cluster good researchers together. I really like this concluding remark:

The characteristic of a selection system is that once a decision has been made the principals put faith in the persons selected. Important positions in society (such as top judges and presidents of Central Banks) are elected either for life or for a very long time period without formal evaluations for good reasons. It is questionable why these reasons should not apply to research.

Update. Peter Turney sent me this pointer to Reviewing the Reviewers by Kenneth Church. In it, Church argues that by selecting fewer and fewer researchers and papers, we are discarding many interesting papers because they are not conventional enough.

Update 2. My own point of view is that we should mimick the Atmospheric Chemistry and Physics journal and move to open peer review (as described in this Nature article.) I am not quite certain how this would work with grant and job reviews, but I think we must move toward more modern systems. I think we overestimate people’s fear toward transparent review systems. After all, if I am ever to be convicted of a crime, I expect to see the juries face to face.

4 thoughts on “Are we destroying research by evaluating it?”

  1. There is no doubt that the current system has flaws, but I don’t think “leave them alone” is the answer. Let’s ignore conferences for a moment and focus on journals. In my experience as a reviewer of journal papers, most papers are neither accepted “as is” nor rejected; instead, they are accepted with major revisions or rejected with encouragement to resubmit. In my opinion, the second version of a paper is always quite a bit better than the first version, and it is usually accepted. I believe that many of my own papers have been very much improved by this reviewing process, so I would not want to see it eliminated. Also, papers that are rejected even after revision tend to appear later, in some less prestigious journal. In summary, I’m sympathetic to these criticisms, but “leave them alone” is not the answer.

  2. This is something that has bothered me about a possible career in research. When I started graduate work, I was pretty sure I wanted a research position. As time has progressed, I’ve found that the ideas I want to explore don’t fit well into the “salami” model of publishing, which has greatly altered my career path.

    Not only that, I find it difficult to bring myself to publish the same thing over and over. Some people I’ve seen get incredible traction out of a single idea and the papers written by them on the subject rarely build on the idea; instead, they rephrase it. I can’t say I like that approach very much. It doesn’t sit well with me.

  3. This is a very serious issue. What is at stake is not only the validity of concepts and theories, but also the social ethics that lay behind any social system based on evaluation and judgement on or about one’s works and actions. But the fundamental aspect of peer reviewing is its ability to reproduce parts or totality of a science, just by assessing ideas as conform to accepted models (Kuhn coined the term “paradigm” at the end of the sixties, which describes this reproductive function of normal science, in the perspective of science’s social organization). I think that if researchers tend to view their productions as salami that can be sliced ad infinitum, or almost as an ever ending story to surf on, it is easy to see that the reason of this behaviour is the fact that the peer review system is probabilistic in its structure: with few reviewers picking up ideas in an “urn” like stock of ideas (the total amount of submitted articles added to the total amount of actually published articles which is a priori lesser than the first sum), it is likely that the ideas that will be picked up will tend to be conditioned by their correspondence to the total sum of available ideas. For new ideas and theories are rare, and difficult to develop but more importantly to grasp and understand, and this is also why so many people write commentaries on other’s ideas, not texts offering fresh ideas (I am not even talking of revolutionary ideas, ones that change the course of a discipline or domain…). So, what is there to be done? First, I think that the reaffirmation of research’s fundamental ethical principles should help (not the morality of science as when it comes to use animals or human subjects for instance); and this can only be done by introducing a compulsory formation in ethics. And reviews that don’t respect these principles should be denounced. No one would then be associated with an organization that doesn’t respect deontology and ethics. And to reproduce only what is already known is certainly not a good principle from which derive some editorial guidelines.

  4. I remember when we were trying to publish material on interval constraints – about which there had been a sum total of 2 publications in both math and C.S. and having all the trouble in the world getting anybody to understand what we were trying to say and why it was interesting.

    Failing the peer-review method we went about on a person-person campaign to explain our ideas and did some “intellectual lobbying”… after which, these ideas started spreading and more stuff got published by good people.

    I think your point Daniel is a very good one. The problem is that if the subject matter isn’t already in the realm of “normal science” (an accepted problem / solution space in the community) then it’s very hard to actually get new stuff out with the existing peer-review process and incentives. Not that it’s always bad, as Peter points out. It’s just not always good from the point of view of creative, new ideas.

Leave a Reply

Your email address will not be published. Required fields are marked *