Publish or perish? Let them perish!

There has been much ink spilled on the evils of public or perish, that is, the way professors and would-be professors are mostly gauged by what they wrote and especially, how much they wrote.

Most recently, David Lorge Parnas, one of the most prolific authors in Computer Science (I found 242 papers in his name) has published an article on this topic: stop the numbers game (Communications of the ACM, Volume 50, Number 11 (2007), Pages 19-21). It starts like this:

As a senior researcher, I am saddened to see funding agencies, department heads, deans, and promotion committees encouraging younger researchers to do shallow research.

Here are the evils of the numbers game according to Parnas:

  • It encourages superficial research.
  • It encourages overly large groups.
  • It encourages repetition.
  • It encourages small, insignificant studies.
  • It rewards publication of half-baked ideas.

He concludes as follows:

Sadly, the present evaluation system is self-perpetuating. Those who are highly rated by the system are frequently asked to rate each other and others; they are unlikely to want to change a system that gave them their status.

What Parnas says is not new. See my posts Productivity measures are counterproductive?, Are we destroying research by evaluating it? and On the upcoming collapse of peer review.

What Parnas misses is that the publication process itself is changing. While Parnas is a prolific author, he does not publish in open archives and he does not appear to have a blog. He has predicted the failure of Wikipedia as an encyclopedia in 2005 because it lacks a classical peer review process. I think he is dead wrong: our current classical peer review process is not the only one that can work, and it is not the optimal system.

But whether you agree with him or not on the evils of the counting game, I do not think you can easily reject this last recommendation:

When serving on recruiting, promotion, or grant-award committees, read the candidate’s papers and evaluate the contents carefully. Insist that others do the same.

I actually just reviewed a couple of grant applications and in both instances, I drilled down to the papers the researcher wrote and reviewed them. Sometimes you have pleasant surprises (results are as strong or stronger as the researcher claimed), but you also get bad surprises (an article touted as the cornerstone of one’s research is a thin 1-pager).

Recruiting is a bit of a tougher issue. If the candidate has single-author papers, then you can probably do a decent job if you know the field, but most candidates will only have multiple-author papers. Reading the papers may not be a good predictor for the candidate’s ability.

(Thanks to Sébastien Paquet for a thoughtful discussion.)

2 thoughts on “Publish or perish? Let them perish!”

  1. It encourages overly large groups

    On the other hand, there are many situations in which multi-author papers are counted less than single-author ones. This has been cited as an impediment for interdisciplinary collaboration.

    Perhaps this evaluation of research is like Consumer Reports evaluating chocolate (which they’ve done): it’s relatively straightforward to recognize crap, but after that it really depends on the consumer. This is excepting the Nobel, Fields, and Turing level stuff, which it’s likely won’t be recognized until after the publication in question is no longer relevant for evaluation purposes (at least, that’s what I keep telling myself).

  2. Michael, for your reference:

    When Parnas refers to overly large groups, he explains “Academics with large groups who spend little time with each student, but put their name on all of their students’ papers, will rank above those who work intensively with a few students.”

Leave a Reply

Your email address will not be published. Required fields are marked *

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax