Trading latency for quality in research

I am not opposed to the Publish or Perish mantra. I am an academic writer. I am what I publish. We all think of researchers as people wearing laboratory coats, working on exotic devices. And my own laboratory includes a one-million-dollar computer cluster with a SAN server as large as a fridge. I also generate much software. But you know what? The writing is what matters.

And publishing is easy. Write and submit many papers  conforming to the expectations of the editors. Eventually, some of your work will be accepted. And there are thousands of journals, conferences and workshops. Just write a lot.

Yet, don’t publish everything you write—even when what you wrote looks like a research paper. Hold on to it.  Because, publishing everything that looks like a research paper leads to what Feynman famously described as Cargo Cult Science. Indeed, there is a real danger that we become so good at faking science that we are no longer doing science at all! We become dishonest.

In our haste to be published…

  • we cut corners in our experiments, when we validate our ideas at all;
  • we pretend that our work is applicable in the real world, when it isn’t;
  • we don’t take the time to reproduce and reflect on known results;
  • we give the positive aspects of our research while omitting to mention the negatives;
  • we complexify the issues so that our research looks fancier;
  • we get lost in abstract nonsense.

If you want your work to really matter, you should be honest. You should not fool yourself and others. So what do we do? Maybe we should publish carefully. While barely reducing our output rate as academic writers, we can introduce extra steps to keep us more honest. What do we need?

  • Diverse point of views: it is easy to fool a small group of like-minded experts, but comparatively more difficult to fool the readers of my blog.
  • Time to reflect: if you read what you wrote months ago, and you don’t feel the urgency to communicate it more broadly, maybe it wasn’t all that good to begin with?

The problem is that once a paper is published in a journal or a conference, we tend to move on. Anyhow, we cannot easily revise our published work. Are there other models? Economists regularly publish working papers—commonly known in Computer Science as technical reports. But the difference between computer scientists and economists is that economists revise their working papers. And only when their work has stood the test of time, that is, has been available freely for months or years, do they submit it to conventional peer review.

This year, I will try the following experiment. Both on this blog and on my publication page, I will “publish” working papers and specifically ask readers to be critical of my work. Only after a couple of months have passed (or more) will I submit my work to a journal or conference.

This will introduce some latency in my publication output. Can I trade latency for quality? I plan to report back in a year on this (very public) experiment.

Further reading: Time for computer science to grow up by Lance Fortnow.

10 thoughts on “Trading latency for quality in research”

  1. Great initiative, I’m curious to see how it works out!

    W.r.t. to plagiarism: I don’t think direct plagiarism will be a problem, as it is will be obvious that the ideas were published here first. There’s no problem with arXiv pre-prints either. Besides, the readers of this blog are your witnesses :)

  2. Hi Daniel, it is a good idea,
    I think that you could also crosspost to
    PS: I don’t think palgiarism would be a problem. No decent scientist would publish somebody else’s paper. Besides, if it is an experimental one, you need software to reproduce results. That is you can be easily caught and banned from the scientific community.

  3. One thing to consider is what happens if a paper is improved substantially by comments on the blog. This could also result in 20-author papers. Not necessarily a bad thing for science, but something to think about.

  4. Johnatan,
    Though I would share some of your concern, but … reviewers do improve the quality of the paper and often considerably, but that does not make them co-authors. The primary goal of such an exposure is to find errors.

  5. @Jonathan

    That’s a good point.

    But I think that it is not a problem if you are the primary author of the paper in question and you are willing to share the credit.

    But, yeah, I expect to have to think this through some more as I do it.

  6. @Itman What about indecent scientists?

    Has there been attempts at authoring a paper using something like a wiki, or is that maybe going too far?

    Regardless, I’m all for this experiment.

  7. @Geoff Several papers have been written using wikis and blogs. See for example:

    I’m not sure it has been wildly successful yet, but we are still at the early stages.

    And I would certainly participate in such a project if I found one that was a good match for my research interests. And I hope I will one day.

  8. The problem is that sometimes people are “measured” for the quantity of papers they publish. Specially here in Brazil, where the number of papers you write can change the salary you get.

    That is why we sometimes have lots of bad quality works here …

  9. The Polymath project relied mainly on blogs, much like what Daniel proposes; for details, see the October 2009 Nature article by Tim Gowers and Michael Nielsen, the two main players in the project. In that sense, it was not exactly collaborative writing: someone had to put together all the pieces and actually write the paper.Authorship was one of the issues in the project; see Is massively collaborative mathematics possible on Tim Gower’s blog. The first paper produced through this project, which is still at the preprint stage, uses a pseudonym, D.H.J. Polymath, without giving the name of any of the 27 contributors. Even if it’s technically possible to evaluate individual contributions (espectially if a wiki is used instead of blogs), one can easily understand that this collaboration mode could be deemed inappropriate by many (especially junior researchers) in the “publish or perish” culture.

    A different so-called “next-generation” wiki (see, which “links every word to its corresponding author” has been proposed to address this issue. It seems to have been used only in one project, called wikigenes, which doesn’t aim at producing scientific papers. Thus, to my knowlodge, we have still to see an example of an entirely wiki-based collaborative production of a scientific paper, starting from scratch.

Leave a Reply

Your email address will not be published. Required fields are marked *