The perils of filter-then-publish

Why do I prefer the publish-then-filter system, which dominates social media such as blogs, to the traditional filter-then-publish system used by scientific journals? Because the conventional peer review system (filter-then-publish) has disastrous consequences:

  1. In the conventional peer review system, you seek to please the reviewers who in turn try to please the editor who in turn is trying to guess what the readers want. It should not be a surprise that the papers are optimized for peer review, not for the reader. While you will eventually get your work published, you may have to drastically alter it to make it pass peer review. A common theme is that you will need to make it look more complicated. In a paper I published a few years ago, I had to use R*-trees, not because I needed them, but because other authors had done so. When I privately asked them why they had used R*-trees, the answer was “it was the only way to get our paper in a major conference”. So my work has been made more complicated for the sole purpose of impressing the reviewers: “look, I know about R*-trees too!” Several times, during the course of peer review, I was asked to remove material which was judged to be “textbook material”: didactic material is frowned upon in many circles (hint: it is not fancy enough).  Be warned: if you find an easy way to prove a result, and it ends up looking trivial in retrospect, your work may become unpublishable. You will need to invent complex related problems to pass peer review. It explains why several  important results appear as remarks in long and complicated papers. Either purposefully, or by habit, people will write in a way to make their paper pass peer review even if it makes the work inaccessible. Do you think research papers have to be boring? If so, you have been brainwashed.
  2. The conventional system is legible: you can count and measure a scientist’s production. The incentive is to produce more of what the elite wants. In a publish-then-filter system nobody cares about quantity: only the impact matters. And impact can mean different things to different people. It allows for more diversity in how people produce and consume science. Thus, if you think it would be better if we stopped counting research papers, then you should reject conventional peer review and favor the publish-then-filter system.
  3. The difference between filter-then-publish and publish-then-filter is analogous with the difference between Soviet central planning and a free market. You either let a select few decide, or you let the market decide. You can either trust that the people will be smart enough, or you can delegate the selection to a few trusted experts.
  4. The conventional peer review system pretends to delegate the assessment of scientists to review boards. Instead of reading each other, we trust brands. The net result is that people hire and promote each others without reading the work. Thus, the conventional system kills any incentive to build a coherent and interesting body of work: you are just a machine that produces research papers as commodities. You know how you succeed in science these days? Take a few ideas, then try every small variation of these ideas and make a research paper out of each one of them. Each paper will look good and be written quickly, but your body of work will be highly redundant. Instead of working toward deep contributions, we encourage people to repeat themselves more and more and collect many shallow contributions. We sacrifice scholarship for vanity.

Further reading: Become independent of peer review and Three myths about scientific peer review.

Source: This post was inspired by a comment made by Sylvain Hallé.

Daniel Lemire, "The perils of filter-then-publish," in Daniel Lemire's blog, May 18, 2011.

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

13 thoughts on “The perils of filter-then-publish”

  1. Hey Daniel,

    I agree that science does have its inconveniences. But I don’t think your idea is that good either.

    A problem -and a big one- of blogs and social media is that the quality standards are sub-par. Most of the publishers lack a writing background and an editorial filter. Pieces are ill-written. And -like Zinserr puts it in his book- modern writing is just rubbish. We can accept people blogging about their dogs like that. But can we accept people writing scientific ideas like that.

    If you look at some Chinese Journals, authors get accepted for friendship. The papers have terrible errors. Ideas are often mixed and ill posed and the authors have little idea on the concepts they’re using. Not to mention their English terrible.

    On the other hand, a good reviewer may be your best friend. I have found much help from kind reviewers. Who propose reading references and point out writing errors. My papers are usually enhanced by their ideas. (One would be incredibly arrogant to think no one could give ideas to improve their papers)

    And on the markets,

    The QWERTY keyboard exists because the markets decided it should. Yet, it is far from being the ideal keyboard layout. Few people could argue that windows is the best OS, yet it rules the market. The best product don’t rule the markets. Is the most popular products the ones that does.

  2. I just think Leon missed the point of your post.

    Anyways, While what you say sounds decent and interesting a blog post outlining how the “publish-and-then-filter” approach would be implemented.

    The blog was excellent and made for a very interesting read. It’s refreshing to hear such perspectives.

  3. The other huge problem is that peer-review is a justification for putting academic research behind pay-walls. (Despite the fact that reviewers aren’t paid.)

    Good academic research is now one of the least accessible kinds of information in the world, while paid propaganda and dumb speculations are rife.

    The publishers with their captive market have no incentive to make it more visible. They know the universities have to keep subscribing to the journals.

    The academics are usually decent enough, but as their objective is to please the reviewers and not piss off the publishers, and they have no incentive to gain a mass readership, they don’t systematically make their work visible as a matter of course.

    And, of course, the work isn’t popular enough for anyone to make available on bittorrent.

    The result is that academic writing and the conversation in the academy is kept artificially distinct from popular / internet discourse.

    In an ideal world, good academics, who were identified and credited by the system, would also become influential public intellectuals by virtue of being *linked*.

    As it is, their most important work is almost never visible to the web, never linked from wikipedia or discussions. And so has far less influence than it deserves.

  4. Still lots of friction to adoption of a publish-then-filter model in academia. The arguments I hear most often

    – – – –

    1) Publish-then-filter makes it impossible to design a much-needed metric to allocate research funds.

    This argument holds some merit, but IMHO it’s backwards. The reward system should be designed AFTER the publication system, not the other way around. If we seriously move to a publish-then-filter model (and if there is a real need for a metric) then I’m convinced we are creative enough to find one. Sure, it’s frightening, sure poor researchers who are able to game the current metric risk doing poorly, but I’m convinced we can design one that would reward researchers who truly work for the greater good of our society — what the reward system should have been about in the first place.

    – – – –

    2) Blogs are crappy, so any content following a publish-then-filter model will be crappy.

    Wrong in so many ways:
    – A lot of blogs are not crappy, those I run into given my online social filter tend to be much better (to me, at least) than a random journal paper.
    – Journal paper hunting also requires filtering (your colleage, your advisor, a conference).
    – If good writers start to publish-then-filter quality will increase.

    If you insist on using this argument, please at least point out a structural reason why blogs would be of lower quality. Personally, I can’t think of any.

    – – – –

    I’d be really interested in hearing any other arguments against this model. I think we need to start advocating it seriously in academia, and a list of counter-arguments would be a good place to start.

  5. I think all systems will have some flaws.
    Here are some thoughts that come to mind for me:
    (a) The “publish and filter” system already exists. Anyone can publish their work on their web page, in technical reports, workshops, in arXiv, and other fora, and many do.
    (b) The branding of well-known journals and conferences can be hugely useful. If you are an unknown author and you make a grand break-through, publishing in a well known reviewed venue potentially gives you a huge leg up, where otherwise your article may languish in obscurity or be adopted by others without due credit.
    (c) Social filtering mechanisms introduce biases of their own. If famous scientist XYZ and unknown author xyz each come up with the same idea at the same time, a formal reviewing process can at least strive to put them on an equal footing. In a social filtering model, ‘branding’ exists in any case, only it happens at the level of the individual. Formal reviewing can at least strive to separate fame from content.

    There are of course good aspects to publish-then-filter. But network effects introduce their own (huge) biases — I would argue that evaluation of research under this model depends too much on connectivity rather than content.

  6. Michiel, very interesting input. I beg to disagree though, for a couple of reasons.

    – The current academic system, with regard to impact, is filter-then-publish-then-filter again. That’s the only way I can explain how some Siggraph papers have impact while others don’t. As such, they are still pretty much driven by network effects.

    – The scenario you make up (one famous and one less famous person publishing the same paper) can occur in a F-t-P model and 9 times out of 10 the famous person will win and the other wont get anything. That’s because reviewers are not only the idea, but the ability to express it well in a fashion that is pleasing to reviewers. In a P-t-F model at least your stuff is up there and it has all chances of sparking a discussion.

    – The network effects I see on the web today tend to reward people who contribute original content, people who cite their sources, people who connect with others. To me, the idea that fame is enough to gain you a lasting audience is deeply rooted in the dynamics of pre-web media. In a web community fame is important to make you a hub node, but if you fail to give credit where credit is due, the network’s backlash tend to be pretty harsh.

    – Finally, it’s not enough that you can already do publish-then-filter today. If your community doesn’t adopt it, you have no one to connect with. You can’t be alone in your separate model, especially since it’s largely incompatible with today’s model, as we’ve often discussed.

    Anyway, I believe most of academia is in too deep… What will happen is that parallel communities of researchers build on a P-t-F model will slowly emerge. Then we’ll be able to verify if you’re right or if I am, using the good old experimental scientific method. 😉

  7. Hi Daniel,

    Thank you very much for a candid account of the “over-complication” of papers. It is really tiring to sort through the mass of over-complicated statements intended to elucidate a pretty straightforward point.

    It is really cathartic to know that other people face this frustration too!

  8. @Itman

    In my experience, peer reviewing is a useful thing, which may significantly improve the quality of the paper.

    I entirely agree.

    You cannot create the system that lacks favoritism unless you improve the people themselves!

    People behave differently depending on the system they are in. An over-competitive, top-down system will generate one type of behavior. An open, bottom-up system will generate another type of behavior.

    in the publish-then-filter model, nobody will point out to your errors. Nobody will improve your writings! Nobody will suggest relevant references!

    On my blog there are many people pointing out my mistakes (you!), there are many people pointing out my poor writing (I get nasty emails at almost every typo) and people suggest a lot of references to me.

    (…) by arguing that they were inappropriate (…)

    The traditional peer review process leaves very little room for the author to argue and authors have a strong incentive to give to the reviewers exactly what they expect. This is even more true in conferences where you almost never have a chance to defend your paper. With acceptance rates at 10% or lower, every bit matters.

    (1) If the paper is rejected, you will never get a chance to argue, period.

    (2) You can write back to the reviewers, to explain that you disagree, but it is a lot of effort and there is a good chance that the reviewers will dismiss your arguments. It is a lot easier to go along.

  9. Phillipe,
    commmon, the web fame is deeply flamed. You cannot create the system that lacks favoritism unless you improve the people themselves!

    All models have drawbacks: in the publish-then-filter model, nobody will point out to your errors. Nobody will improve your writings! Nobody will suggest relevant references!

  10. Daniel,
    First of all: yohohoho, they want everybody to use R-trees or similar methods. I could have gotten around this, by arguing that they were inappropriate in my case (which they probably were).

    As to the system themselves, I am pretty sure that
    1) Pure publish-then-filter system wont work;
    2) Scientists should have blogs now, where they promote their work. In particular, explain it to the public;
    3) Publish-then-filter and filter-then-publish should co-exist. The balance, however, matters. It will probably shift slightly towards to publish-than-filter.

    In my experience, peer reviewing is a useful thing, which may significantly improve the quality of the paper. My experience also says that peers would suggest to simplify the paper rather than to make it more complex.

    I agree that the system may foster incrementalism. Yet, I also believe that truly big ideas will find their way.

  11. O yes, conferences are bad. In regard to pointing out to the errors: yes, some people will highlight your mistakes, but not necessarily experts. To fix errors, you have to attract a lot of experts. Some authors will succeed at this, but most – will not. I am afraid that experienced and well-connected authors will get most experts.

Leave a Reply

Your email address will not be published.

You may subscribe to this blog by email.