Ten years ago, a team lead by Irina Conboy at the University of California at Berkeley showed something remarkable in a Nature paper: if you take old cells and put them in a young environment, you effectively rejuvenate them. This is remarkable work that was cited hundreds of times.
Their work shows that vampire stories have a grain of truth in them. It seems that old people could be made young again by using the blood of the young. But unlike vampire stories, this is serious science. So maybe there are “aging factors” in the blood of old people, or “youth factors” in the blood of young people. Normalizing these factors to a youthful level, if possible, could keep older people from getting sick. Even if that is not possible in the end, the science is certainly fascinating.
So whatever happened to this work? It was cited and it lead to further academic research… There were a few press releases over the years…
But, on the whole, not much happened. Why?
One explanation could be that the findings were bogus. Yet they appear to be remarkably robust.
So why did we not see much progress in the last ten years? Conboy et al. have produced their own answer regarding this lack of practical progress:
If all this has been known for 10 years, why is there still no therapeutics?
One reason is that instead of reporting broad rejuvenation of aging in three germ layer derivatives, muscle, liver, and brain by the systemic milieu, the impact of the study published in 2005 became narrower. The review and editorial process forced the removal of the neurogenesis data from the original manuscript. Originally, some neurogenesis data were included in the manuscript but, while the findings were solid, it would require months to years to address the reviewer’s comments, and the brain data were removed from the 2005 paper as an editorial compromise. (…)
Another reason for the slow pace in developing therapies to broadly combat age-related tissue degenerative pathologies is that defined strategies (…) have been very difficult to publish in high impact journals; (…)
If you have not been subject to peer review, it might be hard to understand how peer comments can slow down researchers so much… and even discourage entire lines of research. To better understand the process… imagine that you have to convince four strangers of some result… and the burden is entirely on you to convince them… and if only just one of them refuses to accept your argument, for whatever reason, he may easily convince an editor to reject your work… The adversarial referee does not even have to admit he does not believe your result, he can simply say benign things like “they need to run larger or more complicated experiments”. In one project I did, one referee asked us to redo all the experiments in a more realistic setting. So we did. Then he complained that they were not extensive enough. We extended them. By that time I had invested months of research on purely mundane tasks like setting up servers and writing data management software… then the referee asked for a 100x extension of the data sizes… which would have implied a complete overhaul of all our work. I wrote a fifteen-page rebuttal arguing that no other work had been subjected to such levels of scrutiny in the recent past, and the editor ended up agreeing with us.
Your best strategy in such case might be to simply “give up” and focus on producing “uncontroversial” results. So there are research projects that neither I nor many other researchers will touch…
I was reminded of what a great computer scientist, Edsger Dijkstra, wrote on this topic:
Not only does the mechanism of peer review fail to protect us from disasters, in a certain way it guarantees mediocrity (…) At the time, it is done, truly original work—which, in the scientific establishment, is as welcome as unwanted baby (…)
Dijkstra was a prototypical blogger: he wrote papers that he shared with his friends. Why can’t Conboy et al. do the same thing and “become independent” of peer review? Because they fear that people would dismiss their work as being “fringe” research with no credibility. They would not be funded. Without funding, they would quickly lose their laboratory, and so forth.
In any case, the Conboy et al. story reminds us that seemingly innocent cultural games, like peer review, can have a deep impact on what gets researched and how much progress we make over time. Ultimately, we have to allocate finite resources, if only the time of our trained researchers. How we do it matters very much.
Thankfully, since Conboy et al. published their 2005, the world of academic publishing has changed. Of course, the underlying culture can only change so much, people are still tailoring their work so that it will get accepted in prestigious venues… even if it makes said work much less important and interesting… But I also think that the culture is being transformed. Initiatives like the Public Library of Science (PLoS) launched in 2003 have shown the world that you could produce high impact serious work without going through an elitist venue.
I think that, ultimately, it is the spirit of open source that is gaining ground. That’s where the true meaning of science thrived: it does not matter who you are, what matters is whether you are proposing works. Good science is good science no matter what the publishing venue is… And there is more to science than publishing papers… Increasingly, researchers share their data and software… instead of trying to improve your impact through prestige, you can improve your impact by making life easier for people who want to use your work.
The evolution of how we research may end up accelerating research itself…
Great article.
Is it a matter of having the right incentives for researchers? Instead of “publish or perish”, “publish, produce, share and/or teach”?
An example of a researcher having a wide impact is Conrad Sanderson through his C++ Armadillo library. I suspect producing this software while also managing to publish extensively in peer reviewed journals requires a combination of dedication and ability that goes beyond the capability of most people.
If producing counted as much as writing papers, then this model would be more accessible to the average researcher who may not otherwise be capable of publishing enough papers.
Will the academic world naturally move away from reviewed publication as the primary measure of performance as the culture changes, or is reform needed?
I don’t know Conrad Sanderson but I would not presume that working on software necessarily slows down his academic work. The thing is… productivity is not a zero-sum game.
Will the academic world naturally move away from reviewed publication as the primary measure of performance as the culture changes, or is reform needed?
Ultimately, it is a resource allocation problem. There are only so many research grants, jobs and so forth. People only have so much time. So people are pushed to certain things, and they are discouraged from doing others. No system is even perfect, but we have to somehow decide that some people should be funded before other people. There is no way not to make this choice. I don’t think that using publications as a measure of performance is necessarily bad. I don’t think that people refuse to share data and software so that they can publish more (though this might be their excuse), it is all deeply rooted in the culture.
The problem we are facing here is that Conboy et al. felt that it was in their own best self-interest to delay by about ten years work that looks to me to be highly important for all of us. They could have published their work elsewhere, but they chose not to do so. Instead, they express frustration at the fact that prestigious venues would not publish their work. Clearly, they held on to their work and tried to get it published in prestigious venues but they kept failing. I understand their frustration well. At the same time, they could have taken their work to PLoS Biology or some other alternative journal. Or heck! They could have posted the work as a technical report on their home page. They make it seem like their work would then not have counted. This might have been reasonable from their point of view… but… I think it is less and less reasonable.
It is getting harder and harder to stand by the belief that unless works appear in a highly prestigious venue, it does not count. That’s no accident. Smart and talented people have invested much of their time in the last ten years to make it happen.
And this matters very much because it means that it is less likely today that people like Conboy will hold on to important work in the hope of scoring a prestigious paper. I am not saying it won’t happen… it still does all the time… but we have made progress. And it is progress that matters very much.
A well done peer review is very useful. However, this is becoming a rare commodity as reviewer received (almost) no reward for doing an honest peer review.
The other important point is that MOST paper in biomedical science are wrong, because of the poor research practices. It would be even worst without peer review to filter the field. The corollary is that you can always manage to publish a paper if you are ready to move to another less prestigious journal.
The quest for prestige and the quest for truth are different things.
In the open-source world, anyone can claim to have software that does miraculous things. And indeed, people make extravagant claims all the time. Yet our civilization has come to rely critically on open-source software. We have no need for the equivalent of Nature. Do we have peer review? Oh yes! We do. The reason Linux is being used for mission-critical purposes is that there is an extensive review process of each and every contribution… a process that is far more thorough than the process of submitting a paper to Nature.
Science should be vetted in a similar manner. Before I can trust your work, I need to see extensive peer review of it… But I should not care if some powerful editor vet it or not.
I agree. But, unless you have a strong mechanism that enforce good vs bad work, you are likely to end up with a selection process that enforce popular vs impopular. This is how the pseudoscience work by the way.
Open source softwares are ongoing a permanent testing process. This is not the case of most scientific papers, because they are very hard to replicate.
A large fraction of the scientific controversy a simple product of poor research practice. And, outside the scientific community, users of the scientific knowledge have very little opportunity to test it.
Open source softwares are ongoing a permanent testing process. This is not the case of most scientific papers, because they are very hard to replicate.
Yes. And that is exactly the problem we must solve.
The old incentives in science were almost backward. You wanted your competitors to have a hard time reproducing your results, precisely so that you alone could pursue them (and to make sure they could not contradict you). Given that nobody can reproduce your work how do they trust you? Because you have written your work in such a way that people in high places will like it and recommend that it be accepted in a prestigious venue. It is not quite as bad as all that, but some of what I wrote is true.
It is really a pre-scientific, authority-based process.
Think about it… conceptually, there is no difference between accepting a truth because it comes from the Pope and accepting a truth because it comes from Nature.
Scientists should shy away from reasoning on the basis of authority.
The whole idea that without prestigious venues we would not know what to believe is just sad.
From my own experience, peer review is significantly reduce the number of rubbish results published. Overall, I reject roughly 50% of the paper I review because they are pure garbage. Hence peer review is certainly useful.
As a researcher, I seek references in Google Scholar, and whether it appeared in a pretigious venue or just arXiv is not a factor in my decision to check out a reference.
What traditional peer review is good at is filtering work that is obviously wrong. But that’s hardly our problem, is it?
Traditional peer review will not and cannot protect you from people who make up data, either intentionally or not. The only protection we have against that is people going “hmmm… funny, this does not work here… why is that?” And then, with a twist of irony, traditional peer review makes it impossible to publish a paper saying that “Smith et al. got it wrong, we can’t reproduce their work” (you hardly ever see such papers as they are tremendously difficult to get beyond peer review).
Personally, I have seen many papers saying that article of X is wrong. I written that a few time myself.
Peer review is not the cause of the lack of reproducibility. Research practice is.
Peer review essentially stopped me dead in the waters. I have a Ph.D. in theoretical physics but have been blocked by peer review on every attempt. I went into physics because physicists were the only teachers who would make a serious attempt to explain their views (my desire was to understand the universe I found myself in). One of the responses I often received was “you will have to find someone more educated than I as I really don’t understand the explanation.”
So I went to graduate school to learn the foundations of their beliefs. I am now approaching 80 years and I have never met a person with an advanced education who had any interest in thinking about their beliefs. They are all interested in “being authorities”. I have come to be convinced that modern science is a religion and not a science.
I would love to encounter someone interested in understanding the universe. I could show them things they have never even considered which provide answers beyond common belief.
Einstein’s theories of “space-time” are as erroneous as belief in a flat earth!
I agree with the spirit of your comment.