I like stories where prestigious professors screw up spectacularly. It reminds us that everybody gets it wrong some of the time. The Reinhart-Rogoff story is one such case.
Two very reputed professors at Harvard University, Carmen Reinhart and Kenneth Rogoff, published online a working paper in 2010 called Growth in a time of debt. The paper has since been cited in the scientific literature about 500 times, or about once every two days. They later wrote a best-seller book on the same topic. They built a web site to promote their ideas where they shared some of their data.
Their research paper is remarkably easy to understand: countries with large debt tend to have lesser economic growth. This claim is entirely credible. On the one hand, countries with poor economies may tend to grow large debts because they have weak revenues. On the other hand, large debts may lead to tight fiscal policies (e.g., high taxes) or lax monetary policies (e.g., printing money) that may reduce economic growth.
The part that is less credible is that Reinhart and Rogoff were able to find a magical threshold (debt-to-GDP ratio of 0.9) that triggered low growth. In effect, they implicitly claimed that any debt-to-GDP ratio in excess of 0.9 would reduce your economic growth. This gave politicians a scientific justification for austerity measures (e.g., raising taxes and cutting back on government services).
The story then becomes interesting. Thomas Herndon, a graduate student, grabbed data from the Reinhart-Rogoff web site and tried to reproduce their results. He couldn’t. He then asked the authors for help. Reinhart and Rogoff shared the Excel spreadsheet they used with Herndon. He then promptly found basic flaws in their data processing. For example, the two professors ran sums over the wrong cells. It seems that they made several odd choices when processing the data. His paper is freely available online.
Things took a turn for the worse when, faced with this opposition, Reinhart-Rogoff balked and asserted that these errors did not impact their work. Thankfully, they did not deny their mistakes.
This whole incident shows the importance of sharing data and software. Reinhart and Rogoff were almost exemplary regarding data in this case: they widely shared their data. Their mistake was in not widely distributing their software (in the form a spreadsheet) earlier. Had the spreadsheet been available from the start, they would be in a much better position.
Of course, another version of the story could be that, had Reinhart and Rogoff not shared their data and software, they would have plausible deniability and their work could still be credible. But this only means that you, as a reader, should put more trust in work where the data and software are available.
In any case, by keeping the software private, the best that Reinhart and Rogoff could have hoped for was to delay the inevitable: once your work is the basis for public policy, it is bound to come under intense scrutiny and significant mistakes will be discovered. By sharing your software, you establish your good faith.
There are other minor points that I find interesting in this story:
- All this work is posted online. To my knowledge, no journal has been directly involved.
- Usually, negative results are unpublishable: journals are not interested. It is unclear that the paper by Herndon et al. can be published in a conventional journal even though it is obviously important work.
- It seems that Reinhart and Rogoff are credited for much of the suffering due to the austerity measures in Europe. This seems entirely ridiculous to me. I think that Reinhart and Rogoff acted in good faith.
Further reading: Influential Reinhart-Rogoff economics paper suffers spreadsheet error (via Scott Guthery) and My take on Reinhart and Rogoff by David Henderson.
This is a classic case of confirmation bias. The start from a position that they want to prove, and find the data accomodating.
You even demonstrate the bias in your description of the work, with statements like “entirely credible”. How do we know? Now that the specific methodology has been proven flawed, do we really know that larger debt equals poorer growth? Is there any statistical correlation at all? Is there a correlation between limits? Perhaps debt to GDP is a factor, but it could be dwarfed by other more significant factors or local norms.
In Canada’s case (for example), we could carry a high debt ratio but do well if commodity (oil, iron, aluminium, potash, wheat, lumber) prices are high. Whereas we could have poor growth if we have no debt but commodity prices are very low.
The bottom line here is that while I agree in a general way that servicing debt has a cost, each circumstance has to be weighed in its context. In some cases debt may be important – even crippling. In others – it may be a useful tool to attain other factors which will create positive growth – which may in turn wipe out debt. This is something the Clinton administration was ceanly aware of. Look at economic performance and debt ratios in those terms. Clinton used growth to slay debt, not the other way round.
“I think that Reinhart and Rogoff acted in good faith” is an interesting statement, and quite possibly true.
But how did the (certainly correct) analysis that “large debts may lead to tight fiscal policies that may reduce economic growth” lead to the (I think clearly absurd) conclusion that artificially pushing the number below the magic threshold (by enacting exactly those harmful policies) would solve the problem?
@Jens
There is nothing in their paper to support the thesis that austerity will boost a floundering economy.
This is maybe what Reinhart and Rogoff believe and advocate but, as far as I can tell, this is not what they wrote in their research.
It’s a great case study in scientific journalism. From what I’ve read, it doesn’t sound like the excel error significantly changes the results. What does make a large difference is R&R’s decision to exclude data for some countries in the years following WWII, and an unusual weighting scheme that doesn’t take time spent in a debt category into account (so 19 years of high UK debt and moderate growth count the same as the 1 year New Zealand experienced high debt and low growth).
So the issues that really skewed the results are questions of data appropriateness/cherry picking, and how to account for serial correlations between years. If the formula had been correct, the actual issues with the data would have been the same, but the public would never have cared. This would have been debated in esoteric journals. Instead, because an unrelated but relatable and unambiguous error was also there, the narrative is that the paper is fundamentally incorrect.
I guess the lesson is, if you’re going to screw up, screw up in a boring academic manner =)
I’m really wondering why everyone is so upset against the original authors: so they did a mistake, their reasoning was flawed by all those errors in their spreadsheet, ok. It remains that the ones to blame are those who used those results as a basis for their policy and claimed their decisions were sound because some scientists said so.
It’s called Argument from authority and it’s the first trick taught in Rhetoric 101. Or whatever it was called by the ancient greeks.
Djamé