I was recently on a review committee for a PhD proposal. The student was brilliant. His proposal sounded deep and engaging. The methodology looked scientific: build a model, program the software, gather data, compute the metrics. Yet the student’s hypothesis could never be realistically be proven false. The project was not conducive to telling us anything about the world. At best, we could learn something about an abstract construction. This is, I fear, all too common in computer science. To make it worse, I believe that most computer scientists are unaware of this methodological failing.

In medicine, medical doctors read scientific papers or, at least, executive summaries of said papers. The papers contribute useful knowledge. However, even the best software practitioners can go years without reading any research, assuming that they ever read any. I believe that it is because they rightly feel that the papers will not teach them much about the real world.

You would probably be upset if you learned that your medical doctor is unaware of the latest clinical research concerning your condition. However, how concerned would a manager at Facebook be if he learned that a software engineer is not up-to-date with computer science research?

The problem comes from the fact that computer scientists rarely work with models or, rather, that they are confused about what a scientific model is.

A model in science is an algorithm that enables you to make falsifiable predictions about the real world. In computer science, it might go as follows:

Software X will be faster than software Y on data Z.

A model can be falsified. In my computer-science example, you should be able to run X and Y and you check which is faster. Because they make actual predictions about the world, models are tremendously useful.

Not all scientific models are cleanly falsifiable. For example, natural evolution is a scientific model. The belief that unit testing makes software more reliable is part of another. However, scientific models are always sustained by real-world observations as opposed to our own mental constructions.

But that is not what computer science offers you typically. Instead, computer scientists tend to make statement that avoid scientific falsifiability:

According to some cost model that may or may not be indicative of actual running speed, algorithm X is better than algorithm Y on data Z.

Oldberg proposed to reserve the word model for the genuine scientific models, and to use the French word modèle for the other kind.

You can compare a modèle with reality (what Oldberg calls evaluating), but you can never prove it wrong. A modèle is true as long as it is logically consistent, irrespective of reality.

Computer scientists love their modèles!

The problem is made worse by the fact that researchers working on modèles more easily get the upper hand. They are never wrong. They can endlessly refine their modèles and re-evaluate them. As long as there is no actual problem to be solved, the modèles will tend to displace the models. Cargo cult science wins.

Of course, the reverse phenomenon may exist within industry. People working with modèles are at a disadvantage. They can’t make useful predictions. They can only explain, in retrospect, what is observed. All their sophistication fails to help them when real-world results are what matters.

Though it does not get much press, one of the great social and technological innovations of the last 30 years has been open source software. To about 90% of the population, this remains a mystery. Are the people producing open source code communists?

The truth, of course, is that open source software is just another example of the invisible college. Whenever some occupation requires advanced knowledge, its practitioners tend to network and freely exchange. This is often done under the radar.

This is not to say that people won’t withhold information or be fiercely competitive. But rather that, on the whole, people will tend to share freely with their guild. It is not altruism: this collaboration is key to long-term success as individuals. In science, we have formalized this process: to exist as a scientist is to share your results regularly in the form of papers.

Well before there was anything known as open source, programmers would share with other programmers. At some point, the likes of Bill Gates asserted their copyright over code which prevented the free exchange of code. However, not the free exchange of ideas: if you can get a Microsoft engineer in a bar, and you are an established programmer, he will typically share internal ideas with you. In fact, as long as it cannot be traced by to them by their employer, most programmers will freely share with other programmers, even if they are from competing companies.

In this sense, open source code is just the visible part of the invisible college of programmers. Because open source is an expression of the invisible college, we have that open source code is meant to benefit or impress other programmers. For example, the most popular projects on GitHub right now is Bootstrap which has as its motto: “by nerds, for nerds”.

And that is the key to be effective at open source. You have to be a programmer and you have to address yourself to other programmers.

Starting an open source browser or word processor meant for end-users is not the best choice. Failure probability is high. You need many ressources beyond coding that are expensive. This means that you need a business strategy. Got an MBA? However, you can build a faster JavaScript engine, a better XSLT engine, and so on. These will be highly valued by your (programmer) peers. In effect, do not confuse business activities based on open source, with what open source is. Genuine open source is about the invisible college. It is about helping other programmers.

Recently, many successful businesses are just based on a collection of open source tools put together quickly. And it is exactly why I started this post by saying that open source is a great innovation. In effect, no other community of practitioners has established such a rich invisible college. This makes programmers some of the most effective employees from a cost-value ratio.

Hence, despite a sluggish recovery, software engineers enjoy a 2% unemployment rate and some of the best salaries an engineer can get. And I suspect that open source code and the free exchange of ideas among programmers has a lot to do with these good results.

Of course, programmers who are not participating in this invisible college risk being left in the dust. If you work in isolation, never sharing… you will have a harder time leveraging the strength of the community.

Further reading: Alexia Gaudeul, Open Source Licensing in Mixed Markets, or Why Open Source Software Does Not Succeed, 2008. Eric S. Raymond, The Social Context of Open-Source Software (chapter from the Cathedral and the Bazaar).

One of the better known problems in Computer Science is the P versus NP problem. It is often related to the following question: do all problems for which we can check the correctness of a solution quickly can also be solved quickly.

Most computer scientists believe that P is different from NP. In colloquial terms, this means that we believe that there are problems whose solution can be checked quickly but such that it is very difficult to find the solution.

One relevant problem is the Hamiltonian path problem: given a set of roads and cities, is there a path that visits every city exactly once? It is easy to check any given solution, but it is thought to be quite hard (in general) to find such a solution.

A typical application for our belief that P is different from NP is to show that a given problem is difficult. Proving that something is difficult does not sound very important and practical at first, but it can save your job!

There is a million dollar prize to whoever can resolve the P is equal to NP question. This fact alone has attracted much attention to the problem. Whoever solves it would get not only instant fame, but quite a bit of wealth as well.

Lance Fortnow has spent a few years on a book that recounts all the fascinating adventures related to this problem. He explains the problem in depth in an accessible book entitled The Golden Ticket. The book is thoroughly researched and reviewed. Anyone from a smart high school student to a computer scientist is sure to get a lot of this book. The presentation is beautiful. There are few formulas but lots of facts. The book is of course kept simple, so hard-core computational complexity expert might be disappointed, but I think many of them will enjoy the stories Fortnow offer.

Disclosure: I received a free review copy of the book. I am otherwise unrelated to Fortnow.

Following the Reinhart-Rogoff case, where famous scientists go formulas wrong in the Excel spreadsheet that supported their research, a lot of people commented on the adequacy of a spreadsheet tool for important work.

Excel does have one tremendous benefit: it is accessible. Most people using spreadsheets don’t even realize that they are programming. In the Reinhart-Rogoff case, this accessibility was a great virtue: it allowed a regular PhD student to verify the computations.

However, there are several critical problems with a tool like Excel that need to be widely known:

• Spreadsheets do not support testing. For anything that matters, you should validate and test your code automatically and systematically.
• Spreadsheets make code reviews impractical. To inspect the code, you need to look at every cell. In practice, this means that you cannot reasonably ask someone to read over your formulas to make sure that there is no mistake.
• Spreadsheet encourage redundancies. Spreadsheets encourage copy-and-paste. Though copying and pasting is sometimes the right tool, it also creates redundancies. These redundancies make it very difficult to update a spreadsheet: are you absolutely sure that you have changed the formula throughout?

Unfortunately, spreadsheet programming is far more common in research than we would like to admit. I keep reviewing research manuscripts where the figures were obviously made with Excel. It is also very widespread in business: decisions worth millions (if not billions) of dollars are taken on the basis of a spreadsheet all the time.

Professionals should avoid spreadsheets for activities where mistakes matter. Reinhart and Rogoff should have used a bona fide programming language with proper testing, code review and documentation.

Further reading: Lotus Improv was an early attempt to build a spreadsheet tool that did not have some of these problems. It was a market failure. (Credit: Preston L. Bannister)

I like stories where prestigious professors screw up spectacularly. It reminds us that everybody gets it wrong some of the time. The Reinhart-Rogoff story is one such case.

Two very reputed professors at Harvard University, Carmen Reinhart and Kenneth Rogoff, published online a working paper in 2010 called Growth in a time of debt. The paper has since been cited in the scientific literature about 500 times, or about once every two days. They later wrote a best-seller book on the same topic. They built a web site to promote their ideas where they shared some of their data.

Their research paper is remarkably easy to understand: countries with large debt tend to have lesser economic growth. This claim is entirely credible. On the one hand, countries with poor economies may tend to grow large debts because they have weak revenues. On the other hand, large debts may lead to tight fiscal policies (e.g., high taxes) or lax monetary policies (e.g., printing money) that may reduce economic growth.

The part that is less credible is that Reinhart and Rogoff were able to find a magical threshold (debt-to-GDP ratio of 0.9) that triggered low growth. In effect, they implicitly claimed that any debt-to-GDP ratio in excess of 0.9 would reduce your economic growth. This gave politicians a scientific justification for austerity measures (e.g., raising taxes and cutting back on government services).

The story then becomes interesting. Thomas Herndon, a graduate student, grabbed data from the Reinhart-Rogoff web site and tried to reproduce their results. He couldn’t. He then asked the authors for help. Reinhart and Rogoff shared the Excel spreadsheet they used with Herndon. He then promptly found basic flaws in their data processing. For example, the two professors ran sums over the wrong cells. It seems that they made several odd choices when processing the data. His paper is freely available online.

Things took a turn for the worse when, faced with this opposition, Reinhart-Rogoff balked and asserted that these errors did not impact their work. Thankfully, they did not deny their mistakes.

This whole incident shows the importance of sharing data and software. Reinhart and Rogoff were almost exemplary regarding data in this case: they widely shared their data. Their mistake was in not widely distributing their software (in the form a spreadsheet) earlier. Had the spreadsheet been available from the start, they would be in a much better position.

Of course, another version of the story could be that, had Reinhart and Rogoff not shared their data and software, they would have plausible deniability and their work could still be credible. But this only means that you, as a reader, should put more trust in work where the data and software are available.

In any case, by keeping the software private, the best that Reinhart and Rogoff could have hoped for was to delay the inevitable: once your work is the basis for public policy, it is bound to come under intense scrutiny and significant mistakes will be discovered. By sharing your software, you establish your good faith.

There are other minor points that I find interesting in this story:

• All this work is posted online. To my knowledge, no journal has been directly involved.
• Usually, negative results are unpublishable: journals are not interested. It is unclear that the paper by Herndon et al. can be published in a conventional journal even though it is obviously important work.
• It seems that Reinhart and Rogoff are credited for much of the suffering due to the austerity measures in Europe. This seems entirely ridiculous to me. I think that Reinhart and Rogoff acted in good faith.

Further reading: Influential Reinhart-Rogoff economics paper suffers spreadsheet error (via Scott Guthery) and My take on Reinhart and Rogoff by David Henderson.