On the quality of academic software

Software is eating the world. Despite a poor year, Facebook has a market capitalization of $65 billion. This little company with barely 2000 developers is worth as much as a car marker.

Students should take notice. I would expect countless students to come to college demanding top-notch software training. I would expect graduate students to produce gorgeous software programs.

Yet software produced in universities and colleges is awful, and it is not getting better. I have an explanation:

  • Most professors spent little time programming. And because they don’t program, they do not enforce good practices such as code reviews and unit testing. Don’t believe me? Try to look up your favorite professor on GitHub. Does he proudly display the code he produced?
  • Because few professors program, it should not come as a surprise that there are few, if any, publication outlets for academic researchers who want to present their software products. In turn, it means that if you produce high quality software, few of your academic peers will even know about it.
  • Most academic software is written by students who lack the experience and the incentives to produce good software. You would think that after spending four years in college and attending countless classes where they have programming assignments, most computer science and engineering students would be decent programmers. That is not my experience. And part of the reason is that schooling is a process by which you emulate your teachers. I am sure that if you attend four years of schooling with Linus Torvalds, you would become a decent kernel programmer. However, Linus is not, nor is he ever likely to be, a college professor. College professors don’t spend much time programming and neither do their students.

So, academic software is awful because academic folks aren’t that great at programming. But there are also other factors involved:

  • Programming can be as easy as cooking eggs or building a bird house. That is, it is not very difficult to write a PHP script to display the content of a database inside a browser. But producing novel software requires the programmer to act as creator. While your programming classes might teach you how to cook an egg (metaphorically speaking), they fall short of teaching you how to design a new dish for a 4-star restaurant.

    And the bar for novelty is lower than you might expect. Given an algorithm taken out from a research paper, dozens of implementations are possible, most of them inefficient. It takes a lot of work and experience to come up with the right design even if the full pseudo-code is provided to you. And making sure you implemented it properly can be much harder than one expects.

    There is very little appreciation for this fact among theoreticians who often believe that the hard part is designing an algorithm with nice theoretical properties. They fail to recognize that there might be orders of magnitude differences in speed between two algorithms having the same computational complexity.

    In effect, suppose you were provided a summary of Stephen King’s next novel. Could you beat King to it and produce as good a novel? Doubtful. Software is similar. Execution is everything. A clever plot is worthless.

  • Most academic researchers write software for themselves. As Cook put it: “People who have only written software for their own use have no idea how much work goes into writing software for others.” Cooking your own food is a lot easier than being a chef in a restaurant. The difference between the two is at least an order of magnitude, if not two.

    There is very little appreciation for this fact in academia. The default is to write throw-away code: you write the code, you use it and you forget about it. Issues like maintenance and documentation are discussed at length in some classes, but it is rarely put in practice within academia.

So, what about the future? I remain pessimistic regarding academic software. There will always be exceptions, but as a rule, I expect mediocrity to remain. Hence, when I review a research paper or a thesis, I expect the software supporting it to be awful. And I expect whoever hires new programmers to think the same. And there is a practical consequence to my pessimism: perhaps if you hope to get a job as a Facebook engineer, you ought to spend more time on GitHub and less time in the classroom. If you are lucky, you might convince your school to grant you a few credits for your open source work.

Further reading: In a recent blog post, Google’s Matt Welsh states that he wasted millions of dollars as a Harvard professor. How? By producing software prototypes that were later thrown away. Maybe his code could have stood the test of time if only the university had better supported its commercialization. I have my doubts.

You might also want to read In the long run… by Suresh Venkat for a description of the distance between software practice and theoretical computer science.

Daniel Lemire, "On the quality of academic software," in Daniel Lemire's blog, June 18, 2012.

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

22 thoughts on “On the quality of academic software”

  1. It takes years for a theoretical idea to mature into quality code. The first implementations of an algorithm (and the first papers about it) tend to be bad, as you probably had no clear understanding of the algorithm when writing them. Yet if you continue working, you will eventually understand the algorithm well enough, so that you can describe it in simple terms and write a proper implementation of it.

    In bioinformatics, new tools tend to use algorithms from 5 to 10 years ago. This is the time it takes first for the original authors to get the algorithm right, and then for the others to notice the algorithm, understand it, and see what it is good for.

  2. It sounds like you’re mixing up “high quality software” with “software that effectively solves a real problem.” These are different and I doubt that very much of Facebook’s software is gorgeous.

  3. Back when I studied computer science, there really was very little training in terms of programming. People assumed that you already knew how to program. Later on there were classes on software engineering, but these were not mandatory (and pretty theoretical, too). There was also a strong feeling that programming is not “academic” enough. I doubt that that has changed in the mean-time, unfortunately.

  4. Aren’t you overgeneralising?

    In Academy, there are not only professors and students. There are also post-doc researchers, research assistants, collaborators. You could even hire developers for academy-related projects.

  5. Well, the purposes of software development is different: software house build programs to sell it to a third part, thus it should be usable for final users. Researchers/PhD students, as explained, develop only for their purposes: the paper (or more romantically to grow human knowledge!). In my todo list there is always review code, but then other software modules have to be written and the priority of review drops!
    The solution is to hire in academia software engineers that could coordinate research group coders and teach how to efficiently code.
    I’ve read code that is less understandable than hieroglyphs, in my career at university!!


  6. A great, but all too rare counter-example is Maple. It was released commercially at the end of the eighties (and is still popular these days) after I think 10 years of being built by the professors and students in the Symbolic Computation Lab in the University of Waterloo.

    I worked with the code twenty years ago to port it to many different platforms. It was a grand example of abstraction and elegance, I really do wish that they’d publish it someday.


  7. The article is generalizing a bit but there is value in what was written. Some universities do not prepare students for practical, modern day software development. Students exit with their degree with an excellent theoretical background and have little or no hands-on practice to producing excellent software. You cannot exclusively blame the universities. You’ll see a wide variety of curricula available that are highly theoretical, mix of theory and practical, and little theory + lots of hands-on. Students need to investigate before committing to a university.

    It is a little unfair to paint a wide brush over professors saying they don’t code either. Some of the best “coders” I know are professors. Many universities own the content created by professors and thus cannot be published publicly. Similar to working for a closed-source software company. In the same vein, there is a competitive advantage for a stellar professor to keep her curricula available to paying students. I agree, however, if there is anything that could be shared then please do stick it up on Github!! 🙂

    Furthermore, many universities make attempts to connect students with software companies to give them some practical skills via internships, co-ops and work experience. I encourage students to look for higher education institutions that provide those as part of the degree.

    Finally, the point of the article could be used as an incentive to light a flame on existing universities and professors that don’t care. That is the sad part. I’ve worked with amazing institutions and professors that are super talented and nobody sees their awesomeness! But there some who do not care and do not want to progress where necessary. I think those are the types of people the author is referring to.

    To read more on my thoughts about software in education you can read more on my blog: http://blog.dudeblake.com/2009/09/improving-traditional-software.html

  8. You have a point, of course, but there are some very promising counter-examples, where a community has changed the incentives: if cleaning up and documenting that ‘throw-away code’ will land you a high-impact publication, even academics will do an effort. And then the paper review process improves it further.

    This is what the open-source track of JMLR has done for the field of machine learning: http://jmlr.csail.mit.edu/mloss/.

  9. After working 12 years in computer science research, I can only wholeheartedly agree that academic software is bad for the most part. Students have too little experience and time. Academic teams are composed entirely differently than industrial teams. Profs feel they need implementation to establish their ideas and naively believe they are producing software industry can just take over.

  10. There is another problem with poor quality academic software. In fields where efficiency or accuracy of algorithms is compared between different methods, the software that is not released, or that cannot be easily run by other people, leads to non-replicable results and therefore may inhibit scientific progress.

    Ted Pedersen had a great short piece on it in Computational Linguistics

    Empiricism is not a matter of faith
    Computational Linguistics,34(3), Pages 465-470

    He proposes that releasing code as opensource should become a requirement. Sadly, I think this proposal would be unrealistic now because the time and effort that has to go into making code usable by other people is not appreciated in academia. And yes, we could hire software engineers. We do sometime. But there is rarely money for it, because funders value publications above anything else.

  11. Daniel, what’s the incentive for researchers and professors to spend time writing good software instead of studying the new big idea that might make them publish another paper?

    As others have pointed out, there are two main reasons why a professor might be interested in good software: to contribute to an open source project or to build a prototype that can be marketed in the outside world (either for commercial purposes or for disseminating the results of one’s research).

    None of these reasons are valued/supported by universities.

  12. Sounds like academic anything. Academic marketers, advertisers, and entrepreneurs are no different. They don’t test ads, start legit business, our are accountable to a customer.

  13. As John noted quality software and software that someone uses is a difference.

    But yes, at least some unit tests here in there would be nice – especially when new students come into the team.

    But another reason that professors don’t code “openly” is that they need to show that they are the first when publishing a paper. Or they fear that someone could steal their ideas or their code base and will outperform himself.

    Also I can see a minor movement that researchers are on GitHub …

  14. Great points Daniel. Here’s a favorite quote of mine from Donald Knuth on developing the typesetting program TeX:

    “The most important lesson for me, was that software is hard; and it takes a long time. From now on I shall have significantly greater respect for every successful software tool that I encounter. My original plan was to spend one year working on algorithms for typography; but that was the spring of 1977, so you can see that it has taken more than eight years to finish the job.

    Software creation not only takes time, it’s also much more difficult than I thought it would be. Why is this so? I think the main reason is that a longer attention span is needed when working on a large computer program than when doing other intellectual tasks. A great deal of technical information must be kept in one’s head, all at once, in high-speed random access memory somewhere in the brain. I found to my dismay that I could not be writing large computer programs while teaching my regular classes; I simply couldn’t do justice to activities simultaneously, nor could I be happy if the programs were left unwritten. So, I reluctantly took occasional leaves of absence from university teaching. In this sense I believe that program writing is substantially more demanding than book-writing.”

    One note of optimism though. Software matters a lot in our world, and open source and collaboration are having an accelerating impact on culture and innovation in many, many ways. The next big ideas and advances are often possible only by leveraging and contributing to well crafted open source tools and codes. Proper academic credit for software craftsmanship seems inevitable to me, because the academy becomes less relevant as it is sidelined. Eventually, as universities get left in the dust in this respect and funding agencies make software quality and availability more central to grant review sections, the institutions will grudgingly admit that software craftsmanship is as important as other intellectual tasks, sufficiently codify criteria by which to measure and teach it (and then probably begin to take credit for it). Computer science itself was dismissed as a lesser intellectual activity in its early years. Silliness!

  15. As a PhD student, the burden of writing good software is on the student themselves. If the student doesn’t care about writing good software, the academic software is hopeless. But the goal of a researcher is publications and finding commercial/industry partners, software efficiency and design does not matter. The theoretical knowledge in your publications and new avenues of development of product matters the most. At the end of the day, researchers answer “Does this concept/theory work?” and the question “Is this software efficient and fast?” is highly irrelevant. Researchers work under the assumption that if they make their research into a product — they will follow software engineering principles. Personally, I hate looking at poorly written code but you do deal with a lot in Academia.

  16. @Ankur

    Software efficiency and design does not matter.

    This might be true in the practice of the business of computer science, but it does raise some questions:

    • If your code is ugly, untested and undocumented… how do you know that it does what you claim that it does?
    • Many papers and thesis say something like this “we have improved the speed by a factor of X”. If your code is all around inefficient, how can this “X” be meaningful? What does it mean that one inefficient and poorly tested piece of code is faster than another inefficient and poorly tested piece of code?
    • If you don’t know how to code properly, and you can’t find an academic position, what kind of job can you hope for?
  17. @Dan

    I don’t think PhD Students are software engineers but having software engineering skills do make them stand out. PhD student’s purpose is to do research and not come up with brilliant scalability paradigms that software engineers deal with. They are just performing concept/theory checks.

    1. The code can surely be ugly and undocumented, but not untested. If the results are not right based on simple use-cases and model-based cases (such as phantoms and datasets that you know the truth for), the code is wrong or your concept for your algorithm is wrong. The code always does what it is suppose to do i.e. produce the result consistent with the theory in the publication. If the code is untested, your results will not be well understood in the peer-review process and thus, no publication. Academia works based on reputation and no one is willing to harm their reputation by publishing suboptimal results based on incorrect code.

    2. If a paper claims a “speedup of a factor X,” then this entails that you do care about software efficiency and thus, you have made coding choices to gain that efficiency. Inefficient code can almost never lead to a speedup, and researchers will tarnish their reputation if they claim something that is just not true.

    3. If you don’t know how to code properly, and but your research is great, you can still land an academic position as the typical Professor who has lost their touch with software. If you do know how to code and are bad at research, you can land a job in any software firm. If you are bad at both, become a consultant.

  18. Academia works based on reputation and no one is willing to harm their reputation by publishing (…) based on incorrect code.

    But they don’t publish their code! Who is going to tell that the code is incorrect?

    Inefficient code can almost never lead to a speedup, (…)

    Sure it can, because researchers typically implement both algorithm A and algorithm B. Even if they are both inefficient, one can be faster than the other.

    If you don’t know how to code properly, and but your research is great, you can still land an academic position

    Absolutely, but it is a lot easier to find a good job as a developer than as an academic researcher.

  19. “PhD student’s purpose is to do research and not come up with brilliant scalability paradigms that software engineers deal with”

    which means no real-life implementation of your ideas and algorithms.

    and that’s why the great majority of academic research, it may be brilliant, but won’t make any impact in the industry.

  20. Great article!
    As PhD student myself, I have to admit that in the beginning of my grad career, I wrote pretty awful code that I thought would only be seen and run by myself. I’ve learned a great deal since then, after doing an internship and reading far better code from my fellow peers and seniors.

    Based on my experience, I think that grad students in systems research such as architecture/distributed systems/low-level databases tend to be better programmers than those in machine learning/NLP , where prediction accuracy, precision, recall are the numbers to report, as opposed to efficiency metrics. Building big and efficient systems usually requires good architectural design, which means students must be planning well beforehand before they start to build anything. Added to that, students in systems research usually tend to work together on big umbrella projects funded by advisors, which means that they have to share their components/modules to avoid duplication; at the very least, their code has to be readable.

    Overall, I think a grad student’s time is better spent on doing just research-related activities, i.e. reading/writing papers, discussing ideas and collaborating, rather than on improving prototype code. For most students, we only have 5-6 years of being in academic environment, and at least on my part, I’d prefer to spend that time improving skills that will make me a better researcher rather than engineer. This is really not to stay that programming skills are not important (because I know I will be a future engineer), just secondary in my priorities at the moment.

  21. I totally agree that research coode should have production-like quality from the get-go. In particular, all kind of testing is a must?

    Why? Because it directly relates to the quality of research. I don’t wanna read result based on faulty software. In addition, building good-enough software saves time when research code is reused in other projects.

    I’d note however that most so called industry code is ugly and often times way uglier than research prototypes. It is not easy to beautify this crap, because it is fragile.

Leave a Reply

Your email address will not be published.

You may subscribe to this blog by email.