Why building software is hard

Why is building software difficult? Why do so many projects fail? I recently had an argument with a colleague who thinks that the problem is that the software industry is unable to follow due process… to take the requirements, make up a plan and follow it.

Well. There is no such thing as “a project with system requirements and promised characteristics.” The engineers at Google or Microsoft are not given out a sheet of requirements in 2005 and told to deliver a new product meeting these requirements in 2007. The requirements change every week and they poorly define the expectations of whoever is paying you.

There are boring software projects where the requirements are static, for example, making available a MySQL database through a web interface, adding a tool to the intranet where people can submit documents, and so on. Those can be managed pretty well and they are actually. Failure rates are small.

Making analogies with other industries, you should observe that pharmaceutical companies who design new medications typically fail most of the time. What Boeing sells is an already designed and prototyped airplane. They design very few new ones that are not minor variants on existing planes, and when they do have to build a new airplane from scratch, it is tricky and I can assure you that they hire the very best engineers they can for the job.

And before you think that these are bad analogies… Read the story of how Microsoft Vista came about:

Microsoft started work on their plans for “Longhorn” in May 2001, prior to the release of Windows XP. It was originally expected to ship sometime late in 2003 as a minor step between Windows XP (codenamed “Whistler”) and “Blackcomb” (now known as Windows “Vienna”). Gradually, “Longhorn” assimilated many of the important new features and technologies slated for “Blackcomb,” resulting in the release date being pushed back a few times. Many of Microsoft’s developers were also re-tasked with improving the security of Windows XP. Faced with ongoing delays and concerns about feature creep, Microsoft announced on August 27, 2004 that it was making significant changes. “Longhorn” development basically started afresh, building on the Windows Server 2003 codebase, and re-incorporating only the features that would be intended for an actual operating system release. Some previously announced features, such as WinFS and NGSCB, were dropped or postponed, and a new software development methodology called the “Security Development Lifecycle” was incorporated in an effort to address concerns with the security of the Windows codebase. (Source: wikipedia)

So, you see that the model where the engineers are given a set of requirements, fully describing the product the be built, and then they go out to build it, is very, very far from reality. And do not blame whoever runs Microsoft. It is simply an utopia to think that you can specify the requirements of a new software product. Could you have specified the requirements for the Google search engine back in 1995? Software is designed, it is not built like a house. Programming is a craft, not a science.

Yes, there are programming techniques to be learned, and there are tricks to help you keep a large software project on its rails. Unit testing, computational complexity, all these things are very important. But saying that software projects fail for lack of engineering is like saying that the latest Stephen King’s novel is boring because he forgot to draw a UML diagram of the book.

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

2 thoughts on “Why building software is hard”

  1. I agree with your post. I think alot of the misunderstanding arises because people equate software development with project management where the cardinal sin is not to meet the deadline. IMO, this should be replaced with a “quest for value” where software developers and other stake holders communicate regularly about whether the feature set is sufficiently rich, coherent and debugged enough to trigger it’s roll out. It is a process that needs a certain amount of open-endedness to produce the “value” that all parties are ultimately “questing” after and will “discover” when the project starts to to come together. The metaphor of a “quest” is probably more descriptive of the software development process than the metaphor of a “project” governed by deadlines.

  2. My comment to developers has been that if they see their job as to simply implement what’s on the card, their job might as well go to the cheapest offshore supplier

Leave a Reply

Your email address will not be published. Required fields are marked *

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax