Category: Audio podcast

Why is building software difficult? Why do so many projects fail? I recently had an argument with a colleague who thinks that the problem is that the software industry is unable to follow due process… to take the requirements, make up a plan and follow it.

Well. There is no such thing as “a project with system requirements and promised characteristics.” The engineers at Google or Microsoft are not given out a sheet of requirements in 2005 and told to deliver a new product meeting these requirements in 2007. The requirements change every week and they poorly define the expectations of whoever is paying you.

There are boring software projects where the requirements are static, for example, making available a MySQL database through a web interface, adding a tool to the intranet where people can submit documents, and so on. Those can be managed pretty well and they are actually. Failure rates are small.

Making analogies with other industries, you should observe that pharmaceutical companies who design new medications typically fail most of the time. What Boeing sells is an already designed and prototyped airplane. They design very few new ones that are not minor variants on existing planes, and when they do have to build a new airplane from scratch, it is tricky and I can assure you that they hire the very best engineers they can for the job.

And before you think that these are bad analogies… Read the story of how Microsoft Vista came about:

Microsoft started work on their plans for “Longhorn” in May 2001, prior to the release of Windows XP. It was originally expected to ship sometime late in 2003 as a minor step between Windows XP (codenamed “Whistler”) and “Blackcomb” (now known as Windows “Vienna”). Gradually, “Longhorn” assimilated many of the important new features and technologies slated for “Blackcomb,” resulting in the release date being pushed back a few times. Many of Microsoft’s developers were also re-tasked with improving the security of Windows XP. Faced with ongoing delays and concerns about feature creep, Microsoft announced on August 27, 2004 that it was making significant changes. “Longhorn” development basically started afresh, building on the Windows Server 2003 codebase, and re-incorporating only the features that would be intended for an actual operating system release. Some previously announced features, such as WinFS and NGSCB, were dropped or postponed, and a new software development methodology called the “Security Development Lifecycle” was incorporated in an effort to address concerns with the security of the Windows codebase. (Source: wikipedia)

So, you see that the model where the engineers are given a set of requirements, fully describing the product the be built, and then they go out to build it, is very, very far from reality. And do not blame whoever runs Microsoft. It is simply an utopia to think that you can specify the requirements of a new software product. Could you have specified the requirements for the Google search engine back in 1995? Software is designed, it is not built like a house. Programming is a craft, not a science.

Yes, there are programming techniques to be learned, and there are tricks to help you keep a large software project on its rails. Unit testing, computational complexity, all these things are very important. But saying that software projects fail for lack of engineering is like saying that the latest Stephen King’s novel is boring because he forgot to draw a UML diagram of the book.

Duck typing is the concept in programming by which an object is interchangeable with any other object as long as it can run through the same code while providing a sufficiently complete interface¹. It is a direct application of a popular idea: If it walks like a duck and quacks like a duck, it must be a duck. That is, you do not try to obtain proof that you have a duck, you just observe and see that, for the limited time your observation lasted, it appeared to be a duck. It might not be a duck, but you do not care. It is present in popular programming languages like Ruby, JavaScript and Python. Considering that Python is the prototyping language of choice of companies such as Google and JavaScript is ubiquitous on the Web, duck typing has become quite a common paradigm among programmers.

For the non programmers… Imagine you run a routine meant for ducks. First you weight the object, if it is over 2 kilograms, then you check the color of the feathers, if not you terminate the routine (because the duck is not ready to be eaten yet). Then you continue the routine based on the color of the feathers. Now, in this routine, a small rock will pass off as a duck, because you never check its feathers, but if you have a larger rock, then you will encounter a problem in your routine because you can’t check the color of the feathers of a large rock.

Comparatively², many languages (Java, C#, C++, C) first check whether the object is a duck. They may allow it to be a special type of duck, but before the routine even runs, you need to prove you have duck first.

I have come to realize recently that duck typing is a central concept in artificial intelligence and philosophy. Peter Turney argues again and again for the application of this principle, though he would not care to call it duck typing. What is the Turing test, if not an example of duck typing? If you can substitute the human being by a machine, and things still work, then, you might as well say that the machine is human (for the purposes of determining whether it has human intelligence). If you can replace part of my brain by a computer, then isn’t a computer as good as neural matter?

Imagine a similar problem in programming. You are a Python, JavaScript or Ruby programmer and you are asked whether this object can replace this other object. Or, to simplify the task, whether this function is a substitute to this other function. Specifically, you want to know if, once you have made the substitution, you will encounter any missing attributes.

How would you do it? Well, I might try switching one function for the other in some program. And if it works, I might be happy, but I have not really shown that one function can pass for the other, have I? It could still fail in another piece of code. I can falsify the question, but never get a truly positive answer. And no, in general, I cannot exhaust all possible tests to prove my case.

Strongly typed programming languages like Java, C++, C, C#, on the other hand, provide you a way to make sure, at compile-time, before the code even runs, that the interface is complete. And that is why duck typing is fundamentally different from static-typed polymorphism³.

This brings about a few questions. In other words, how long must the machine fool the human being before we conclude that the Turing test is a success?

This is not just a theoretical concern. For example, my life is made slightly miserable because of all the spam I receive. For a time, the combined spam filters ACM and Google Mail made it ok. I was pleased with the result and I would have rated the filtering on par with what a bored human assistant could do. As of a few months ago, these spam filters no longer pass my little Turing test.

This is very common in Machine Learning and Artificial Intelligence. You see the textbook example and you think to yourself wow! this is extraordinary! But then, any long term, real-life exposure with the technique, and you realize how stupid it is. Sometimes it can remain useful (I wouldn’t go without spam filters!), but it no longer fools you into thinking it is “intelligent.”

Similarly, if you use a large collection of text to determine the semantics of a word or a phrase, or to study analogies, you might get decent results for a time, but when the conditions change, things might go to hell. The Semantic Web is similarly plagued: you can never demonstrate that you have an accurate representation, but you can hope to eventually falsify it.

(This reminds me of the No Free Lunch theorems. The best solutions are always local and contextual. )

I’m sure that this is a very common objection to the Turing test or to Natural Language processing: there is no possible exhaustive testing. I think that the only remaining option is to limit the scope of these tests. That is, instead of using the somewhat ill-defined Turing test, you describe a very specific experiment, a very narrow one that can be reproduced exactly. The problem with this approach is that machines already pass such narrow tests.

In other words, as far as duck typing goes, there are many instances where you can already replace a human being by a machine.

Ah! So, I would argue that the Turing test is not a scientific test, really, but just a rather general paradigm, no different than duck typing.

Well, there is something even deeper, I think. Suppose that a machine could pass all my specific Turing tests, except for one of them. But I have an infinite number of tests, and only one of them could show the machine for what it really is. Then what? I have a probability of 0% of discovering the problem considering that the universe will not be around forever, let alone the human race. Would we conclude that the machine has human intelligence then?

To this Stevan Harnad might answer that it is arbitrary to ask for more from a machine than I ask from a person, just because it’s a machine. I do not buy this argument. I am not certain whether I am the only intelligent person in the universe and you are all part of a conspiracy to fool me into thinking there are several of me. Yes, I’m not insane and I tend to believe that other human beings roughly think and feel the way I do. But this is not arbitrary: human beings look a lot like me, down to their inner workings.

Another objection might come from relating the problem to the physical sciences. How do I know that gravity works? Yes, it has worked for many years, but how do I know that gravity always hold true. Well, I do not. It could be that there are part of the universe where gravity is absent or it could weaken one day. Why not?

All I know is that Newton’s laws are useful. All I know is that my spam filters are useful. Going from “this is a useful spam filter” to “this spam filter has human intelligence” is a step I am not yet ready to make, even in principle. And why is that even a useful step to take? Even if we create machines that can pass the Turing test, will we know what intelligence is? It does not necessarily follow. People were able to create highly radioactive materials before they understood what it was. Will these machines be more useful that other machines? Would a machine that can pass 100,000 different Turing tests, necessarily be more useful that the machine that can only pass 99,000 Turing tests?

This has very concrete consequences if you accept my ideas. Research in Computer Science should therefore be focused on making machines useful. Whether or not they can pass some specific Turing tests, even a large number, seems totally secondary to me. Bring me a machine that can filter my spam mail with human-like ability, and I will be happy. Don’t bother me by trying to prove to me that this machine is actually “intelligent.” I do not see why this is a useful concept.

Hence, Computer Science should focus on usefulness criteria and reject other criteria.

(Yes, I am fishing for your objections.)

—

1– Duck typing is a bit more complicated to define than what I did here. There is no specific interface to check. For example, consider this function:


def f(o):
if(o.hasLegs):
o.danceWithMe()


In this instance, any object o not having the attribute “hasLegs” will fail. But any object having the attribute “hasLegs” with value false whenever it gets passed to this function will also do (whether it has the danceWithMe method or not). You see how it can be complicated to determine if a given object can be used with function f. It is probably equivalent to the halting problem.

It is not the same thing as polymorphism because it requires that there be no static typing: the interface does not have to be complete, only sufficiently complete to run through the parts of the code that apply.

2– As an aside, languages supporting duck typing are far more powerful than others in my opinion. Languages with static typing are based on the assumption that catching bugs earlier is better. They were largely influenced by the belief that we should prove our code to be correct. That is, that programming is like proving theorems. Except that they got it wrong. Designing algorithms is like proving theorems; programming is like sketching the plans of a building. Architects do not prove that their buildings are pretty and work. Architects design their buildings to be pretty and functional. There is a huge difference in spirit between proving and designing, but both are hard work.

3– … and a poor model for reality.

It has become evident that, in the realm of Web Services, the REST paradigm is taking over while the Service-oriented Architecture Protocol (SOAP) is progressively being forgotten except in some academic circles and by some companies interested in selling tools to alleviate the pain¹.

Here is what Clay Shirky was saying in 2001:

This attempt to define the problem at successively higher layers is doomed to fail because it’s turtles all the way up: there will always be another layer above whatever can be described, a layer which contains the ambiguity of two-party communication that can never be entirely defined away.

No matter how carefully a language is described, the range of askable questions and offerable answers make it impossible to create an ontology that’s at once rich enough to express even a large subset of possible interests while also being restricted enough to ensure interoperability between any two arbitrary parties.

The sad fact is that communicating anything more complicated than inches-to-millimeters, in a data space less fixed than stock quotes, will require AI of the sort that’s been 10 years away for the past 50 years.

The main reason being put forward is that SOAP is simply too complex. But does complexity means here? The Web is something incredibly complex if you consider how many parts it has, yet, we consider it to be simple.

How to recognize a simple technology? The first criteria any engineer would use is the number of points of failures. SOAP architectures can break in many more ways than REST architectures, and so they are more complex. Meanwhile, theoretical computer science teaches us that something is more complex if it requires more CPU cycles to run. Well, SOAP architectures are also more complex in this light as well, as there is simply a lot more XML going around and the requests are far more verbose.

I’d like to propose that there is another criteria for complexity. And that’s semantics. One should always aim for the simplest possible solution… and providing lots of semantics is not a simple feat. SOAP architectures necessarily include semantics to define the meaning of terms used in the description and interfaces of the service. This is totally absent from REST architectures. It is not so much that there is no semantics in the REST paradigm, but it is kept extremely simple: you only need to know about the semantics of the main HTTP operation (POST, GET, PUT and DELETE). In fact, the wikipedia REST entry includes the following citation attributed to Roy Fielding:

REST’s client-server separation of concerns simplifies component implementation, reduces the complexity of connector semantics (…)

I think this is fundamental. What makes REST simple is that it reduces the amount of semantics the software has to worry about.

Why would semantics be a bad idea? Well, simply because semantics implies coupling, and too much coupling makes a system too complex. Without any coupling, we cannot do anything, but when we throw too much, we harm the system. What type of coupling are we talking about? Well, if I pass the variable x to the function f, there is relatively little coupling. All I do is that I establish a relationship between the function f and the variable x. But what if x is mean to be the cost of a product? Then x must be tied explicitly to the product ID, to some price identifier, and so on. This makes the system harder to maintain, harder to debug, and more failure-prone.

Fundamentally, software design is about communication. But not communication between machines… rather communication between developers. And communications between distributed folks works much better when the message they need to send to each other is kept very simple. That is why the SOAP philosophy is fundamentally flawed.

So, when you design software, you should include as little semantics as possible as this will make your system simpler, and thus, easier to manage.

This is, of course, contrary to what AI enthusiasts do.

—

1. See recent posts by Larry O’Brien and Nick Gall.