Why I still program
People expect that, as you grow older, you give up practical jobs such as programming for more noble tasks such as managing a team and acquiring funding. This especially true in academia where “real professors” delegate the details, keeping only the “big picture stuff”. In other words, organizations are geared toward vertical collaboration: a hierarchical structure where people on top supervise other (cheaper) employees. In research, this means that the senior scientists have the ideas which junior scientists implement. Over time, the senior scientists may become unable to do what the junior scientists do, but they will become experts at acquiring funding. This model can scale up: the senior scientist can direct middle-level scientists who then supervise the younger scientists, and so on. Jorge Cham referred to this model as the Profzi scheme because it works best when funding is abundant and ever increasing.
The counterpart is horizontal collaboration. In this model, the senior scientists do everything, from having the big idea to executing it. They prefer to automate or avoid busy work when possible. Collaboration is mostly used to get a different point of view and complementary expertise. This model still works when funding is scarce, but it fails to scale up with respect to the number of people involved: horizontal collaboration is necessarily intimate.
The type of work that each model supports best differs. I conjecture that vertical collaboration favors long-term plans and predictable results. I believe horizontal collaboration favors serendipity and “wild” ideas.
As a sign that I favor horizontal collaboration, I still program even though I am old. This is unusual. It is so unusual as to get people to raise eyebrows. Some programming takes time, a lot of time. I can spend two or three months a year programming. Presumably, my time is too valuable to be spent on a lowly task like programming that can be best done by people earning a fraction of my income. So why do I still program?
Maybe my best advocate would be the master himself, Donald Knuth:
People who discover the power and beauty of high-level, abstract ideas often make the mistake of believing that concrete ideas at lower levels are relatively worthless and might as well be forgotten. (…) on the contrary, the best computer scientists are thoroughly grounded in basic concepts of how computers actually work, and indeed that the essence of computer science is an ability to understand many levels of abstraction simultaneously.
But I also have my own arguments:
- I want my work to be significant, to have impact. Yet even the widely cited research papers are rarely read. Very few research papers have significant impact. However, it is comparatively easier to do work that matters with software. For example, recently a team from Facebook integrated one of my compressed bitmap index library in Apache Hive: the Hadoop-based framework for data warehousing. I am willing to bet good money that nobody at Facebook read the original paper for which I wrote this software.
- Time and time again, implementing my ideas has forced me to understand them better. A common scenario is that something that sounded reasonable on paper suddenly feels unwieldy when you must implement it. I also often discover bugs in my mathematical arguments through implementation. Could I outsource this work to others? Maybe. But the process would not be as fruitful.
- You do get better at programming over time. I have been building up my expertise for decades. It is enjoyable to start from scratch and solve a difficult problem in days when you know that others would take weeks or months to do the same.
If my arguments are reasonable, and if even Donald Knuth is on my side, why does it still surprise people when I admit to be a programmer-scientist? I believe that the rejection of programming as a lower activity can be explained by the Theory of the leisure class. In effect, we do not seek utility but prestige. There is no prestige in tool-making, cooking or farming. To maximize your prestige, you must rise up to the leisure class: you work must not be immediately useful. Thus, there is more prestige in being a CEO or a politician, than in being a nurse or a cook. Scientists who supervise things from afar have more prestige. Programming is akin to tool-making, thus people from the leisure class won’t touch it. People will call themselves engineer or analyst or developer, but rarely “programmer” because it is too utilitarian.
Warning: Not everyone should be programming. It is a time consuming activity. Because I program so much, there are many other exciting things I cannot do.
Further reading: Brooks, The Design of Design, 2010.
Montreal, Canada 
Follow on
Couldn’t agree with you more! I was almost afraid of moving from my high-level world of abstract ideas back to being a coding software engineer at Google. Yeah, Google is cool, I thought, but “it’s only code”. That was a year ago, before I started actively coding again. When I begun, I felt I was being frowned upon by some researchers (or maybe I was just frowning upon myself?). After a year, though, I can totally see Knuth’s point. I have not given up on powerful and beautiful high-level abstract ideas for one second, but now I go all the way to coding them. At all time I strive to keep that multi-layered vision of the software. The intellectual challenge surpasses that of sticking only to the high-level ideas, and the result is far closer to being usable.
To me, programming is the “unit test” of a high-level idea and I feel researchers should write their own unit tests. As you say, it brings insight back into the high-level ideas. Also, it makes it easy to “refactor” the high-level ideas, as any good unit test do. It also gives you a petri dish where you can settle arguments about high-level ideas in a convincing manner; way too often is authority use to impose ideas on students. Finally, it ensures you never start off a tangent that is too far from applicability for what you want to accomplish.
Also: congratulations on your code being used in Hive! That’s no small achievement.
Comment by Philippe Beaudoin — 6/6/2011 @ 9:43
@Beaudoin
way too often is authority used to impose ideas on students
Good point that I missed.
Comment by Daniel Lemire — 6/6/2011 @ 9:46
I too have noticed the same thing as Philippe, an analogue to this in industrial software development. We would design large new systems and sometimes spend months in the design phase when we should have gotten to work on the coding sooner. The act of coding inevitably exposes things we didn’t think about during design, which might be minor technicalities or might require reconsiderations of parts of the design. This is one reason it can be helpful to solicit the advice of people ‘lower down’ on the prestige hierarchy, because they may have more recent coding experience that may lead them to find those issues sooner.
We did exactly that when making animated feature films (so why not do it when developing software): we design the production of our films, get to work on storyboarding, layout, rough animation, etc. But as soon as possible, we produce final rendered finished frames for some scenes, maybe even before the whole screenplay is in its final form, because it’s important to know as soon as possible how easy it is to get all the way to the end for some scenes. Then we broaden and continue to flesh out the remainder of the production, but now that we know the path through the forest, we can make much more reliable estimates as to how long it will take to get there.
Comment by Allan Rempel — 6/6/2011 @ 11:32
I couldn’t agree more with your post Daniel. It is very easy to get into the habit of not coding in academia. In fact, administrative duties can pretty much take every free minute if you’re not careful. I don’t really want to be a manager. I enjoy getting my hands dirty too much.
So, I created a little system to remind myself of what is important to me. I try to do 3 things every working day: read, write, & code. I call these three tasks my “Research Trinity”, and I put a little X in my calendar each day I complete the trifecta. It turns out to be surprisingly hard to do all three in a day (and some days its hard to manage even 1!). Still, good research requires a mix of all three, and they all bring me tremendous personal satisfaction.
Comment by JSC — 6/6/2011 @ 17:02
As a young professor, I am struggling to keep some time to keep programming. I like it and feel that I want to keep some oversight over the work done by my students (quality control), but I don’t have much time, and get easily sucked in by all the administrative work. Also, as pointed out regularly on this blog, making tools is not as highly rewarded as writing papers.
I am in the process of figuring that out and would hate to let my skills rust…
Comment by Nicolas Saunier — 6/6/2011 @ 17:37
I used to tease my boss when he fired up R or Stata that his time was too valuable to waste on programming and number crunching, and that as a bank SVP he should touch no technology more sophisticated than Outlook or PowerPoint.
Now as I keep being assigned more subordinates and spending more time managing and less time coding, I don’t find those jokes quite so funny as I did before. I pine after the days of individual contribution and horizontal collaboration.
Comment by Michael P. Manti — 6/6/2011 @ 18:38
I started programming in 1965 (FORTRAN). I am very fortunate that I have been able to earn a living all my life by writing code. I would have it no other way.
I have some wonderful challenges ahead if me. What other profession can still elicit whoops of joy after forty five years?
See bio: http://kyber.biz/rants/UNIVAC%20history.php
Comment by Pierre Clouthier — 6/6/2011 @ 20:10
Excellent.
I’m a young professor in one of the Russian universities. Older colleagues say that I must stop wasting time on programming, or else I will never have time to earn money. I had nothing to say to excuse my position, since I really have no time and no money.
Now, when I have read you post, I feel much better. I want to translate your post into Russian and publish it in my blog (providing a link to the original text). Would you allow me to do this?
Comment by Anton — 7/6/2011 @ 1:06
@Anton My posts are available under a Creative Commons license. You certainly can translate them and publish the translation.
Comment by Daniel Lemire — 7/6/2011 @ 7:20
Great post. In my opinion, orogramming is also very important and under-appreciated in mathematics research –
see Zeilberger’s opinion on Birds, Frogs and Beavers:
http://www.math.rutgers.edu/~zeilberg/Opinion95.html
Comment by z — 7/6/2011 @ 12:30
Hey dude how old are you? You look pretty young.
I’m a 29 year old programmer.. I’ve worked with so few programmers at my level in the commercial world, managing a team of people barely able to understand me frightens me.
I wish I could just spend my life in the open source world where so many bright and talented programmers live.
Comment by James — 9/6/2011 @ 12:18
@James
I was joking a bit when writing that I am old. I am older than you, but I am likely younger than your dad.
Comment by Daniel Lemire — 9/6/2011 @ 18:14
I am programmer, not scientist. I agree that real implementation boosts adoption of ideas.
Comment by Bubak — 11/7/2011 @ 14:26
I rather think that programming is not any more what it used to be due to software “commoditization”.
Nevertheless this rather applies to the mass of average programmers in corporate organization not to master programmers like you and a few others
Comment by AgileOrWaterfall — 11/7/2011 @ 19:08
You’re in good company. There’s a famous quote from Richard Feynman: “That which i cannot build i cannot understand”. One of my favorites.
Comment by John — 20/9/2011 @ 12:00
@John
Did Feynman build stuff? I thought he was a pure theoretician?
Comment by Daniel Lemire — 20/9/2011 @ 12:49
That epigram was one of the last things he wrote on his office blackboards before he died. (Note: substitute “create” for “build”.) If you google the quote, you’ll find many links to a photograph.
Directly below he wrote “Know how to solve every problem that has been solved.” The two quotes were written inside adjacent boxes drawn around them.
If those two quotes express related thoughts, I’d be inclined to think the idea behind the first is that one doesn’t truly understand a theory until one’s worked out the mathematical details with full rigor.
If that’s right, it would answer your question. But not knowing more, I can’t say for sure.
Comment by John — 20/9/2011 @ 14:03
Let’s add also to the above arguments that (IMHO) programming helps (like Sudoku for example) to maintain brain agility, and the capacity to put your thoughts in order.
In French, the word for “computer” is “ordinateur” created after some proposition from IBM France. The word “ordinateur” came from the latin root “ordinat” which means “order”. So, a “computer/ordinateur” is a tool which produces [an] order. Well, in fact, the computer obeys first to the developer, so it’s the developer that puts things in order. So, programming helps to rationalize things and points of view.
For example, when programming in C, you learn not to confuse a pointer’s value with the data referenced by this pointer (sounds alike the “type theory” from Bertrand Russell). It sounds like the Allegory of the Cave (Plato). Then, again, you see, programming is all about mental agility, putting things in order, or in perspective if you prefer (like Plato pointed it), to bring light and not confusion. It’s a kind of training.
One could mention also that programming is related programmer’s point of view about (code) beauty, and is about to exercise that concern. I won’t dig that point here.
Comment by Dominique De Vito — 27/9/2011 @ 11:50
I enjoyed reading this thread. Found this link containing some interesting statistics:
http://improvingsoftware.com/2009/05/19/programmers-before-you-turn-40-get-a-plan-b/
“[S]ix years after finishing college, 57 percent of computer science graduates are working as programmers; at 15 years the figure drops to 34 percent, and at 20 years — when most are still only in their early 40′s — it is down to 19 percent. In contrast, the figures for civil engineering are 61 percent, 52 percent and 52 percent.”
Comment by Simon Hardy-Francis — 25/2/2012 @ 14:17
@Simon
Thanks. This matches my intuition. Programming is viewed as a low status occupation.
Comment by Daniel Lemire — 27/2/2012 @ 14:14