Why I still program

People expect that, as you grow older, you give up practical jobs such as programming for more noble tasks such as managing a team and acquiring funding. This especially true in academia where “real professors” delegate the details, keeping only the “big picture stuff”. In other words, organizations are geared toward vertical collaboration: a hierarchical structure where people on top supervise other (cheaper) employees. In research, this means that the senior scientists have the ideas which  junior scientists implement. Over time, the senior scientists may become unable to do what the junior scientists do, but they will become experts at acquiring funding. This model can scale up: the senior scientist can direct middle-level scientists who then supervise the younger scientists, and so on. Jorge Cham referred to this model as the Profzi scheme because it works best when funding is abundant and ever increasing.

The counterpart is horizontal collaboration. In this model, the senior scientists do everything, from having the big idea to executing it.  They prefer to automate or avoid busy work when possible. Collaboration is mostly used to get a different point of view and complementary expertise. This model still works when funding is scarce, but it fails to scale up with respect to the number of people involved: horizontal collaboration is necessarily intimate.

The type of work that each model supports best differs. I conjecture that vertical collaboration favors long-term plans and predictable results. I believe horizontal collaboration favors serendipity and “wild” ideas.

As a sign that I favor horizontal collaboration, I still program even though I am old. This is unusual. It is so unusual as to raise eyebrows. Some programming takes time, a lot of time. I can spend two or three months a year programming. Presumably, my time is too valuable to be spent on a lowly task like programming that can be best done by people earning a fraction of my income. So why do I still program?

Maybe my best advocate would be the master himself, Donald Knuth:

People who discover the power and beauty of high-level, abstract ideas often make the mistake of believing that concrete ideas at lower levels are relatively worthless and might as well be forgotten. (…) on the contrary, the best computer scientists are thoroughly grounded in basic concepts of how computers actually work, and indeed that the essence of computer science is an ability to understand many levels of abstraction simultaneously.

But I also have my own arguments:

  • I want my work to be significant, to have impact. Yet even the widely cited research papers are rarely read. Very few research papers have significant impact. However, it is comparatively easier to do work that matters with software. For example, recently a team from Facebook integrated one of my compressed bitmap index library in Apache Hive: the Hadoop-based framework for data warehousing. I am willing to bet good money that nobody at Facebook read the original paper for which I wrote this software.
  • Time and time again, implementing my ideas has forced me to understand them better. A common scenario is that something that sounded reasonable on paper suddenly feels unwieldy when you must implement it. I also often discover bugs in my mathematical arguments through implementation. Could I outsource this work to others? Maybe. But the process would not be as fruitful.
  • You do get better at programming over time. I have been building up my expertise for decades. It is enjoyable to start from scratch and solve a difficult problem in days when you know that others would take weeks or months to do the same.

If my arguments are reasonable, and if even Donald Knuth is on my side, why does it still surprise people when I admit to be a programmer-scientist? I believe that the rejection of programming as a lower activity can be explained by the Theory of the leisure class. In effect, we do not seek utility but prestige. There is no prestige in tool-making, cooking or farming. To maximize your prestige, you must rise up to the leisure class: your work must not be immediately useful. There is more prestige in being a CEO or a politician, than in being a nurse or a cook. Scientists who supervise things from afar have more prestige. Programming is akin to tool-making, thus people from the leisure class won’t touch it. People will call themselves engineer or analyst or developer, but rarely “programmer” because it is too utilitarian.

Warning: Not everyone should be programming. It is a time consuming activity. Because I program so much, there are many other exciting things I cannot do.

Further reading
: Brooks, The Design of Design, 2010.

Subscribe to this blog
in a reader
or by Email.

Daniel Lemire, "Why I still program," in Daniel Lemire's blog, June 6, 2011.

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

42 thoughts on “Why I still program”

  1. Couldn’t agree with you more! I was almost afraid of moving from my high-level world of abstract ideas back to being a coding software engineer at Google. Yeah, Google is cool, I thought, but “it’s only code”. That was a year ago, before I started actively coding again. When I begun, I felt I was being frowned upon by some researchers (or maybe I was just frowning upon myself?). After a year, though, I can totally see Knuth’s point. I have not given up on powerful and beautiful high-level abstract ideas for one second, but now I go all the way to coding them. At all time I strive to keep that multi-layered vision of the software. The intellectual challenge surpasses that of sticking only to the high-level ideas, and the result is far closer to being usable.

    To me, programming is the “unit test” of a high-level idea and I feel researchers should write their own unit tests. As you say, it brings insight back into the high-level ideas. Also, it makes it easy to “refactor” the high-level ideas, as any good unit test do. It also gives you a petri dish where you can settle arguments about high-level ideas in a convincing manner; way too often is authority use to impose ideas on students. Finally, it ensures you never start off a tangent that is too far from applicability for what you want to accomplish.

    Also: congratulations on your code being used in Hive! That’s no small achievement.

  2. I too have noticed the same thing as Philippe, an analogue to this in industrial software development. We would design large new systems and sometimes spend months in the design phase when we should have gotten to work on the coding sooner. The act of coding inevitably exposes things we didn’t think about during design, which might be minor technicalities or might require reconsiderations of parts of the design. This is one reason it can be helpful to solicit the advice of people ‘lower down’ on the prestige hierarchy, because they may have more recent coding experience that may lead them to find those issues sooner.

    We did exactly that when making animated feature films (so why not do it when developing software): we design the production of our films, get to work on storyboarding, layout, rough animation, etc. But as soon as possible, we produce final rendered finished frames for some scenes, maybe even before the whole screenplay is in its final form, because it’s important to know as soon as possible how easy it is to get all the way to the end for some scenes. Then we broaden and continue to flesh out the remainder of the production, but now that we know the path through the forest, we can make much more reliable estimates as to how long it will take to get there.

  3. As a young professor, I am struggling to keep some time to keep programming. I like it and feel that I want to keep some oversight over the work done by my students (quality control), but I don’t have much time, and get easily sucked in by all the administrative work. Also, as pointed out regularly on this blog, making tools is not as highly rewarded as writing papers.

    I am in the process of figuring that out and would hate to let my skills rust…

  4. I used to tease my boss when he fired up R or Stata that his time was too valuable to waste on programming and number crunching, and that as a bank SVP he should touch no technology more sophisticated than Outlook or PowerPoint.

    Now as I keep being assigned more subordinates and spending more time managing and less time coding, I don’t find those jokes quite so funny as I did before. I pine after the days of individual contribution and horizontal collaboration.

  5. I couldn’t agree more with your post Daniel. It is very easy to get into the habit of not coding in academia. In fact, administrative duties can pretty much take every free minute if you’re not careful. I don’t really want to be a manager. I enjoy getting my hands dirty too much.

    So, I created a little system to remind myself of what is important to me. I try to do 3 things every working day: read, write, & code. I call these three tasks my “Research Trinity”, and I put a little X in my calendar each day I complete the trifecta. It turns out to be surprisingly hard to do all three in a day (and some days its hard to manage even 1!). Still, good research requires a mix of all three, and they all bring me tremendous personal satisfaction.

  6. Excellent.

    I’m a young professor in one of the Russian universities. Older colleagues say that I must stop wasting time on programming, or else I will never have time to earn money. I had nothing to say to excuse my position, since I really have no time and no money.

    Now, when I have read you post, I feel much better. I want to translate your post into Russian and publish it in my blog (providing a link to the original text). Would you allow me to do this?

  7. Hey dude how old are you? You look pretty young.

    I’m a 29 year old programmer.. I’ve worked with so few programmers at my level in the commercial world, managing a team of people barely able to understand me frightens me.

    I wish I could just spend my life in the open source world where so many bright and talented programmers live.

  8. I rather think that programming is not any more what it used to be due to software “commoditization”.

    Nevertheless this rather applies to the mass of average programmers in corporate organization not to master programmers like you and a few others 🙂

  9. You’re in good company. There’s a famous quote from Richard Feynman: “That which i cannot build i cannot understand”. One of my favorites.

  10. That epigram was one of the last things he wrote on his office blackboards before he died. (Note: substitute “create” for “build”.) If you google the quote, you’ll find many links to a photograph.

    Directly below he wrote “Know how to solve every problem that has been solved.” The two quotes were written inside adjacent boxes drawn around them.

    If those two quotes express related thoughts, I’d be inclined to think the idea behind the first is that one doesn’t truly understand a theory until one’s worked out the mathematical details with full rigor.

    If that’s right, it would answer your question. But not knowing more, I can’t say for sure.

  11. Let’s add also to the above arguments that (IMHO) programming helps (like Sudoku for example) to maintain brain agility, and the capacity to put your thoughts in order.

    In French, the word for “computer” is “ordinateur” created after some proposition from IBM France. The word “ordinateur” came from the latin root “ordinat” which means “order”. So, a “computer/ordinateur” is a tool which produces [an] order. Well, in fact, the computer obeys first to the developer, so it’s the developer that puts things in order. So, programming helps to rationalize things and points of view.

    For example, when programming in C, you learn not to confuse a pointer’s value with the data referenced by this pointer (sounds alike the “type theory” from Bertrand Russell). It sounds like the Allegory of the Cave (Plato). Then, again, you see, programming is all about mental agility, putting things in order, or in perspective if you prefer (like Plato pointed it), to bring light and not confusion. It’s a kind of training.

    One could mention also that programming is related programmer’s point of view about (code) beauty, and is about to exercise that concern. I won’t dig that point here.

  12. I enjoyed reading this thread. Found this link containing some interesting statistics:

    “[S]ix years after finishing college, 57 percent of computer science graduates are working as programmers; at 15 years the figure drops to 34 percent, and at 20 years — when most are still only in their early 40′s — it is down to 19 percent. In contrast, the figures for civil engineering are 61 percent, 52 percent and 52 percent.”

  13. Good work Dainel. From a huge list of programming and web development list, your blog catches my mind and like the title of your post.

  14. As an 8 year art major who wandered across the campus to the keypunch room (run by the business school as at that time there was no computer department) I sat down, learned to program and discovered that there was (at an abstract level at least) no difference between say sculpting and programming. Design is design. More importantly I got just as much of a ‘creative fix’ from the one activity as the other. Now more than 35 years later, I have found nothing to change my mind. The creative element is truly why I keep programming (I’ll also admit to a slight addiction to learning new languages 🙂 ) Nice article—just thought I’d explain why I’m still at it almost 4 decades in!

    Hugh S. Myers

  15. As an 8 year art major who wandered across the campus to the keypunch room (run by the business school as at that time there was no computer department) I sat down, learned to program and discovered that there was (at an abstract level at least) no difference between say sculpting and programming. Design is design. More importantly I got just as much of a ‘creative fix’ from the one activity as the other. Now more than 35 years later, I have found nothing to change my mind. The creative element is truly why I keep programming (I’ll also admit to a slight addiction to learning new languages 🙂 ) Nice article—just thought I’d explain why I’m still at it almost 4 decades in!

    Hugh S. Myers

  16. Thanks for you article, and I whole heartily agree. I gave up a management position 7 years ago, at that time I had been away from heads down programming for 5 years (always dabbled though). It took a while but was able to come back up to speed with the current technology, and now am happily coding full time.
    For me, although I love to code and solve puzzles, it is that at the end of the day you can stand back and say that you actually built something, and hopefully that something will help someone and endure.
    BTW I will turn 56 this summer.
    Thanks again.

  17. You say you are old. You don’t look old. I am 67 and I can still program our robots faster than our engineers and so I often do. I have the ability (and opportunity) to see what the customer wants, what they are capable of doing and what is achievable and implement that without bells, whistles or bloatware. I keep one eye on processor efficiency another on usability that younger programmers neglect. Another thing we older coders do better is assess design risk, keep it simple, don’t put in stuff just to impress.

  18. Thanks Sir for this post. Very inspiring.

    In companies like Adobe, i have heard senior managers coding.
    Coding for a telecom thinker should be an essence of life.

  19. Yeah… programming a lot of time unable you yo do another things, but, can i do if programming is one of the things that i love to do day by day?. Good article, i liked it!.

  20. I can totally relate. After leading 2 successful start-ups I don’t have to work and am officially retired. But I do like to program and at 60 look for and implement meaningful programming projects.
    Thanks for the article.

  21. Great article! I think you really hit an important point about the “prestige” involved in different jobs. As we age, we feel embarassed if we cannot tout a job title further up the vertical chain.

    Now I just need to figure out how to go back down the chain and still make good $$$. 🙂

  22. After reading your article I feel vindicated in my choice to continue programming after 41 years. I enjoy it. And I realized after a short stint as a manager that I’d prefer doing the work over talking about the work or directing others.

  23. Absolutely brilliant article.
    Good to know I am not alone in my desire to continue working on low-level constructs.

  24. I was partner in a service bureau doing software programming in assembly and Fortran in the late 60’s. Loved it but I was not making a living. Left it to work for others primarily doing hardware design, but sometimes supporting and testing those designs with software (that I hesitate to admit was Basic and later QuickBasic as the PC revolution took hold.) Retired in the late 90’s, so I remained a dinosaur ossified with Basic variants up through Visual. I try programming for fun with new languages, but never get far without projects to drive the goal. I stayed in the hands-on position, and loved my job and stayed out of management. I agree with your comments, the “Ponzi” scheme exists in hardware design, too.

  25. I’ve been programming since 1967. I’ve tried management and failed a few times at retirement. I always come back to system design and programming for the challenge and the creativity. I like the “research trinity” mentioned by JSC in #4 above; I’ve got to try that. I’ve never had the feeling that I or Programming was viewed as “low status”. Good article, Thanks

  26. I received my MS in CompSic in 1971. I was able to start my own software company, and did quite well, up until 2005. Unfortunately, change, after change, big business, and big govenment has made it virtually impossible to be a small software company. As one business consultant told me: It costs just as much to sell a $5,000 app as it does a $50,000 app.

    What’s really killing programming and other occupations is the management style that divides work into small compartimentalize pieces. Double the people, pay them half, and they are all expendable – ready to be outsourced if need be.

    I could never understand why people think being a ‘manager’ is a higher stature position than a programmer. At one time, your manager knew your job, now your manager is nothing more than a worried status reporter who may have previously managed a shoe store.

    I advise all young people to either get a degree with a license behind it (like CPA, lawyer, doctor) or go into the trades. Programming in the work place has turned into living hell in a cube farm.

  27. With a working background in electronic design, I became seduced by the magic of ‘making my hardware designs perform’ (programming). Later moved into management, got burned out and went back to programming, as a consultant.

    It may be implicit in our competitive nature to aspire to managing others, but I have found managing others (usually against their will) to be a soulless endeavor that saps all meaning from life.

    Even today, working in high level language abstractions of the underlying hardware, I long for the simplicity and power that comes from developing a functioning device from concept to completion.

    I find it interesting that the forward thrust of society would seem to be to eliminate all work so that we all might join the leisure class. Yet there would seem to be no more unhappy individuals that those who have nothing to do.

    Thanks for a thought provoking article that gives cause for the re-examination of what we consider to be valuable.

  28. I too still program and have no intention of stopping.

    I look positively forward at an industry with mature professionals who care about the code details, quality but also the human factors. Gone are the days of production line hacking – at least for progressive companies, I think.

    However, back to my main observation, we have the tools now to create powerful languages in code (ala internal DSL) which means that although you are still “coding” you are not actually writing tedious boilerplate code (“suitable for the masses to code..”) but code at a high-level in-terms of leverage. This is an entirely different world of programming, I think.

  29. Good Article. I’m 67 and agree with David Sands (26). I have a clipping from an article in Network Computing March 2002 by Fritz Nelson that I have pinned to my cabinet door to remind me of what matters.
    “Many view your job as a ‘trade.’ Don’t let what you do become trivialized by the media, by the vendors who attempt to sell around you or by the business people in your organization who don’t understand what you do. Our industry has become the catalyst for progress and differentiation, and you have become the enabler of the infrastructure that will usher in a new era of business advantage in your organizations. If that’s not a profession, I don’t know what is. Treat yourselves like professionals and never apologize.”
    Those words are still true today a decade later.

  30. Since I became a freelance consultant, I mainly work as a project leader; and when I have collected enough money, I can quit doing this boring management job. Then I’m able to spend few months programming and studing whatever I like. I’m 44, discovered ASM at 14, and can’t stop each day about thinking ideas and instructions.
    From Paris

  31. I recently turned 50 and have been programming for 29 years of that. I started out using programmable calculators that allowed you to solve a + b * c type problems and moved on to other languages…GWBASIC and C anyone? The thought of taking a step backwards and becoming a ‘manager’ sends a shiver down my spine. I have always said that if I couldn’t get paid to program then I would do it for free!
    I am lucky enough for programming to be both my vocation and my hobby for which I am immensely thankful.

  32. Thank you for the excellent post.
    I really loved it and translated it into Japanese on my blog:
    I believe it’ll inspire programmers and researchers in Japan.
    Let me know if anything inconvenient to you.

    BTW, I noticed that tlanslating ideas is like implementing ideas for me.
    It also forces me to understand them deeper.
    Maybe that’s one of reasons why I like translating good articles.

  33. During the 1970s and 1980s, it was often argued that computer science was not an experimental science. This was stated most explicitly by Juris Hartmanis in his 1993 Turing Award speech. This prompted a debate which led to the Volume 27, Number 1, March 1995 issue of ACM Computer Surveys. I contributed a single-page article entitled “Computer Science Is an Experimental Science” (http://doi.acm.org/10.1145/214037.214050), which argues that programming is in fact the experimental component of our science.

Leave a Reply

Your email address will not be published.

You may subscribe to this blog by email.