Scalable Web Development

Here is a list of slides of various major Web projects where scalability had to be addressed. I am starting to think about a course project where this would be addressed specifically. Since I am not into software engineering, Web programming is nothing but a technical task for me, but if you throw in high scalability, the topic becomes very interesting and very database-related. In English, the course title is a given (Scalable Web Development), but I am unsure how to call it in French. I do not know exactly how to render the word scalable in French: vague terms are typically difficult to translate. Maybe a title like Programmation web et mise à l’échelle. It is a bit long, but it is the best I can do.

Scalability means three things, at least. First, you have to the number of hits that your server takes every second. That’s IO and CPU scalability. Then you have user-scalability (going from 3 users to 3,000,000) and everything it implies (abuse and spam). Finally, you have design scalability: how fast can you come up with the application, and how fast can you update it as things are crashing and burning.

Topics covered would include:

  • Memcache
  • Ruby on Rails
  • Python
  • PHP
  • MySQL
  • Custom data structures (flat files)
  • Partitions
  • ECMAScript and AJAX

Am I missing anything? Yes, Java is absent.

Students would conclude the course with a project that would have to be design to take a lot of abuse, or at least, design in such a way that if it did get abused, people could come in and fix it easily. For example, students would have to explain how they would partition it and so on. Maybe they’d have to run tests on how it scales (but that’s not entirely satisfying). I’d certainly allow students to come up with sites that they can turn into a business.

It would probably be a course from hell. But then, I am convinced that challenging courses is not what kills students. Boredom and couldn’t-care-less instructors do.

Update. Maybe facebook applications and map reduce should be included.

4 thoughts on “Scalable Web Development”

  1. That sounds like a very good course. Many aspects of scale are hard to capture with simulations and tests; perhaps you can find a heavily used service in the university and test the implementations on that.

    Note that Java does have its place in scaling discussions; programs written in Java can scale well, and some of the most exciting new directions involve hosting dynamic languages in the JVM (eg. RonR with JRuby).

  2. This would be a good course but you’re missing a few essential elements of scalability. Firstly it’s vital that you talk about maintainability both in terms of how easy it is to change your application as well as how easy it is to look after the application once it’s in a production environment.
    You’re also missing the distributed computing perspective that Verner Wogels is always emphasising: handling component/machine failure, distributed consensus, monitoring etc. The job description here is highly educational: http://www.allthingsdistributed.com/2007/07/job_opening_for_a_senior_resea.html

  3. Développement d’application web à déploiement massif?

    Sounds lame a bit.

    Anyhow, Building Scalable Websites (the Flickr Way) by Cal Henderson published by Oreilly should be required reading.

    PS. your spam “captcha” is so bad, I *almost* decided not to post (no, I am not that bad with roman numerals or math, but mixing the two is evil!)

  4. Hi, I hadn’t been here in a while, glad to see you’ve kept the pace (or place?). Today I mentionned collaborative filtering in a post and somehow it reminded me of you, so here I am.

    D’après le Grand dictionnaire ( http://granddictionnaire.com/ ), on peut traduire scalability par extensibilité, même si, dans l’usage courant, extensibilité a une signification plus large (disons que le mot est scalable 😉

    On pourrait traduire par Développement web extensible (ou programmation, mais je préfère le mot “développement” en général).

    Définition :
    Aptitude d’un service à augmenter ou à diminuer son niveau de performance et ses coûts pour répondre aux changements dans la capacité de production ou dans la demande.

    I tried finding other source to translate the term but like you, I came back empty handed.

    I’d also touch on the subject of samples and standard deviations. Very often, we generate random data or run a few tests with no clue about their distribution.

    Anyhow, it sounds like a much needed course. I’d skip anything to do specifically with facebook though, but map reduce is essential I think.

    P.S.: Sylvain, the captcha is much better then what used to be served here. And I was lucky enough a simple concatenation worked on my test. Try that with arabic with all numbers

Leave a Reply

Your email address will not be published. Required fields are marked *