Personalization: the TiVo case

Michael Pazzani offers a course on Web Personalization. I have no time to check it further, but it looks like a very interesting course and the slides are available online.

What’s more interesting is that he offers his students a Word version of a 2004 TiVo paper (TiVo is a company that sells a TV show recording device). The paper is also available as a PDF file through ACM (but you need to be a member).

We describe the TiVo television show collaborative recommendation system which has been fielded in over one million TiVo clients for four years. Over this install base, TiVo currently has approximately 100 million ratings by users over approximately 30,000 distinct TV shows and movies. TiVo uses an item-item (show to show) form of collaborative filtering which obviates the need to keep any persistent memory of each user’s viewing preferences at the TiVo server. Taking advantage of TiVo’s client-server architecture has produced a novel collaborative filtering system in which the server does a minimum of work and most work is delegated to the numerous clients. Nevertheless, the server-side processing is also highly scalable and parallelizable. Although we have not performed formal empirical evaluations of its accuracy, internal studies have shown its recommendations to be useful even for multiple user households. TiVo’s architecture also allows for throttling of the server so if more server-side resources become available, more correlations can be computed on the server allowing TiVo to make recommendations for niche audiences.

Now, together with Verizon and Amazon, this is the third large company to use item-item collaborative filtering as in our Slope One algorithm used by inDiscover.net (which we licensed/sold to Bell/MSN). The TiVo paper itself is not so interesting: the details are conveniently hidden away.

Remember my post about the use of mathematical notations and how it was very useful in making papers precise? Well, the TiVo paper hardly use any mathematical notation, so it looks friendly, but try to really understand what they do and how they do it precisely. Maybe I’m just not smart enough, but I can’t really figure out. If they had expressed themselves in clearly stated and detailed equations, it would be clear. Now, all you have to go on is “we used a linear weighted average”. It is enough to understand the big idea, we could approximatively reproduce their work, we could, but we would have to guess our way through.

Maybe I’m just an old geek complaining too much…

The Social Software Revolution is happening!

What’s the social software revolution? Well, I’m not expert in social software, go see Seb Paquet or Stephen Downes. But my understanding is as follows.

Software has long been seen as a way to support business processes or automate number crunching. Software has also been tool for providing services, such as entertainment (think video games) or telecommunication (think email or voice over IP), but also learning (see the Wavelet forum I host).

So, a dating service can be seen as a service. I met my wife through an Internet dating service. Actually, more like a free posting board. This was back in 1998 and shortly after meeting we adopted a cat called Yahoo.

But back then, my wife and I were a bit strange. It was still slightly shameful to use software to meet someone. Now, it is taken for granted.

Using software to support one’s social life, to support a community, is what I call using social software. It includes posting boards, blogs, wikis, chat rooms, and anything where people can meet and organize themselves. But just like a market is not a “service” per se, but rather a part of the community, social software finally becomes part of the community is no longer a service either. The provider/client paradigm doesn’t really apply. If whoever owns the posting board closes it down or start charging money, the community will move to another posting board. The service provider is just not very important. Nobody cares who pays for my blog hosting or who actually hosts it, these things are invisible in the social software sense.

So, if that’s social sofware, what is the social software revolution? Well, it is the transition between a world where using software to support your social life is a bit strange, to a world where it is the thing to do. It is when social software becomes ubiquitous.

I’ll give you an example… back in 1998, if you wanted to buy something on the Web, you had a really hard time to do so. It took effort. Remember that Google wasn’t around. It is still hard to buy commercial products on the Web other than some electronics and books (at least, in Canada): this is one of the failures of the dot-com era, it did not manage to replace brick-and-mortar. However, it has become tremendously easy to find people, used products and custom services through the Web, through posting boards an other tools. My wife just sold an exercise bike we had in the basement. It was out of the question to either pay for a ad (the expected profit is too small) or start asking people around. All my wife had to do was post a little note in a free local community web site, and within days, someone came over and bought the exercice bike for $40. She didn’t think twice about the fact that she used the Web to sell the bike. She didn’t once say “Web”.

Now, this example is just the tip of the iceberg. Non-commercial trade has just become immensely easier and common (I claim) thanks to social software. Of course, you could almost do this in 1970. Almost. You had the technology to do so, but you didn’t have the momentum. We didn’t have the momentum in 1998: I don’t have any data, but I claim that the person-to-person trade supported by software has grown exponentially since 1998 and there is no reason to believe that the dot-bust has impacted this underground movement.

This is maybe the most important thing to remember. Business is not technology. If technology companies go under, if the programmers are out of a job, it doesn’t mean that the technology is not working. It could very well that the technology is gaining ground at an exponential rate, but that business is unable to make a profit out of it: maybe individuals are making all of the profit. When I find my wife on a free posting board, nobody profits from it but me (and my wife). In fact, by not using a commercial service, it could even be that I contribute to a downward economic trend.

The economy could go to hell due to new technology, but the people, overall, could still be much better off. I think that is what we see. While life is not getting any easier, technology, in this case social software, is making our life better and easier. We just can’t easily measure it in terms of jobs or taxes.

Simple Recipe for Debugging Web Services

Standard Deviations gives a Simple Recipe for Debugging Web Services (where he defines “Web Services” as SOAP as does my friend Yuhong) and it involves downloading a compiling a small command line utility called netcat.

Cool! I like it!

Only requirement is that you’ve got to be running Linux, but surely, if you are reading my blog you are an enlightened person and you use Linux, right?

How to Start a Startup

Paul Graham has done it again. He wrote a beautiful article on How to Start a Startup:

You need three things to create a successful startup: to start with good people, to make something customers actually want, and to spend as little money as possible. Most startups that fail do it because they fail at one of these. A startup that does all three will probably succeed.

I did start what one could call a startup and I failed. I did work like hell for a few short years. I made some good money, but all of it is long gone. However, I learned a lot.

One issue was that we didn’t have the right people. The other issue is that my love of money was not great enough. You’ve got to actually badly want to make money. All I wanted was to find a way to get paid to do interesting work.

These days, the money suck (a professorship is no way to make a living), but I pretty do what I like to do.

I think I still mostly want to be left in peace to my own ideas and work with collaborators I like. I think I’m succeeding at getting what I want. I will never become a big shot professor running a large laboratory (I would hate it), I will never become a filthy rich industry consultant (though I wouldn’t mind doubling my salary) because I will always pick contracts out of interest more than out of greed.

Lesson? I don’t know. Don’t listen to me, listen to Paul Graham and go start a startup, become filthy rich. It sounds like a great plan. I’m a lost cause.

How to Read Mathematics

Through Tall, Dark and Mysterious, I found this web page on How to Read Mathematics by Shai Simonson and Fernando Gouvea. This quote says it all:

Students need to learn how to read mathematics, in the same way they learn how to read a novel or a poem, listen to music, or view a painting.

Some of my papers are said to be “hard to read” because I make ample use of mathematical notations, even when it is not strictly needed. Some reviewers, though they won’t admit to it, don’t like my papers because they can’t read them. Of course, there is no shortage of Ph.D.s who can read mathematics, so there is no reason to stop using mathematical notations. To me, a paper with a strict adherence to mathematical conventions and a thorough use of mathematical notations is far easier to work with than a paper which tries to describe everything in English. English is an ambiguous and dangerous language, especially if you are not trained in philosophy and have English as your third language. Mathematics, on the other hand, is a true universal language and it can be extremely precise if needed. And that’s what I try to give my readers: a precise description of what is and what is not, not just vague impressions. I’ll keep the vague impressions for this blog.

Now, as of students, last time I taught Calculus, I was told that I was using too many Greek letters. Well, I didn’t apologize for using Greek letters. I probably kept using them. Students: learn to work with Greek letters, there is nothing wrong with them and if you do a lot of complex work, you’ll find that 26 letters are not enough and that borrowing from other alphabets does improve the clarity of your documents. Plus, there are some universal conventions you just can’t get around. I could say that “e” is a small quantity, but “e” is Neper’s number. So, when I want to refer to a small quantity, I use the Greek letter epsilon. So do at least another million of crazy geeks.

The real question is, when will I start using mathematical notations on my blog? That’s coming soon… I’ll get MathML on this very page in the near future. With plenty of Greek letters!!!

A survey of Eigenvector Methods for Web Information Retrieval

In the latest issue of SIAM Review (volume 47, no 1) (articles to be available online soon), I read a great paper for those who like mathematics and want a deeper understanding of how Google works. I knew how the PageRank algorithm worked, roughly, but I never imagined it was a true Linear Algebra algorithm. Of course, it is relatively simple as far as Linear Algebra goes, but still…

Becoming a gmail expert

Gmail is a really smart email client. I found out it is even better than I thought. If you want to quickly find unread messages in your inbox, just search for “label:inbox is:unread”.

The trick also is to “star” the messages you want to come back to. This is much better than leaving them unread which is really a hack if you think about it. You build a todo list, always one click away.

Don’t Become a Scientist!

Yuhong reacts to the paper Don’t Become a Scientist! by Jonathan I. Katz. Here’s what she had to say:

Whenever I met students who want to have a ph.d., I would ask them, do you really want it if I tell you the truth? Many students tell me that a scientist can be free to think anything and achieve what they think so that is the ideal life for them. But the truth is this career has too much harshness to be free. My Friend Daniel, a pessimist, has many posts recently about it. I agree with him on this issue though.

The paper she reacts to is also very entertaining. It goes after a few myths. One of them is that if you get a Ph.D. and finally do get a professorship, life will be great:

Suppose you do eventually obtain a permanent job, perhaps a tenured professorship. The struggle for a job is now replaced by a struggle for grant support, and again there is a glut of scientists. Now you spend your time writing proposals rather than doing research. Worse, because your proposals are judged by your competitors you cannot follow your curiosity, but must spend your effort and talents on anticipating and deflecting criticism rather than on solving the important scientific problems. They’re not the same thing: you cannot put your past successes in a proposal, because they are finished work, and your new ideas, however original and clever, are still unproven. It is proverbial that original ideas are the kiss of death for a proposal; because they have not yet been proved to work (after all, that is what you are proposing to do) they can be, and will be, rated poorly. Having achieved the promised land, you find that it is not what you wanted after all.

What can be done? I advocate that we inform potential students of the job prospects. Katz suggest we get funding agencies to stop funding so many Ph.D.s:

If you are in a position of leadership in science then you should try to persuade the funding agencies to train fewer Ph.D.s. The glut of scientists is entirely the consequence of funding policies (almost all graduate education is paid for by federal grants). The funding agencies are bemoaning the scarcity of young people interested in science when they themselves caused this scarcity by destroying science as a career.

I like his solution, but I’ve got no idea how I could ever have an impact on the policies of funding agencies. I’m not enough of a big cheese, but if you are, please help.

I conclude with this remark Katz made in his paper:

I have known more people whose lives have been ruined by getting a Ph.D. than by drugs.

The professoriate is likened to a slime mold

This post makes the point that university professors are essentially self-organizing:

(…) universities are valuable primarily as the habitat of academic disciplines, whose self-organizing systems cannot be brought under central command. Bossing academics is like herding ants — creatures that are relatively feeble unless permitted to exercise their collective genius for self-organization.

So! That’s what universities are? Habitats for disciplines… Hmmm…

Best Degree to Pair w/ a B.Sc. in Computer Science?

Slashdot is running a discussion on the Best Degree to Pair w/ a B.Sc. in Computer Science. The answers are interesting and range from a MBA to Mathematics. These two seem to be the most frequent answers. I was not surprised to the MBA there, but Mathematics was a bit more suprising. Here are two quotes from the posts:

If you want to be a tech for the long haul, perhaps a degree in mathematics.

Most pairable degree with Computer Science: Mathematics. Affinity for math tells employers you’re capable of high level, abstract thought.

This being said, I have a lot of respect for management people, but if everyone becomes a manager, who will be left to manage? The dumb people? Also, with increasingly flat hierarchies, the need for managers should not be so great. Of course, a MBA teaches you more than just management skills (I hope).

Academia: am I too much of a pessimist?

I’ve had quite a number of posts lately about how going for a Ph.D., if you don’t have all the facts about job prospects, can be a frustrating experience for many people. Sometimes I wonder if I’m not too much of a pessimist. The same way some people paint the Ph.D. track as rosy, I could equally well paint a much darker picture than needed.

Well, so far, I have yet to hear from one person who got a Ph.D. in recent years (last 15 years) who doesn’t seem to, at least silently, agree with me. Not everyone seems to agree that this knowledge should spread however. One counterargument is that the students ought to know and if they don’t, then too bad for them. Also, there is the whole line of argument about jobs being for small people: universities don’t have to worry about employment because they are above money, jobs and the economy. Well, wake up call: most people go to universities for job-related reasons. Unless you are independently wealthy or don’t mind living on welfare for the rest of your life, chances are that you are going to university to improve your job prospects. That’s the cold, hard truth.

Ok, I’ll go on to other topics soon, I promise!

Wanted: Really Smart Suckers

I missed this article by Anya Kamenetz which has a provoking subtitle: “Grad school provides exciting new road to poverty” Here’s the introduction:

Here’s an exciting career opportunity you won’t see in the classified ads. For the first six to 10 years, it pays less than $20,000 and demands superhuman levels of commitment in a Dickensian environment. Forget about marriage, a mortgage, or even Thanksgiving dinners, as the focus of your entire life narrows to the production, to exacting specifications, of a 300-page document less than a dozen people will read. Then it’s time for advancement: Apply to 50 far-flung, undesirable locations, with a 30 to 40 percent chance of being offered any position at all. You may end up living 100 miles from your spouse and commuting to three different work locations a week. You may end up $50,000 in debt, with no health insurance, feeding your kids with food stamps. If you are the luckiest out of every five entrants, you may win the profession’s ultimate prize: A comfortable middle-class job, for the rest of your life, with summers off.

Warning: I don’t know what “having summers off” is supposed to mean. At least in Canada, most professors have no more than 4 weeks of vacations. Maybe some really cool schools give you 5 weeks off… You may not have students around you, but you have to do grant applications, research, teaching preparation, administrative duties, and so on.

There are some beautiful quotes in this article, check this one out:

“Top undergraduates are arrogant; they lack perspective,” says Benton. “They’ve been fawned over all their lives, and they think grad school is there to help them realize their potential, not to use them up and toss them out.”

This one is also beautiful:

“The best phrase I’ve heard for us is the intellectual lumpenproletariat,” he says, using the Marxist term for the ground-down members of the underclass who lack the class consciousness for revolt. “If something happened to empower those people, there would be an incredible efflorescence of culture in this country, because there’s more of them now than there ever has been. But they are too busy scuttling around getting shitty jobs.”

Household opera had this to say about the article:

Seven years ago, when I entered graduate school, people were still predicting that, while the academic job market admittedly sucked like a giant Hoover vacuum, there’d eventually be tenure-track positions opening up as all the older professors finally retired. Now that it’s becoming increasingly clear that many of those positions are being replaced by adjunct jobs, I wonder if the old “they’ll retire sooner or later” argument is still in use. If (as the Voice article reports), “[g]rad school applications are up slightly over the last decade, as unemployed college grads seek a haven from the job market,” I suspect yes — though this also suggests that a lot of bright young people see grad school as a way of sitting out economically tough times.

Anyhow, I come back to my old, tired argument: be forthcoming about job possibilities. This quote from the article says it all: “I didn’t know what I was getting into. It would have been different if I had known. You’re committed to your subject. You don’t think of yourself as a 40-year-old trying to support a family.”

Using vegetables as learning objects

Tall, Dark, and Mysterious talks about Vegetables of revolution:

Well, I continued, You have a function, like say this one – at this I traced an outline of the squash with my index finger – you rotate it around the x-axis, so it traces out a three-dimensional shape, like this squash. I turned the squash in my hand, and said, That’s what I said before, “symmetric vegetables”. This one is perfectly even, see? – turn it around and it doesn’t wobble. Same with this carrot.

At this his eyes widened. Oh, yeah! he exclaimed. Cool! I remember that stuff now. He paused, and then said, less confidently, Well, kind of. He thought about it for a moment again, and then added, Probably would remember more if we’d used vegetables.

That’s it. Next time I have to teach about volumes of revolution, I’ll use vegetables.

Real knowledge management

Sean McGrath (the Propylon guy, not the New Brunswick fellow I worked with) has a nice quote about knowledge management:

Blogs and WIKIs are real knowledge management at work. Add in a change-control repository with sane URLs (like CVS with ViewCVS or Subversion for the new generation). Then add a mailling list manager with sane URLs like GNU Mailman…What have you got? A powerful set of tools for knowledge management, collaboration and information integration that use URIs the way the Gods intended – persistent, semantic identifiers of resources.

The key thing here is that we already have excellent solutions around… so if you work in knowledge management and are blind to blogs and wikis, you are lost to me. CVS is also underutilized. It is amazing how many knowledge management researchers have never heard of CVS and are constantly reinventing the idea behind it and the implementation.

Primed for Numbers

This article in the Chronicle takes on the debate as to whether women are genetically predisposed to do poorly in mathematics. I like this quote:

The researchers found that, in general, mathematically gifted females had broader abilities than did mathematically talented males.

If you ask me, this rings true. Math girls do exist: they just happen to also be good at other things than math.

And the question that never gets asked: why do we necessarily want more women in science and mathematics? Isn’t it cool that women are increasingly present in law and medecine?

In Quebec, we have more women than men in medicine, education, law and so on… would we also want to have more women in science and mathematics?

The only answer I can give is that if we had more women in science and mathematics, we would have more people altogether in science and mathematics. This would be a good thing for science and mathematics professors (including me).

Alternative careers for Ph.D. holders

Frogs and Ravens writes about an article published in the Chronicle on alternative careers for Ph.D. holders. She cites the following passages from the article:

Is there a chance that the alternative-careers movement (which in many ways I laud and admire) has unwittingly sold humanities Ph.D.’s yet another professional pipe dream? Could it be that all of us — both those still “in” academe (that is, in the professoriate) and those in the nonacademic realm — still share a misguided optimism about the marketability of a humanities Ph.D.?

(…)

I’m not sure yet about the answers to those questions. But what I am seeing is that for people like me who finished a traditionally configured graduate program, the cold truth is that translating a humanities Ph.D. into well-paying and personally satisfying employment beyond the faculty ranks is a difficult slog with an unpredictable outcome.

That’s a very insightful statement which I think can be extended to all Ph.D.s not just humanities.

The cold hard truth is that, at least in Canada, there is very little room for Ph.D. holders. There is little room in industry as well as in universities. Little room compared to what? Compared to the number of Ph.D.s produced. For example, in 2002, UQÀM had almost a hundred students in some Ph.D. programs. If everything goes well, this means a single school in Canada can produce 10 Ph.D.s a year in a given field: I doubt many fields have much more than 10 openings a year in all of Canada.

There appears to be more jobs, maybe, in government, but that’s cyclic. One day the government will be looking for 5 researchers on aluminium technology, the other day they will be looking for 5 researchers on artic climate… If you are looking for a job, I suggest looking for openings in your government. But it won’t solve all your problems: there might not be any job for you and there might be stiff competition for these jobs. Or you might not like a government job.

Now, some people have been saying that Ph.D.s should look for alternative jobs (outside universities). I’ve complained before in this pages that I failed at getting an industry job when I graduated with my Ph.D. (Nothing like using a blog to talk about one’s failures.) I think I’m a reasonably bright guy, so, no, companies are not starving for fresh Ph.D.s That’s a myth.

In fact, I was having lunch with a colleague the other day who spent many year in industry and he admitted to hiding away his Ph.D. in order to get a job. I got this advice several times.

Is this sane?

How do we fix this? Information! We have to get the word around to potential Ph.D. students about how the job market is. If you are a professor, like myself, tell your student about it. I know I did by posting on my blog and Steven (a M.Sc. student I supervise) definitively read some of my posts about jobs being hard to come by with a CS degree. Before you accept a Ph.D. student, have good talk with him. If you know people considering a Ph.D. path, tell them about what you know.

I’m not saying we should close down Ph.D. programs! But we should be honest and forthcoming about the job prospects. A university is not a Walmart. We don’t have to sell education at all cost.

Robert Paterson’s Weblog: Going Home – Our Reformation

From Downes’, I got to Going Home, an incredible blog post, read this extract:

I believe that Blogging, and its wider family of Social Software tools, will not only affect education but will shake our entire society to the core. I believe that our descendants will look back at its arrival the same way that we now look back at the advent of the printing press.

And see how you can relate it to this other extract:

Imagine, it is midnight and you have won your seat in Parliament. Everyone around you is jubilant. But you are depressed. You went into politics to make a difference. You thought that it would be all about the issues. But to win, you had to become a spin expert like all the others. Worse, you know now that you are good at it.

Or this one:

Imagine you are in hospice in Charlottetown. You are scared. You look back at your life. You did all that was expected of you. You have been a pretty good husband and dad. You had the career that your father so wanted you to have. You did him proud, ending up a senior executive of a bank. But you are so sad. You are so sad. You always loved working with wood. After you retired, you discovered that you were a cabinetmaker. And what about Jean? She was your great love but you chose duty instead and backed away. Who have you betrayed the most? You lived all those other people’s plans for your life and you have missed your own.