What makes recommender systems work?

Why can we predict tastes? There are several possible explanations:

  • Intrinsically, individuals have predictable tastes. To test this theory, we would need to isolate each individual. Collect their opinions. Then attempt to make predictions. (You also need to prevent the recommender system from giving feedback to the user!)
  • People are influenced by the perceived popularity of items. People tend to like a movie more when they think it is a blockbuster. A  more general statement is: Preferences are constructed in the process of elicitation.

Jon Dron linked to a 2007 New York Times article by Duncan J. Watts which claims that the second factor can easily dominate. The article is based on experiments. However, I have several arguments to support this claim.  Highly cited research papers are often no more interesting than regular papers. Except that you must also cite them, otherwise people will complain that you are missing a reference. The same is true with music or books. I often read novels based because many people read them: that is how, for example, I found la compagnie des glaces or Dune.

Hence,  collaborative filtering is a circular process:

  • People like certain items (mostly) because they perceive that similar people like them.
  • Collaborative filtering matches people with such items.

Of course, that is a bit pessimistic: people are also influenced by the intrinsic values. However, how can you differentiate the social perception from the intrinsic value of an item? Is this novel entertaining, or is it popular? Outside a laboratory, we cannot tell these factors apart!

How well would collaborative filtering work if individuals were isolated? It would work poorly. There are millions of books and research articles. Many are quite good. Without a social component to push us in some directions, we would have diverse tastes.

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

2 thoughts on “What makes recommender systems work?”

  1. I disagree in the last paragraph. Diversity is essential to predict tastes. Having a “delta function” in the data (ratings -> correlation etc.) means no information, hence no prediction power. A broad distribution of tastes always leads to better results. To predict blockbusters is an easy task, but not very useful. I claim: collaborative filtering works because of divergent tastes. The more divergent the better.

Leave a Reply to marcel Cancel reply

Your email address will not be published. Required fields are marked *

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax