How many users are needed for an efficient collaborative filtering system?

  • You can build an effective recommender system with as little as two people.
  • As you have more users, you tend to have more training data. Hence, you may have more accurate recommendations.
  • More accurate recommendations may not be important to your users.
  • The exact count of your users may not matter as much as the diversity of your users.
  • A good rule of thumb is that you should have many more users than you have items to recommend.
  • Given the right algorithms, your accuracy will improve monotonically with the number of users and the amount of training data.
  • The users may enter feedback data to correct the assumptions of your recommender system and thus, improve it over time.

Explanation: The title of my blog post is the subject of an email I got recently. A very popular question.

Acknowledgment: Andre inspired me to write this post.

2 thoughts on “How many users are needed for an efficient collaborative filtering system?”

  1. Daniel:

    I am interested in how you use ‘accuracy’ here – since there is no ‘right’ answer for a recommender, accuracy is hard to measure, let alone improve. I suspect that you are really talking about predicting ratings (such as one can do for the Netflix prize).

    I think that the rating prediction accuracy is a vastly overrated metric for evaluating recommender systems. This metric ignores all sorts of aspects of recommendation that can add or detract from the quality of recommendation: novelty, transparency, resistance to hacking and shilling, diversity all contribute to the quality of a recommendation.

    The canonical wisdom for CF systems is that more data is better – and if you are just predicting ratings, then I agree, but I think we’ve seen many examples of recommendation in the wild where more users result in poorer recommendations. Just look at the diversity of recommendations at sites like Digg or Last.fm. As their user base goes up, the diversity of recommendations goes down, the recommender hacks goes up, and the overall recommender experience gets worse. Look at the top 10 tracks at last.fm this week. As the size Last.fm user base has increased it has become a very homogenized music site.

    http://www.last.fm/music/+charts/track/

    (well, sorry for the rant, thanks for the interesting and provocative list).

  2. I am interested in how you use ‘accuracy’ here – since there is no ‘right’ answer for a recommender, accuracy is hard to measure, let alone improve. I suspect that you are really talking about predicting ratings (such as one can do for the Netflix prize).

    Yes. I am. And I agree with you. A friend of mine, Peter Turney, who also reads this blog, might answer something along the line that an incomplete metric is better than no metric at all.

    I think that the rating prediction accuracy is a vastly overrated metric for evaluating recommender systems. This metric ignores all sorts of aspects of recommendation that can add or detract from the quality of recommendation: novelty, transparency, resistance to hacking and shilling, diversity all contribute to the quality of a recommendation.

    I agree 100%. I have written about this on my blog in the past.

    The canonical wisdom for CF systems is that more data is better – and if you are just predicting ratings, then I agree, but I think we’ve seen many examples of recommendation in the wild where more users result in poorer recommendations. Just look at the diversity of recommendations at sites like Digg or Last.fm. As their user base goes up, the diversity of recommendations goes down, the recommender hacks goes up, and the overall recommender experience gets worse. Look at the top 10 tracks at last.fm this week. As the size Last.fm user base has increased it has become a very homogenized music site.

    Very interesting comment. And I agree.

Leave a Reply

Your email address will not be published. Required fields are marked *