Measuring the diversity of recommended lists, at last

For a number of years, algorithm researchers in collaborative filtering and recommender systems have focused on accuracy as the sole performance metric.

Imagine that you bought a couple of albums from Celine Dion and you liked them a lot. Then the best answer might be to suggest you buy all the other Dion albums. Or is it?

No. You do not want to optimize accuracy above all else. You need to balance accuracy and diversity. To any user, it is obvious. Researchers often prefer to ignore diversity because it is harder to measure.

Several people, me included, have argued in favour of diversity, but metric proposals were still missing. I have had on my to-do list to write a paper on measuring the diversity of recommender systems. Unfortunately, I cannot cope with more than a few projects at any one time. Fortunately, it looks like I will not have to write this paper. Zhang and Hurley have done a good job at it:

Zhang, M. and Hurley, N. 2008. Avoiding monotony: improving the diversity of recommendation lists. In Proceedings of the 2008 ACM Conference on Recommender Systems (Lausanne, Switzerland, October 23 – 25, 2008). RecSys ’08. ACM, New York, NY, 123-130.

Do not be put off by the mathematics: they are a tad formal, but the right ideas are there, just read slowly sections 3 and 4. Basically, diversity is measured as the average dissimilarity between items. That is a standard form of diversity measure. This strategy to measure diversity is not novel, but to my knowledge, they are the first to apply it to collaborative filtering.

What is next? You are looking for a paper idea?

  • Take Zhang and Hurley’s class of diversity measures, and apply them to existing recommender systems. Show that there is an accuracy-precision trade-off. All you need is a dissimilarity measure between items.
  • Do user studies to prove people prefer a balance between diversity and accuracy.

Requirement: if you steal anyone of these ideas, you have to email me a copy of your paper once it is written.

Daniel Lemire, "Measuring the diversity of recommended lists, at last," in Daniel Lemire's blog, November 14, 2008.

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

One thought on “Measuring the diversity of recommended lists, at last”

  1. Your seems related to a book I just started reading called “The Cult of Statistical Significance.” The authors say we should ask “how much” questions more often than “whether” questions. So I think they would agree that asking how sure you are that someone likes Celine Dion is not as important as asking how much they like her, or in terms of diversity, how much they like artists like her.

Leave a Reply

Your email address will not be published.

You may subscribe to this blog by email.