Rethinking Recommendation Engines
Wednesday, April 2nd, 2008
Over two years ago, Netflix announced a Recommendation Engine contest - anyone who invents an algorithm that does 10% better than their current recommendation system will win $1 Million dollars. Many research teams raced to attack the problem, excited by the unprecedented amount of data available. Initially quite a lot of progress was made, but then slowly the progress stalled and now teams are stuck at around the 8.5% improvement mark.
In this post we argue that the improvement in recommendation engines is not an algorithmic problem, but rather a presentation issue. Respinning recommendations as filters and delivering them without setting high expectations is more likely to yield progress than crunching more data faster.Building a recommendation engine is a complex endeavor, which we discussed here a year ago. But in addition to being a technical challenge, there are also fundamental psychological questions: do people want recommendations and if so, then when are they open to them? Perhaps an even bigger question is: what happens when the user receives one or more bad recommendations? How tolerant will they be?
(more…)
