How to devise the perfect recommendation algorithm

AT LAST YEAR’S consumer-electronics show in Las Vegas, Reed Hastings, the CEO of Netflix, set out an ambitious goal for serving his customers: “One day we hope to get so good at suggestions that we’re able to show you exactly the right film or TV show for your mood when you turn on Netflix.”

But what is exactly the right show? Mr Hastings’s company has been a pioneer in the science of recommendation algorithms, dating back to its days as a humble DVD-by-mail company. Netflix’s thinly sliced classifications of films and TV shows, and its equally finely graded assessments of customers’ viewing preferences, established the standard for product suggestions.

Still, algorithms take some of the adventure and serendipity out of hunting for new entertainment, and rarely nudge a customer towards anything way off his radar. This is a challenge for independent producers of music, literature or film, who already find it extremely difficult to get noticed amidst so much choice. Recommendation software can make the problem worse.

Suggestion algorithms can exploit what customers are known to like by pushing similar fare, or they can encourage them to explore things they might be less familiar with. Typical algorithms tend to exploit known preferences more than encourage exploration. When a customer buys a book, for instance, Amazon will recommend books on similar subjects that previous shoppers have also bought. Netflix will suggest a show based on the choices of other people with similar viewing histories.

Some recommendations are fairly crude, as when an online store repeatedly offers more of something the consumer has already bought and is unlikely to want more of, like an umbrella. Others are more adventurous, encouraging the customer to try something new. But if they go too far, they risk putting him off. “It’s more predictable to use similar recommendations over and over to get the engagement,” says Robert Kyncl, chief business officer of YouTube. “But the pay-off is much greater when you introduce something that is a bit of an odd choice and it works.”

Spotify, a music-streaming service, offers a different model with its Discover Weekly playlist, which it produces for more than 100m customers. It analyses billions of users’ playlists to find songs that others with similar interests have liked. These tracks are combined into a playlist of 30 songs (perhaps including some familiar as well as new ones) delivered each Monday.

The company says the service is used by tens of millions of listeners and gives a significant boost to thousands of performers each week. By limiting the list to a couple of hours-worth of listening, and by setting an expiration date each week, Spotify creates a sense of scarcity to keep listeners engaged.

Music lends itself well to this treatment. Streaming services have catalogues of around 30m songs each, compared with mere thousands of film and TV titles. And songs, unlike films, are short enough for a poor recommendation not to matter much. Spotify’s experience shows that algorithms can occasionally nudge people away from hits and expand their horizons.

Stay tuned. If you liked this article, read the next one.