Spotify: Music Discovery in a World of Discover Weekly
Machine learning has underpinned Spotify's foray into personalized recommendations, distinguishing from earlier players in the arena. How good are its formulae and what are the broader implications of programming taste?
The music industry today is almost unrecognizable when compared against the norm prior to the turn of the millennium. Sales of physical records gave way to piracy. Piracy gave way to digital distribution (notably championed by Apple’s iTunes). Digital distribution gave way to streaming.
With 191 million total monthly active users, including 87 million premium subscribers, as of 2018 Q3 [1], Spotify is the undisputed leader of music streaming today. But in the face of stiff competition from online radios like Pandora and Last.FM, and other deep-pocketed streamers like Apple Music or even YouTube, Spotify’s early mover advantage is not in and of itself sufficient to guarantee its future success.
Finding an “Edge” in the Brave New World of Streaming
While music is not a commodity, the provision of music can be likened to one; and the various music streaming platforms are ultimately undifferentiated so long as their catalogues are similar. As the music streaming industry matured and became more widely accepted by both producers and consumers, music streamers have managed to sign up the same suite of major record producers (Universal Music, Warner Music, and Sony Music).
This makes finding one’s “competitive edge” of critical importance for the players in the music streaming industry. On the one hand, this could take the form of competitive chicken a la aggressive pricing or service bundling, or occasional early releases or exclusives with a popular artist. On the other hand, streamers could provide a better service in the form of music curation, i.e. recommending songs to users based on their listening habits.
In the 2000s and early 2010s, this was done manually: For example, Songza manually curated playlists using human “music experts” that users could then listen to; Pandora manually tagged song attributes to each song so that similar songs could be bundled. [2] The downside to human-driven recommendations, however, is that they cannot preclude the curators’ biases, and a manually-intensive process ultimately also means a highly costly process.
Enter Spotify
Spotify, however, has taken the lead in using machine learning techniques to generate highly personalized recommendations tailored to each user, most notably through its Discover Weekly function, first released in July 2015. The foundation of Spotify’s personalized recommendation engine lies in a few key technologies.
Spotify handles this problem using two methods. Echo Nest, a music analytics Spotify acquired in 2014, uses natural language processing to analyze the verbal content of songs and how artists and songs are described online. [4] Convolutional neural networks (“convnets”) applies deep learning technology used in image recognition to analyze audio tracks, thereby identifying high-level “traits” in each song such as tempo, key, mode, time signature, and many others. The result is a prediction for how a song could perform under collaborative filtering, even without there being any historical usage data. [5]
Using machine learning techniques, a “taste profile” is developed for every user, and every song is classified into one or more “microgenres”. The output from these analyses is then cross-referenced with collaborative filtering in another layer of algorithms. This is the foundation for Spotify’s personal recommendation methodology.
What are the results of Spotify’s personalized recommendations? For starters, users seem to love it. A quick search through social media platforms reveals a bounty of effusive praise.
Others approached this from a more critical lens. In a 2017 empirical study on Spotify Radio, it was found that liking, disliking, and skipping do not seem to materially impact the recommendation queue. Moreover, similar artists and songs tend to reappear repeatedly in any given radio, and this is reflected in more anecdotal feedback on the product. Clearly, Spotify has a ways to go in improving the quality of this service. [6]
In another 2017 study on gendered differences in Spotify’s personalized recommendations, it was noted that, despite gender being a mandatory input for account creation, there did not appear to be any difference in the proportion of male/female artists recommended to male/female bots; nor were there any major differences in the artists being recommended (regardless of artist gender). However, 80% of recommended artists were male, whereas just 15% were female. This broadly tracks the music industry’s gender imbalance. [7] Should gender be a variable in making recommendations? Does Spotify have a role in correcting the massive gender imbalance in the music production industry? More broadly speaking, what other factors should Spotify be considering or controlling for in its recommendation algorithms?
(800 words)
Citations:
[1] Spotify, 2018 Q3 Quarterly Report (Luxembourg: Spotify, 2018).
[2] “The magic that makes Spotify’s Discover Weekly playlists so damn good.” Quartz. 21 Dec 2015.
[3] “Ever Wonder How Spotify Discover Weekly Works? Data Science.” Galvanize. 22 Aug 2016.
[4] “How Does Spotify Know You So Well?” Medium. 10 Oct 2017.
[5] “Recommending music on Spotify with deep learning.” Sander Dieleman. 5 Aug 2014.
[6] Pelle Snickars. (2017). More of the Same – On Spotify Radio. Culture Unbound: Journal of Current Cultural Research, 9(2), 184-211.
[7] Eriksson, Maria, & Johansson, Anna. (2017). Tracking Gendered Streams. Culture Unbound. Journal Of Current Cultural Research, 9(2), 163-183.
Other sources:
Fleischer, Rasmus, & Snickars, Pelle. (2017). Discovering Spotify – A Thematic Introduction. Culture Unbound. Journal Of Current Cultural Research, 9(2), 130-145.
Tang, M., & Yang, M. (2017). Evaluating Music Discovery Tools on Spotify: The Role of User Preference Characteristics. Tushu Zixun Xuekan, 15(1), 1-16.
“Machine learning at Spotify: You are what you stream.” O’Reilly. 7 Dec 2017.
“How did Spotify Get So Good at Machine Learning?” Forbes. 20 Feb 2017.
Thank you for sharing the views on machine learning in Spotify. I really enjoyed the article. I found it very interesting how Spotify is able utilize machine learning to differentiate their offerings in the Discover Weekly playlist. I hadn’t heard of the gender bias in Spotify’s results until reading this article. I wonder if this highlights an area where human supervision is necessary to oversee all areas of machine learning.
This is a very well written and informative overview of Spotify’s approach to machine learning, especially the challenge / current (partial) solution to the Cold Start problem. Thank you!!
Regarding the issue of gender imbalances – if I was a manager at Spotify, I would be concerned that “artificially” increasing gender representation to proportions not represented in the supply of musicians has the potential to decrease customer satisfaction. I would think that increasing gender diversity at the top of the funnel would be a more natural way of improving gender diversity and would be more likely to be successful with consumers long-term.
Thanks for sharing! Like Ennis, I also had not heard of this problem in Spotify’s recommendations and am surprised at how large the gender imbalance in the music industry is. It seems like a problem that machine learning cannot necessarily tackle, at least in this context. Perhaps machine learning can begin to be used by record labels to predict whether new/emerging artists will be successful, but since there is already a gender imbalance I’m not sure that would help. I think an appropriate next step for Spotify could be to create more playlists featuring female artists so that listeners are conditioned to listen and enjoy more female voices, eventually generating more demand for female vocals.
This is a great example of how machine learning does lead to smarter products with not-so-smart outcomes. To address the gender bias, I would think the teams are trying to compensate and attract a more evenly split amount of listeners to have more representative data. While gender seems to bias the information, imagine all the other elements and factors that inform music preferences…and whether or not those are being misrepresented! Furthermore, the high reliance on machine learning for Spotify may be a liability in the long term, especially given the rising competition in the space. In order to remain competitive, I wonder how Spotify will grow its business to remain relevant — will it look into building hardware? or expand into other forms of media like movies?
This is a very interesting question regarding gender biases. This is an issue that has plagued the music industry for a long time, and as Farrah mentioned, it does need to start at the top of the funnel with the industry giving more opportunities to women at a young age. However, I think Spotify has the opportunity to take a strong social stance on this topic by altering its algorithm to balance the gender ratio. While this may have an impact on customer satisfaction in the short term, this would be a long term move that is likely in line with the values of Spotify’s younger demographic and the trend towards more female artists. Especially as other players in the space commoditize the provision of music, building a brand that resonates with consumers is crucial.