Can You Guess the Artist? AI-Generated Music Catering to your Tastes
Spotify is pushing the boundaries of the AI-frontier with AI-generated playlists and even original new songs
Spotify, a music streaming app with 182 million subscribers and 422 million users that use the app at least once a month, is no stranger to the power of AI and data. Spotify is looking to change the music industry with AI generated songs, but this is not the only way the company is incorporating AI as it seeks to differentiate from other music streaming services. As a surprise to me, AI use in the music industry dates back to 2017! Sony created a hit song “Daddy’s Car” in 2017 which has to date 2.9 million views on YouTube alone. The song, created through a program called “Flow Machine” created a new original sound with features akin to the Beatles as programmed in the algorithm. Following the apparent success of this song, Spotify poached the Sony artist and producer who worked in Sony’s Computer Science Lab to head Spotify’s Creator Technology Research Lab division, its own internal AI lab. The purpose of this division, established in 2017, was to focus on making tools to help artists in their creative process.
AI through Acquisitions
Spotify gathers millions of data points on its users from which songs are streamed by users, which ones are saved, which artists/songs/albums are searched by users, etc. This data collection enables the company’s recommendation features which have over time become even more accurate reflecting user preferences. While AI’s applicability in the field of music was still nascent in the mid-2010s, companies such as Google (Magenta) and IBM (Watson) were looking into how to teach computers to recreate human creativity.
Spotify, not one to stay behind in the streaming game, has pursued a number of acquisitions since 2013 to beef up its AI capabilities. In 2013, it acquired Echo Nest bringing in-house expertise in acoustic analysis and machine learning. This led to the music curation features through personalized playlists such as Release Radar, Discover Weekly, and Daily Mix, leveraging the extensive data the company was already collecting on its millions of users. Additionally, Spotify acquired in 2017 a startup called Niland, a music technology company that provides music search and discovery engines based on deep learning & machine listening algorithms.
The integration of these acquisitions helped Spotify leverage three kinds of AI models to produce robust personalized playlists. These playlists create value for the company through increased number of streams as well as visibility to new artists by exposing them to users that might have similar music preferences to the content they produce.
AI Models Used by Spotify
1. Collaborative filtering (comparing user preferences like Netflix’s algorithms does through ratings but instead through how many users save songs, number of times a song is played or how many times users click to the artist page)
2. Natural language processing (AI scans a song’s metadata and blogposts about musicians, articles on the internet and social media to see their relevance), and
3. Audio models (analyze and categorize songs)
Evolution of AI Music and Creating Tools for Creators
Flow Machine, developed by Sony, is an AI system, that works by first analyzing a database of songs, and then following a particular musical style to create similar compositions. However, the final composition still has a touch of human skill in the arrangement of the song and the writing of the lyrics. This was the program used to create the “Daddy’s Car” song and proof that AI-generated music could in fact become a hit song, regardless of the lack of association with a known artist.
Spotify dived in head straight into the world of AI bringing in over the Sony veteran to lead its very own in-house research lab. This move likely reflected the company’s willingness to explore how AI can enhance value creation for collaborators, artists and more, beyond helping Spotify fine tune its playlist recommendations. Additionally, it means the company understood the need to have a dedicated division within the company to explore different use cases for AI and run different experiments before bringing them online into the app.
Spotify has its own online music-making studio, Soundtrap, and in 2022, the company launched an open-source AI-powered tool called Basic Pitch to help users upload audio recordings and help them convert it into MIDI (musical digital standard used by electronic musical instruments). In 2022, Benoit Carré (AI-generated music veteran) released his new album Melancholia, an 11-track collection created using the tools developed by Spotify’s lab.
Challenges Ahead
Spotify faced controversy in the music community and press after it was found many of its playlists included songs associated with fake artists that were traced back to the Swedish company. A big issue regarding AI-generated music goes back to the royalties and ability to protect intellectual property. After all, if the AI algorithm created the underlying melody, is it copyrightable? Can the artist whose “similar” melody was fed into the algorithm able to argue in a lawsuit that it deserves royalties related to those streams?
Many have criticized Spotify’s venture into AI-generated music as a way to circumvent royalties by generating songs which it would own the rights to. With the amount of data the app gathers on users, it can easily use the AI tools it is developing to create songs that would be instant hits and perhaps employ a couple of in-house songwriters to write compelling enough lyrics to accompany the AI-melodies.
Spotify is now also looking to encourage regular users to access AI tools to create their own music. For other platforms such as TikTok, Facebook and Instagram, user generated content has been key to their growth. Questions remain whether this is really going to create additional value for Spotify’s user base and the company itself.
Challenges remain in the road ahead for Spotify’s development of new use cases for AI and ML to enhance its offerings to users and artists. However, the biggest question is how Spotify will navigate the consequences of implementing these use cases vis a vis music studios and artists’ interests and the implications regarding copyright when AI models are trained on existing songs and melodies and asked to produce something similar to something that already exists.
Sources:
https://musically.com/2022/09/02/spotify-launches-ai-music-tool-basic-pitch/
https://musically.com/2022/03/22/benoit-carres-new-ai-music-album-made-with-spotify-tools/
https://www.fastcompany.com/40439000/why-did-spotify-hire-this-expert-in-music-making-ai
https://outsideinsight.com/insights/how-ai-helps-spotify-win-in-the-music-streaming-world/
Really enjoyed reading your post! I had not heard about Spotify facing controversy from playlists with songs associated with fake artists. I wonder how royalties will be paid out in the future if AI is used to make music and generate playlists. Maybe it will be a first to market allocation where the musician who produces a song with the beat / melody first gets the majority of the payments. I’ll be interested to see!
Thank you Isabella for this article! This is a space I am so curious about. I remember seeing the Michael Jackson Hologram and really being fascinated by technology. I think by piecing together all these articles, the metadata about the customers and their listening patterns and countless hours of songs these legendary artists have created, you can potentially use AI to come up with a new album from some of the greatest bands/musicians ever lived ( including Beatles, Pink Floyd etc.) but then the question still remains – will it completely replace artists and their ‘secret sauce’ ?
I think another application that Spotify can get through this data is to look at who is going to be the ‘next big artist’. They have hours and hours of data of musicians and listening behavior ( when you forward, when you rewind, how many times did you play, what specific words/instruments triggered that?) – this can be used to scrape the internet and get new artists hired to Spotify platform – in-fact in a way becoming a studio
Thanks for the post!
As a fan of Craiyon, I would love to create my very own mix of music. I’m sure Spotify is serious about its AI studio as current studios just take such a big chunk of revenue. This is unfair and borderline exploitative. Will Spotify be a better Patron?
In another discussion, would such a generative AI be a tool such as a keyboard or would the rights have to be split with the programmers of such a platform?
Super interesting post, Isabella! The fact that AI can now generate music is awesome, if not wholly unexpected. A few years back, there was an MRI study done by psychologists that showed how our brains light up to different types of music, and how that data could be fed into an algorithm to select already existing songs to queue up next. I think the use of AI is particularly clever by Spotify because, as you said in your post, it helps them avoid the massive royalties they pay the artists (something that has kept Spotify in the red until recent years).
This is awesome! I loved listening to the linked songs– thanks for sharing!
I was really struck by the idea of AI-produced or AI-enabled music. My husband is currently doing a masters in music production at the Berkelee College of Music. He’s a classical and jazz pianist and guitarist, but through this program has entered into a whole new world of electronic instruments and engineered sounds. Based on his experience, I can see so much potential for AI-produced music in the behind-the-scenes songwriting industry. Most better-known musicians don’t write their own music or lyrics. Instead, teams of musicians work together to create and effectively pitch songs for artists. Most of their pitched songs don’t go anywhere but the ones that do have great payouts. Your post makes me wonder if this songwriting industry will be replaced by AI in the next 10-20 years as these technologies improve to the point where they don’t need any human intervention to create a strong piece. Or if the skillset of those musicians creating and pitching songs will massively shift to prioritize tech skills/AI competency over traditional musicianship.
Such an interesting post, Isabella! To your question, “After all, if the AI algorithm created the underlying melody, is it copyrightable?” I feel that it definitely cannot be copyrightable. If the algorithm is parsing out popular chord sequences, melodic patterns, and/or beats from existing songs to create music, it seems to me to be a form of plagiarism… Though it’s true that artists are always inspired by each other in creating music, if the algorithm simply cannot generate music without being fed existing songs, I feel it’s hard to argue that it’s an “innocent” form of plagiarism as opposed to artists, who strive to come up with original music rather than piecing together parts that “work” from existing songs.
I’ve been an avid Spotify user since 2013, and the way its suggestions have improved since the early days is amazing. Anecdotally it feels like it used to be really hit or miss, whereas now it seems to have learned my preferences and every recommendation is good.
The question you raise about whether the artists being used to “train” the suggestion model should get royalties is a super interesting one. How does pricing for creative goods change in the face of AI?
Thank you for writing about this Isabella! I always use Spotify’s daily mixes and suggested playlists, but I had no idea what was going on behind the scenes. I’m also worried about their AI-created music.. I feel that music creation should be left to individual artists and songwriters to preserve authenticity and our humanity! I would hate to have generic music created by AI dominate the music scene in 10-15 years. Usually AI is very exciting, but this particular use case is frightening to me.