Making Music Streaming Smarter

Music streaming services are, by their nature, technologically sophisticated. And thanks to recent advancements, they’re now even better able to discern your musical tastes and serve up the songs you want to hear. Here are the latest developments in that field.

AI

Services such as Spotify^®, Apple Music^®, Amazon Music, Qobuz, TIDAL and Deezer offer catalogs in the neighborhood of 100 million songs. With so much music, it can often be challenging for users to decide what to listen to, which is why customized suggestions are so helpful. That’s where artificial intelligence (AI) comes in.

Some streaming services provide playlists curated by humans, but those are not personalized for each user. More commonly, they employ AI algorithms that rely heavily on the user’s listening histories to create suggestions. Recent improvements in AI, particularly Machine Learning (ML) — a subset of AI — have substantially increased the accuracy of such recommendations.

Machine Learning

Machine Learning involves training computers to analyze vast amounts of data, look for patterns, and make human-like inferences and predictions. A machine-learning algorithm continually improves its abilities as it learns more data, without requiring additional programming from a human.

Streaming services use several different ML techniques to create user recommendations. One is called collaborative filtering. This involves analyzing users’ listening patterns for commonalities, then making song and album suggestions based on them. A simplified example of the logic used in collaborative filtering would be: “Because most listeners of band A also listen to band B, recommend songs by band B to anyone who listens to band A.”

Another Machine Learning technique that Spotify and other streaming services use to categorize music before making recommendations is Natural Language Processing (NLP). This takes the metadata associated with each song (such as artist, genre, songwriter, lyrics, release date, production credits, etc.) and then searches the web for keyword matches in articles, blog posts and other text. NLP sorts through what it finds to identify connections between songs or artists that help inform its recommendations.

Yet another way streaming services implement AI is by analyzing a song’s audio for specific characteristics, a technique known as feature extraction. Tempo, genre, key, mood and instrumentation are some of the characteristics it considers. This analysis helps the streaming service accurately categorize songs, and its data adds to the precision of its recommendations.

Your streaming service also learns more about you whenever you listen to a song. It remembers your activities, such as searching for an artist, following an artist, rating a song, adding a song to a playlist, or choosing to listen to a recommended song or album. Not only does the streaming service use that data to refine its recommendations to you but also to inform its collaborative filtering algorithms.

MIIR: Finding the “Chill Phrase”

If all that sounds pretty sophisticated, just wait. An AI-based technology on the horizon will soon be making recommendations even more accurate and impactful.

This development was recently announced by a new enterprise called MIIR Audio Technologies (MIIR is an acronym for Music Intelligence Impact Retrieval), formed by a group of scientists, producers, musical artists and others. The company claims to have discovered how to find the moments in a song that affect listeners the most. They call these “Chill Phrases” — the lines or musical phrases in a song that gives the listener the greatest emotional response. “Most people can think of a phrase in a song that gives them the chills,” explains Roger Dumas, PhD, Chief Science Officer and co-inventor of the technology. MIIR finds that Chill Phrase, along with a secondary one, and then rates both according to a Chill Index.

This data will help streaming services recommend the most impactful music to their listeners, increasing customer satisfaction, but MIIR also has plenty of other potential applications. For example, the company touts its possibilities for maximizing the effectiveness of music in TV commercials and even for helping patients with dementia.

Generative AI

Other than the “big brother” ramifications of having your music-listening activity uploaded to a database, the AI techniques and applications described here are relatively benign and help the services present you with music that more accurately fits your taste.

However, there is one area in which AI and streaming music services intersect that may negatively impact artists who legitimately make royalties (small as they are) from their music getting streamed. It involves another form of artificial intelligence called Generative AI.

That technology, available to the public on websites such as Boomy, allows users to create AI-generated songs based on a few descriptive choices such as genre and mood. Such compositions are controversial for a couple of reasons.

First, these songs are created using all preexisting music as their reference. The songwriting, lyrics, arrangements and mixes of everything that’s come before become the raw materials for the generative engines. Both artists and the record industry argue that such activity is tantamount to copyright infringement.

Secondly, unscrupulous people have used AI-generated songs and bots posing as listeners to generate royalties, an activity called “artificial streaming” — something that Spotify terms a “longstanding, industry-wide issue that [we are] working to stamp out across our service.” The company recently purged many AI-generated songs from its collection, suspecting they were being used in such schemes. (Read more about the controversy.)

AI’s impact on music streaming will only increase as technology evolves. As with AI in general, we have to hope its benefits outweigh the problems it creates. One sure bet is that personalized recommendations from streaming music services will continue to get more sophisticated and accurate in the months and years ahead.