GSoC’23: Integrating Apple Music with ListenBrainz

Hello everyone!

My name is Vardan and I am a student at University of Alberta. I’ve always had a thirst for knowledge and a deep desire to make contributions to projects that have a global impact. I like listening to music and the idea of contributing to an adjacent music metadata project was captivating for me. I eagerly embarked on this journey, contributed my best, and am happy to have made a meaningful impact. This experience not only aligns perfectly with my academic pursuits but also presents a chance to collaborate with some of the brightest minds in the field.

Objective

The primary objective of this project is to extend the functionality of ListenBrainz by integrating Apple Music, a music streaming service, into its ecosystem. The goal is to allow users to link their Apple Music accounts with ListenBrainz and utilize the integrated BrainzPlayer to play their music selections.

This project aims to meet the growing demand for broader music service integration within ListenBrainz, enhancing user experience and expanding the range of personalized music recommendations. Through successful implementation, this project will contribute to the platform, by catering to a wider audience and fostering a richer musical discovery experience.

Project Overview

We can divide the project mainly in two parts i.e.,

1. Music Streaming Integration: Apple provides MusicKit.js library to add Apple Music playback facilities to a website. MusicKit JS handles all authentication on its own. Neither Apple Music nor MusicKit JS provide a way to refresh expired tokens. Due to lack of proper OAuth support in Apple Music, I had to jump quite a few hoops for an implmentation that ensures smooth user experience.
   
2. Building an Apple Music Content Resolver: The metadata content resolver is used to store metadata of the entire catalog present in an external service’s database. Currently, ListenBrainz has such a resolver for Spotify. As a part of this project, I intend to add a similar content resolver for Apple Music as well.

Music Streaming Integration

1. Connect account with Apple Music Frontend: To initiate the integration of Apple Music into ListenBrainz, the first step is configuring MusicKit JS on the frontend. I added an option for users to choose to link their Apple Music account. When the user clicks the button to link their account, the MusicKit library is loaded on the webpage.
   
   
2. Developer Token and MusicKit Configuration: To configure MusicKit on the frontend, a developer token needs to be generated on the backend and sent to the frontend. After including the MusicKit library, you can configure the MusicKit instance using the generated developer token:

function music_kit_loaded_listener(token) {
  const configuration = {
    developerToken: token.access_token,
    debug: true,
    app: {
      name: "ListenBrainz",
      build: "latest",
    },
  };
  return function listener() {
    window.MusicKit.configure(configuration).then((_) => {
      const instance = window.MusicKit.getInstance();
      instance
        .authorize()
        .then((music_user_token) =>
          submit_music_user_token(
            token.access_token,
            music_user_token,
            token.expires_at
          )
        );
    });
  };
}

3. User Authentication and Token Submission: The authorize method provided by MusicKit can be utilized to authenticate the user and gain access to their Apple Music account. Upon successful authorization, the obtained Music-User-Token is sent to the server’s callback URL. However, it’s important to note that Apple Music does not require a client secret for OAuth; instead, the process is carried out client-side using MusicKit.
   
4. Backend Processing: The callback URL endpoint for music services receives the authorization code for other services. In the case of Apple Music, the OAuth process is slightly different due to the absence of a client secret. MusicKit is used to handle authorization client-side, and once the user grants access, the music.authorize().then(musicUserToken => {…}) block is executed. The retrieved Music-User-Token is then submitted to the server at the callback URL.
   
5. Storing Token and Storefront: To ensure a seamless experience, the obtained token and the user’s storefront (region) should be stored in the database for future use. This data will be crucial for accessing and customizing the user’s Apple Music content.
   
6. Integrate Apple Music in BrainzPlayer: The BrainzPlayer, which is a web-based music player used in ListenBrainz, can be extended to support Apple Music playback. MusicKit is integrated into the player in a manner similar to its inclusion in the OAuth frontend setup.
   
7. Player Controls and Event Handling: Once the BrainzPlayer is loaded with MusicKit, various playback controls like play, pause, etc., can be accessed using the MusicKitInstance. Event callbacks can also be added to respond to changes in playback state through the playbackStateWillChange and playbackStateDidChange events.
   
8. User Authorization Check: Before implementing event callbacks, it’s essential to verify whether the user has authorized Apple Music. This step ensures that only authorized users can interact with the Apple Music integration in the player.
   
9. Track Enqueue and Playback: To play tracks from Apple Music, the MusicKit player allows tracks to be enqueued using the setQueue or playNext method. The Apple Music Identifier of the desired track is used to facilitate playback.
   
10. Search and Playback: In cases where a specific track lacks an Apple Music Identifier in the ListenBrainz database, the search API can be employed to retrieve relevant tracks. Subsequently, tracks can be enqueued from the search results into the player for seamless playback within the BrainzPlayer interface.

Building an Apple Music Content Resolver

The Apple Metadata Content Resolver queries Apple Music Catalog APIs with artist/album/track identifiers to gather information about them and store it in ListenBrainz database. Its modeled on the existing Spotify Metadata Content Resolver. It took multiple passes of refactoring to get to the current implementation which shares a large amount of with Spotify Content Resolver. Hence, helping reduce redundant code.

Working with lucifer, I identified a common set of minimal functions that differed in Spotify and Apple resolvers. The rest of the code was separated into a runner or base class. In its latest implementation, Apple Music Resolver only needs to implement 6 custom methods for its entire functioning.

1. get_seed_albums – queries the Apple Music Charts API to find newly released albums ids to seed the content resolver and assist it in finding new releases.
2. The method get_items_from_listen extracts album IDs from a given “listen” data and creates a list of JobIteminstances.
3. The method get_items_from_seeder takes a message containing Apple album IDs and creates a list of JobIteminstances.
4. transform_album – converts a raw response from the Apple Music API to an Album object for storage in the ListenBrainz database. It removes a lot of the redundant relationship data and also organises tracks and artists appropriately with the Album.
5. The method fetch_albums queries Apple Music Albums API using the given Apple Music album identifiers. If needed, it uses the pagination for tracks to fetch the remaining tracks in the album. Then it tries to find new seeds for the resolver by calling discover_albums on the album artists’ identifier.
6. The method discover_albums given an Apple Music Artist identifier it queries the Apple Music Artists API to find all other albums of the artist and enqueues the discovered albums to the job queue.
   

Overall, the AppleCrawlerHandler class is designed to interact with the Apple Music API to fetch and process album and artist data, and generate job items for further processing. The class is part of a larger system or application responsible for crawling and caching Apple Music metadata.

Check out the code here Apple Music Content Resolver.

My experience & learnings

I’m truly grateful for the mentorship I received from riksucks, lucifer, and the entire MetaBrainz team. These individuals are not only excelling in their respective fields but also excelling as mentors. Contributing to ListenBrainz has been an immensely rewarding experience for me. The collaborative nature of Metabrainz team brings fun and dynamic perspectives of everyone into every discussion and suggestions flow freely.

Being part of a project that holds genuine significance is a programmer’s aspiration, I feel fortunate to have fulfilled that dream. The opportunity to contribute to a widely used project, alongside a community of seasoned contributors, has been invaluable. Working on an expansive codebase has stretched my capabilities and boosted my confidence in open-source skills.