GSoC 2019: Recording Similarity Indexing for AcousticBrainz

For Starters… Who Am I?

My name is Aidan Lawford-Wickham, better known as aidanlw17 on IRC, and I’m entering my second year of undergraduate study in Engineering Science at the University of Toronto. This summer, I had the opportunity to participate in my first Google Summer of Code with the MetaBrainz Foundation. Working on the AcousticBrainz project under the mentorship of Alastair Porter (alastairp), I used previous work on measuring track to track similarity as the basis for a similarity pipeline using the entire AB database.

How Did I Get Involved?

When I started applying for GSoC, I needed to find an organization that paired a challenging learning environment with a project of personal interest. Given my own passion for listening to music, playing music, and exploring its overlap with culture, MetaBrainz quickly became my top priority. I jumped on the #metabrainz IRC channel for the first time, and I’ve been active daily ever since!

From there, the whole community welcomed me with open arms and responded thoughtfully to my questions about setting up my local development environment. I made my first pull request for AcousticBrainz, AB-387, which added the ability to include dataset and class descriptions when importing datasets as CSV files. This allowed me to work alongside my soon-to-be mentor for the first time and further acquaint myself with the acousticbrainz-server source code.

I was excited about my first PR and wanted to contribute more. Not only was this a project related to my passions, but it had already begun to teach me about technologies that I hadn’t used before. I was struck by the possibility to contribute more, and work with great people on a non-profit, open source project. I quickly decided that MetaBrainz was the only place I would apply for GSoC and began to think about proposals. I read through the previous work on recording similarity done by Philip Tovstogan, which was based upon a PostgreSQL solution with shortcomings in terms of speed. With a strong supporting background, high community interest, and my own dreams of the possibilities to come from predicting similar tracks, I created a proposal to build a similarity pipeline using Spotify’s nearest neighbours library, Annoy. The timeline and tasks shown on the full proposal were adjusted throughout the summer, but the general objectives were maintained. Looking back on the summer now, the basic requirements for the project were as such:

  • Using the previous work, define metrics for measuring similarity that will translate recording features from the AB database into vectors. Compute and store these vectors for every recording in the database.
  • Create an Annoy index for each of these metrics, adding the metric’s vector for each recording to the index.
  • Develop methods of querying an index, such as outputting nearest neighbours (similar recordings) to a specific recording or many recordings, or finding the similarity between two recordings.
  • Allow users to query the indices via an API.
  • Create an evaluation that allows us to measure the success of our indices in the public eye, fine tune our parameters, and display index queries via a graphical user interface.

Community Bonding Period

After losing sleep before the announcement, and a huge sigh of relief on May 6th, I was ecstatic to get started.

There was plenty of required reading, and I familiarized myself with the different elements of building similarity into AB. After discussing with Rob (ruaok) and Alastair and cementing our decision to use Annoy as the nearest neighbours algorithm of choice, I took to reading through Annoy documentation and making a small implementation to grasp the concepts. Annoy works blazing fast, and uses small, static files – these are points that would prove advantageous for us in terms of querying indices many times, as quickly as possible. Static index files allow for them to be shared across processes and could potentially make them simple to redistribute to others in the future – a major benefit for further similarity research.

I studied Philip’s previous work, gained an understanding of the metrics he used in his thesis, and reimplemented all of his code to better grasp the concepts and use them as a basis for the summer. Much of Philip’s work was built to be easily expandable, and flexible to different types of metrics. Notably, when integrating it with a full pipeline including Annoy, priorities like speed meant that we lost some of this flexibility. I found this to be an interesting contrast between the code structure for an ongoing research purpose, and the code ready to be deployed in production on a website.

All the while, I kept a frequent dialogue with Alastair to gel as a team, clarify issues with the codebase, and further develop our plans for the pipeline. To build on my development skills, learn more about contributing guidelines and source control, and improve the site, I worked on some exciting PRs during the bonding period. Most notably, I completed AB-406 over a series of 3 PRs, which allowed us to introduce a submission offset column in the low-level table to handle multiple submissions of a single recording. This reduced the need for complexity in queries to the API, decreasing the load on the server. Additionally, I added some documentation related to contributions and created an API endpoint that would allow users to only select specific features rather than an entire low-level document for a recording – aiming at reducing server load.

Last but not least, I got really involved with the weekly meetings at MB! We have meetings every Monday on #metabrainz to give reviews of the last week, and discuss any other important community topics. I love this aspect of the community. Working remotely, it creates a strong team atmosphere and brings us all a bit closer together – even if we’re living time zones apart. During one meeting, we discussed whether or not past GSoC proposals should be available to students. What do you think? This prompted me to share my own experience with the application process at MetaBrainz and look into if/how we could improve it.

… And so it began, we dove into the first coding period.

The Key Components, a Deeper Look

Computing Similarity Metrics

Having explored the previous similarity work from Philip, I used his definitions of metric classes and focused on developing a script to compute metrics for each recording in the database incrementally. Recognizing that we would also need a method of computing metrics for a single recording on submission, I made this script as open ended as possible. After successfully computing all metrics for the first time, we went through an iterative process of altering the logic and methodology to dramatically improve its speed. Ultimately, we used a query to get the batch of low-level recordings that haven’t had similarity computations, complete with their low-level data and all high-level models. Though we revised and found bugs in this script time and time again, I’m confident in saying that with perseverance we finally got it working.

Prior to the beginning of the project I had limited experience working with SQL databases, and this objective pushed me to develop new ways to approach problems, and gave me a much deeper understanding of PostgreSQL.

Building Annoy Indices

With all that vectorized recording data from the metrics computation, nothing sounds better than adding it to an ultra-fast index built for querying nearest neighbours! Feeding the data into an index and watching it output similar recordings in milliseconds became the most satisfying feeling. The Annoy library is a platform for nearest neighbours of all sorts, and it is generally simple: define the index, add items with an identifier and a vector, built the index, save it for later use, load it up, and then use its built-in methods to query for similar items. Easy, right? The added challenge is making this interface with recordings from our database as items, and meeting our needs in terms of speed and alterability when new items are added. Annoy is built without checks in many places, and we required a custom cycle of building, loading, and saving indices to ensure they were operable for our purposes (once an index is built, new items may not be added). At this point, the index model is open to saving new indices with different parameters, which allows us to tune as we further develop the pipeline.

After wrapping the index in a class that interfaced with our needs, we added scripts to build all indices and save them, and scripts to remove indices if need be. Currently, the project has 12 indices, one for each metric in use:

  • MFCCs
  • Weighted MFCCs
  • GFCCs
  • Weighted GFCCs
  • BPM
  • Key
  • Onset Rate
  • Moods
  • Instruments
  • Dortmund
  • Rosamerica
  • Tzanetakis

API Endpoints

Making API endpoints available was a high priority activity and was an exciting aspect of the project since it would allow users to interact with the data provided by a similarity pipeline. Using the index model, I created three API endpoints:

  • Get the n most similar recordings to a recording specified by an (MBID, offset) combination.
  • Get the n most similar recordings to a number of recordings that are specified (bulk endpoint).
  • Get the distance between two recordings.

For each endpoint, a parameter indicates the metric in question, determining which index should be used. Currently, the endpoints also allow varying index parameters, such as the distance type (method of distance calculation) and number of trees used in building the index (precision increases with trees, while speed decreases).

A full explanation of the API endpoints is documented in the source code.

Baseline Evaluation

As I said, an index can be altered using multiple parameters that impact the build speed, query speed, and precision in finding nearest neighbours. Assessing the query results from our indices with public opinion is a top priority, since it gives us valuable data for understanding the quality of similarity predictions. With the evaluation we will be able to collect feedback from the community on a set of similar recordings – do they seem accurate, or should a recording have been more or less similar? What recording do you think is the most similar? With this sort of feedback, we can measure the success of different parameters for Annoy, eventually optimizing our results.

Moreover, this form of evaluation provides a graphical user interface to interact with similar recordings, as a user-friendly alternative to the API endpoints. Written using React, it feels snappy and fast, and I feel that it provides a pleasing display of similar recordings. At this point in the project I was glad to accept a frontend challenge which differed from the bulk of my work thus far.

Documentation and Project Links

Similarity pipeline related:

Additional work:

Going Forward

This summer allowed for us to build on previous similarity work to the point of developing a fast, full pipeline. At this point, there is still a vast amount of work to be continued on the pipeline and I am eager to see it through. In the upcoming year I plan to continue contributing to AcousticBrainz and the MetaBrainz Foundation as a whole. These are areas that I’m interested in continuing to develop for the recording similarity pipeline:

  • Parameter tuning on Annoy indices
  • Adding more metrics to cover other recording features
  • Adding support for hybrid metrics that consider multiple features (this was started by Philip and should be integrated to provide more holistic similarity)
  • Making indices available for offline use
  • Creating statistics and visualizations of vectors for each metric

Wrapping Up

To say the least, this has been a highly rewarding experience. MetaBrainz is a community full of extraordinary, thoughtful, and friendly developers and enthusiasts. I will be forever thankful for this opportunity and the lessons that I gained this summer. I am so excited to meet everyone at the summit this September! I’d like to personally thank my mentor, Alastair Porter (alastairp), for his perceptive guidance, his support, his friendship, and his own contributions to the project. Thanks to Robert Kaye (ruaok) for his support, thoughts, and enthusiasm towards this project, as well as for his dedication to MetaBrainz. Thanks to Google for making this all possible – SoC is a highly unique opportunity to learn about open source software and make new connections! Cheers.

GSoC 2019: JSON Web API for BookBrainz

The time has come to wrap up the very productive and learning summer of the last 3 months as a GSoC student with MetaBrainz.

Hello Everyone!!

I am Akhilesh Kumar, a recent graduate from the National Institute of Technology, Hamirpur, India. I have been working on BookBrainz for MetaBrainz Foundation Inc. as a participant in the Google Summer of Code ’19. It has been an amazing experience and I’ve learned a lot over the summer. I was mentored by Nicolas Pelletier (Mr_Monkey on IRC) during this period. This post summarizes my contributions to the project and the experiences that I had throughout the summer.

Continue reading “GSoC 2019: JSON Web API for BookBrainz”

Picard 2.1 release

MusicBrainz Picard 2.1 is finally here. This version includes a lot of fixes and improvements over previous stable version (2.0.4). It is recommended to upgrade of course.

Most notably, it includes following changes:

  • support for MusicBrainz genres
  • new convenient script functions: $title and $delete
  • new command line option (-P, --no-plugins) allowing to skip plugin loading, which may help when a plugin is crashing Picard at start and to identify if an issue is due to a plugin or the main program
  • improve tags support, better compatibility with other applications, wider range of audio files supported
  • better stability and performance

I’d like to thank all developers, translators, testers and users who contributed to this version, kudos to Philip Wolfer (phw).

Binary packages from OSx and Windows are available from Picard Github releases page and from Picard website.

Here is the complete change log:

  • [PICARD-105] – Picard won’t load non-album tracks from fingerprints
  • [PICARD-421] – Releases in private collections are not shown as being in them
  • [PICARD-518] – Sliders without labels in “Options – Metadata – Preferred Releases”
  • [PICARD-637] – $matchedtracks is broken
  • [PICARD-875] – AIFF does not support any of the compatid3 tags
  • [PICARD-949] – Track can be placed in the incorrect spot on the release after using Scan
  • [PICARD-1013] – False file save error in specific circumstances
  • [PICARD-1060] – Collections menu not displayed correctly anymore
  • [PICARD-1112] – Cannot save tags that were previously deleted from file
  • [PICARD-1133] – Plugins list doesn’t load automatically after setting proxy
  • [PICARD-1162] – Solo vocals are tagged wrong
  • [PICARD-1219] – Picard creating empty ID3 TIPL / TMCL / IPLS frames
  • [PICARD-1245] – Set field “Grouping” doesn’t work as expected
  • [PICARD-1275] – After uninstalling a plugin Picard needs to be restarted for it to be reinstalled
  • [PICARD-1281] – Picard has wrong version string on macOS
  • [PICARD-1320] – Black text on a dark theme
  • [PICARD-1332] – Deleted tags for matched files stay deleted
  • [PICARD-1336] – MP4 reports “bpm” as unsupported tag
  • [PICARD-1339] – Removing unclustered files can be very slow
  • [PICARD-1340] – File info doesn’t display Mono / Stereo in Channels field anymore
  • [PICARD-1341] – Cluster track order misinterprets disc/track numbers
  • [PICARD-1346] – Move additional files fails if multiple patterns match
  • [PICARD-1348] – Keyboard shortcuts broken due to localization
  • [PICARD-1350] – Drag and drop on cover image box does not always work as expected
  • [PICARD-1355] – Setting or unsetting album for non-album tracks does not work
  • [PICARD-1359] – Crash with tagger integration when using DuckDuckGo Privacy Essentials
  • [PICARD-1364] – picard.exe has no version tag
  • [PICARD-1368] – Info messages are not shown on logging level Info
  • [PICARD-1369] – Crash on Python 3.7.0 opening URLs
  • [PICARD-1370] – Windows installer to add “Quick Launch” icon no longer supported on Win10
  • [PICARD-1371] – Windows installer does not warn when installing on 32 bit system
  • [PICARD-1373] – Source distributions are unusable
  • [PICARD-1374] – Picard crashes while typing a regular expression in some cases
  • [PICARD-1375] – Metadata sanitation before move-script execution fails
  • [PICARD-1376] – Error saving Ape files with tag marked for deletion that does not exist
  • [PICARD-1381] – Test results depend on execution order of tests
  • [PICARD-1397] – Do not save tags marked as unsupported
  • [PICARD-1398] – Snap package is missing locale files
  • [PICARD-1405] – Pasting formatted text into scripting window shows formatting
  • [PICARD-1410] – Loading Vorbis file with invalid rating value fails
  • [PICARD-1412] – Deleting tag counts not as important metadata change
  • [PICARD-1414] – Image errors lead to crash in info dialog
  • [PICARD-1415] – Open Containing Folder and Open with MusicPlayer does nothing for UNC paths
  • [PICARD-1418] – Display localized default dialogs and keyboard shortcut hints
  • [PICARD-1420] – Can not save wma file. TypeError: sequence item 0
  • [PICARD-1428] – Removing tags which are only in original file metadata not possible
  • [PICARD-1430] – “Authentication required” dialog does not trigger authentication
  • [PICARD-1431] – Some ID3 frames gets deleted even if the corresponding tags are shown as unchanged
  • [PICARD-1434] – Tag acoustid_id can not be removed or deleted in script
  • [PICARD-1436] – Text extraction of “title” and “label” for translation.
  • [PICARD-1437] – After reload file is being shown as changed
  • [PICARD-1438] – Message box buttons Yes/No aren’t translated
  • [PICARD-1439] – Newline character in cover art naming script causes exception on saving

New Feature

  • [PICARD-490] – Allow tagging AAC/ADTS files with APEv2 tags
  • [PICARD-1043] – Support reading & writing iTunes Classical tags
  • [PICARD-1045] – Check for new version
  • [PICARD-1268] – Support concertmaster recording relationships as performer:concertmaster
  • [PICARD-1273] – Add an option to exclude new cover art type “Raw / Unedited”
  • [PICARD-1319] – Provide cover art metadata to cover image naming script
  • [PICARD-1344] – Add $delete function
  • [PICARD-1352] – Add a command-line option to skip plugin loading
  • [PICARD-1354] – Allow using vocals and instruments as credited
  • [PICARD-1367] – Allow opening searches in browser when using search dialogs
  • [PICARD-1384] – Add AppStream data
  • [PICARD-1386] – Add $title function
  • [PICARD-1395] – Support genres from MusicBrainz
  • [PICARD-1440] – Support loading and renaming Standard MIDI Files (SMF)

Task

  • [PICARD-1333] – Run CI tests agaist oldest supported mutagen
  • [PICARD-1347] – Refactor script.py to avoid code duplication
  • [PICARD-1365] – Allow building with PyQt 5.11 or later
  • [PICARD-1442] – Support new Audio Play secondary type

Sub-task

  • [PICARD-1407] – Save originalalbum / originalartist to ASF/WMA
  • [PICARD-1408] – Save originalalbum / originalartist to APE

Improvement

  • [PICARD-664] – When dragging a recording, show the actual file name instead of the path
  • [PICARD-792] – Package a start menu tile for Windows 10 on the windows version
  • [PICARD-1039] – Use forward delete instead of delete button on macOS
  • [PICARD-1049] – Picard should use TXXX:WORK rather than TXXX:Work
  • [PICARD-1068] – Picard should use MP4 ©wrk for Work rather than generic text field
  • [PICARD-1244] – Refresh list of plugins after uninstalling or installing a local plugin
  • [PICARD-1285] – There is no Close menu item in Picard 2.0 on macOS
  • [PICARD-1313] – Refactor plugin UI
  • [PICARD-1325] – Allow disabling new version update checking for packagers
  • [PICARD-1338] – Picard should be more resilient if it gets invalid responses from servers
  • [PICARD-1358] – Use macOS style widgets in the user interface of the macOS version of Picard
  • [PICARD-1363] – AcoustId submission for matched files is impossible when musicbrainz_recordingid is unset
  • [PICARD-1366] – Show Python version in about
  • [PICARD-1379] – Port astrcmp to new Python C Unicode API
  • [PICARD-1383] – Use MCN / barcode read from disc to improve DiscId lookup
  • [PICARD-1393] – Change the application ID
  • [PICARD-1416] – Should store ID3 Artists field as TXXX:ARTISTS not TXXX:Artists
  • [PICARD-1417] – Only show plugins with compatible API version
  • [PICARD-1424] – Translate AppStream data
  • [PICARD-1425] – Support all movement tags for APE, Vorbis and MP3
  • [PICARD-1426] – Map musicbrainz_originalalbumid and musicbrainz_originalartistid to MP4 and WMA
  • [PICARD-1443] – Sort secondary release types in UI alphabetically

Import your listens to ListenBrainz from Spotify!

Hullo!

We’ve been working on a system to import listens automatically to ListenBrainz from Spotify and we’ve recently deployed it to the ListenBrainz beta site. We would really appreciate it if you could help us test it out!

Please note that this is still beta software, there is a (very small) chance that we might miss a listen or two. So if you’re using this, please make sure that ListenBrainz is not the only service where you’re archiving your listens.

Another thing to note is that importing the same listens from two different sources such as Last.FM and Spotify may cause the creation of duplicates in your listen history. If you opt into our automatic Spotify import, please do not use the Last.FM import or submit listens from other ListenBrainz clients. This is a temporary limitation while we find better ways to deduplicate listens.

That’s it for the caveats, please go ahead and use the new shiny Spotify Importer. And feel free to report bugs on tickets.metabrainz.org or on IRC in #metabrainz on Freenode.

Thanks!

How to set up a ListenBrainz development environment

One of the first rites of passage when working on a new project is creating your development environment. It always seems simple, but sometimes there are bumps along the way. The first activity I did to begin contributing to ListenBrainz was create my development environment. I wasn’t successful with the documentation in the README, so I had to play around and work with the project before I was even running it.

The first part of this post details how to set up your own development environment. Then, the second half talks about the solution I came up with and my first contribution back to the project.

Continue reading “How to set up a ListenBrainz development environment”

MetaBrainz team changes, autumn 2018

Hello!

The only constant in the world is change, right?

First off, the somewhat sad news: Sambhav, AKA samj1912, has left MetaBrainz the team as a contractor and has moved to London. The upside of this news is that he will continue to work on Picard for us and will remain a part of our team as a volunteer, but his presence will not be quite as intense as before. Thank you for your hard work these past months, especially for finishing the impossible Solr search project!

With Sambhav’s departure and our improved finances, I’m proud to announce that we’re taking on two new contractors!

Nicolas Pelletier AKA Monkey: You may remember the talented Monkey from when we designed our new logos. He was the designer who created the logos and our new bootstrap theme that adorns most of our pages now. Working with Monkey was straightforward, effective and the results were great, so when he expressed interest in working on BookBrainz, I was pleased to hear this news. Monkey will be working for us full time and spending 75% of his time on BookBrainz and 25% of his time to help with design and UX work for the rest of our projects. In the next blog post I’ll talk more about BookBrainz and what we can expect from that project in the future.

Nicolás Tamargo AKA Reosarevok: Reosarevok is no stranger to our community — he’s made 1.7M edits to MusicBrainz, is our Style BDFL and answers all of our support@ emails. He’s been learning more programming and asked to be part of the MusicBrainz team part time. We agreed to give this a go and in the short term he will be focusing on genre support and helping with the React migration among other tasks. If this trial run works out, we’ll see about expanding his scope on our team.

Welcome on board Monkey and good luck with the new position, Reo!