End-of-life for external access to MusicBrainz search server

For MusicBrainz mirrors, MusicBrainz server development, and custom applications, there used to be a direct access to search.musicbrainz.org which was actually inherited from the previous search infrastructure until 2018.

Keeping it available in the current infrastructure uses a compatibility layer which has been unintentionally broken last year. This issue has been reported only once. The alternative and much more sensible solution which is to install a local search server has improved in reliability and simplicity in the meantime. Polling the community shown that only one person would be interested which isn’t enough to justify the maintenance overhead for this replaceable service.

Therefore it has been decided to close this access for good.

Thanks to everyone who made this transitioning service to work and handle about 40k requests a day on average!

MetaBrainz Summit 2022

The silliest, and thus best, group photo from the summit. Left to right: Aerozol, Monkey, Mayhem, Atj, lucifer (laptop), yvanzo, alastairp, Bitmap, Zas, akshaaatt

After a two-year break, in-person summits made their grand return in 2022! Contributors from all corners of the globe visited the Barcelona HQ to eat delicious local food, sample Monkey and alastairp’s beer, marvel at the architecture, try Mayhem’s cocktail robot, savour New Zealand and Irish chocolates, munch on delicious Indian snacks, and learn about the excellent Spanish culture of sleeping in. As well as, believe it or not, getting “work” done – recapping the last year, and planning, discussing, and getting excited about the future of MetaBrainz and its projects.

We also had some of the team join us via Stream; Freso (who also coordinated all the streaming and recording), reosarevok, lucifer, rdswift, and many others who popped in. Thank you for patiently waiting while we ranted and when we didn’t notice you had your hand up. lucifer – who wasn’t able to come in person because of bullshit Visa rejections – we will definitely see you next year!

A summary of the topics covered follows. The more intrepid historians among you can see full event details on the wiki page, read the minutes, look at the photo gallery, and watch the summit recordings on YouTube: Day 1, Day 2, Day 3

OAuth hack session

With everyone together, the days before the summit proper were used for some productive hack sessions. The largest of which, involving the whole team, was the planning and beginning of a single OAuth location – meaning that everyone will be sent to a single place to login, from all of our projects.

A great warmup for the summit, we also leapt forward on the project, from identifying how exactly it would work, to getting substantial amounts of code and frontend elements in place.

Project recaps

“I broke this many things this year”

To kick off the summit, after a heart-warming introduction by Mayhem, we were treated to the annual recap for each project. For the full experience, feast your eyeballs on the Day 1 summit video – or click the timestamps below. What follows is a eyeball-taster, some simplistic and soothing highlights.

State of MetaBrainz (Mayhem) (4:50)

  • Mayhem reminds the team that they’re kicking ass!
  • We’re witnessing people getting fed up with streaming and focusing on a more engaged music experience, which is exactly the type of audience we wish to cater to, so this may work out well for us.
  • In 2023 we want to expand our offerings to grow our supporters (ListenBrainz)
  • Currently staying lean to prepare for incoming inflation/recession/depression

State of ListenBrainz (lucifer) (57:10)

  • 18.4 thousand all time users
  • 595 million all time listens
  • 92.3 million new listens submitted this year (so far)
  • Stacks of updates in the last year
  • Spotify metadata cache has been a game changer

State of Infrastructure (Zas) (1:14:40)

  • We are running 47 servers, from 42 in 2019
  • 27 physical (Hetzner), 12 virtual (Hetzner), 8 active instances (Google)
  • 150 Terabytes served this year
  • 99.9% availability of core services
  • And lots of detailed server, Docker, and ansible updates, and all the speed and response time stats you can shake a stick at.

State of MusicBrainz (Bitmap) (1:37:50)

  • React conversion coming along nicely
  • Documentation improved (auto-generated schema diagrams)
  • SIR crashes fixed, schema changes, stacks of updates (genres!)
  • 1,600 active weekly editors (stable from previous years)
  • 3,401,467 releases in the database
  • 391,536 releases added since 2021, ~1,099 per day
  • 29% of releases were added by the top 25 editors
  • 51% of releases were added with some kind of importer
  • 12,607,881 genre tag votes
  • 49% of release groups have at least one genre
  • 300% increase in the ‘finnish tango’ genre (3, was 1 in 2021)

State of AcousticBrainz (alastairp) (21:01:07)

  • R.I.P. (for more on the shut down of AB, see the blog post)
  • 29,460,584 submissions
  • 1.2 million hits per day still (noting that the level of trust/accuracy of this information is very low)
  • Data dumps, with tidying of duplicates, will be released when the site goes away

State of CritiqueBrainz (alastairp) (2:17:05)

  • 10,462 total reviews
  • 443 reviews in 2022
  • Book review support!
  • General bug squashing

State of BookBrainz (Monkey) (2:55:00)

  • A graph with an arrow going up is shown, everyone applauds #business #stonks
  • Twice the amount of monthly new users compared to 2021
  • 1/7th of all editions were added in the last year
  • Small team delivering lots of updates – author credits, book ratings/reviews, unified addition form
  • Import plans for the future (e.g. Library of Congress)

State of Community (Freso) (3:25:00)

  • Continuing discussion and developments re. how MetaBrainz affects LGBTQIA2+ folks
  • New spammer and sockpuppet countermeasures
  • Room to improve moderation and reports, particularly cross-project

Again, for delicious technical details, and to hear lots of lovely contributors get thanked, watch the full recording.

Discussions

“How will we fix all the things alastairp broke”

Next (not counting sleep, great meals, and some sneaky sightseeing) we moved to open discussion of various topics. These topics were submitted by the team, topics or questions intended to guide our direction for the next year. Some of these topics were discussed in break-out groups. You can read the complete meeting minutes in the summit minutes doc.

Ratings

Ratings were added years ago, and remain prominent on MusicBrainz. The topic for discussion was: What is their future? Shall we keep them? This was one of the most popular debates at the summit, with input from the whole spectrum of rating lovers and haters. In the end it was decided to gather more input from the community before making any decisions. We invite you to regale us with tales of your useage, suggestions, and thoughts in the resulting forum thread. 5/5 discussion.

CritiqueBrainz

Similar to ratings, CritiqueBrainz has been around for a number of years now and hasn’t gained much traction. Another popular topic, with lots of discussion regarding how we could encourage community submissions, improvements that could be made, how we can integrate it more closely with the other projects. Our most prolific CB contributor, sound.and.vision, gave some invaluable feedback via the stream. Ultimately it was decided that we are happy to sunset CB as a website (without hurry), but retain its API and integrate it into our other projects. Bug fixes and maintenance will continue, but new feature development will take place in other projects.

Integrating Aerozol (design)

Aerozol (the author of this blog post, in the flesh) kicked us off by introducing himself with a little TED talk about his history and his design strengths and weaknesses. He expressed interest in being part of the ‘complete user journey’, and helping to pull MetaBrainz’ amazing work in front of the general public, while being quite polite about MeB’ current attempts in this regard. It was decided that Aerozol should focus on over-arching design roadmaps that can be used to guide project direction, and that it is the responsibility of the developers to make sure new features and updates have been reviewed by a designer (including fellow designer, Monkey).

MusicBrainz Nomenclature

Can MetaBrainz sometimes be overly-fond of technical language? To answer that, ask yourself this: Did we just use the word ’nomenclature’ instead of something simpler, like ‘words’ or ‘terms’, in this section title? Exactly. With ListenBrainz aiming for a more general audience, who expect ‘album’ instead of ‘release group’, and ‘track’ instead of ‘recording’, this was predicted to become even more of an issue. Although it was acknowledged that it’s messy and generally unsatisfying to use different terms for the same things within the same ‘MetaBrainz universe’, we decided that it was fine for ListenBrainz to use more casual language for its user-facing bits, while retaining the technical language behind the scenes/in the API.

A related issue was also discussed, regarding how we title and discuss groupings of MusicBrainz entities, which is currently inconsistent, such as “core entities”, “primary entities”, “basic entities”. No disagreements with yvanzo’s suggestions were raised, the details of which can be found in ticket MBS-12552.

ListenBrainz Roadmap

Another fun discussion (5/5 – who said ratings weren’t useful!), it was decided that for 2023 we should prioritize features that bring in new users. Suggestions revolved around integrating more features into ListenBrainz directly (for instance, integrating MusicBrainz artist and album details, CritiqueBrainz reviews and ratings), how to promote sharing (please, share your thoughts and ideas in the resulting forum thread), making landing pages more inviting for new users, and how to handle notifications.

From Project Dev to Infrastructure Maintenance

MetaBrainz shares a common ‘tech org’ problem, stemming from working in niche areas which require high levels of expertise. We have many tasks that only one or a few people know how to do. It was agreed we should have another doc sprint, which was scheduled for the third week of January (16th-20th).

Security Management / Best Practices

Possible password and identity management solutions were discussed, and how we do, and should, deal with security advisories and alerts. It was agreed that there would be a communal security review the first week of each month. There is a note that “someone” should remember to add this to the meeting agenda at the right time. Let’s see how that pans out.

Search & SOLR

Did you know that running and calibrating search engines is a difficult Artform? Indeed, a capital a Artform. Our search team discussed a future move from SOLR v7 to SOLR v9 (SOLR is MusicBrainz’ search engine). It was discussed how we could use BookBrainz as a guinea pig by moving it from ElasticSearch (the search engine BB currently runs on) to SOLR, and try finally tackle multi-entity search while we are at it. If you really like reading about ‘cores’, ‘instances’, and whatever ‘zookeeper’ is, then these are your kind of meeting minutes.

Weblate

We currently use Transifex to translate MusicBrainz to other languages (Sound interesting? Join the community translation effort!), but are planning to move to Weblate, an open-source alternative that we can self-host. Pros and cons were discussed, and it seems that Weblate can provide a number of advantages, including discussion of translation strings, and ease of implementation across all our projects. Adjusting it to allow for single-sign on will involve some work. Video tutorials and introducing the new tool to the community was put on the to-do list.

Listenbrainz Roadmap and UI/UX

When a new user comes to ListenBrainz, where are they coming from, what do they see, where are we encouraging them to click next? Can users share and invite their friends? Items discussed were specific UI improvements, how we can implement ‘calls to action’, and better sharing tools (please contribute to the community thread if you have ideas). It was acknowledged that we sometimes struggle at implementing sharing tools because the team is (largely) not made up of social media users, and that we should allow for direct sharing as well as downloading and then sharing. Spotify, Apple Music, and Last.FM users were identified as groups that we should or could focus on.

Messages and Notifications

We agreed that we should have a way of notifying users across our sites, for site-user as well as user-user interactions. There should be an ‘inbox-like’ centre for these, and adequate granular control over the notification options (send me emails, digests, no emails, etc.), and the notification UI should show notifications from all MeB projects, on every site. We discussed how a messaging system could hinder or help our anti-spam efforts, giving users a new conduit to message each other, but also giving us possible control (as opposed to the current ‘invisible’ method of letting users direct email each other). It was decided to leave messaging for now (if at all), and focus on notifications.

Year in Music

We discussed what we liked (saveable images, playlists) and what we thought could be improved (lists, design, sharing, streamlining), about last years Year in Music. We decided that this year each component needs to have a link so that it can be embedded, as well as sharing tools. We decided to publish our Year in Music in the new year, with the tentative date of Wednesday January 4th, and let Spotify go to heck with their ’not really a year yet’ December release. We decided to use their December date to put up a blog post and remind people to get their listens added or imported in time for the real YIM!

Mobile Apps

The mobile app has been making great progress, with a number of substantial updates over the last year. However it seems to be suffering an identity crisis, with people expecting it to be a tagger on the level of Picard (or not really knowing what they expect), and then leaving bad reviews. After a lot of discussion (another popular and polarising topic!) it was agreed to make a new slimmed-down ListenBrainz app to cater to the ListenBrainz audience, and leave the troubled MusicBrainz app history behind. An iOS app isn’t out of the question, but something to be left for the future. akshaaatt has beaten me to the punch with his blog post on this topic.

MusicBrainz UI/UX Roadmap

The MusicBrainz dev and design team got together to discuss how they could integrate design and a broader roadmap into the workflow. It was agreed that designers would work in Figma (a online layout/mockup design tool), and developers should decide case-by-case whether an element should be standalone or shared among sites (using the design system). We will use React-Bootstrap for shared components. As the conversion to React continues it may also be useful to pull in designers to look at UI improvements as we go. It was agreed to hold regular team meetings to make sure the roadmap gets and stays on track and to get the redesign (!) rolling.

Thank you

Revealed! Left to right: Aerozol, Monkey, Mayhem, Atj, lucifer (laptop), yvanzo, alastairp, Bitmap, Zas, akshaaatt

On behalf of everyone who attended, a huge thanks to the wonderful denizens of Barcelona and OfficeBrainz for making us all feel so welcome, and MetaBrainz for making this trip possible. See you next year!

MusicBrainz Android App: Adding BrainzPlayer in Android App

Greetings, Everyone!

I am Ashutosh Aswal (IRC nick yellowhatpro), pursuing my bachelor’s from Punjab Engineering College Chandigarh, India. As a Google Summer of Code’22 contributor, I worked for MetaBrainz, on the MusicBrainz Android app and added a music playback feature to the app, which we call BrainzPlayer.

During the GSoC period, I was mentored by Akshat Tiwari (akshaaatt). Through this post, I will be summarizing my journey throughout the summer with MetaBrainz.

Let’s begin!! ( •̀ ω •́ )✧

Project Description

The project’s target was to introduce BrainzPlayer, a local music playback feature, into the MusicBrainz Android app. After this feature integration, users can play locally saved music directly from the app.

My pull requests.

My commits.

Coding Journey

We started with setting up the Music Service, Exoplayer, and the related Media APIs, which allow playback to be possible on the device, even when the app is in the background.

After this, we defined the Media Source, which accesses our local storage to search the media items and make them accessible within the app.

After accomplishing this, we worked on the notifications feature, which shows the metadata of the currently playing media item, and lets us control the playback, like seek, play, pause, etc., directly from the notification panel without opening the app.

Notification Panel

Now we worked on a service connector class that contains the functions to deal with the playback commands within the app.

After this, our app was ready to play songs. Now was the time to add some cool UI.

The UI is written in Jetpack Compose, Android’s latest toolkit for building awesome UI. Using Compose we worked on the Player Screen, which contains the playback features.

Now that we have the music playback feature, we worked on different entities: song, album, artist, and playlist.

To achieve this, we introduced a local database within the app. We introduced the various entities, including the required data and logic layer.

We wrote multiple database queries and added repositories for the entities in the data layer. Then we worked on the logic part and created functions that took in account the data layer and would show the result in the UI.

After working on the data and logic layer, we focused on creating the UI for the different entities. Each entity has its screen, from where the user can play songs. For this, we coordinated with aerozol, and I would thank him for coming up with beautiful designs and our BrainzPlayer logo. Then finally, with the designs in hand, we could execute them in compose.

By the end of the program, we were able to add some animations, and find out bugs and fix them.

Finally, the BrainzPlayer feature is merged with the master branch, so we can expect it to go into production soon. \^o^/

Preview of the upcoming feature:

Acknowledgement:

I want to thank my mentor, akshaaatt, for his immense support and guidance. Under his mentorship, I could learn, experiment, and improve my code quality over the time.

I am also indebted to the MetaBrainz team for their kind and supportive behavior, which made the journey incredible and unforgettable, and makes me motivated to work with them even beyond.

That’s it from my side.
Thank you for having me !! ヾ(≧▽≦*)o

My Google Summer of Code 2022 summary

What and for whom

Organization: MetaBrainz Foundation
Project: MusicBrainz Picard
Mentors: Laurent Monin (zas) & Philipp Wolfer (phw)
Main focus: Introducing single-instance mode in Picard 3.0
GSoC website: Link

What has been done: TL;DR edition

  • Picard works in single-instance mode by default, allowing to force-spawn a new instance
  • Picard accepts not just file paths but also URLs, MBIDs and commands as command-line arguments
  • The command-line arguments are sent to the existing instance (and processed by it) if possible
  • Picard can execute commands passed by the command-line interface; e.g. save all files, show the Picard window or close the app
  • Picard can also load the commands from a text file

List of pull requests

Single-instance mode

  • Picard#2116: A big commit where the whole single-instance mode for Picard was designed and introduced (only for file paths though)
  • Picard#2135: Fixed problems with exiting the app, caused by Picard#2116
  • Picard#2130: Supported URLs (with MBIDs) and mbid:// links (documented there) can be passed with file paths via CLI to an existing (or to a new one) instance
  • Picard#2137: Supported commands (like QUIT or SHOW) can be passed via CLI to an existing instance

Picard remote commands enhancements

  • Picard#2141: REMOVE_EMPTY & REMOVE_UNCLUSTERED commands added
  • Picard#2142: LOAD command, extending the positional arguments’ functionality, added
  • Picard#2143: FROM_FILE command, executing a command pipeline from a given file, added
  • Picard#2144: CLEAR_LOGS command added
  • Picard#2145: Fixed errors with the FROM_FILE command
  • Picard#2146: WRITE_LOGS command, allowing to save Picard logs into a file, added

Code refactoring

  • Picard#2080: Code explicitly marked as deprecated got removed, my initial commit to get to know the Picard’s codebase and workflow
  • Picard#2127: Minor patch, unparsed args are now ignored as they were not used anywhere
  • Picard#2139: Refactored the whole process of passing arguments to Picard, replaced ‘%’-formatted strings with f-strings, more than one arguments can be passed correctly to a command

Other

What have I learnt during GSoC 2022

  • How to work with other people on GitHub
  • How to improve my git experience (e.g. hooks)
  • How one can handle inter-process communication, basically I have researched:
    • pipes
    • named pipes
    • sockets
    • dbus
  • How to use Windows API with Python
  • Differences between Windows and Unix pipes
  • \0 is the only character that is prohibited on both Windows & Unix in path names
  • /tmp is not the recommended way to store non-persistent app data on *nix
  • os._exit might be useful when pythonic threads get broken
  • Importing a tuple in Python is underrated. git diff gets cleaner, as one sees only the additions

Some personal thoughts

  • Python is a really decent language that helps with starting one’s programming journey but the deeper I went, the more annoyances I have encountered (that is why I ended up starting to work as a C++ dev)
  • Ultra-safety is a double-edged sword: good luck terminating Pythonic futures/threads with file operations
  • CI/CD and testing in general is as important as decent codebase
  • If one can plan their time well, flexible work hours make their work both more effective and more enjoyable
  • Python sometimes change for worse or breaks the code without any reason (e.g. they have switched from using a mode into w on pipes, ref: LINK)
  • I will not start any new personal project in Python (especially one using multi-threading, multiple processes etc.), unless forced to do so. Nu for scripting, filling the niche & exploring the functional programming, some statically-typed languages for bigger projects, games, research, etc.
  • Impostor syndrome is just an another excuse to procrastinate. Do not be scared to learn & do new things but also ask smart questions. Everyone makes mistakes but if you made it to this org, you are a good fit and have enough qualifications

Special thanks

The whole MetaBrainz community is awesome and I am glad I have become a part of it, but I would like to express my special gratitude to the people I have directly worked with in any way 🙂 (alphabetical order by github username)

MusicBrainz Server update, 2022-09-06

This release mostly consists of small bug fixes and improvements. One bigger bug fix (MBS-12497) involves an issue where it was impossible to apply any edit which would cause an artist credit with any redirects pointing to it to be removed. Sorry about your stuck edits, people! They should now pass.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to chaban, HibiscusKazeneko, jesus2099, Mineo and sammyrayy for having reported bugs and suggested improvements. Thanks to ikerm2003 and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2022-09-06.

Continue reading “MusicBrainz Server update, 2022-09-06”

MusicBrainz Server update, 2022-08-22

It’s time for more fixed bugs and refactoring/maintenance tasks.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to CatQuest and otringal for having reported bugs and suggested improvements. Thanks to hamaryns, ikerm2003 and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2022-08-25-hotfixes.

Continue reading “MusicBrainz Server update, 2022-08-22”

MusicBrainz Server update, 2022-08-08

This is a small release since we’ve been resting (and sometimes fully on holiday) for the summer. The most visible change for website users is that the “Ratings” tab for ratable entities is now a “Reviews” tab, and it also includes ratings and reviews from CritiqueBrainz. We used to only display these for release groups – now you can see them for every entity that supports reviews (artists, events, labels, places, recordings, release groups and works). We’re hoping having this reminder that things can be reviewed will encourage more users to have a say and let us know about their favourite – or less favourite – music!

Also: for people running a mirror server who updated to our May schema change and want to use tags and genres: sorry, we broke your tags table. Please check our separate blog post on how to run a script to fix it.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to chaban and rdswift for having reported bugs and suggested improvements. Thanks to mfmeulenbelt and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2022-08-08.

Continue reading “MusicBrainz Server update, 2022-08-08”

Steps to fix missing genre and tag data on MusicBrainz mirror servers

If you recently updated your mirror server to the 2022-05-16 schema change release, we’re sorry to say that a bug in our upgrade script caused aggregate genre and tag data (if you had imported any) to be deleted. If you need this data, it can be re-imported from a recent dump, and we’ve written a script to help automate that.

You can safely ignore this post if

To restore the genre and tag data, follow these steps:

  1. Ensure you’ve replicated up to the most recent replication packet available. If you’re not sure, run ./admin/replication/LoadReplicationChanges. If you’re up-to-date, it should log “Replication packet … is not available.”
  2. Run git checkout production && git pull origin production.
  3. Turn off any cron jobs that update the database, including for replication.
  4. Run ./admin/sql/updates/20220720-mbs-12508.sh.
  5. Restart any cron jobs that you disabled.

You can verify that this process worked by checking the number of tags in the database: echo 'SELECT count(*) FROM tag' | ./admin/psql READWRITE. It should be over 200,000.

Sorry for the inconvenience, and let us know if you encounter any further issues.

MusicBrainz Server update, 2022-07-18

This release is very small, mainly consisting of minor improvements related to genre relationships. We have quite a few of those now, including links to Wikidata and Rate Your Music for pretty much any genre that has matching entries in those sites and links between many genres and their country of origin. We’re also fixing one bug with event setlist display that had probably been there since we introduced events – whoops!

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to chaban and sound.and.vision for having reported bugs and suggested improvements. Thanks to mfmeulenbelt, outsidecontext and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2022-07-18.

Continue reading “MusicBrainz Server update, 2022-07-18”

MusicBrainz Server update, 2022-07-03

This is a small release in number of tickets, but it finally adds the functionality to add relationships to genres. As such, you can now find out what għana or harsh noise wall actually are by simply reading their Wikipedia extracts. We can also store aliases for genres now, so while we still don’t do anything with that information directly, we do indicate that “hiphop” and “hip-hop” are alternative ways to write “hip hop“. Eventually, the plan is to combine these with the main genre in smart ways.

NB: As a side effect of adding new edit types for genres and genre aliases, the daily subscription emails failed for a few days. After this release the emails are working again, and you should have received the edits for all the missing days in your first email after the update.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to chaban and selflessself for having reported bugs and suggested improvements. Thanks to Mellthas, mfmeulenbelt, rinsuki and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2022-07-03.

Continue reading “MusicBrainz Server update, 2022-07-03”