MetaBrainz Summit 2022

The silliest, and thus best, group photo from the summit. Left to right: Aerozol, Monkey, Mayhem, Atj, lucifer (laptop), yvanzo, alastairp, Bitmap, Zas, akshaaatt

After a two-year break, in-person summits made their grand return in 2022! Contributors from all corners of the globe visited the Barcelona HQ to eat delicious local food, sample Monkey and alastairp’s beer, marvel at the architecture, try Mayhem’s cocktail robot, savour New Zealand and Irish chocolates, munch on delicious Indian snacks, and learn about the excellent Spanish culture of sleeping in. As well as, believe it or not, getting “work” done – recapping the last year, and planning, discussing, and getting excited about the future of MetaBrainz and its projects.

We also had some of the team join us via Stream; Freso (who also coordinated all the streaming and recording), reosarevok, lucifer, rdswift, and many others who popped in. Thank you for patiently waiting while we ranted and when we didn’t notice you had your hand up. lucifer – who wasn’t able to come in person because of bullshit Visa rejections – we will definitely see you next year!

A summary of the topics covered follows. The more intrepid historians among you can see full event details on the wiki page, read the minutes, look at the photo gallery, and watch the summit recordings on YouTube: Day 1, Day 2, Day 3

OAuth hack session

With everyone together, the days before the summit proper were used for some productive hack sessions. The largest of which, involving the whole team, was the planning and beginning of a single OAuth location – meaning that everyone will be sent to a single place to login, from all of our projects.

A great warmup for the summit, we also leapt forward on the project, from identifying how exactly it would work, to getting substantial amounts of code and frontend elements in place.

Project recaps

“I broke this many things this year”

To kick off the summit, after a heart-warming introduction by Mayhem, we were treated to the annual recap for each project. For the full experience, feast your eyeballs on the Day 1 summit video – or click the timestamps below. What follows is a eyeball-taster, some simplistic and soothing highlights.

State of MetaBrainz (Mayhem) (4:50)

  • Mayhem reminds the team that they’re kicking ass!
  • We’re witnessing people getting fed up with streaming and focusing on a more engaged music experience, which is exactly the type of audience we wish to cater to, so this may work out well for us.
  • In 2023 we want to expand our offerings to grow our supporters (ListenBrainz)
  • Currently staying lean to prepare for incoming inflation/recession/depression

State of ListenBrainz (lucifer) (57:10)

  • 18.4 thousand all time users
  • 595 million all time listens
  • 92.3 million new listens submitted this year (so far)
  • Stacks of updates in the last year
  • Spotify metadata cache has been a game changer

State of Infrastructure (Zas) (1:14:40)

  • We are running 47 servers, from 42 in 2019
  • 27 physical (Hetzner), 12 virtual (Hetzner), 8 active instances (Google)
  • 150 Terabytes served this year
  • 99.9% availability of core services
  • And lots of detailed server, Docker, and ansible updates, and all the speed and response time stats you can shake a stick at.

State of MusicBrainz (Bitmap) (1:37:50)

  • React conversion coming along nicely
  • Documentation improved (auto-generated schema diagrams)
  • SIR crashes fixed, schema changes, stacks of updates (genres!)
  • 1,600 active weekly editors (stable from previous years)
  • 3,401,467 releases in the database
  • 391,536 releases added since 2021, ~1,099 per day
  • 29% of releases were added by the top 25 editors
  • 51% of releases were added with some kind of importer
  • 12,607,881 genre tag votes
  • 49% of release groups have at least one genre
  • 300% increase in the ‘finnish tango’ genre (3, was 1 in 2021)

State of AcousticBrainz (alastairp) (21:01:07)

  • R.I.P. (for more on the shut down of AB, see the blog post)
  • 29,460,584 submissions
  • 1.2 million hits per day still (noting that the level of trust/accuracy of this information is very low)
  • Data dumps, with tidying of duplicates, will be released when the site goes away

State of CritiqueBrainz (alastairp) (2:17:05)

  • 10,462 total reviews
  • 443 reviews in 2022
  • Book review support!
  • General bug squashing

State of BookBrainz (Monkey) (2:55:00)

  • A graph with an arrow going up is shown, everyone applauds #business #stonks
  • Twice the amount of monthly new users compared to 2021
  • 1/7th of all editions were added in the last year
  • Small team delivering lots of updates – author credits, book ratings/reviews, unified addition form
  • Import plans for the future (e.g. Library of Congress)

State of Community (Freso) (3:25:00)

  • Continuing discussion and developments re. how MetaBrainz affects LGBTQIA2+ folks
  • New spammer and sockpuppet countermeasures
  • Room to improve moderation and reports, particularly cross-project

Again, for delicious technical details, and to hear lots of lovely contributors get thanked, watch the full recording.

Discussions

“How will we fix all the things alastairp broke”

Next (not counting sleep, great meals, and some sneaky sightseeing) we moved to open discussion of various topics. These topics were submitted by the team, topics or questions intended to guide our direction for the next year. Some of these topics were discussed in break-out groups. You can read the complete meeting minutes in the summit minutes doc.

Ratings

Ratings were added years ago, and remain prominent on MusicBrainz. The topic for discussion was: What is their future? Shall we keep them? This was one of the most popular debates at the summit, with input from the whole spectrum of rating lovers and haters. In the end it was decided to gather more input from the community before making any decisions. We invite you to regale us with tales of your useage, suggestions, and thoughts in the resulting forum thread. 5/5 discussion.

CritiqueBrainz

Similar to ratings, CritiqueBrainz has been around for a number of years now and hasn’t gained much traction. Another popular topic, with lots of discussion regarding how we could encourage community submissions, improvements that could be made, how we can integrate it more closely with the other projects. Our most prolific CB contributor, sound.and.vision, gave some invaluable feedback via the stream. Ultimately it was decided that we are happy to sunset CB as a website (without hurry), but retain its API and integrate it into our other projects. Bug fixes and maintenance will continue, but new feature development will take place in other projects.

Integrating Aerozol (design)

Aerozol (the author of this blog post, in the flesh) kicked us off by introducing himself with a little TED talk about his history and his design strengths and weaknesses. He expressed interest in being part of the ‘complete user journey’, and helping to pull MetaBrainz’ amazing work in front of the general public, while being quite polite about MeB’ current attempts in this regard. It was decided that Aerozol should focus on over-arching design roadmaps that can be used to guide project direction, and that it is the responsibility of the developers to make sure new features and updates have been reviewed by a designer (including fellow designer, Monkey).

MusicBrainz Nomenclature

Can MetaBrainz sometimes be overly-fond of technical language? To answer that, ask yourself this: Did we just use the word ’nomenclature’ instead of something simpler, like ‘words’ or ‘terms’, in this section title? Exactly. With ListenBrainz aiming for a more general audience, who expect ‘album’ instead of ‘release group’, and ‘track’ instead of ‘recording’, this was predicted to become even more of an issue. Although it was acknowledged that it’s messy and generally unsatisfying to use different terms for the same things within the same ‘MetaBrainz universe’, we decided that it was fine for ListenBrainz to use more casual language for its user-facing bits, while retaining the technical language behind the scenes/in the API.

A related issue was also discussed, regarding how we title and discuss groupings of MusicBrainz entities, which is currently inconsistent, such as “core entities”, “primary entities”, “basic entities”. No disagreements with yvanzo’s suggestions were raised, the details of which can be found in ticket MBS-12552.

ListenBrainz Roadmap

Another fun discussion (5/5 – who said ratings weren’t useful!), it was decided that for 2023 we should prioritize features that bring in new users. Suggestions revolved around integrating more features into ListenBrainz directly (for instance, integrating MusicBrainz artist and album details, CritiqueBrainz reviews and ratings), how to promote sharing (please, share your thoughts and ideas in the resulting forum thread), making landing pages more inviting for new users, and how to handle notifications.

From Project Dev to Infrastructure Maintenance

MetaBrainz shares a common ‘tech org’ problem, stemming from working in niche areas which require high levels of expertise. We have many tasks that only one or a few people know how to do. It was agreed we should have another doc sprint, which was scheduled for the third week of January (16th-20th).

Security Management / Best Practices

Possible password and identity management solutions were discussed, and how we do, and should, deal with security advisories and alerts. It was agreed that there would be a communal security review the first week of each month. There is a note that “someone” should remember to add this to the meeting agenda at the right time. Let’s see how that pans out.

Search & SOLR

Did you know that running and calibrating search engines is a difficult Artform? Indeed, a capital a Artform. Our search team discussed a future move from SOLR v7 to SOLR v9 (SOLR is MusicBrainz’ search engine). It was discussed how we could use BookBrainz as a guinea pig by moving it from ElasticSearch (the search engine BB currently runs on) to SOLR, and try finally tackle multi-entity search while we are at it. If you really like reading about ‘cores’, ‘instances’, and whatever ‘zookeeper’ is, then these are your kind of meeting minutes.

Weblate

We currently use Transifex to translate MusicBrainz to other languages (Sound interesting? Join the community translation effort!), but are planning to move to Weblate, an open-source alternative that we can self-host. Pros and cons were discussed, and it seems that Weblate can provide a number of advantages, including discussion of translation strings, and ease of implementation across all our projects. Adjusting it to allow for single-sign on will involve some work. Video tutorials and introducing the new tool to the community was put on the to-do list.

Listenbrainz Roadmap and UI/UX

When a new user comes to ListenBrainz, where are they coming from, what do they see, where are we encouraging them to click next? Can users share and invite their friends? Items discussed were specific UI improvements, how we can implement ‘calls to action’, and better sharing tools (please contribute to the community thread if you have ideas). It was acknowledged that we sometimes struggle at implementing sharing tools because the team is (largely) not made up of social media users, and that we should allow for direct sharing as well as downloading and then sharing. Spotify, Apple Music, and Last.FM users were identified as groups that we should or could focus on.

Messages and Notifications

We agreed that we should have a way of notifying users across our sites, for site-user as well as user-user interactions. There should be an ‘inbox-like’ centre for these, and adequate granular control over the notification options (send me emails, digests, no emails, etc.), and the notification UI should show notifications from all MeB projects, on every site. We discussed how a messaging system could hinder or help our anti-spam efforts, giving users a new conduit to message each other, but also giving us possible control (as opposed to the current ‘invisible’ method of letting users direct email each other). It was decided to leave messaging for now (if at all), and focus on notifications.

Year in Music

We discussed what we liked (saveable images, playlists) and what we thought could be improved (lists, design, sharing, streamlining), about last years Year in Music. We decided that this year each component needs to have a link so that it can be embedded, as well as sharing tools. We decided to publish our Year in Music in the new year, with the tentative date of Wednesday January 4th, and let Spotify go to heck with their ’not really a year yet’ December release. We decided to use their December date to put up a blog post and remind people to get their listens added or imported in time for the real YIM!

Mobile Apps

The mobile app has been making great progress, with a number of substantial updates over the last year. However it seems to be suffering an identity crisis, with people expecting it to be a tagger on the level of Picard (or not really knowing what they expect), and then leaving bad reviews. After a lot of discussion (another popular and polarising topic!) it was agreed to make a new slimmed-down ListenBrainz app to cater to the ListenBrainz audience, and leave the troubled MusicBrainz app history behind. An iOS app isn’t out of the question, but something to be left for the future. akshaaatt has beaten me to the punch with his blog post on this topic.

MusicBrainz UI/UX Roadmap

The MusicBrainz dev and design team got together to discuss how they could integrate design and a broader roadmap into the workflow. It was agreed that designers would work in Figma (a online layout/mockup design tool), and developers should decide case-by-case whether an element should be standalone or shared among sites (using the design system). We will use React-Bootstrap for shared components. As the conversion to React continues it may also be useful to pull in designers to look at UI improvements as we go. It was agreed to hold regular team meetings to make sure the roadmap gets and stays on track and to get the redesign (!) rolling.

Thank you

Revealed! Left to right: Aerozol, Monkey, Mayhem, Atj, lucifer (laptop), yvanzo, alastairp, Bitmap, Zas, akshaaatt

On behalf of everyone who attended, a huge thanks to the wonderful denizens of Barcelona and OfficeBrainz for making us all feel so welcome, and MetaBrainz for making this trip possible. See you next year!

Stepping up on our UX: Welcoming Simon Hartman to the team

Hello!

I am pleased to announce that long time contributor and complainer about our UI/UX, Simon Hartman, AKA aerozol has joined our team as a part time designer!

While we are starting with a very modest 3 hours of his time per week, we feel that this marks a rather important step forward for our team. While we now have two team members who have UX/design skills (Monkey and Akshat), they also carry a significant load of engineering tasks working on their respective projects.

Having Simon as part of our team will allow us to carve out concrete design tasks for him to focus on. Simon and Akshat will also revive our long dormant design system, which lets us create UI components that are intuitive and consistent. Our engineering team will be able to re-use these components across our sites, simplifying the future development of new pages. We hope that this shared design system will improve the user interface across all of our sites, with a strong focus on bringing the MusicBrainz UI into the modern age.

Having concrete help on the design front has been needed badly for a long time, which makes me very excited to welcome Simon to our team. Welcome!

Welcoming Akshat Tiwari to the MetaBrainz team!

I’m pleased to announce that we are continuing our long tradition of hiring our best Google Summer of Code participants — I’d like to warmly welcome Akshat Tiwari to the MetaBrainz team!

Akshat has been working on our Android App, continuing the work from last summer to improve the app and to add new features. He has been doing great work and demonstrating the fact that he understands user interfaces and has an eye for design as well as coding. This is a rare combination of talents and since we’ve been in dire need for improving the UI/UX for the MusicBrainz web site since forever, this was the time to finally get this project moving seriously.

Akshat has joined us on a trial contract through the end of the year with the goal of creating a new home page for MusicBrainz (and more hopefully) — the current home page is still stuck in the early 2000s and hasn’t evolved as our projects have evolved.

Our hope is to have Akshat become a permanent member of the MusicBrainz team and once the home page is completed, that he will continue on the UI/UX revamp that Chhavi started several years ago.

Welcome to the team Akshat!

Kartik Ohri joins the MetaBrainz team!

I’m pleased to announce that Kartik Ohri, AKA Lucifer, a very active contributor since his Code-in days in 2018, has become the latest staff member of the MetaBrainz Foundation!

Kartik has been instrumental in rewriting our Android app and more recently has been helping us with a number of tasks, including new features for ListenBrainz, AcousticBrainz as well as breathing some much needed life into the CritiqueBrainz project.

These three projects (CritiqueBrainz, ListenBrainz and AcousticBrainz) will be his main focus while working for MetaBrainz. Each of these projects has not had enough engineering time recently to sufficiently move new features forward. We hope that with Kartik’s efforts we can deliver more features faster.

Welcome to the team, Kartik!

Playlists and personalised recommendations in ListenBrainz

Just in time for Christmas we are pleased to announce a new feature in our most recent release of ListenBrainz, the ability to create and share your own playlists! We created two playlists for each user who used ListenBrainz containing music that you listened to in 2020. Check out your lists at https://listenbrainz.org/my/recommendations. Read on for more details…

With our continuing work on using data in ListenBrainz to generate recommendations, we realised that we needed a place to store lists of music. That sounded like playlists to us, so we added them to ListenBrainz. As always, we did this work in the public ListenBrainz repository. You can now create your own playlists with the web interface or by using the API. Recordings in playlists map to MusicBrainz identifiers. If you’re trying to add something and can’t find it, make sure that it’s in MusicBrainz first.

Once you have a playlist, you can listen to it using our built-in BrainzPlayer, or export it to Spotify if you have an account there. If you have already linked your Spotify account to ListenBrainz you may have to re-authenticate and give us permission to create playlists on your behalf. Playlists can also be exported in the open JSPF format using the ListenBrainz API.

Over the last year we’ve started thinking about how to use data in MetaBrainz projects to generate recommendations of new music for people to listen to. For this reason, we started the Troi recommendation framework. This python package allows developers to build pipelines that take data from different sources and combine it in order to generate recommendations of music to listen to. We have already developed data sources using MusicBrainz, ListenBrainz, and AcousticBrainz. If you are a developer interested in working on recommendations in the context of ListenBrainz we encourage you to check it out.

Now that we can store playlists we needed some content to fill them with. Luckily we have some great projects worked on by students over the last few years as part of MetaBrainz’ participation in the Google Summer of Code project, including this year’s work on statistics and summary information by Ishaan. Using Troi and ListenBrainz statistics, we got to work. Every user who has been contributing data to ListenBrainz recently now has two brand new 2020 playlists based on the top recordings that you listened to in 2020 and the recordings that you first listened to in 2020. If you’re interested in the code behind these playlists, you can see the code for each (top tracks, first tracks) in the troi repository.

If you’re a long-time user of ListenBrainz you may be familiar with the problem of matching your listens to content in MusicBrainz to be able to do things with it. We’ve been working hard on a solution to this problem and have built a new tool using typesense to provide a quick and easy way to search for items in the MusicBrainz database. You are using this tool when you create a playlists using the web interface and search for a recording to add. This is still a tech preview, but in our experience it works really well. Thanks to the team at typesense for helping us with our questions over the last few weeks!

This work is still in its early days. We thought that this was such a great feature that we wanted to get it out in front of you now. We’re happy to take your feedback, or hear if you are having any problems. Open a ticket on our bug tracker, come and talk to us on IRC, or @ us. Did we give you a bad jam? Sorry about that! We’d love to have a conversation about what went well and what didn’t in order to improve our systems. In 2021 we will start generating weekly and daily playlists for users based on your recent listens using our collaborative filtering recommendations system.

Merry Christmas from the whole MetaBrainz team!

The ODI publishes two reports on Sustainable Data Institutions

The Open Data Institute has just published two reports: Designing Sustainable Data Institutions and Designing Trustworthy Data Institutions which include insights provided by us regarding our MusicBrainz project.

When I was starting out MusicBrainz and was trying to work out how to make the project sustainable, I would’ve given just about anything to have access to these reports. I am proud that, nearly 20 years later, I was able to contribute to these reports so that others may benefit from our hard work.

I find the section Suggestions for those scoping, designing and running data institutions on page 40 of the PDF version of Designing Sustainable Data Institutions quite enlightening:

  1. Ensure your revenue model aligns with your organisational goals
  2. Understand how your revenue sources will change during your institution’s lifecycle
  3. Consider both financial and non-financial aspects of sustainability
  4. Identify and mitigate future risks
  5. Learn from others

Each of these points represent a whole collections of small lessons that I’ve learned by (often painful) experience of the past years. Also, I feel that these points are not strictly limited Data Institutions, but many also apply to making open source projects sustainable. If you’re in the business of running a data or open source organzation, I would strongly encourage you to read this paper!

Also very interesting is the second report about Designing Trustworthy Data Institutions:

For example, the representative from MusicBrainz said, “[A culture of honesty] builds trust, and this trust builds sustainability”

Compared to sustainability, the concepts of trust were much more clear to me from the beginning. However, that doesn’t make this report any less relevant — especially in current times, I welcome an emphasis on trust!

Thank you to the ODI for including MusicBrainz and doing all of the hard work on these reports!

 

Welcoming Paula LeDieu to our board of directors!

Late in 2019, we finally filled our one vacant spot on our board of directors — we had been holding out until we found the right person and we finally have! And then towards the end of the year we all got distracted by holidays and world events and never got around to formally announcing that we have a new addition to our board.

With great pleasure I would like to announce that Paula LeDieu, an amazingly connected person who seems to know everyone, has joined the MetaBrainz Foundation Board of Directors! I first met Paula when I was bootstrapping MusicBrainz and pondering how to setup a foundation for the project — we would continuously bump into each other at various conferences in the world. Paula’s professional history includes a lot organizations that early on shaped the internet, including the BBC, iCommons and Mozilla. Paula’s professional experience and connections will be a great asset to our organization.

Paula lives in Sydney, Australia, stretching our board of directors across 3 continents and far too many time-zones. Thank you for agreeing to join our board of directors and welcome to the team, Paula!

Thank you for your continued support, Google!

We’ve recently received our annual $30,000 support from Google. The brings the total amount donated by Google’s Open Source Programs Office to us to over $470,000 — hopefully next year we’ll cross the half million dollar threshold!

I can’t quite express my gratitude for this level of support! Without Google’s help, especially early on, MetaBrainz may never have made it to sustainability. Google has helped us in a number of ways, including Google Code-In and Summer of Code — all of these forms of support have shaped our organization quite heavily over the past 15 or so years.

Thank you to Google and everyone at the Google Open Source Programs Office — we truly appreciate your support over the years!

Thank you Microsoft!

Microsoft reached out to us back in early 2018 in order to use our data in Bing — we followed the normal sort of on-boarding procedure that we use for our supporters. During one of these on-boarding calls we were asked if there was more that Microsoft could do to help us and support our mission. Soon thereafter I provided them with a list of things that would be useful to us. Sadly, the request to buy a major record label and then to give it to us to manage was turned down for being too expensive. 😦

However, Microsoft did like two items on our list and agreed to support us — they were:

1) Azure hosting credits — we’re always looking for more hosting capacity and these credits will allows us to provide virtual machines to our team and to close collaborators who are doing good work, but might be lacking the computing power to push their projects forward. This contribution is of direct benefit to our community — often times our projects contain quite a lot of data and thus have some heavy processing requirements. We’re currently using our hosting credits to do some large data set crunching and some testing for the Virtual Machine that we provide to users who wish to get up and running with MusicBrainz data quickly.

2) Sponsoring our summit — our annual team meeting and foundation summit happens at the end of each September, normally in Barcelona where we have our main office. Microsoft’s sponsorship allows us to invite more people to the event, since we have the means to cover their expenses. Our summits have traditionally been our annual forum for meeting the other team members and volunteers and to take a breather from the normal course of business. At the event we see a more human side of each other and we’re more easily able to discuss our challenges and the vision for the future.

We really appreciate our supporters who go above and beyond the normal levels of support for us — these contributions really sweeten the deal of hacking on open source software!

Thank you so much to Microsoft and everyone at Microsoft who helped move this contribution forward!

State of the Brainz: 2019 MetaBrainz Summit highlights

The 2019 MetaBrainz Summit took place on 27th–29th of September 2019 in Barcelona, Spain at the MetaBrainz HQ. The Summit is a chance for MetaBrainz staff and the community to gather and plan ahead for the next year. This report is a recap of what was discussed and what lies ahead for the community.

Continue reading “State of the Brainz: 2019 MetaBrainz Summit highlights”