If you’ve been lurking on our forums, you will know that our style guidelines have been receiving lots of attention lately! Both MusicBrainz docs and BookBrainz docs have been getting new additions and updates, and there’s still plenty being discussed. As I type these very words, there are greased-up editors in what’s known to us only as “the pit” – pitting their words, experience, and muscular bodies against each other in a bid for dominance over typographical punctuation.
Okay, okay, so there’s not really a pit, but here’s a rundown of some of the style discussions that are taking place, or have taken place, this year:
*proposals have been fully or partially implemented into the official style guidelines
For ongoing updates, keep an eye on the forums, and consider subscribing to the new forum tag ‘styleguide-proposal’. These community discussions do ultimately affect editing and your collections, so you may want to input as well. Slap on that grease, and get in The Pit! (I have to note, again, that there are no actual pits involved in any MeB discussions. A sad state of affairs.)
A big thank you to all the community members who shape our databases via their guideline input.
After a two-year break, in-person summits made their grand return in 2022! Contributors from all corners of the globe visited the Barcelona HQ to eat delicious local food, sample Monkey and alastairp’s beer, marvel at the architecture, try Mayhem’s cocktail robot, savour New Zealand and Irish chocolates, munch on delicious Indian snacks, and learn about the excellent Spanish culture of sleeping in. As well as, believe it or not, getting “work” done – recapping the last year, and planning, discussing, and getting excited about the future of MetaBrainz and its projects.
We also had some of the team join us via Stream; Freso (who also coordinated all the streaming and recording), reosarevok, lucifer, rdswift, and many others who popped in. Thank you for patiently waiting while we ranted and when we didn’t notice you had your hand up. lucifer – who wasn’t able to come in person because of bullshit Visa rejections – we will definitely see you next year!
A summary of the topics covered follows. The more intrepid historians among you can see full event details on the wiki page, read the minutes, look at the photo gallery, and watch the summit recordings on YouTube: Day 1, Day 2, Day 3
OAuth hack session
With everyone together, the days before the summit proper were used for some productive hack sessions. The largest of which, involving the whole team, was the planning and beginning of a single OAuth location – meaning that everyone will be sent to a single place to login, from all of our projects.
A great warmup for the summit, we also leapt forward on the project, from identifying how exactly it would work, to getting substantial amounts of code and frontend elements in place.
To kick off the summit, after a heart-warming introduction by Mayhem, we were treated to the annual recap for each project. For the full experience, feast your eyeballs on the Day 1 summit video – or click the timestamps below. What follows is a eyeball-taster, some simplistic and soothing highlights.
Continuing discussion and developments re. how MetaBrainz affects LGBTQIA2+ folks
New spammer and sockpuppet countermeasures
Room to improve moderation and reports, particularly cross-project
Again, for delicious technical details, and to hear lots of lovely contributors get thanked, watch the full recording.
Next (not counting sleep, great meals, and some sneaky sightseeing) we moved to open discussion of various topics. These topics were submitted by the team, topics or questions intended to guide our direction for the next year. Some of these topics were discussed in break-out groups. You can read the complete meeting minutes in the summit minutes doc.
Ratings were added years ago, and remain prominent on MusicBrainz. The topic for discussion was: What is their future? Shall we keep them? This was one of the most popular debates at the summit, with input from the whole spectrum of rating lovers and haters. In the end it was decided to gather more input from the community before making any decisions. We invite you to regale us with tales of your useage, suggestions, and thoughts in the resulting forum thread. 5/5 discussion.
Similar to ratings, CritiqueBrainz has been around for a number of years now and hasn’t gained much traction. Another popular topic, with lots of discussion regarding how we could encourage community submissions, improvements that could be made, how we can integrate it more closely with the other projects. Our most prolific CB contributor, sound.and.vision, gave some invaluable feedback via the stream. Ultimately it was decided that we are happy to sunset CB as a website (without hurry), but retain its API and integrate it into our other projects. Bug fixes and maintenance will continue, but new feature development will take place in other projects.
Integrating Aerozol (design)
Aerozol (the author of this blog post, in the flesh) kicked us off by introducing himself with a little TED talk about his history and his design strengths and weaknesses. He expressed interest in being part of the ‘complete user journey’, and helping to pull MetaBrainz’ amazing work in front of the general public, while being quite polite about MeB’ current attempts in this regard. It was decided that Aerozol should focus on over-arching design roadmaps that can be used to guide project direction, and that it is the responsibility of the developers to make sure new features and updates have been reviewed by a designer (including fellow designer, Monkey).
Can MetaBrainz sometimes be overly-fond of technical language? To answer that, ask yourself this: Did we just use the word ’nomenclature’ instead of something simpler, like ‘words’ or ‘terms’, in this section title? Exactly. With ListenBrainz aiming for a more general audience, who expect ‘album’ instead of ‘release group’, and ‘track’ instead of ‘recording’, this was predicted to become even more of an issue. Although it was acknowledged that it’s messy and generally unsatisfying to use different terms for the same things within the same ‘MetaBrainz universe’, we decided that it was fine for ListenBrainz to use more casual language for its user-facing bits, while retaining the technical language behind the scenes/in the API.
A related issue was also discussed, regarding how we title and discuss groupings of MusicBrainz entities, which is currently inconsistent, such as “core entities”, “primary entities”, “basic entities”. No disagreements with yvanzo’s suggestions were raised, the details of which can be found in ticket MBS-12552.
Another fun discussion (5/5 – who said ratings weren’t useful!), it was decided that for 2023 we should prioritize features that bring in new users. Suggestions revolved around integrating more features into ListenBrainz directly (for instance, integrating MusicBrainz artist and album details, CritiqueBrainz reviews and ratings), how to promote sharing (please, share your thoughts and ideas in the resulting forum thread), making landing pages more inviting for new users, and how to handle notifications.
From Project Dev to Infrastructure Maintenance
MetaBrainz shares a common ‘tech org’ problem, stemming from working in niche areas which require high levels of expertise. We have many tasks that only one or a few people know how to do. It was agreed we should have another doc sprint, which was scheduled for the third week of January (16th-20th).
Security Management / Best Practices
Possible password and identity management solutions were discussed, and how we do, and should, deal with security advisories and alerts. It was agreed that there would be a communal security review the first week of each month. There is a note that “someone” should remember to add this to the meeting agenda at the right time. Let’s see how that pans out.
Search & SOLR
Did you know that running and calibrating search engines is a difficult Artform? Indeed, a capital a Artform. Our search team discussed a future move from SOLR v7 to SOLR v9 (SOLR is MusicBrainz’ search engine). It was discussed how we could use BookBrainz as a guinea pig by moving it from ElasticSearch (the search engine BB currently runs on) to SOLR, and try finally tackle multi-entity search while we are at it. If you really like reading about ‘cores’, ‘instances’, and whatever ‘zookeeper’ is, then these are your kind of meeting minutes.
We currently use Transifex to translate MusicBrainz to other languages (Sound interesting? Join the community translation effort!), but are planning to move to Weblate, an open-source alternative that we can self-host. Pros and cons were discussed, and it seems that Weblate can provide a number of advantages, including discussion of translation strings, and ease of implementation across all our projects. Adjusting it to allow for single-sign on will involve some work. Video tutorials and introducing the new tool to the community was put on the to-do list.
Listenbrainz Roadmap and UI/UX
When a new user comes to ListenBrainz, where are they coming from, what do they see, where are we encouraging them to click next? Can users share and invite their friends? Items discussed were specific UI improvements, how we can implement ‘calls to action’, and better sharing tools (please contribute to the community thread if you have ideas). It was acknowledged that we sometimes struggle at implementing sharing tools because the team is (largely) not made up of social media users, and that we should allow for direct sharing as well as downloading and then sharing. Spotify, Apple Music, and Last.FM users were identified as groups that we should or could focus on.
Messages and Notifications
We agreed that we should have a way of notifying users across our sites, for site-user as well as user-user interactions. There should be an ‘inbox-like’ centre for these, and adequate granular control over the notification options (send me emails, digests, no emails, etc.), and the notification UI should show notifications from all MeB projects, on every site. We discussed how a messaging system could hinder or help our anti-spam efforts, giving users a new conduit to message each other, but also giving us possible control (as opposed to the current ‘invisible’ method of letting users direct email each other). It was decided to leave messaging for now (if at all), and focus on notifications.
Year in Music
We discussed what we liked (saveable images, playlists) and what we thought could be improved (lists, design, sharing, streamlining), about last years Year in Music. We decided that this year each component needs to have a link so that it can be embedded, as well as sharing tools. We decided to publish our Year in Music in the new year, with the tentative date of Wednesday January 4th, and let Spotify go to heck with their ’not really a year yet’ December release. We decided to use their December date to put up a blog post and remind people to get their listens added or imported in time for the real YIM!
The mobile app has been making great progress, with a number of substantial updates over the last year. However it seems to be suffering an identity crisis, with people expecting it to be a tagger on the level of Picard (or not really knowing what they expect), and then leaving bad reviews. After a lot of discussion (another popular and polarising topic!) it was agreed to make a new slimmed-down ListenBrainz app to cater to the ListenBrainz audience, and leave the troubled MusicBrainz app history behind. An iOS app isn’t out of the question, but something to be left for the future. akshaaatt has beaten me to the punch with his blog post on this topic.
MusicBrainz UI/UX Roadmap
The MusicBrainz dev and design team got together to discuss how they could integrate design and a broader roadmap into the workflow. It was agreed that designers would work in Figma (a online layout/mockup design tool), and developers should decide case-by-case whether an element should be standalone or shared among sites (using the design system). We will use React-Bootstrap for shared components. As the conversion to React continues it may also be useful to pull in designers to look at UI improvements as we go. It was agreed to hold regular team meetings to make sure the roadmap gets and stays on track and to get the redesign (!) rolling.
On behalf of everyone who attended, a huge thanks to the wonderful denizens of Barcelona and OfficeBrainz for making us all feel so welcome, and MetaBrainz for making this trip possible. See you next year!
I am Ansh Goyal (ansh on IRC), an undergraduate student from the Birla Institute of Technology and Science (BITS), Pilani, India. This summer, I participated in Google Summer of Code and introduced a new feature, CritiqueBrainz reviews for BookBrainz entities.
I was mentored by Alastair Porter (alastairp on IRC) and Nicolas Pelletier (monkey on IRC) during this period. This post summarizes my contributions made for this project and my experiences throughout the journey.
Book reviews are a glimpse into a world you may or may not choose to enter. Reviews give books greater visibility and a greater chance of getting found by more readers. BookBrainz and CritiqueBrainz should enable users to rate and write reviews directly through the website using the CritiqueBrainz API.
For GSoC ’22, I made a proposal that aimed at adding CritiqueBrainz reviews for BookBrainz entities.
During the community bonding period, my mentors, Alastair Porter and Nicolas Pelletier, helped me create a streamlined pathway to move along with the project. We also worked on various tickets like CB-433, CB-434, and CB-410 and added multiple features.
So we decided first to complete reviewing BookBrainz edition groups thoroughly from CritiqueBrainz as well as BookBrainz, and then extend the project for all other entities like literary work, author, and Series.
We discussed the various database and structural changes involved in the project, like adding BookBrainz Database in CritiqueBrainz, adding tables in BookBrainz DB, etc., the page designs and overall improvements.
Now that the database was set up and ready for us to work on, it was time to write SQL queries for fetching the edition groups and all the other associated information, like identifiers and relationships. I made the code reusable to prevent duplication while fetching data for different entity types. So I opened a PR for it.
So now it was the time to allow users to review an edition group in CritiqueBrainz. For this, I made a few changes in the database, allowed BB entity types, and then added pages for their reviews in this PR. Then I worked on showing the information fetched from the BookBrainz database to the users on their entity pages.
Then to allow users to search edition groups, I worked on adding an option to search BB entities with the help of the search API already present in BB. This feature was implemented in this PR.
After adding support for Edition Groups, it was time to add support for other entity types. This expansion was very smooth because of the reusable components created by then. So I added support for Literary Works, Authors and Series. Later we discovered that the series items were not being ordered correctly, so this was fixed in #460.
During this process, my mentors and I discovered some improvements and refactoring, which were done in #445, #451 and #456.
After adding support for all the entities, I added support for showing the relationships between the entities on the respective entity pages. These included showing Author-Work, Work-Edition Group, and Work-Work relationships.
CritiqueBrainz Author’s Page
While enabling CritiqueBrainz to review entities, I was also working on BookBrainz to allow users to view reviews and ratings and post them from the entity’s page.
So I started adding support to fetch and display reviews and ratings for edition groups which involved creating a route which would handle getting and pushing reviews to CritiqueBrainz.
After this step, it was time to connect BookBrainz with CritiqueBrainz. This involved authentication using OAuth login. To add this feature, I first added a table ‘external_service_oauth’ in the database and then in the ORM.
Then I added routes to allow user login to CritiqueBrainz, handled the callback, and saved the tokens in the database. After that, the next thing was to allow users to post reviews from the entity page. For that, I create a modal similar to the one in ListenBrainz (to maintain consistency).
After completing my project, I began working on my stretch goals and starting with unifying reviews for entities common in both BookBrainz and MusicBrainz databases. We decided that if an entity exists in both databases, we show the reviews for all the entities on the entity page (PR).
I am incredibly grateful to my mentors for their constant guidance and support throughout my project. I learned a lot of technical concepts and improved the quality of my code during this journey. I had a wonderful time interacting with the amazing folks at MetaBrainz and exchanging valuable thoughts during our weekly meetings.
I would love to thank Google and the MetaBrainz Foundation for providing me with this great opportunity!
Hi everyone, I am Akash Gupta, currently pursuing my undergraduate from Kalinga Institute of Industrial Technology. This summer, I participated in Google Summer of Code and developed a new feature — Series Entity— for the project BookBrainz.
I was mentored by Nicolas Pelletier (monkey on IRC) during this period. This post summarizes my contributions to the project and the experiences that I had throughout the summer.
Hi everyone, I am Prabal Singh currently studying in Indian Institute of Technology, Guwahati. This summer I participated in Google Summer of Code and developed a new feature – User Collections – for the project BookBrainz.
Today we come with a big BookBrainz website update that allows you to merge duplicate entities!
Being able to clean up the database is an essential step towards importing public bibliographic records and catalogs from partner websites. As with MusicBrainz, you can visit an entity page on BookBrainz and click on a button to add an entity to a merge queue. You can merge multiple entities in one go easily.
After clicking the merge button you will be presented with a page that lets you review and select the correct information in case of conflicting data. The revision history of merged entities is preserved, and in the near future you’ll be able undo merges.
After many years as a community driven project and often under-staffed, the BookBrainz project has always been the red-headed step child of our projects. A few weeks ago I asked if the community felt that we should make BookBrainz an official project of the foundation and got a very positive response.
After that, we started informally seeking developers to take on this position, leading to the hire of Monkey, who will now be the lead of the BookBrainz project, taking over for Ben Ockmore. Ben will take on a contributor role to BookBrainz going forward and remain on the project! Thanks for all of your hard efforts in the past, Ben!
While Monkey comes up to speed on the codebase, we’ve been brainstorming what features he should focus on first . The short term focus on BookBrainz will be on bringing it into our hosting setup at Hetzner, which means making the codebase ready for running inside of docker with all of the MetaBrainz specific hosting quirks. Part of this project will be to remove elastic search and to utilize our new Solr based search system that we recently released for MusicBrainz.
After getting BookBrainz moved to our hosting facility that focus will be to create a minimally viable product. What exactly does this mean? One of the frequent complaints I’ve received about BookBrainz is that it is missing core functionality of a proper metadata project. Core functionality means that a user should be able to view and edit all of the metadata that is in BookBrainz and then retrieve this data from the BookBrainz API. It should include full data dumps with incremental data dumps being added a bit later.
What do you think the missing core features of BookBrainz are?
Finally, we’re in discussions with the OpenLibrary team, wondering how to best work together and not to duplicate efforts — we’ll post more about this once we’ve reached an agreement with the OpenLibrary team on how we should proceed.
First off, the somewhat sad news: Sambhav, AKA samj1912, has left MetaBrainz the team as a contractor and has moved to London. The upside of this news is that he will continue to work on Picard for us and will remain a part of our team as a volunteer, but his presence will not be quite as intense as before. Thank you for your hard work these past months, especially for finishing the impossible Solr search project!
With Sambhav’s departure and our improved finances, I’m proud to announce that we’re taking on two new contractors!
Nicolas Pelletier AKA Monkey: You may remember the talented Monkey from when we designed our new logos. He was the designer who created the logos and our new bootstrap theme that adorns most of our pages now. Working with Monkey was straightforward, effective and the results were great, so when he expressed interest in working on BookBrainz, I was pleased to hear this news. Monkey will be working for us full time and spending 75% of his time on BookBrainz and 25% of his time to help with design and UX work for the rest of our projects. In the next blog post I’ll talk more about BookBrainz and what we can expect from that project in the future.
Nicolás Tamargo AKA Reosarevok: Reosarevok is no stranger to our community — he’s made 1.7M edits to MusicBrainz, is our Style BDFL and answers all of our support@ emails. He’s been learning more programming and asked to be part of the MusicBrainz team part time. We agreed to give this a go and in the short term he will be focusing on genre support and helping with the React migration among other tasks. If this trial run works out, we’ll see about expanding his scope on our team.
Welcome on board Monkey and good luck with the new position, Reo!
I’m Shivam Tripathi, an undergraduate student from the Indian Institute of Information Technology, Una. I interned for the MetaBrainz foundation under the Google Summer of Code programme for the year 2018 and worked on the BookBrainz project. I was mentored by Ben Ockmore during this period. This post summarizes my contributions to the project and experiences that I had throughout the summer trying to solve various problems related to the implementation of the project.
The original proposal I submitted to Google underwent some modifications as the project progressed, details of which can be found later in this post.
Summer of Code started with the community bonding period – during which I attended the regular Monday meetings at MetaBrainz’s IRC channel #metabrainz and interacted with the MB community members. I added multiple new entities to the BookBrainz’s website and helped some users with BookBrainz related queries on the community page (intended for support/general QA related to all of MetaBrainz projects).
Also during this period, my mentor Ben Ockmore and I discussed and finalized the architecture of the importer application. It was decided to split the entire importer into two microservices: one for producer (which reads the data dumps and produces generic objects for each record using BookBrainz data storage format) and the other of consumer (which reads and validates the generic object and then insert them into the BB database). It was decided to connect these microservices using a message broker queue (RabbitMQ was finalized). In addition to this, the code repository architecture was decided to be such that we should be able abstract away the entire message broker logic, so that later it would be possible to swap out RabbitMQ with any other hosted service later (like pubsub).
Fig. 1: Initial design of the intended importer application. For more information, visit the original document.
The program coding period kicked off with making changes to the existing BookBrainz schema to enable it support our new imports. The initial design as discussed here was later updated to include views as well for imports per entity to enable simpler queries.
Following this, I started working on the bookbrainz-data repository to add some basic functions for aiding the import process. I started work in accordance with one of the existing roadmaps for the BookBrainz project which was to shift all database logic from bookbrainz-site to bookbrainz-data – adding features on a per-function basis. Initially it was decided to use Immutable.js for all data flowing in and out of bookbrainz-data-js, but very soon we realized that it was not practical to follow this approach. After some discussions, we finally settled on this repository design change to incorporate new function-oriented functionality. We named this new sub-module func.
Once I had basic functions to handle database transactions in place, I started working on the importer architecture. It was decided to create multiple instances of the producer process each with the ability to run asynchronous operations on it’s own. Similarly, we should be able to fork multiple consumer process, each capable of fetching data from queue and sending it off to the database.
To address this problem, I started working on a module which given a function would make it possible to run multiple processes running multiple instances of the given function. It should be such that we can generate the arguments dynamically for each process and along with some set-up and tear-down actions before and after we fork the process.
To get a better grasp of underlying functionality, one can read the final API and documentation. It’s a generic module which can be used for any functionality. The diagram for it’s execution flow can be found below:
Fig 2. AsyncCluster module execution flow. For more details, see the complete documentation.
While developing producers, I first designed the generic producer object structure for all entity types – an object skeleton which all producers need to create from the read records to be pushed into the queue. This object structure was to be enforced across all data sources, and this helps the consumers to expect an object of fixed nature on which they can later run automated validation tests prior to adding to the database.
As the data dumps were of considerable size, I used data streams to read the data from the flat files and parsed them to create generic entity objects which used BookBrainz data storage format. After parsing each record, I pushed the records into the queue.
Parsing required thorough analysis of the data dumps, and manually mapping each key-value pair in the data dump to the generic object structure. All the data which did not fit the present BB schema (and hence had to be excluded from the generic producer object structure) was added to a metadata field associated with the import. This metadata field is stored as a bjson in the database, so that we can individually query and index any of the fields in the metadata later.
While developing the consumers, I initially set up a validation module. Much of it was adapted from the existing validators on BookBrainz site, which I was able to use without much alteration thanks to the generic producer object structure. The validation modules in the bookbrainz-site have been written to quickly validate the form submitted by the editor post creation/editing of an entity. To use them in the import process, I wrote a converter which transforms the generic producer object to form sections understood by the validation module. Apart from this, I added better error handling to ensure all errors are caught and reported in case a something goes wrong.
Error handling was another important aspect of the import process, apart from the validation process. Being a command line application, tracking errors was central to ensuring that all components were running as intended. At the same time we had to ensure that no record which could be potentially imported into the database was missed. To address these problems, I decided to discard the record if it fails the data integrity validation tests (which means the data is most probably corrupt). But in the case of any transaction error, we give a fixed number of chances to the record before discarding it (by acknowledging the message). A future goal for this process could be to push the erred record into another queue for analysis and replaying of those messages from this queue back to the original queue when the problem gets sorted.
Once the importer was in place, I focused on building up the func.imports module with more functions for the import entities – like discard and approve. I also added functions to fetch recent imports, and a lot other helper code for the imports. I also ensured that all errors occur loudly and never silently slip away. With the help of my mentor, I also migrated most of the functions required for data transactions on the bookbrainz-site. This was crucial to my project – as in many instances the existing functions could not be used due to them initiating their own database transaction for each action. I split all these actions into functions, and bound them with the transaction object they received rather than initiating their own transaction. I also ensured we use modern ES6 features – which made the adapted code much more sleek and compact. It was a long process, as I had to read almost entire of existing code for data transactions on the bookbrainz-site and adapt each of them correctly. All the code finally came together in the create-entity module – which can now be used for entity creation as well as upgrading the imports to entities.
The work on bookbrainz-site and bookbrainz-data mostly happened side by side. First I added a recent imports page – which would fetch most recent imports from the database and display them inside the React component. The recent imports is designed as a single page application which dynamically loads the paginated records and renders them on-screen. The working of the recent page application is as follows:
Fig. 3: Recent imports execution flow
Next, I added import-entity display pages for all five entity types. They were supposed to display the entity attributes along with links to approve/discard/edit and approve functionality. Approving the import-entity was done so that the user gets redirected to the newly created entity. The import-entity display page for work is as follows:
Fig. 4: Work Import Entity Page. Similarly, pages were added for Creator, Entity, Publication and Publisher.
In case of a discarded import, I added an extra page similar to existing confirm deletion page – which asks the user to confirm the action and then waits until the entity is deleted before redirecting the user to the home page. The discard page looks as follows:
Fig. 5: Discard Import Entity Page
Next, I implemented the editing imports prior to approval. For this, I wrote two modules – one to transform the import to the structure used by the editing form and one to convert form data to an entity. When a user wishes to edit an import, the import is transformed to the form and rendered on the screen. The user can then edit the import. When the user clicks submit, we transform the form data to a new entity type and use the create-entity function to create a new entity in the BookBrainz database. The user is then redirected to the newly created entity page. The code for rendering the form and editing the entity was completely reused for imports with minimal changes. I then added functionality to add imports to the ElasticSearch index, and display them in present results. The final search page is as follows:
Last three months have been a fantastic experience for me. Not only did I get to learn a lot of new technologies and write some exciting software, but also I got to brush up my existing skills and interact with the completely awesome MetaBrainz community. Such an opportunity comes truly once in a lifetime, and I extend my sincerest gratitude to Google for running such a great and extremely inclusive programme which allows students from all over the world to avail such an opportunity. Special thanks to my mentor Ben Ockmore for always being patient and helping me out whenever I felt stuck.
Thank you MetaBrainz community for your continuous guidance and support!