MetaBrainz Foundation Community Blog
Hi everyone! As you might remember, our last post was the end of a series of entries meant to summarize everything that had happened in instrument-land so far. So now that catching up is done, here’s what has happened lately!
This post is rather large, so if you only read one thing about instruments today, make it our new guide for requesting instruments! (More details about this in the last section below.)
Continue reading “Interlude: Ensembling!”
One of the things various people have asked MusicBrainz for time and time again has been genres. However, genres are hard to do right and they’re very much subjective—with MusicBrainz dealing almost exclusively with objective data. It’s been a recurring discussion on almost all of our summits, but a couple years ago (with some help from our friend Alastair Porter and his research colleagues at UPF), we finally came to a path forward—and recently Nicolás Tamargo (reosarevok) finally turned that path forward into code… which has now been released! You can see it in action on e.g., Nine Inch Nails’ Year Zero release group.
For now genres are exactly the same as (folksonomy) tags behind the scenes; some tags simply have become the chosen ones and are listed and presented as genres. The list of which tags are considered as genres is currently hardcoded, and no doubt it is missing a lot of our users’ favourite genres. We plan to expand the genre list based on your requests, so if you find a genre that is missing from it, request it by adding a style ticket with the “Genres” component.
As we mentioned above, genres are very subjective, so just like with folksonomy tags, you can upvote and downvote genres you agree or disagree with on any given entity, and you can also submit genre(s) for the entity that no one has added yet.
A bunch of the people asking for genres in MusicBrainz have been application developers, and this type of people are usually more interested in how to actually extract the genres from our data.
The method to request genres mirrors that of tags: you can use inc=genres to get all the genres everyone has proposed for the entity, or inc=user-genres to get all the genres you have proposed yourself (or both!). For the same release group as before, you’d want https://musicbrainz.org/ws/2/release-group/3bd76d40-7f0e-36b7-9348-91a33afee20e?inc=genres+user-genres for the XML API and https://musicbrainz.org/ws/2/release-group/3bd76d40-7f0e-36b7-9348-91a33afee20e?inc=genres+user-genres&fmt=json for the JSON API.
Since genres are tags, all the genres will continue to be served with inc=tags as before as well. As such, you can always use the tag endpoint if you would rather filter the tags by your own genre list rather than follow the MusicBrainz one, or if you want to also get other non-genre tags (maybe you want moods, or maybe you’re really interested in finding artists who perform hip hop music and were murdered – we won’t stop you!).
You can parse the tag→genres from entities.json in the root of the “musicbrainz-server” repository which will give you a list of what we currently consider genres. Then you can simply compare any folksonomy tags from the %_tag tables.
One thing to keep in mind for any data consumers out there is that, as per our data licensing, tags—and thus also genres—are not part of our “core (CC0-licensed) data”, but rather part of our “supplementary data” which is available under a Creative Commons Attribution-ShareAlike-NonCommercial license. Thus, if you wish to use our genre data for something commercial, you should get a commercial use license from the MetaBrainz Foundation. (Of course, if you’re going to provide a commercial product using data from MusicBrainz, you should always sign up as a supporter regardless. :)).
We are hoping to get a better coverage of genres (especially genres outside of the Western tradition, of which we have a very small amount right now) with your help! That applies both to expanding the genre list and actually applying genres to entities. For the latter, remember that everyone can downvote your genre suggestion if they don’t agree, so don’t think too much about “what genres does the world think apply to this artist/release/whatever”. Just add what you feel is right; if everyone does that we’ll get much better information. 🙂
In the near future we’re hoping to move the genre list from the code to the database (which shouldn’t mean too much for most of you, other than less waiting between a new genre being proposed for the list and it being added, but is much better for future development). Also planned is a way to indicate that several tags are the same genre (so that if you tag something as “hiphop”, “hip hop” or “hip-hop” the system will understand that’s really all the same). Further down the line, who knows! We might eventually make genres into limited entities of a sort, in order to allow linking to, say, the appropriate Wikidata/Wikipedia pages. We might do some fun stuff. Time will tell!
At the end of the last version I was getting into the groove—and feeling determined.
I had managed to vittle down the instrument backlog to half! I though surely I could do even better! Let’s start digging into the meat of things and, get serious.
Now we were going to do things seriously, so therefore this was a pretty big version!
It’s called:
I started by looking into a few issues during autumn:
However right in the middle of this version came GCI!
I was part of the contest as a mentor this year (for the first time) so naturally I had some instrument tasks ready for our participants. They were pretty popular, with a lot of tickets being added or improved by the participants. So, this is how serious we got for GCI:
After GCI we still had quite a bit of seriousness left:
The majority of these where done during autumn ’17 and the first half of ’18. The whole thing would have been finished in the summer, hadn’t May, June and July ’18 been the hottest summer ever! I just couldn’t think, so I took a break until August and then I finished dealing with the last few stragglers.
This version ended up including 58 tickets: the largest to date!
What happens now? Well, at the end of the “Let’s get serious” version I noticed some issues with families and the like, but that’s a topic for the next blog post!
It’d been almost 2 years since I started, and I’ve managed to get quite a lot done! Here is a graph that illustrates the progress (notice the the jump in added tickets during GCI period, when the participants actually added them faster than I could close them!):
And with this we have finally caught up! So hopefully all future blogs from now on will be posted soon after the relevant versions are released.
After many years as a community driven project and often under-staffed, the BookBrainz project has always been the red-headed step child of our projects. A few weeks ago I asked if the community felt that we should make BookBrainz an official project of the foundation and got a very positive response.
After that, we started informally seeking developers to take on this position, leading to the hire of Monkey, who will now be the lead of the BookBrainz project, taking over for Ben Ockmore. Ben will take on a contributor role to BookBrainz going forward and remain on the project! Thanks for all of your hard efforts in the past, Ben!
While Monkey comes up to speed on the codebase, we’ve been brainstorming what features he should focus on first . The short term focus on BookBrainz will be on bringing it into our hosting setup at Hetzner, which means making the codebase ready for running inside of docker with all of the MetaBrainz specific hosting quirks. Part of this project will be to remove elastic search and to utilize our new Solr based search system that we recently released for MusicBrainz.
After getting BookBrainz moved to our hosting facility that focus will be to create a minimally viable product. What exactly does this mean? One of the frequent complaints I’ve received about BookBrainz is that it is missing core functionality of a proper metadata project. Core functionality means that a user should be able to view and edit all of the metadata that is in BookBrainz and then retrieve this data from the BookBrainz API. It should include full data dumps with incremental data dumps being added a bit later.
What do you think the missing core features of BookBrainz are?
Finally, we’re in discussions with the OpenLibrary team, wondering how to best work together and not to duplicate efforts — we’ll post more about this once we’ve reached an agreement with the OpenLibrary team on how we should proceed.
Thanks!
Hello people,
Thank you so much for reporting bugs in our Picard 2.0.0beta2 release. We fixed most of the critical bugs that you guys and gals reported. You can find the beta3 release with the fixes here – Picard 2.0.0.beta3
If you have been following our Picard related blogs, you will know that we decided to release a new stable version of Picard before the beginning of the summer.
To help us, advanced users, translators and developers are encouraged to:
Note – If any of you are seasoned Windows/macOS devs and have experience with PyInstaller, we need some help with PICARD-1216 and PICARD-1217. We also need some help with code signing Picard for OSX. Hit us up on #metabrainz on freenode for more information. We will be very grateful for any help that you may offer!
A simplified list of changes made since 1.4 can be read here.
Be aware that downgrading from 2.0 to 1.4 may lead to configuration compatibility issues – ensure that you have saved your Picard configuration before using 2.0 if you intend to go back to 1.4.
Hello people,
Thank you so much for reporting bugs in our Picard 2.0.0beta1 release. We fixed most of the critical bugs that you guys and gals reported. You can find the beta2 release with the fixes here – Picard 2.0.0.beta2
If you have been following our Picard related blogs, you will know that we decided to release a new stable version of Picard before the beginning of the summer.
To help us, advanced users, translators and developers are encouraged to:
Note – If any of you are seasoned Windows/macOS devs and have experience with PyInstaller, we need some help with PICARD-1216 and PICARD-1217. We also need some help with code signing Picard for OSX. Hit us up on #metabrainz on freenode for more information. We will be very grateful for any help that you may offer!
A simplified list of changes made since 1.4 can be read here.
Be aware that downgrading from 2.0 to 1.4 may lead to configuration compatibility issues – ensure that you have saved your Picard configuration before using 2.0 if you intend to go back to 1.4.
MetaBrainz sponsored Music Hack Day London 2014 and we had agreed to provide a prize for one of the winners. We thought that Thomas Bronte from MuseScore had the best hack and offered him a choice of a few prizes that were appropriate for hack day winners. Thomas declined and instead asked if he could pen a guest blog entry on our blog when they were ready to reveal their new project. We immediately agreed to do that, since open source projects need to stick together and help each other out. Finally, this is the blog post that Thomas and crew penned—read on to find out about their excellent new project!
OpenScore is a new crowdsourcing initiative to digitise classical sheet music by composers whose works are in the public domain, such as Mozart and Beethoven. Massive crowdsourced projects like Wikipedia, Project Gutenberg and OpenStreetMap (not to mention MusicBrainz!) have done wonders for the democratisation of knowledge, putting information and power in the hands of ordinary people. With OpenScore, we want to do the same for music.
OpenScore’s aim is to transform history’s most influential pieces from paper music into interactive digital scores, which you can listen to, edit, and share. This will be of huge benefit to orchestras, choirs, ensembles, and individuals looking for materials from which to practise music, but it doesn’t end there! All OpenScore sheet music editions will be freely distributed under Creative Commons Zero (CC0). This means there are no restrictive copyright terms, so everyone will be free to use the files for any purpose. We want to maximize the benefit to music education and research, and inspire composers and arrangers to produce new content.
The advantages of digital sheet music are huge. OpenScore Editions will be available in the popular MusicXML format which can be read by most music notation programs. The files can also be parsed by software tools for research and analysis, and can even converted to Braille notation for blind musicians. Digital scores can also be easily adapted into non-standard forms of notation for use in education, accessibility, or gaming; or turned into artistic visualisations. The works will be stored in an online database, accessible via a REST API. Each work will be associated with its composer’s MusicBrainz and WikiData IDs to enable cross referencing with existing online content.
OpenScore is the result of a partnership between two of the largest online sheet music communities: MuseScore and IMSLP. Since 2006 the IMSLP community has been searching for out-of-copyright musical editions, scanning and uploading them to create one of the world’s largest online archives of public domain sheet music in PDF format. MuseScore has a dedicated community of millions of people around the world, who use MuseScore’s website and open source notation software to compose, arrange, practise and share digital sheet music. OpenScore will harness the power of these communities to transcribe the IMSLP editions, which are currently just pictures of pages, into interactive digital scores by typing them up, one note at a time, into MuseScore’s sheet music editor.
OpenScore starts with a Kickstarter campaign to liberate 100 of the greatest classical pieces. This will help us to start developing the necessary systems to scale up to liberating all public domain music. Backers can help pick the pieces to be liberated, so if you love classical music and you wish to liberate a composer or a specific work, make sure you support the Kickstarter campaign and help spread the word about OpenScore and digital sheet music!
Continuing our story from last time…
Eventually I mean to come back to this group of instruments (and in fact this is what I am doing currently), expanding the family tree I’ve been working on, which is why this “fix” version is still open.
Tamburica: 2017 January 6th to the 25th:
Initially, what I found to be the hardest part of adding instruments was actually the addition of aliases. It was a cumbersome, time consuming and annoying part, and I often clamoured for someone to write some sort of Batch Alias script. Then, in March, loujine came in like a superhero with a batch alias script! Now thanks to this script I can add aliases en masse, which has helped me tremendously!
After reosarevok fixed INST-723, I could link hybrids with the instruments they were hybrids of, therefore a batch of “hybrid” instruments came next.
Hybrids: 2017 February to March:
By this time a community discussion came out about the use of disambiguations and descriptions (instrument exclusive fields, which, unlike disambiguations, are translatable), with some users suggesting they were too long, some pointing out issues with translating and yet others suggesting the description field should be dropped altogether. While this discussion didn’t reach a clear resolution, it’s definitely something we should revisit later.
Stay tuned for part three!
If your replicated slave threw an error trying to apply packet #104949 (showing the message ERROR: duplicate key value violates unique constraint "artist_alias_idx_primary"), then you can un-break things by doing the following:
master branch:git checkout master && git pull origin master (OR, if you don’t want to update your code, clear the dbmirror_pending tables instead:echo 'TRUNCATE dbmirror_pending CASCADE; TRUNCATE dbmirror_pendingdata CASCADE;' | ./admin/psql READWRITE)./admin/replication/LoadReplicationChangesYou can also re-import from dump 20170605-031203 or later, and replicate from there. We’re very sorry for the inconvenience.
The issue here was caused by a bug in our alias merge code that interacted strangely with dbmirror. Since that code went untouched for years, the trigger for this issue must have been extremely rare. I’ve put in place a fix for the merge logic to ensure it doesn’t happen there again, and am investigating dbmirror’s behavior to see why it didn’t sequence the updates correctly.
Hi everyone! I’m CatQuest, the Endeavouring MusicBrainz Instrument Inserter. I’ve been meaning to write a post or two about this for a while, but now finally everything is in order!
As some of you may know, I have “officially” taken over instrument additions from reosarevok. I suggested this way back in June 2016 because a) reo was already overworked enough between doing Style, handling support, dealing with social media, and other things, and b) as a consequence of this, the Instrument Requests queue had grown unmanageably large.
Initially I started without grouping, components, or “fix versions”, but later I’ve been allowed to organise everything to my liking and so, I’ve gone back and retroactively added tickets for initial batches, modelled on freso’s ORG tickets.
So without further ado: Here is a list of the instrument tickets I solved from my initiate appointment in July 2016 to roughly October:
Very first batch:
It felt natural to separate this at Pipe and Tabor, so the next batch got a different “fix” version, from October to (taking a Christmas break) January 2017:
Second batch:
Stay tuned for part two!