Final sprint towards NewHost

So tomorrow is the day when the old servers at Digital West will be put to rest. Our developers and system administrator have been hard at work over the last few weeks (and especially the last few days!) getting everything ready for a hopefully smooth transition to hosting everything at Hetzner in Germany.

The vast majority of our sites and services are in fact already being served from Germany, but the largest one—musicbrainz.org—remains in the US for another night.

Our transition of the final services, such as musicbrainz.org, has already started, but the very last sprint of moving the remaining bits over will start tomorrow, November 8th, at around 6 AM EST / noon UTC / 13:00 CET and will follow the plan laid out in this Google document:
https://docs.google.com/document/d/1MgqZ4hiKC0MZJ400ZCOD8JlcjBjh4GNpakauf51YKQQ/edit?usp=sharing

Expect some downtime, plenty of read-only time, and general wonkiness as we get the last gears set in place to hopefully keep us running smoothly for the next several years to come!

September, October, and November 2015 Community Recap

Jeez. This has been overdue! So a lot of things happened, and this Community Recap series kind of got put on the back burner, which obviously means a lot of things have piled up, so if I forgot something you thought should have been mentioned, please share it in the comments below. 🙂 I will be doing a few months at a time until I’ve caught up to the now().

With that said, let’s proceed!
Continue reading “September, October, and November 2015 Community Recap”

We’re actually really going to take the HTTPS plunge!

Closing in on three years after stating that “We’re going to take the HTTPS plunge!”, we’re actually really going to do it now. 🙂

Most of our sites have forced HTTPS for some time (metabrainz.org, critiquebrainz.org, bookbrainz.org, listenbrainz.org), but there are still a couple of stragglers, notably musicbrainz.org and acousticbrainz.org.

For MusicBrainz, our beta site is now all HTTPS, web service and all. The main, non-beta musicbrainz.org will be going HTTPS-only except for what’s under /ws/ (ie., the web service) to allow taggers and other programs not currently using HTTPS some transition time. We do not currently have an ETA for when we will make the final jump to HTTPS-only on the MusicBrainz web service, as that partly depends on feedback from our web service users, which leads me to:

If you’re currently using the MusicBrainz web service, please try and switch your program to using beta.musicbrainz.org and see whether your program breaks or not and let us know the status of it. We are aware that some Python versions and MusicBrainz libraries do not support our setup, so while your program may fail now, it might simply be because of dependencies of your program not being updated yet and you might not need to do anything specifically on your end – however, some programs/libraries might need some updates, so the more people test and report back, the better we’ll be able to judge when we can go all-HTTPS-only on musicbrainz.org.

For AcousticBrainz, we now have a shiny new Let’s Encrypt certificate on https://acousticbrainz.org thanks to our systems administrator Zas! As a result, we are going to start redirecting all HTTP traffic to HTTPS on the AcousticBrainz website, including API queries.

In order to give everyone time to verify that their scripts correctly recognise and validate our Let’s Encrypt certificate, we are going to delay the redirect until July 1, 2016. On this date, any HTTP query will automatically be redirected to HTTPS. We will also enable HSTS, so that compliant browsers will redirect to HTTPS on the client-side.

If you have any questions about either the MusicBrainz or the AcousticBrainz transition, please ask.

Temporary autoeditor election procedure

Hello people,

In former days, new autoeditor elections were announced on the autoeditor mailing list that all autoeditors were automatically subscribed to. However, all our mailing lists, including the autoeditor one, died in September 2015 when the server they were hosted on took its last breath. This effectively halted the election of new autoeditors. It was always the plan that our new forums should be able to handle this, but our recent issues have meant development on the features necessary to completely replace the autoeditor mailing list has been slow. Thus Nicolás (reosarevok) and I had a talk about how to handle elections going forward, and we came up with this procedure:

  1. The proposer nominates the editor normally (starting an election), and then adds a post in the Autoeditors category of the forum linking to the election and asking for seconders. Like with the mailing list, this should also contain the proposer’s reasoning for proposing the candidate.
  2. Nicolás (reosarevok) will then mail out to all autoeditors with a link to the election topic (and possibly a link to this blog post).
  3. The proposer, Nicolás or me (or another autoeditor, if they’re faster) will update the thread once a 2nd seconder is found and the voting has started, and again when the voting has ended and the results are in.

We have added most autoeditors who are already signed up on the new forums to the @MB_Autoeditors group, but not all autoeditors have signed in to Discourse yet, and some have spaces or other “weird” characters in their username that make Discourse not able to parse them. If you find that you’re not in the @MB_Autoeditors group and you think you should be, please write a message to me (Freso) or Nicolás (reosarevok) via MusicBrainz and ask us to add you to the group. (Sending your message via MusicBrainz will let us know that it is indeed you/your account, so please don’t poke us on IRC or elsewhere about it.)

This is all obviously intended to be temporary, just until we’re able to get the process fully automated again. If you have any Ruby experience/know-how and would like to help out, please check out OTHER-248 (and possibly OTHER-254). There’s also MBS-8836 on the MusicBrainz server side for the Perl-istas.

Let us know if you have any concerns or questions about this (reminder: temporary) approach, either in the comments or on the forums. I personally hope it’ll work well enough to carry us through for a while longer until everything is ready.

GSoC 2016 students and projects

Google announced the final list of Google Summer of Code 2016 students and their projects yesterday. The list of MetaBrainz’ projects can be seen at our page on the GSoC site, but just for good measure, here’s the rundown:

MusicBrainz
Jeff Weeks (weeksio) returns to finish up the SOLR search server. We’re really hoping that this will be the end of our current search server woes. He will be mentored by the German duo of Ulrich Klauer (chirlu) and Rob Kaye (ruaok).
MusicBrainz Picard
Rahul Raturi (rahulr) will be working on improving searching MusicBrainz from within Picard, mentored by MusicBrainz’ senior developer Michael Wiencek (bitmap).
BookBrainz
Max Prettyjohns (QuoraUK) is going to try and take on adding gamification to our fledgling book/literature database. He will be supervised by the BookBrainz project leads and lead developers Ben Ockmore (LordSputnik) and Sean Burke (Leftmost).
ListenBrainz
Pinkesh Badjatiya (armalcolite) has pledged to tackle adding a much requested feature for our youngest project: implementing a Last.FM compatible submission API. Robert Kaye (ruaok) will be the one guiding him along.
AcousticBrainz
Daniele Scarano (hellska) will be spending the summer writing a toolkit for creating datasets, which should help researchers using AcousticBrainz. He will be mentored by MetaBrainz software engineer Roman Tsukanov (Gentlecat).
Kartik Gupta (kartikgupta0909) has set out to create an offline client for computing AcousticBrainz dataset evaluations. Alastair Porter (alastairp), the AcousticBrainz project lead, will be their mentor.
Goran Cetusic (cetko), our final student of this year, will be exploring how AcousticBrainz data can be utilised within Google’s BigQuery storage under the guidance of Alastair Porter.

Congratulations and good luck to all our students! We’re looking forwards to following your progress over the summer and see what you end up with. 🙂

For all the students that applied but did not get accepted: we appreciate your applications, and even if you did not make the cut this year, we hope that you will stick around and apply with us again next year when we know you better – and you know us better.

For now, let the community bonding… begin! 🙌

Deprecating MBIDs

This post is an April Fools joke. Rest assured, we have no intention of changing the MBID system that MusicBrainz currently uses.

But, like all good parody news items, there is an element of truth behind this post. The announcement of the Echo Nest API shutdown is real, and with this change you will no longer be able to use the Echo Nest IDs to look up information. This particularly hurts users of the Million Song Dataset, which maps each track to the Echo Nest ID. The new Spotify API isn’t even providing any compatibility api or ID mapping, leaving users to look up 1 million Spotify IDs in the remaining months that the the old Echo Nest api will remain available.

At MusicBrainz, we understand the importance of a stable identifier system. That’s why, 16 years ago, we picked these unwieldy-looking UUID identifiers – that have since proven to have stood the test of time, with room to continue growing. You can look up an MBID made 16 years ago, today – and it will still work another 16 years in the future.

Hello all,

Following Echo Nest’s bold announcement that Echo Nest ids are being replaced by Spotify IDs, we figured it was time to make our own ID change public as well – MBIDs were a fantastic idea 16 years ago, but let’s face it, they’re not the most beautiful thing around, so our MBIDs will now also be replaced by Spotify IDs to help with a proper mapping across tools. Anything without a Spotify mapping will simply get purged. This should greatly simplify the data we have and remove any doubt for some releases whether they exist or not – if they’re on Spotify, they clearly exist!

We would like to commend Echo Nest on their brave leadership in this, giving us the courage to move on from our ancient heritage and try new things. With the speed technologies evolve in this digital age, it can be hard to keep up with things and keep things fresh, but Echo Nest is showing the way forward, and we’re delighted to be able to follow so quickly in their path.

I hope you all will welcome this bold move by our team. We hope to have it ready by next schema change. We know we’re excited! 😀

PS. No, we will not provide a mapping between MBIDs and the new Spotify IDs. We trust our data users to be capable to set things up on their own. Happy hacking! 🙂

Say hello to Discourse!

Hello all members of the *Brainz community, I have got something in store just for you!

Some people may have noted talks and whispers about a grand and glorious move to use Discourse for various discussions related to MusicBrainz and all other MetaBrainz projects. The intention of it is to replace and unify both the now-dead mailing lists (R.I.P.) and our current forums. Guess what? The day has come at last!

The MetaBrainz Community Discourse can be found at https://community.metabrainz.org/ and is our new home for all discussions about MusicBrainz, BookBrainz, AcousticBrainz, and whatever other kind of *Brainz you want to talk about.

One of its major features is that it does not require yet another user (like the current forums, our ticket tracker, the wiki, …). When you press “Sign Up” or “Log In” it will ask you to authenticate with MusicBrainz to access some basic information. Once given permission, it will direct you back to the Discourse site and you’re logged in. (You can revoke the permission at a later point, should you need to.) No more having a dozen username/password combinations, just to participate in the community!

The site does still have some rough edges though, and various things are likely to get tweaked over the coming weeks, but today being the 1st day of (N. hemisphere) spring, I thought we should enjoy this season of “rebirth, rejuvenation, renewal, resurrection and regrowth” with this new baby of the MetaBrainz community.

A couple of people have already gone and started some discussions there, but feel free to go there yourself and start your own discussion. If you have started a discussion on the current/old forums, now is also a good time to restart/continue/move that discussion to the Discourse site as the forums will be put into read-only mode any day (posts will not be moved over).

If you don’t know where to start, start with reading the FAQ and after that, you could post to the introduction thread and introduce yourself. To get an overview about what’s going on, https://community.metabrainz.org/categories has a list of the categories currently in use and https://community.metabrainz.org/tags has a list of the tags in use. You could also just go to the front page, https://community.metabrainz.org/, and see what discussions are active right now and join in there. 🙂

I hope to see a lot of lively and friendly and constructive discussion going on there, so head over there and start making it happen. 😉

Your friendly neighbourhood community manager,
Freso