The General Data Protection Regulation is a complex EU regulation that stipulates many points for protecting private data of users on the Internet. Even though this is an EU regulation, it has a worldwide impact due to the nature of the Internet. This regulation comes into effect today, May 25, 2018 and is the reason why so many companies have sent you mail in the past few weeks about updating their privacy policies.
The MetaBrainz Foundation with its collection of projects is also affected by this regulation. We’ve been learning and adapting our sites to be compliant with the regulation – sadly this regulation isn’t entirely black and white and there is an incredible amount of room left for interpretation of these rules.
The good news is that this regulation is roughly in line with our established practices: We’ve always held private information in a high regard and applied the sort of rules to ourselves as we wish to have our own private data treated. Luckily, this makes our compliance effort considerably easier. We’ve made two significant changes to how we treat your data and also adopted terminology as used in the GDPR in order to use the same languages that many other sites are now adopting. Please keep reading to find out the exact details of what we are doing to comply.
However, we do ask for your compassion and help in our process of complying with the GDPR. As we already mentioned, the GDPR is a complex set of rules that are not fully clarified yet. We’ve taken action on the steps that are clear to us and we’re following ongoing conversations on points that are in gray zones or unclear to us. We’ve made our best initial effort on compliance and promise to keep working on it as the picture becomes more clear. If you believe that we could improve our compliance, please contact us and let us know what we can do to improve. It would also help us if you could provide concrete discussion or examples to help us understand and take action on your suggestion.
Finally, below is the link to our GDPR compliance statement, implementing the regulations as we understand them and how they affect your data in our ecosystem. Where possible, we provide links for deeper understanding, links for you to examine our relevant code and links to tickets to follow the process of improving our compliance.
MetaBrainz’ GDPR Compliance Statement
With the release of the Next Generation Schema in May of 2011 we officially deprecated the use of version 1 of our XML API. Now, 6 years later, we feel that we can finally pull the plug on this version of the API — it receives less than 1% of our Web Service traffic.
On the first release after 1 August 2018 we’re going to remove the Web Service version 1 (ws/1 API endpoint) support. If you are one of the few authors of software that has not updated your software to use the newer ws/2 endpoint, your software’s MusicBrainz functionality will cease to work after 1 August.
We think more than 6 years is enough time for people to upgrade their tools. 🙂
We plan to have some brief downtime on Thursday, September 14 at around 15:00 UTC for essential upgrades and maintenance across our servers. Thanks for your understanding!
Edit: Maintenance complete! Let us know if you experience any issues.
The Google Open Source Programs office is amazing!
In the past few years Google has sponsored our summit where we gather a pile of MusicBrainzers into one room and talk shop (and a bit of play) for a whole weekend. But, due to extenuating circumstances and our move to NewHost, we opted for a much smaller and lower key developer gathering in Barcelona. As a result we didn’t spend much money and I opted to not pester Google to support our summit as they have in the years past.
Then yesterday the lovely Cat Allman from OSPO send us a PO for $5000 and a stern reminder to send an invoice soon. After explaining the situation to Cat, she still felt it was appropriate to collect the money and to “use the money as best serves the project.”
Wow, thank you!
The question I have, for all of our readers, contributors, editors, hackers and advocates: How do you think we should spend $5000 to best serve MusicBrainz and its sister projects?
I’ll leave it wide open to everyone to chime in — feel free to put suggest anything reasonable and meaningful in the comments. Please do the skip the “send it to me!” type comments. 🙂
Finally, I want to thank Google for its continued support of our projects. Through its annual support of $40,000, Summer of Code, Code-In, the paid data license for its Knowledge Graph and the support of our summit, Google is the biggest supporter of the MetaBrainz Foundation. By far: Google has donated more than $366,000!
THANK YOU GOOGLE. Your support really helps us along!
I’m pleased to report that our nightmare of finding/reconstructing the missing replication packets is finally over!
Through many heroic hours of work, Bitmap and Chirlu have reconstructed the missing replication packets. All clients should now be on their way to being up to date. We’ve learned a number of lessons (some good, some bad — that’s life, right?) in this ordeal and we hope to avoid these issues in the future.
An integral part of this recovery process were a number of people from our community who helped us: Users mbcz, rembo10 and xeam sent us their complete DB dumps! Bitmap used these to sanity check and diff several other database to finally extract the missing packets. Thank you for dropping what you were doing and sending us a few GB of data over blazingly fast connections. Without you this would not have been possible; and this is not an exaggeration. Thank you!
After some more rest we’re going to continue to put out smaller fires that remain from the move to NewHost, but for now, the big fires are put out. Just in time for the weekend!
In the 11 year history of the replication stream we’ve had to have users restart their stream about 3-4 times because of problems on our end. Zero would’ve been nicer, but I’m proud that we’ve been able to make this system work for so long. On a daily basis we seem to have about 400 replicated copies of MusicBrainz running all over the world. Clearly this part of our service is well used and I sleep a little better at night knowing that our most critical data is backed up across the globe.
Just for fun, here is a graph of the replication API usage over the last 6 months:
Towards the end the graph shows the week plus long break, then a small blip as some of our replicas got unstuck yesterday and the much larger spike shows the rest of the replicas getting unstuck. Now, as to what caused the blip in mid-October — I have no idea.
Anyways, please accept my apologies for the replication stream outage and keep replicating!
In my post from yesterday I talked about our continuing struggle to fix our replication stream. Overnight we learned two new things:
- The lengthy hard drive recovery process has failed and yielded no useful results. 😦
- We have a working DB diff program in place that allows us to create missing replication packets from two DBs at known packet numbers.
#2 is a great step forward in fixing our replication stream, but we’re missing a specific replicated database at a specific point in time. If you have a replicated database, please read on and see if you can help us:
- Is your database at replication packet 99847? The way to find out your current replication sequence is to look in slave.log in your server directory or to issue the query
select current_replication_sequence from replication_control;
at the SQL prompt.
- Do you have a complete replicated MusicBrainz database, including the cover_art_archive schema?
- Are you willing to make a dump of the DB and send it to us?
- Do you have a fast internet connection to make #3 possible?
If you’ve answered yes to all of the above, please send email to support at metabrainz dot org.
It has been a long week since our move to the new hosting provider in Germany. Our move across the Atlantic worked out fairly well in the grand scheme of things. The new servers are performing well, the site is more stable and we have a modern infrastructure for most of our projects.
However, such moves are not without problems. While we didn’t encounter many problems, the most significant one we did encounter was the failure to copy two small replication packets off the old servers. We didn’t notice this problem until after the server in question had been decommissioned. Ooops.
And thus began a recovery effort that is almost worthy of a bad Hollywood B-movie plot. Between myself traveling and the team finishing the most critical migration bits, it took 2 days for us to realize the problem and find a volunteer to fetch the drives from the broken server. Only in a small and wealthy place such as San Luis Obispo, could a stack of recycled servers sit in an open container for 2 days and not be touched at all. My friend collected the drives and immediately noticed that the drives were damaged in the recycling process, which isn’t surprising. And we can consider ourselves really lucky that this drive didn’t contain private data — those drives have been physically destroyed!
Since then, my friend has been working with Linux disk recovery tools to try and recover the two replication packets off the drive. Given that he is working with a 1TB drive, this recovery process takes a while and must be fully completed before attempting to pull data off the drive. For now we wait.
At the same time, we’re actively cobbling together a method to regenerate the lost packets. In theory it is possible, but it involves heroic efforts of stupidity. And we’re expending that effort, but so far, it bears no fruit.
In the meantime, for all of the people who use our replicated (Live Data) feed — you have the following choices:
- If you need data updates flowing again as soon as possible, we strongly recommend importing a new data set. We have a new data dump and fresh replication packets being put out, so you can do this at any time you’re ready.
- If the need for updates is not urgent yet and you’d rather not reload the data, sit tight. We’re continuing our stupidly heroic efforts to recover the replication packets.
- Chocolate: It really makes everything better. It may not help with your data problems, but at least it takes the edge off.
We’re terribly sorry for the hassle in all of this! Our geek pride has been sufficiently dinged that our chocolate coping mechanisms will surely cause us to put on a pound or two.
UPDATE 1: The first recovery examination has not located the files, but my friend will do a second pass tomorrow and turn over file fragments to us that might allow us to recover files. But that won’t be for another 8 hours or so.