Server Update 2013-04-22 and a Notice Regarding Passwords

We’ve just finished deploying a new update to the MusicBrainz website, but before covering release notes we’d like to clarify a few points regarding the recent password leak.

There have been a few reports suggesting that MusicBrainz was attacked or compromised, but we can assure that this is not the case. As stated in our original blog post, we accidentally included private information in our otherwise routine public backups. As the majority of MusicBrainz data is available under the Creative Commons BY-NC-SA or CC0, these backups are intended to be public – the mistake was the human error in accidentally including password hashes.

A few of you also mentioned that we didn’t respond in a timely fashion in alerting users via emails. We completely agree with these comments, and apologise for not being able to respond quicker. Unfortunately, we did not have the infrastructure to do such mass mailing, and as we didn’t want to rush this out to the point of making yet more errors, it took us a little longer than we all would have liked.

With that out of the way, here’s what we’ve done in the last fortnight. Many thanks to Michael Wiencek, Nicolás Tamargo, and the rest of the MusicBrainz team for their work on this release.

Bug

  • [MBS-4277] – Guess Case: Keep uppercase option should be clearer
  • [MBS-4468] – “Enter vote” button incorrectly also adds edit note
  • [MBS-5597] – Funky Caps inConsisTency in Edit search paGes (presets or not)
  • [MBS-5832] – Rating not showing on Merge Recordings page
  • [MBS-5967] – No feedback after editing a user
  • [MBS-5998] – Barcodes incorrectly represented in mo RDF
  • [MBS-6077] – Reorder tracks edit not working
  • [MBS-6081] – Viewing MusicBrainz Events and Cover Art information makes the CAA launch event appear to the left of the actual graph
  • [MBS-6082] – “Remove Label” edit page has broken documentation link
  • [MBS-6100] – Internal server error with webservice requests which require authentication
  • [MBS-6111] – Amazon taking precedence over CAA after merging releases
  • [MBS-6121] – Logging in with non-ascii password throws exception
  • [MBS-6122] – Unable to merge release groups when cover art is set
  • [MBS-6169] – Check for edit user uses the wrong sub

Improvement

  • [MBS-4632] – Link to archive.org item from cover art tab
  • [MBS-4658] – Report: duplicate ISWCs
  • [MBS-5261] – Link to How to Add Disc IDs from the Disc ID tab of releases
  • [MBS-5667] – Rename “Other release groups”
  • [MBS-5800] – Remove restriction on entering standalone recordings
  • [MBS-6001] – Make the autoeditor election overview more useful
  • [MBS-6034] – label search is missing country column
  • [MBS-6086] – Setting cover art type (from nothing to something) should be an autoedit
  • [MBS-6109] – Work search doesn’t show type
  • [MBS-6110] – Relate To URL should auto-identify audiojelly.com as “can be purchased for download” relation.
  • [MBS-6119] – Seeing vote counts on closed elections shouldn’t require log in

New Feature

  • [MBS-1891] – Anchors/Permalinks on Edit Notes
  • [MBS-5737] – Notification when RE has created a release

Task

  • [MBS-6096] – Add soundtrackcollector.com to the Other Databases whitelist

The Git tag for this release is v-2013-04-11.

libdiscid 0.5.0 released

A new libdiscid release was made available today.

Changes:

– LIB-29: add read_sparse() for faster reading again
– LIB-35: add HAVE_SPARSE_READ and VERSION_* defines
– LIB-36: hide internal symbols on Linux/Unix
– LIB-34: distmac and distwin32 cmake targets

The important change is the read_sparse() function:

Philipp Wolfer added a read_sparse() function. With this function you can either only read the TOC or specifiy which features of the disc you want to read.
The normal read() also extracts ISRCs starting with 0.3.0.
You might want to change existing applications to use read_sparse if you care about performance and don’t use ISRCs. The TOC is usually cached, so read_sparse() can be faster (0,5 vs. 3 seconds measured).
The difference is only in where the time is spent. It doesn’t really save overall time, but the TOC is read right when the disc is inserted so no additional disc access is performed when using the TOC in your application.
To make it possible to keep using read() when read_sparse() is not available we provide the HAVE_SPARSE_READ define, which can be used like that:

#ifndef DISCID_HAVE_SPARSE_READ
#define discid_read_sparse(disc, dev, i) discid_read(disc, dev)
#endif
…
discid_read_sparse(disc, device, 0)

We also provide defines for the libdiscid version numbers.
However, you should rather test for features/functions in the build files and create specific defines for your use case.
The above define is only provided as a convenience for read_sparse().

There are more details about the other changes in the full announcement mail.

If you didn’t follow the musicbrainz-devel list:
This year brought several new releases for libdiscid, starting with ISRC and MCN support in libdiscid 0.3.0. Applications using libdiscid 0.2.2 (or even lower) still work with libdiscid 0.5.0.

Information, documentation and other links are at:
http://musicbrainz.org/doc/libdiscid
That includes builds for Windows and Mac OS X.

Server Update, 2013-04-08

We’ve just finished pushing out another fortnight of work to the main MusicBrainz servers. While work continues on the forthcoming schema change release, we have fixed a few bugs and added a few improvements. Thanks to Lukáš Lalinský and the rest of the MusicBrainz team for their work on this release! Here’s what’s changed:

Bug

  • [MBS-5524] – Edit relationship type edits are missing information for identifying the relationship type
  • [MBS-5830] – Attributes are not shown in relationship type edits
  • [MBS-5964] – Artist credit’s sort name tooltip doesn’t update after edit
  • [MBS-5980] – Cover art comments can make the cover art page look very unstructured
  • [MBS-5993] – name variation checks happen after HTML entity conversion
  • [MBS-6019] – Some FreeDB data getting wrongly imported from the dumps
  • [MBS-6021] – Track count search in Add Disc uses full release count for multi-disc releases
  • [MBS-6038] – "See all {num} {entity}" is just wrong i18n-wise
  • [MBS-6042] – Release country field in edit search breaks after doing a search
  • [MBS-6043] – In front page "Recent Additions" the text for the cover art pop-up labels (HTML title attribute) is wronlgy escaped
  • [MBS-6075] – An error occured while loading this edit

Improvement

  • [MBS-5613] – Set cover art page does not show enough information
  • [MBS-6039] – Make it possible to display attributes after the link phrase

Sub-task

  • [MBS-4258] – Make dates translatable

The Git tag for this release is v-2013-04-08.

Potential Security Leak

What Happened?

On March 29th 2013 we discovered that one of the MusicBrainz database dumps contained password hashes for a large portion of MusicBrainz accounts. While we don’t believe that these password hashes are either useful or widely distributed, we are requiring all users change their passwords.

What Data Was Leaked?

bcrypt password hashes, with a cost parameter of 8, for all accounts as of March 25th 2013.

Why Did This Happen?

We’ve recently began work on a long standing ticket against MusicBrainz server – MBS-357, “don’t store passwords in clear text”. We’re going to be moving away from clear text passwords, and we’ve decided to use one of the current industry standards for hashing passwords – bcrypt. Using bcrypt means that MusicBrainz will store only the hashes of passwords, which in laymans terms is a “fingerprint” of the password. Hashing means that we never store the actual password, but only the hash. There are many hashing functions available, and bcrypt is designed to be an expensive hash to compute with an adjustable “cost” – this makes it very hard to find out what the original password was via brute force attacks.

While this does mean that it’s hard to extract passwords from the hashes, the initial round of hashing passwords to move away from clear text is time consuming. As such, we built a small program that would gradually hash passwords over the course of a few days in order to make the switch from clear text passwords to secure password hashes done with as little downtime as possible.

This script hashed the password into the bcrypt_password column for all editors, and would also be notified when users changed their password in order to update the hash. Unfortunately, our database dump scripts sanitize this data by excluding data after-the-fact, rather than declaring what data to dump before running the script. As such, it dumped the entire editor table with the new column, as we forgot to add a rule to exclude this column.

Our Response

The database dumps that contain this data were promptly deleted, and have been replaced with correctly sanitized database dumps. Unfortunately logs from this server do show that this database dump was downloaded, and as we have no real indication of where this data now is, we’re treating this seriously. We have adjusted our database dumping scripts to be very specific about exactly which data they should export, so that in the future we will not leak private data by making the same mistake again.

We’re extremely sorry about this mistake, and while we don’t believe this data should allow attackers to retrieve user passwords, we can’t be 100% certain. As such, we require that all users change their password as soon as possible.

Search server fixes released

Last week’s search server release had some bugs that we decided should be fixed sooner than later. Paul Taylor rose to the challenge and fixed 4 important bugs and we just finished releasing the updated code. Thanks for your efforts, Paul!

Release Notes – MusicBrainz Search Server – Version 2013-04-04

Bug

  • [SEARCH-279] – Seach server returning wrong results
  • [SEARCH-280] – Artist search DAVID BOWIE → FRANZ SCHUBERT (score 100) !? Bowie (score 0)
  • [SEARCH-281] – If set explain=true option with dismax search it actually does a non-indexed search

Improvement

  • [SEARCH-267] – Create new rewrite method for Dismax FuzzySearch

Picard 1.2 released

Version 1.2 of Picard has been released today, with some notable features and bug fixes.

As announced recently, PUID support is being removed from MusicBrainz on October 15. This release of Picard has followed suit by removing all support for AmpliFIND/PUID scanning and submission. AcoustID is now the default (and only) method of scanning files. If you’re currently using AmpliFIND as your fingerprinting system, then after installing Picard 1.2 you must enable AcoustID in your options, under the “Fingerprinting” section.

Those who have the Picard cover art plugin installed will likely want to remove it. Picard 1.2 has built-in support for downloading cover images from a variety of sources, without the need for a plugin. Please see the new configuration under Options -> Cover Art.

Something important to note for people who build/package Picard is that Python 2.6 is now the minimum required version.

Special thanks to Wieland Hoffmann, Laurent Monin, Lukáš Lalinský, and everyone who’s helped contribute/report bugs for this release!

Picard 1.2 can be downloaded at:
http://musicbrainz.org/doc/MusicBrainz_Picard

Changes since 1.1:

  • Picard now requires at least Python 2.6
  • Removed support for AmpliFIND/PUIDs
  • Add support for the Ogg Opus file format
  • It’s now possible to download cover images without any plugin. Cover Art Archive images can be downloaded by image type
  • Improved directory scanning performance
  • Prefer already-loaded releases of the same RG when matching files
  • Allow dropping new files onto specific targets
  • Add basic collections management support (PICARD-84)
  • Allow adding custom tags in the tag editing dialog (PICARD-349)
  • Fix replacing of Windows-incompatible characters (PICARD-393)
  • Save both primary and secondary release types (PICARD-240)
  • Handle errors from the AcoustID service better (PICARD-391)
  • Accept HTTPS URLs on drag-and-drop (PICARD-378)

Updated search server now live

We’ve just updated our search servers to the latest version. Thanks to Paul Taylor for his long hard work porting our code to Lucene 4.1. Big thanks also to Murdos and Nikki for helping with this release!

Also, we’ve installed a small gigabit ethernet switch for our search server cluster so that we can move new search indexes around much faster. Hopefully we will see indexes updating in just over 2 hours from now on.

Read on for all the details of this release:

Release Notes – MusicBrainz Search Server – Version 2013-03-29

Bug

  • [SEARCH-239] – Search updater doesn’t update the index last-updated value returned by SEARCH-232
  • [SEARCH-244] – Since the October Schema Change Release, search server is now returning empty join phrases when once doesnt exist , whereas before it didn’t display it all
  • [SEARCH-247] – Runtime Exception: The property or field count on the class org.musicbrainz.mmd2.Medium$TrackList is required to be included in the propOrder element of the XmlType annotation.
  • [SEARCH-263] – Search server applications do not gracefully disconnect from PostgreSQL on termination

Improvement

  • [SEARCH-217] – Allow searching and displaying of folksnomy tags for the release entity
  • [SEARCH-246] – Extend support for searching for blank parameters to ISWC and ISRC

New Feature

  • [SEARCH-228] – Let Dismax Search for Labels search Label Code

Task

Updating our privacy policy

A number of bugs have been piling up pointing out inconsistencies in our privacy policy. We’ve realized that the existing privacy policy is quite brittle with respect to our fast paced server development cycle. I’ve edited the privacy policy in an attempt to future proof the policy while not eroding anyone’s privacy. The revised policy has already had some community review and now I would like to cast the net to a wider audience for review.

However, I would like to set the stage carefully here: This is not a call to re-write our privacy policy or to nit-pick it to pieces. I am trying to nail down the open bugs and ensure that our privacy policy reflects our current actual activities. So, unless you find an actual problem that needs to be addressed, I am not likely to make any further changes to the revised policy.

That said, here is the list of bugs I’m trying to close: MBS-5708, MBS-5709, OTHER-144, OTHER-145, MBS-5942, MBS-5948.

Here is the new privacy policy that should fix the bugs above. Compare that to our current policy and also have a look at the diff of the wiki markup between the two pages.

Server update 2013-03-25

We’ve just finished deploying changes from the last fortnight. This release is mostly bug fixes, but we also have a bit more validation when entering label URLs and some improvements to reports. Many thanks to Alastair Porter, Frederik “Freso” S. Olesen, Lukáš Lalinský, Nicolás Tamargo, Paul Taylor, Pavan Chander and the MusicBrainz team for their work on this release. Detailed changes follow:

Bug

  • [MBS-4364] – Various Artists should not have a gender (and other attributes)
  • [MBS-4868] – Regression : « → view aliases » link disappeared from alias edits
  • [MBS-5947] – Changed AC display doesn’t detect unchanged gaps
  • [MBS-5953] – oauth: web application reauthorization creates table spam
  • [MBS-5958] – Release editor did something very very wrong
  • [MBS-5979] – Automatic redirect to beta clear release editor seeding
  • [MBS-5996] – Release and work level relationships getting mixed up in release pages
  • [MBS-6012] – Internal server error in Wikipedia extract
  • [MBS-6016] – browse works by invalid artist id causes an error

Improvement

  • [MBS-2535] – Lookup labels based on Label Code when entering releases
  • [MBS-4894] – Reject non-BBC Music URLs when using the BBC Music relationship
  • [MBS-5491] – Check for “vs.” and “feat.” in PossibleCollaborations report
  • [MBS-5912] – Report of releases with cover art relationships

Task

  • [MBS-5975] – Add Anime News Network Encyclopedia to “Other Databases” whitelist
  • [MBS-5976] – Add Generasia.com to the “other databases” whitelist

The Git tag for this release is v-2013-03-25.