Schema change release, 2013-05-15

Today we released a schema change update for MusicBrainz. Schema change updates change the format of the underlying MusicBrainz database and allow us to store more information, or model information in a richer/more correct form.

Summary

This release specifically includes some exciting new features:

Areas

A new entity is this release is the “area” entity, which can track countries, subdivisions of countries, cities, and other such location entities (venues, however, will be another entity). While this release primarily only introduces the entity, migrating only our existing list of countries, it’s now also possible to add start and end locations to artists, and to mark works as anthems. Editing of areas, their aliases and annotations, and area-area and area-url relationships, is limited to a new class of “Location Editor”.

ISNI Codes

The International Standard Name Identifier (ISNI) is an ISO standard that identifies public identities of parties. We now have support for storing ISNI codes inside the MusicBrainz database, which will make it easier to cross-reference data in MusicBrainz with other databases.

Multiple Release Events

Releases can now have multiple date and country pairs, whereas previously they could only have one country and one date. This will allow us to more accurately store information about releases that occur in different areas at different dates, but are otherwise the same physical product.

Forthcoming Features

We also began work on the database support for some future MusicBrainz features:

Track MBIDs

All tracks on mediums now have unique identifiers. This will allow people to refer to a specific track in a release in a way that is more resilient to editing than just the track name or position. Currently we have database support for this, but track identifiers are not yet exposed in either the website or the web service.

Dynamic Work Attributes

Dynamic work attributes will let us introduce new attributes to describe works without schema changes

Free Text Relationship Attribute Credits

This feature will let editors specify an alternative name for relationship attributes to specifically exactly which model guitar was used in a recording, rather than the current vague “electric guitar” attribute. Support for this feature is now in the database (in the link_attribute_credit) table, but the UI to do this editing is still to be finished.

Support more formats in the Cover Art Archive

Uploading cover art to the cover art archive will soon support a few other image formats, starting with PNG.

Other Schema Changes

The remaining smaller schema changes are:

  • The wiki transclusion version mapping is now stored in the database, not a flat file.
  • The link_type.short_link_phrase was renamed to link_type.long_link_phrase.
  • The work.artist_credit column was dropped.
  • Collections can now have a description.

Upgrading

If you are currently running a slave database, then you will need to perform a few manual steps to upgrade to the new version:

  1. Take down the web server running MusicBrainz, if you’re running a web server.
  2. Turn off cron jobs if you are automatically updating the database via cron jobs.
  3. Make sure your REPLICATION_TYPE setting is RT_SLAVE
  4. Switch to the new code with git fetch origin followed by git checkout v-2013-05-15
  5. Run carton install --deployment to install any new perl modules.
  6. Run carton exec -Ilib -- ./upgrade.sh from the top of the source directory.
  7. Set DB_SCHEMA_SEQUENCE to 17 in lib/DBDefs.pm
  8. Turn cron jobs back on, if needed.
  9. Restart the MusicBrainz web server, if needed.

Please see the instructions in our more recent blog post instead of these instructions.

Release Notes

This release wouldn’t have possible without help from Alastair Porter, Michael Wiencek, Nicolás Tamargo or the rest of the MusicBrainz team – thank you all for your hard work! As we missed the previous release, there are a few other changes in this release. Here are the full release notes:

Bug

  • [MBS-4703] – Add Medium edit does not correctly display the auto-edit note
  • [MBS-5834] – Users’ votes page mistitled as “edits”
  • [MBS-5851] – Instruments are missing from the JSON webservice responses
  • [MBS-5979] – Automatic redirect to beta clear release editor seeding
  • [MBS-6015] – Edit Artist Credit edits affecting track ACs don’t appear in related release edit histories
  • [MBS-6129] – Relationship editor isn’t correctly parsing attributes
  • [MBS-6149] – Entity merges silently dropping aliases with locales
  • [MBS-6178] – URL page headers are completely inconsistent
  • [MBS-6196] – work/edit_form.tt includes artist credit docs
  • [MBS-6249] – Add Event button broken on add release with cdtoc

Improvement

  • [MBS-1346] – New Report: Artists with 0 subscribers
  • [MBS-2229] – Allow multiple release events per release
  • [MBS-3626] – Display license logos in the sidebar
  • [MBS-3669] – Merge dated and undated relationships
  • [MBS-4115] – Cover art archive: Support .png SQL changes
  • [MBS-4294] – Add a “description” field to collections (SQL/UI)
  • [MBS-4756] – Move the wiki transclusion index to the database
  • [MBS-4866] – URL autoselect: ameblo.jp -> has blog at
  • [MBS-4867] – Timeline graph rate-of-change graph should change its vertical scaling depending on where it’s zoomed
  • [MBS-4925] – Add country of birth and country of death to Artist (person)
  • [MBS-5528] – Change short_link_phrase to long_link_phrase
  • [MBS-5772] – Generate relationship documentation (semi-)automatically
  • [MBS-5848] – Instrument credits (SQL)
  • [MBS-6023] – Track MBID UI changes
  • [MBS-6141] – Add discography page URL matching for universal-music.co.jp, lantis.jp, jvcmusic.co.jp, wmg.jp, avexnet.jp and kingrecords.co.jp
  • [MBS-6142] – Prevent Wikipedia links from being added as discography page relationships
  • [MBS-6188] – Remove rating from work merge page
  • [MBS-6189] – Show work languages on ISWC page
  • [MBS-6190] – Artist credit diffs per word, join phrase per char

New Feature

  • [MBS-799] – Location, venue and event support
  • [MBS-1839] – Track MBID SQL changes
  • [MBS-2417] – Support multiple countries/regions on a single release
  • [MBS-3296] – Add dynamic attributes
  • [MBS-3985] – Support multiple artist countries
  • [MBS-5272] – Create daily and weekly “rollup” replication packets
  • [MBS-5302] – Store International Standard Name Identifier (ISNI, ISO 27729) for artists and labels
  • [MBS-5861] – Dynamic work attributes (SQL)

Task

  • [MBS-5314] – Drop the work.artist_credit column
  • [MBS-6133] – Ensure JPEG uploads still work through CAA
  • [MBS-6167] – Add iTunes links to the sidebar
  • [MBS-6170] – Add Bandcamp links to the sidebar

Sub-task

  • [MBS-4217] – Spotify relationship under the External links section
  • [MBS-5809] – SQL changes for MBS-4294: Add a “description” field to collections
  • [MBS-5917] – transclusion to DB: SQL
  • [MBS-5918] – transclusion to DB: UI
  • [MBS-5919] – locations: SQL
  • [MBS-5920] – locations: UI
  • [MBS-5929] – Schema changes to store relationship type guidelines and example usages in the database
  • [MBS-5930] – Generate documentation automatically/allow managing guidelines in DB
  • [MBS-5933] – Schema changes to support multiple (date, country) pairs on releases
  • [MBS-6187] – UI changes to support multiple country/date pairs on releases

Server Update 2013-04-22 and a Notice Regarding Passwords

We’ve just finished deploying a new update to the MusicBrainz website, but before covering release notes we’d like to clarify a few points regarding the recent password leak.

There have been a few reports suggesting that MusicBrainz was attacked or compromised, but we can assure that this is not the case. As stated in our original blog post, we accidentally included private information in our otherwise routine public backups. As the majority of MusicBrainz data is available under the Creative Commons BY-NC-SA or CC0, these backups are intended to be public – the mistake was the human error in accidentally including password hashes.

A few of you also mentioned that we didn’t respond in a timely fashion in alerting users via emails. We completely agree with these comments, and apologise for not being able to respond quicker. Unfortunately, we did not have the infrastructure to do such mass mailing, and as we didn’t want to rush this out to the point of making yet more errors, it took us a little longer than we all would have liked.

With that out of the way, here’s what we’ve done in the last fortnight. Many thanks to Michael Wiencek, Nicolás Tamargo, and the rest of the MusicBrainz team for their work on this release.

Bug

  • [MBS-4277] – Guess Case: Keep uppercase option should be clearer
  • [MBS-4468] – “Enter vote” button incorrectly also adds edit note
  • [MBS-5597] – Funky Caps inConsisTency in Edit search paGes (presets or not)
  • [MBS-5832] – Rating not showing on Merge Recordings page
  • [MBS-5967] – No feedback after editing a user
  • [MBS-5998] – Barcodes incorrectly represented in mo RDF
  • [MBS-6077] – Reorder tracks edit not working
  • [MBS-6081] – Viewing MusicBrainz Events and Cover Art information makes the CAA launch event appear to the left of the actual graph
  • [MBS-6082] – “Remove Label” edit page has broken documentation link
  • [MBS-6100] – Internal server error with webservice requests which require authentication
  • [MBS-6111] – Amazon taking precedence over CAA after merging releases
  • [MBS-6121] – Logging in with non-ascii password throws exception
  • [MBS-6122] – Unable to merge release groups when cover art is set
  • [MBS-6169] – Check for edit user uses the wrong sub

Improvement

  • [MBS-4632] – Link to archive.org item from cover art tab
  • [MBS-4658] – Report: duplicate ISWCs
  • [MBS-5261] – Link to How to Add Disc IDs from the Disc ID tab of releases
  • [MBS-5667] – Rename “Other release groups”
  • [MBS-5800] – Remove restriction on entering standalone recordings
  • [MBS-6001] – Make the autoeditor election overview more useful
  • [MBS-6034] – label search is missing country column
  • [MBS-6086] – Setting cover art type (from nothing to something) should be an autoedit
  • [MBS-6109] – Work search doesn’t show type
  • [MBS-6110] – Relate To URL should auto-identify audiojelly.com as “can be purchased for download” relation.
  • [MBS-6119] – Seeing vote counts on closed elections shouldn’t require log in

New Feature

  • [MBS-1891] – Anchors/Permalinks on Edit Notes
  • [MBS-5737] – Notification when RE has created a release

Task

  • [MBS-6096] – Add soundtrackcollector.com to the Other Databases whitelist

The Git tag for this release is v-2013-04-11.

Server Update, 2013-04-08

We’ve just finished pushing out another fortnight of work to the main MusicBrainz servers. While work continues on the forthcoming schema change release, we have fixed a few bugs and added a few improvements. Thanks to Lukáš Lalinský and the rest of the MusicBrainz team for their work on this release! Here’s what’s changed:

Bug

  • [MBS-5524] – Edit relationship type edits are missing information for identifying the relationship type
  • [MBS-5830] – Attributes are not shown in relationship type edits
  • [MBS-5964] – Artist credit’s sort name tooltip doesn’t update after edit
  • [MBS-5980] – Cover art comments can make the cover art page look very unstructured
  • [MBS-5993] – name variation checks happen after HTML entity conversion
  • [MBS-6019] – Some FreeDB data getting wrongly imported from the dumps
  • [MBS-6021] – Track count search in Add Disc uses full release count for multi-disc releases
  • [MBS-6038] – "See all {num} {entity}" is just wrong i18n-wise
  • [MBS-6042] – Release country field in edit search breaks after doing a search
  • [MBS-6043] – In front page "Recent Additions" the text for the cover art pop-up labels (HTML title attribute) is wronlgy escaped
  • [MBS-6075] – An error occured while loading this edit

Improvement

  • [MBS-5613] – Set cover art page does not show enough information
  • [MBS-6039] – Make it possible to display attributes after the link phrase

Sub-task

  • [MBS-4258] – Make dates translatable

The Git tag for this release is v-2013-04-08.

Potential Security Leak

What Happened?

On March 29th 2013 we discovered that one of the MusicBrainz database dumps contained password hashes for a large portion of MusicBrainz accounts. While we don’t believe that these password hashes are either useful or widely distributed, we are requiring all users change their passwords.

What Data Was Leaked?

bcrypt password hashes, with a cost parameter of 8, for all accounts as of March 25th 2013.

Why Did This Happen?

We’ve recently began work on a long standing ticket against MusicBrainz server – MBS-357, “don’t store passwords in clear text”. We’re going to be moving away from clear text passwords, and we’ve decided to use one of the current industry standards for hashing passwords – bcrypt. Using bcrypt means that MusicBrainz will store only the hashes of passwords, which in laymans terms is a “fingerprint” of the password. Hashing means that we never store the actual password, but only the hash. There are many hashing functions available, and bcrypt is designed to be an expensive hash to compute with an adjustable “cost” – this makes it very hard to find out what the original password was via brute force attacks.

While this does mean that it’s hard to extract passwords from the hashes, the initial round of hashing passwords to move away from clear text is time consuming. As such, we built a small program that would gradually hash passwords over the course of a few days in order to make the switch from clear text passwords to secure password hashes done with as little downtime as possible.

This script hashed the password into the bcrypt_password column for all editors, and would also be notified when users changed their password in order to update the hash. Unfortunately, our database dump scripts sanitize this data by excluding data after-the-fact, rather than declaring what data to dump before running the script. As such, it dumped the entire editor table with the new column, as we forgot to add a rule to exclude this column.

Our Response

The database dumps that contain this data were promptly deleted, and have been replaced with correctly sanitized database dumps. Unfortunately logs from this server do show that this database dump was downloaded, and as we have no real indication of where this data now is, we’re treating this seriously. We have adjusted our database dumping scripts to be very specific about exactly which data they should export, so that in the future we will not leak private data by making the same mistake again.

We’re extremely sorry about this mistake, and while we don’t believe this data should allow attackers to retrieve user passwords, we can’t be 100% certain. As such, we require that all users change their password as soon as possible.

Server update 2013-03-25

We’ve just finished deploying changes from the last fortnight. This release is mostly bug fixes, but we also have a bit more validation when entering label URLs and some improvements to reports. Many thanks to Alastair Porter, Frederik “Freso” S. Olesen, Lukáš Lalinský, Nicolás Tamargo, Paul Taylor, Pavan Chander and the MusicBrainz team for their work on this release. Detailed changes follow:

Bug

  • [MBS-4364] – Various Artists should not have a gender (and other attributes)
  • [MBS-4868] – Regression : « → view aliases » link disappeared from alias edits
  • [MBS-5947] – Changed AC display doesn’t detect unchanged gaps
  • [MBS-5953] – oauth: web application reauthorization creates table spam
  • [MBS-5958] – Release editor did something very very wrong
  • [MBS-5979] – Automatic redirect to beta clear release editor seeding
  • [MBS-5996] – Release and work level relationships getting mixed up in release pages
  • [MBS-6012] – Internal server error in Wikipedia extract
  • [MBS-6016] – browse works by invalid artist id causes an error

Improvement

  • [MBS-2535] – Lookup labels based on Label Code when entering releases
  • [MBS-4894] – Reject non-BBC Music URLs when using the BBC Music relationship
  • [MBS-5491] – Check for “vs.” and “feat.” in PossibleCollaborations report
  • [MBS-5912] – Report of releases with cover art relationships

Task

  • [MBS-5975] – Add Anime News Network Encyclopedia to “Other Databases” whitelist
  • [MBS-5976] – Add Generasia.com to the “other databases” whitelist

The Git tag for this release is v-2013-03-25.

Server Update, 2013-02-25

Thanks to work from Ben Ockmore, Frederik “Freso” S. Olesen, Michael Wiencek, Nicolás Tamargo, nikki, Sean Burke, and the MusicBrainz team, we’ve just released a new version of the MusicBrainz website. As usual, this is mostly a bug fix release, but we do have one shiny new feature… collection subscriptions! It is now possible to subscribe to your own collections, or other public collections, and receive daily emails summarizing edits recently made that might affect releases in these collections. This feature is new, so if you encounter any problems, please be sure to let us know!

Here’s a full list of what’s changed:

Bug

  • [MBS-3514] – Part of an edit was applied but not shown in preview nor in edit history
  • [MBS-3952] – Last Updated time is wrong on the Statistics page
  • [MBS-4543] – Seeding the create relationship page ignores some parameters
  • [MBS-4774] – Edit to swap two recordings creates a blank edit
  • [MBS-4947] – /edit/open won’t load
  • [MBS-5016] – Buttons misaligned in Firefox
  • [MBS-5309] – Not all artist credit changes are shown in tracklist edits
  • [MBS-5388] – Relationship editor allows linking an entity to itself
  • [MBS-5397] – Edit medium’s diff (insert track) : shows track duration change where there are none
  • [MBS-5402] – Country (and language) dropdown is sorted by English alphabet
  • [MBS-5461] – No space between releases in collection and button to remove them
  • [MBS-5520] – Translation problem on disc IDs page
  • [MBS-5547] – Relationship Editor: Can’t unselect orchestra type
  • [MBS-5560] – Internal server error (after creating recording-work relationship via relation-editor)
  • [MBS-5565] – Link to add new label is missing when there are results
  • [MBS-5677] – URL MBIDs missing from the webservice
  • [MBS-5726] – Internal server error when requesting /ws/2/discid in JSON
  • [MBS-5765] – Very long releases have bad release length parameters
  • [MBS-5787] – i18n: “This beta test server allows testing of new features with the live database” is untranslatable
  • [MBS-5799] – Relationship editor doesn’t allow adding multiple relationships with different attributes
  • [MBS-5815] – Use “Died” in edit artist page like on sidebar
  • [MBS-5819] – Instrument select in relat. editor requires accents
  • [MBS-5821] – i18n: work types are untranslatable inside the relationship editor
  • [MBS-5824] – Internal server error loading Wikipedia extract for page name containing &
  • [MBS-5825] – Internal server error when requesting an artist with a really large ID
  • [MBS-5828] – Internal server error on /chrome_frame
  • [MBS-5836] – Relationship editor accepts POST data containing mismatched entity/link type combos
  • [MBS-5842] – “Position” is untranslatable in add_cover_art.tt
  • [MBS-5880] – Internal server error for Wikipedia extract when Wikipedia page is a redirect
  • [MBS-5889] – Subscription summary test fails, does not check collection subscriptions

Improvement

  • [MBS-1955] – Sort relationship types tree
  • [MBS-2225] – Subscribe to collection
  • [MBS-3990] – Add a user preference for using beta version of MB
  • [MBS-4094] – Make artist credits more compact in edits
  • [MBS-5568] – Work types list should be sorted
  • [MBS-5644] – relationship editor : check all/several recordings and batch-add relationship to same work
  • [MBS-5759] – Make reorder cover art an auto-edit
  • [MBS-5823] – Work alias page doesn’t allow to select the guess case options

New Feature

  • [MBS-5833] – Show off the latest releases on the homepage

Task

  • [MBS-3856] – Revert MBS-1052 “Amazon should lookup cover art by barcode too”
  • [MBS-5802] – Add autoselect + sidebar links to VIAF
  • [MBS-5849] – Adding lyricsnmusic.com to the lyrics whitelist

Sub-task

  • [MBS-4270] – Release editor JS strings should be in text.js or similar so they are translatable
  • [MBS-4997] – Add lookup url to webservice
  • [MBS-5490] – CAA coverart over SSL

The Git tag for this release is v-2013-02-25.

Server update, 2013-02-11 and an important notice regarding edits

We’ve just finished pushing out another two weeks changes to the MusicBrainz web site. While this release is predominantly a bug fix release with a few small improvements, we’ve made a fairly substantial change to the way edits are applied.

As of this release, all subsequent edits entered will have an expiration period of 7 days – a reduction from the previous 14 days. We’ve made this change in order to reduce the time that editors have to wait for changes to be applied, which should lead to an improved user experience; and we’ve also made the change to hopefully try and make the edit queue a little bit more managable. This change is exploratory, so if you find it counterproductive, we’d love to hear your thoughts. IRC, the forums and the mailing lists are all good channels to voice your feedback.

Also, we have finally made the switch to GitHub. While the existing repository URLs will continue to work, they will no longer be updated. If you want to stay up to date with the latest code, make sure to update your checkout information.

Many thanks to Frederik “Freso” S. Olesen, Michael Wiencek, Nicolás Tamargo and the rest of the MusicBrainz team for their work on this release. Here’s what’s new:

Bug

  • [MBS-3457] – Direct search results incorrectly reports a recording as "standalone"
  • [MBS-3962] – Edit search doesn’t really exclude artists with "is not"
  • [MBS-4522] – Empty annotations are not merged correctly
  • [MBS-5144] – The MusicBrainz logo in the top-left corner has a different background color than the rest of the header
  • [MBS-5395] – "Actions" column on alias page is too narrow for translations
  • [MBS-5432] – Internal server error when editing cover art
  • [MBS-5434] – Inconsistent terminology: primary/secondary types and type/extra types
  • [MBS-5506] – Edit search for "My vote is not" does not work as expected
  • [MBS-5525] – Blank edit relationship type edits
  • [MBS-5566] – No more autoedit mark ?
  • [MBS-5567] – Internal server error entering remove cover art edit
  • [MBS-5617] – "new image goes here" is not translatable
  • [MBS-5650] – ISE when attempting to approve already-closed release group edit
  • [MBS-5655] – It’s possible to "Remove [a] release label" although there’s none
  • [MBS-5696] – ModBot can fail to close ‘edit artist’ edits that violate uniqueness on (name, comment).
  • [MBS-5698] – Some move disc ID edits display nothing for the old release
  • [MBS-5700] – Subscribers aren’t removed when a user deletes their account
  • [MBS-5705] – Rows in the appearances section of the artist relationships tab are sometimes one column too short
  • [MBS-5719] – The track number of the last track on some 2-disc releases is missing
  • [MBS-5724] – Sort/copy name missing for work aliases
  • [MBS-5728] – Possible to enter ‘edit release group’ edits that fail to apply due to spaces in artist credits
  • [MBS-5732] – Empty labels don’t warn for deletion pending
  • [MBS-5740] – Useless "select all" checkbox on Release Duplicates tab
  • [MBS-5742] – Editing a work without JS on will enter a silent remove ISWC edit
  • [MBS-5748] – Not possible to "Approve" an edit where yours is the only "No" vote
  • [MBS-5750] – Subscriptions report filtering ignores labels
  • [MBS-5754] – Editing recordings from tracklist does not work if "Release Duplicate" is chosen
  • [MBS-5762] – "Work Type" and "Work Language" incorrectly left blank for "Merge Works" edits when viewing predefined edit searches
  • [MBS-5763] – Edit release edits not properly translated
  • [MBS-5764] – Work language column in the stats links to releases
  • [MBS-5770] – There is no constraint on release_label that either (or both) the label or catalog number are not null.
  • [MBS-5771] – Add utamap.com (うたまっぷ) in the lyrics URL white list
  • [MBS-5774] – "Vote on all edits" is not translatable
  • [MBS-5776] – Cover art types are not being translated on the cover art tab
  • [MBS-5777] – Users’ work ratings page has no title
  • [MBS-5786] – i18n: secondary RG type labels on overview are untranslatable
  • [MBS-5788] – No edit is created when trying to submit/attach a DiscID
  • [MBS-5791] – ISRCs can’t be removed
  • [MBS-5797] – Internal server error if the Wikipedia link is not really a Wikipedia link

Improvement

  • [MBS-1413] – Make profile bios support WikiFormat
  • [MBS-1774] – Webservice: expose UUID for AR type
  • [MBS-3535] – Stack traces should mention which server was handling the request
  • [MBS-4880] – Add Help/Info about How "Merge mediums and recordings" works
  • [MBS-4902] – Add Google+ links to the sidebar
  • [MBS-5505] – Add j-lyric.net to the lyrics whitelist
  • [MBS-5605] – Ratings on collection pages
  • [MBS-5707] – Add autoselect for IMDb URL relationships for labels
  • [MBS-5727] – Disc title missing from edit relationships page
  • [MBS-5767] – Add an index on (editor, id DESC)
  • [MBS-5775] – Update Facebook URL cleanup to use https
  • [MBS-5779] – Remove useless nbsp on common-macros.tt
  • [MBS-5798] – Normalise wikisource URLs to http
  • [MBS-5803] – Add autoselect for SecondHandSongs artist URL

Task

  • [MBS-5757] – Consider diminishing time edits stay open
  • [MBS-5766] – Open up /release/ and /work/ in robots.txt
  • [MBS-5810] – Add Open Library to the other databases whitelist

The Git tag for this release is v-2013-02-11.