Post schema change fix for importing clean data

Yesterday we found a bug that prevents the import of a post schema change update data set. We’ve pushed out a fix for this and tagged it with:

v-2012-05-15-import-fix

If you’re planning on importing a new data set, make sure to check out this tag, rather than the tag mentioned in this entry.

Search server release: 2012-05-15

In case you haven’t gotten enough of release announcements, we have another one for you. Yesterday during the main releases we also released a new search server to match the main server release. Thanks much to Paul Taylor for working on this release to be timed perfectly!

UPDATE: The search server and the MMD schema repositories have been tagged with this tag:

release-2012-05-15

Bug

  • [SEARCH-198] – The artist is getting a lowered score on MBS
  • [SEARCH-199] – Search includes empty annotations
  • [SEARCH-200] – Search on release giving to much boost to matches on CatalogNo
  • [SEARCH-201] – explain option doesnt work if search results contain non ISO-8859-1 characters
  • [SEARCH-216] – Null pointer exception when building freedb

Improvement

  • [SEARCH-157] – Be able to search for a track by its metadata OR its puid
  • [SEARCH-186] – Search Server has hard coded redirect URL
  • [SEARCH-187] – Update Junit Test from 3 to 4
  • [SEARCH-202] – Allow searching for RGs based on their releases’ status
  • [SEARCH-204] – Upgrade codebase to Lucene 3.6
  • [SEARCH-214] – Add release group ID to the web service indexed search results for recordings

New Feature

  • [SEARCH-205] – Search server should return multiple ISWCs for works
  • [SEARCH-207] – Changes due to introduction of ISO-3 language code
  • [SEARCH-208] – Chnages due to Split release group attributes into two types Schema Change
  • [SEARCH-209] – Support for Multiple IPI Artists
  • [SEARCH-211] – Support for new Track ‘Number’ field in a track
  • [SEARCH-212] – Add abiility to index, display and search works by lyrics language as part of schema change
  • [SEARCH-213] – Changes due to MBS-1385:Support unknown end dates

Task

Announcing libmusicbrainz releases 4.0.3 and 5.0.1

Regrettably, a couple of errors were found close to the release of v4.0.2 and v5.0.0. I have just released v4.0.3 and v5.0.1 with the following changes:

– Fix LMB-32 – Correctly ignore unrecognised nodes
– Don’t compile using -Werror when building from tarball

The releases are available:

libmusicbrainz-4.0.3.tar.gz
(MD5 checksum: 19b43a543d338751e9dc524f6236892b)

libmusicbrainz-5.0.1.tar.gz
(MD5 checksum: a0406b94c341c2b52ec0fe98f57cadf3)

Documentation for the new version is available under

http://metabrainz.github.com/libmusicbrainz/

Apologies to all for the need to make this release so soon after the last one.

Andy

Schema change server update, 2012-05-15

Nearly one year after we released NGS, we have another schema change update with lots of new features!

This release contains 9 new features and improvements that take advantage of the new schema. These are:

  • More social user profiles which can now have Gravatars, languages (and the users proficiency) age and country.
  • More expressive aliases for artists, labels and works. Aliases can now have types, sort names and multiple aliases may be used per a locale, along with the ability to mark one alias as ‘primary’ for that locale.
  • Release group types have been separated into primary and secondary types. A release group now has 1 primary type and may have multiple secondary types. This allows us to have ‘remix compilation albums’, for example
  • Works may have multiple ISWCs
  • Artists, labels and relationships may be marked as ‘ended’ to indicate that they have ended, but the exact date is not known
  • Vinyl style/free text track numbers are now supported.
  • Works may have a lyrics language associated with them
  • Artists and labels may have multiple IPIs
  • We have moved to use ISO 639-3 for our language table. While not all languages are exposed at the moment, this gives us a lot more flexibility going forward.

Many thanks to nikki for going way beyond our expectations for testing (and patience!); to Ian McEwen for his continued work on statistics; and to the MusicBrainz team for making this all happen.

If you have a replicated instance of MusicBrainz, please follow these instructions to get your server running on the new schema:

  1. Take down the web server running MusicBrainz, if you’re running a web server.
  2. Turn off cron jobs if you are automatically updating the database via cron jobs.
  3. Make sure your REPLICATION_TYPE setting is RT_SLAVE
  4. Switch to the new code with git fetch origin followed by git checkout v-2012-05-15-schema-change
  5. Run carton install --deployment. If you have not switched your installation to using carton, please read INSTALL.md on how to do this.
  6. Run carton exec -- ./upgrade.sh from the top of the source directory.
  7. Set DB_SCHEMA_SEQUENCE to 15 in lib/DBDefs.pm
  8. Turn cron jobs back on, if needed.
  9. Restart the MusicBrainz web server, if needed.

If you are running a mbslave mirror, check out the latest code and read the upgrade instructions in the README file.

Bug

  • [MBS-3189] – Remove unused ref_count column and related functions
  • [MBS-4616] – Add work language statistics
  • [MBS-4629] – /cover-art page shows no collections
  • [MBS-4637] – Timeline graph won’t graph anything without an entry in statistics/view.js
  • [MBS-4640] – Clicking cover art opens box with “����” (4 U+FFFD)
  • [MBS-4642] – Thickbox CSS interferes with MB CSS
  • [MBS-4647] – Cover art page allows submitting edit with no cover art when JS is off
  • [MBS-4648] – Changing cover art type from “other” to unset causes Internal Server Error
  • [MBS-4678] – upgrade.sh is not ready for testing
  • [MBS-4679] – Internal server error adding secondary types to a release

Improvement

  • [MBS-1485] – Alias types
  • [MBS-1798] – Lyrics language for works
  • [MBS-1799] – Add ISO 639-3 language codes to the database
  • [MBS-1981] – Add blog feed to the home page
  • [MBS-2240] – Aliases: certain locale can be used only once in the list of aliases
  • [MBS-2532] – Allow more than one IPI per artist
  • [MBS-2851] – Timeline graph events should be in the database
  • [MBS-2885] – Allow more than one ISWC per work
  • [MBS-3646] – Split release group attributes into two types
  • [MBS-3788] – Alias improvements
  • [MBS-4625] – Improve wording of cover art tab when cover art comes from relationships
  • [MBS-4676] – Do not allow people entering deprecated relationships

New Feature

  • [MBS-842] – Allow vinyl style track numbers and sides
  • [MBS-1385] – Support unknown end dates
  • [MBS-3704] – Allow adding sort names to artist aliases
  • [MBS-4337] – Make user profile more social: add (optional) fields avatar, gender, birth year, country

Announcing libmusicbrainz4 releases 4.0.1 and 5.0.0

Andy Hawkins says:

Hi,

I am pleased to announce two new versions of libmusicbrainz:

libmusicbrainz-4.0.2 has been updated to take account of changes made to the server on 15th May 2012. Some interfaces are now marked as deprecated, as they have been extended during the work.

Full documentation is available here:

http://metabrainz.github.com/libmusicbrainz/4.0.2/

The release can be downloaded here:

https://github.com/downloads/metabrainz/libmusicbrainz/libmusicbrainz-4.0.2.tar.gz

(MD5 checksum 5ff62abeca00fdad1bb3a8f99065ae61)

libmusicbrainz-5.0.0 has been introduced to enable the library to be more easily included in Debian due to a conflicting package name. It is identical to libmusicbrainz-4.0.2, with the following exceptions:

1. All include files are now in the musicbrainz5 directory

2. You should now link against libmusicbrainz5 (-lmusicbrainz5)

3. All previously deprecated functions have been removed.

Please note that all future work is likely to only occur on the 5.x library, so this should be used wherever possible.

Full documentation is available here:

http://metabrainz.github.com/libmusicbrainz/5.0.0/

The release can be downloaded here:

https://github.com/downloads/metabrainz/libmusicbrainz/libmusicbrainz-5.0.0.tar.gz

(MD5 checksum 3396e0c66cfacfa1f32abc7cfdbcbe13)

As ever, please report any issues in JIRA at

http://tickets.musicbrainz.org

under the project ‘libmusicbrainz’.

If you have any questions, please post them in the musicbrainz-devel mailing list. I will also attempt to be available on the #musicbrainz-devel IRC channel on Freenode.

Search server index updating paused

Tomorrow’s release requires us to update the main server and the search servers at the same time. This presents a bit of a chicken-and-the-egg problem: We need to build new indexes even before we’ve migrated the database to our new schema.

To accomplish this, I’ve stopped index updating and created a separate database that will allow me to build some indexes that will be a few hours out of date when we release tomorrow. They will be slightly old, but at least we will have indexes that work.

I expect the indexes to be up to date about 3 hours of the release is complete. If you find the indexed search out of date, please use the direct search in the meantime.

Sorry for any inconveniences this may cause.

Universal Music UK launches the Artist Gateway… using MusicBrainz!

Universal Music in the UK just launched their Artist Gateway, a site to showcase their artists and to provide lots of detail about their artists. Universal is using MusicBrainz IDs and data throughout their system to give their site the proper structure and links to elsewhere on the net. To launch it, Universal put out a slick video, complete with a plug for MusicBrainz:

[vimeo http://www.vimeo.com/41151572 w=450&h=253]

This is a huge step for MusicBrainz — in the past the labels didn’t even acknowledge our existence and now they are using our data.

Thanks James Cowdery & crew over at Universal in London!

Upgrading our data licenses

A potential customer just nudged me about our use of now deprecated CC licenses and a deprecated Public Domain dedication. We should really address these issues and upgrade our data licenses. We are currently using the following licenses:

I propose that we move to:

The move to the CC license version 3 allows us to use the latest and most robust version of the Creative Commons license. If you’d like lots more detail on what this change means, please read the CC version 3 license “brief” explanation.

The CC0 grant improves a number of aspects of our old Public Domain dedication, especially in jurisdictions outside the US. Our current dedication isn’t sufficient to renounce any copyright over the data in some countries. The CC0 grant is an improved version that maximizes the global coverage of our rejection of copyright for our data.

Summary in plain english: We’d like to move to updated, more robust licenses. We are not changing what data is available under which license, nor are we taking away any rights that end users already have.

If you have questions or comments, please post them here. If there are no objections to this change, I will make it effective with our May 15th release.

UPDATE: Fixed an incorrect link that Mike from the Creative Commons pointed out. Thanks Mike!

Next schema change release: May 15

We’ve talked a bit about our upcoming schema change release, but we hadn’t nailed down the exact date of the release. Now that we’re tangibly close, we’ve settled on the May 15th as the actual release date.

As a reminder, here are the tickets that will change our schema and here are all the tickets scheduled for the 15th.