Month: May 2015

Schema change release, 2015-05-18 (including upgrade instructions)

Our previously mentioned schema change release is finished! Below will be upgrade instructions, including configuration updates for replication access tokens.

This release does not include UI for several of the schema change patches, which will (hopefully) happen for next release on June 1. The incomplete patches are MBS-7489 (credits for artists in relationships), MBS-4145 (tag upvote/downvote), and MBS-8004 (collections for additional entity types). These patches have had their schema change components finished, but the UI was incomplete or needed more work.

Schema Change Upgrade Instructions

These are largely as previous upgrade instructions, using the tag v-2015-05-18-schema-change. The primary difference is the inclusion of configuring an access token for replication.

  1. Make sure your REPLICATION_TYPE setting is RT_SLAVE and your DB_SCHEMA_SEQUENCE is set to 21 in lib/DBDefs.pm. If you’re running a standalone server, you can run the upgrade, but it may be easier to just import a new data dump!
  2. Ensure you’ve replicated up to the most recent replication packet available with the old schema. (if you’re not sure, run ./admin/replication/LoadReplicationChanges and see what it tells you; if you’re ready to update, it should say “Mismatched schema sequence, 21 (database) vs 22 (replication packet)”).
  3. Take down the web server running MusicBrainz, if you’re running a web server.
  4. Turn off cron jobs if you are automatically updating the database via cron jobs.
  5. Switch to the new code with git fetch origin followed by git checkout v-2015-05-18-schema-change
  6. Run ./upgrade.sh (or carton exec -Ilib -- ./upgrade.sh if you’re using carton, with very old setups).
  7. Run cpanm --installdeps --notest . to ensure your perl-based dependencies are up to date. This release adds a dependency on LWP::Protocol::https, for fetching replication packets from the new server; many systems may already have this installed, but it should be verified.
  8. Set DB_SCHEMA_SEQUENCE to 22 in lib/DBDefs.pm as instructed by the output of ./upgrade.sh
  9. Assuming you have been updating your server with replication, it will now be necessary to configure an access token:
    1. Go to https://metabrainz.org/supporters/account-type and choose your account type as applicable. If you’re an individual, non-commercial user of the data, choose “non-commercial”; if not, choose an applicable tier in the “commercial” section. If you’re not sure of the appropriate tier, make your best guess; it can be adjusted if necessary.
    2. Then, from https://metabrainz.org/profile, create an access token, which should be a 40-character random alphanumeric string provided by the site.
    3. Finally, add this token to lib/DBDefs.pm under the REPLICATION_ACCESS_TOKEN configuration option. The final configuration section should look something like sub REPLICATION_ACCESS_TOKEN { "ck3UpgwgOXhWC6SpFcd99rZOTjzfrei3gQlgZZ9z" }.
    4. Don’t reveal your access token! If you do, inadvertently, you can use the MetaBrainz site to generate a new token, invalidating the old one. (The one in the example above is one I created for myself and then invalidated — don’t get any ideas, it won’t work!)
  10. Turn cron jobs back on, if applicable.
  11. Restart the MusicBrainz web server, if applicable. It’s also recommended you restart memcached.

Finally, the list of bugs closed this release:

Bug

  • [MBS-4436] – Medium titles cannot be longer than 255 charaters

Improvement

  • [MBS-1347] – Implement aliases for release groups, releases and recordings
  • [MBS-7906] – maybe don’t show “”≠null diff. in edit pages
  • [MBS-8279] – Remove empty_artists etc. database functions

New Feature

  • [MBS-8302] – Add Live Data Feed access token support

Task

  • [MBS-8266] – Make medium titles VARCHAR NOT NULL
  • [MBS-8278] – Update DB_SCHEMA_SEQUENCE in DbDefs.pm.sample
  • [MBS-8283] – Remove DB constraint that disallows empty event names

Not included in this list but also relevant is MBS-8349, which while fixed for a previous release, in this release is also applied to old slave servers, which may help performance for some queries.

New MetaBrainz site, new look and Live Data Feed access tokens

If you’ve been following this blog, you’ll be aware that we’ve been working on a complete overhaul of our branding and the look and feel of our web sites. While we still have a few minor adjustments to make here and there, we are happy to present you with our newly re-designed MetaBrainz Foundation web site.

MetaBrainz

New features of this site include our new site design, new logos, showcasing our projects and our customers, and HTTPS support. But the most important feature of this new site include online sign-up for commercial use of our data and access token generation for our Live Data Feed.

Two months ago we announced that as of today our Live Data Feed would require an access token in order to download the replication packets. These access tokens are available for free for private use and available on a sliding support scale for commercial use. Starting right now, you may go to the new Sign Up page and associate a new MetaBrainz account with your existing (or new) MusicBrainz account. Once you complete the sign-up process, you will be able to generate an access token to use with our Live Data Feed. We will post details on what to do with this access token after the schema change release takes place later today.

We’ve needed this new site for years and years, but have always been too busy focusing on our flagship project MusicBrainz. But, with help from Roman Tsukanov (aka Gentlecat) and MonkeyDo, we have finally released a web site that both brings us into modern times and streamlines the process for commercial users to start using our data.

If you are curious about what logos we chose for our other projects, see the animation on the MetaBrainz home page. Soon we will roll out the new site design onto CritiqueBrainz, BookBrainz, Cover Art Archive and our Picard site. However, since MusicBrainz is not built on the Bootstrap toolkit it will take us a while to roll the design out onto that site.

Thank you to Gentlecat, Nicolas at MonkeyDo, and everyone who gave feedback on the many logo and site designs that have been floated here for the past few weeks.

Schema upgrade downtime: Monday, 18 May, 2015 @ 17:00 UTC

Our next schema change version will be released on Monday, 18 May, 2015 around 10am PDT/1pm EDT/17:00 UTC/18:00 BST/19:00 CEST. We expect that MusicBrainz will be unavailable for 15 – 30 minutes during this time. We will put up the downtime notification on the site and tweet from @musicbrainz right before the release.

Since we’re total slackers, we still haven’t set up our backup database server since it suffered a hardware failure. This means that we won’t be able to put the site into read-only mode and will require us to do a full downtime. Hopefully for our next schema change we’ll have tackled our backlog of sysadmin duties and will have a backup DB server to make the release easier.

Sorry for any trouble this may cause you.

P.S. Look for another blog post on Sunday for details on where to get your access tokens for the Live Data Feed.

Moving more quickly now, here is yet another round of logos!

Based on the last round of feedback, here is another round:

MusicBrainz_logos_13-05-15

If something isn’t being shown, you can assume that it isn’t really up for discussion anymore. (Except perhaps for color schemes, which this round may change.)

As usual, hit us with your comments, but do it soon as we’re iterating faster now.

Thanks!

Nearly final round of logos!

Most of the logo decisions have been made and we’re ironing out the last details of font placement. Open questions are:

  1. What text placement do we like?
  2. Which of the new AcousticBrainz logos we we like?
  3. For the large MusicBrainz logo, do we want to have the drop shadows?

MusicBrainz_logos_12-05-15_v2

As usual, let us know your thoughts on our blog!

(P.S. Some of the logos have some minor artifacts on them — we’re aware of them and we’ll fix those in the final logos)

Downstream Wikipedia link usage and migration to Wikidata

MusicBrainz has linked to Wikipedia for many years and we now have links to Wikidata as well. Wikidata, however, acts as a central repository for Wikipedia links, so it does not make sense for MusicBrainz to maintain its own separate set of Wikipedia links, especially since Wikipedia URLs are not very stable (because of page moves and deletions) and require a lot of maintenance. Most of our data with Wikipedia links is now also linked to Wikidata, so we plan to start removing Wikipedia links where we have a Wikidata link which has the same Wikipedia link.

What this means for downstream data users:

If you use Wikipedia links, we will provide Wikidata links but you will need to fetch the Wikipedia links you want from Wikidata separately. Wikidata has information on ways to access their data at https://www.wikidata.org/wiki/Wikidata:Data_access

We plan to start removing the links after the schema change this month, starting with the less common languages and entity types. It will take a while to work through the existing links, so we don’t expect to start removing English links from artists until after the Autumn schema change.

We recognise that some people may have code which depends on these links – if you’re using these links and the above sounds problematic, please let us know how you’re using the data (which languages and entity types) and how much time you would need to support Wikidata.

One more round of logos

Thank you for your feedback from the last round of logos! The most important messages that we received from the last round were:

  • The colors were too saturated
  • There logos contained too much detail and when made smaller, would not look good

Nico, our designer from Monkey.Do, has taken this to heart and come up with the next round of logos. In this iteration, he is demonstrating progressive details with a logo: At the lowest level of detail the brain is just the split hexagon as suggested by Aerozol, and as the logo gets larger, more detail appears in the logo.

Have a look:

MusicBrainz_logos_30-04-15

As usual, we’d love your feedback!