Updating MusicBrainz slave instances for 2012-10-15

If you have a replicated instance of MusicBrainz, please follow these instructions to get your server running on the new schema:

  1. Take down the web server running MusicBrainz, if you’re running a web server.
  2. Turn off cron jobs if you are automatically updating the database via cron jobs.
  3. Make sure your REPLICATION_TYPE setting is RT_SLAVE
  4. Switch to the new code with git fetch origin followed by git checkout v-2012-10-15-schema-change
  5. Run carton install --deployment to install any new perl modules.
  6. Run carton exec -- ./upgrade.sh from the top of the source directory.
  7. Set DB_SCHEMA_SEQUENCE to 16 in lib/DBDefs.pm
  8. Turn cron jobs back on, if needed.
  9. Restart the MusicBrainz web server, if needed.

This upgrade requires quite a bit of disk-space to execute; your slave may run into trouble if there is less than 10Gb of disk space free. If you’re on a disk space constrained machine, you may want to consider re-importing the data rather than upgrading in place. The next data dump should be available in about 14-16 hours from now.

Schema change release 2012-10-15

We just finished the latest schema change release. Please stay tuned for instructions on how to upgrade your MusicBrainz slave instances.

This release features work from Nikki, Navap, Bitmap, Freso and the MusicBrainz developers. Thanks for your work everyone! The following changes were included in this release:

Bug

  • [MBS-240] – Attached files not downloading properly in transcluded page
  • [MBS-3610] – Ratings : 502 Bad Gateway
  • [MBS-3853] – Annotation history page allows the same version to be selected for both Old and New
  • [MBS-3963] – Edit search "artist in my subscription" should hide the artist input field
  • [MBS-4033] – Offered and installed search plugin name mismatch
  • [MBS-4206] – Whitespace isn’t removed from the last join phrase
  • [MBS-4220] – Artists shown to last 0 days if begin year = end year
  • [MBS-4268] – "{age} ago" does not work for all languages
  • [MBS-4747] – Wrong diff or wrong edit allowed for an URL
  • [MBS-4812] – Empty "Remove release label" edit was entered
  • [MBS-4929] – Some fields on user’s "Open edits" page are duplicating while merging works
  • [MBS-4978] – Release merges with strategy "Merge mediums and recordings" don’t show up on the affected recordings’ open edits list.
  • [MBS-5047] – Changing secondary RG types is an autoedit for all users
  • [MBS-5051] – Empty artists with pending edits still get the « will be removed automatically » warning
  • [MBS-5090] – Some entities are marked as having pending edits when they don’t
  • [MBS-5177] – Editors can edit after removing their email address
  • [MBS-5253] – Table release_group_secondary_type_join misses a created field
  • [MBS-5264] – Track Parser results in edits changing durations on tracks in mediums with discids in some cases
  • [MBS-5306] – Internal server error when searching for edits by artist country and relationship type
  • [MBS-5321] – Some relationship editor tests are failing.
  • [MBS-5329] – Relationship editor breaks when using translations
  • [MBS-5330] – Medium formats untranslated on release pages and in release editor
  • [MBS-5331] – Edit note label is cut off when translated
  • [MBS-5332] – Last updated timestamp is missing on some details pages
  • [MBS-5333] – cover art types translate incorrectly
  • [MBS-5336] – Relationships get double-translated
  • [MBS-5342] – .pot files are generated with the ‘perl-format’ po flag when they are not being passed to sprintf
  • [MBS-5344] – Edits display page, for an add cover art edit, for a removed release, has invalid link
  • [MBS-5346] – TuneCore blocking blocks valid ISRCs
  • [MBS-5357] – Translations creeping into edit info
  • [MBS-5368] – Changing the link type in the relationship editor causes the dialog to jump
  • [MBS-5372] – AR editor: paste should detect entity type
  • [MBS-5377] – MediaWiki upgrade broke WikiDoc redirects
  • [MBS-5382] – AR editor: "part of" ARs are displayed backwards
  • [MBS-5384] – Relationship editor: track artists only shown when AC has different number of artists
  • [MBS-5385] – Relationship editor shouldn’t merge relationships with different dates
  • [MBS-5390] – Relationship editor isn’t loading related works for some recordings
  • [MBS-5391] – Relationship editor doesn’t support the "ended" checkbox
  • [MBS-5392] – Relationship editor should display dates consistent with the rest of the site
  • [MBS-5396] – relationship editor changes to MB.utility.keys cause it to die on null
  • [MBS-5406] – "Subscriptions" editor tab shows 3 empty <li> inside <ul> between « jesus2099 is subscribed to: » and p.pageselector
  • [MBS-5412] – Some statistics tabs have partially-untranslated headers
  • [MBS-5413] – e.g. "Artist-Artist" and "Artist-Artist relationships" not translating correctly on /statistics/relationships
  • [MBS-5414] – Javascript tests no longer run in older versions of Opera
  • [MBS-5415] – "Types:" (under cover art) should often be in singular
  • [MBS-5417] – WikiDocs: Prevent access to File and Image namespace
  • [MBS-5419] – Chrome frame wasn’t removed completely/properly
  • [MBS-5425] – Some ModBot edit notes end up stored translated

Improvement

  • [MBS-1226] – Address verification email should include the client’s IP address
  • [MBS-1530] – More space/sections on the add disc id page
  • [MBS-1789] – No way to compare discid track length against tracklist length
  • [MBS-2449] – Search page should say when it was last updated
  • [MBS-2492] – Don’t show quality levels in edit documentation where they don’t apply
  • [MBS-2543] – Make the dash at the end of /cdtoc/ and /cdstub/ URLs optional
  • [MBS-2650] – After adding a relationship, go to the overview tab
  • [MBS-2717] – User’s ratings page should be paginated
  • [MBS-3606] – Matched CD TOC page should have a link to the disc id
  • [MBS-3791] – Drop Description from URLs
  • [MBS-4104] – Show release merges under all track/recording artists
  • [MBS-4405] – Report: Artists that may be persons shoudn’t show obvious groups
  • [MBS-4690] – Include links to all image sizes
  • [MBS-5099] – Make release/work language clickable
  • [MBS-5236] – Add stats for work types
  • [MBS-5279] – Try to avoid duplicating links in the sidebar
  • [MBS-5285] – Add NOT NULL constraints to disambiguation & join phrase column
  • [MBS-5290] – CAA replication
  • [MBS-5350] – Add a way to check # of open edits and edits entered in the last day
  • [MBS-5352] – /doc/Edit_Types/38 and similar should link to the documentation.
  • [MBS-5362] – Add cleanup and autoselect for iTunes links
  • [MBS-5364] – Ask the user to confirm navigation away from the relationship editor (mainly for backspace)
  • [MBS-5365] – In the relationship editor, pressing enter while focused in the entity lookup should add the relationship
  • [MBS-5366] – Make it easier to reuse artists in the relationship editor
  • [MBS-5371] – Don’t show ModBot as a limited user
  • [MBS-5375] – Remove whitespace above doctype declaration
  • [MBS-5376] – Add artist credit description
  • [MBS-5404] – Improve the relationship editor’s multiselect plugin

New Feature

  • [MBS-5320] – All data dumps should be signed by MusicBrainz
  • [MBS-5324] – Relationship editor should allow using auto-editor privileges
  • [MBS-5363] – Allow adding/editing release group relationships in the relationship editor

Task

  • [MBS-5182] – Add final controlled_for_whitespace checks
  • [MBS-5289] – Finish/merge log statistic features
  • [MBS-5291] – Move existing timeline stats into its own schema
  • [MBS-5316] – Remove PGOPTIONS/search_path from ./admin/psql
  • [MBS-5354] – Remove drastically out-of-sequence cover art images
  • [MBS-5355] – Remove depedency on Function::Parameters
  • [MBS-5374] – Commit MB logo into repo

The Git commit SHA for this release is 5bd71d211dceb25a7a0d545a14cb3273e3e89f19, tag is v-2012-10-15-schema-change.

Announcing the Cover Art Archive

In a server update last April we quietly said that “we’ve also improved cover art support slightly.” What we actually did was release the first version of the Cover Art Archive, a cooperation between MusicBrainz and the Internet Archive. First, a little background:

Cover art (the images associated with music products) adds a great amount of value to the digital music experience. Many projects and apps on the net use these images to add color and depth to their music tools. However, there isn’t a cleanly organized, publicly available resource where everyone can access these images. You can use Amazon product images, but your project needs to be able to abide by their Terms of Service, which doesn’t work for everyone. Many projects use Google Images to source their cover art, but that is an inexact science since they may not always find the right image.

The Cover Art Archive aims to solve these problems by making these images available to the public. But since we are not lawyers, we can not say what can and can not be done with these. So use them at your own risk! That said, everyone on the internet is using these images anyway and the common understanding is that if you’re selling music you’re pretty safe. We suggest that when you try to figure out what to do, make sure that you respect the artists and their labels and make the music world a better place.

All images in the Cover Art Archive are indexed by the release’s MBID, and all metadata can be parsed by a JSON document. For instance, to fetch the front cover for any given release, construct this URL:

http://coverartarchive.org/release/76df3287-6cda-33eb-8e9a-044b5e15ffdd/front

Once you GET this resource, you will be redirected to the proper Internet Archive URL that yields either an image file or a 404 error if we do not have this image. For lots more details on how to use the Cover Art Archive, please take a look at our API documentation. So far, there are Java, C and Perl bindings to the API.

For some stunning examples of what people have already done with the Cover Art Archive, please take a look at these links:

So far, we’ve collected nearly 100,000 images that are attached to 54,000 releases for a 5% coverage in MusicBrainz. The largest file we have clocks in at 23MB and the largest image is 16,000 x 7842 (125 megapixels!). For all of the juicy stats on this project, check out our cover art statistics page.

We’ve just gotten started and we need your help! Won’t you please consider uploading some images to this archive? To get started, log in with your MusicBrainz account (or create a new one) find your favorite release and then click on the cover art tab to view the existing pieces of art and/or upload new ones. For more details, see our How to add cover art guide.

Thank you to everyone who has worked hard to make this project a reality! And thank you to Brewster Kahle and the Internet Archive for fostering this project!

Server update, 2012-10-01

We’ve just finished pushing out an update today, which will be the last update before the schema change, which is now due in 2 weeks. Sadly, this release doesn’t feature the much anticipated relationship editor, which had to be reverted at the last minute as we still think it could do with more user testing. Hopefully it will make the next release!

This release features work from Aurélien Mino, Nicolás Tamargo, nikki, patate12, Pavan Chander, Ulrich Klauer and the MusicBrainz developers. Thanks for your work everyone!

Bug

  • [MBS-3825] – Release Editor is not displaying the release title on the tracklist tab
  • [MBS-3866] – Disambiguation comments can be the same
  • [MBS-3902] – Database allows artist type group with a gender set
  • [MBS-4614] – Country dropdown is sorted by English name
  • [MBS-4670] – "Relate to…" maybe should not add #relate_to hash to address bar URL
  • [MBS-4861] – Data loss: Changing release artist revert track numbers
  • [MBS-5019] – Release group with no name created
  • [MBS-5089] – Statistics: "unique" number of ISWCs is higher than "all" number of ISWCs
  • [MBS-5111] – Editor statistics data regularly falls out of cache, and cannot be recomputed
  • [MBS-5199] – Recording disambiguation comment is not removed when it should in RE’s recordings tab
  • [MBS-5240] – Adding an IPI code to an artist with IPI pending fails
  • [MBS-5241] – Add Medium/Add Release do not show up in recording edit histories
  • [MBS-5242] – Add Medium edits don’t always link recordings, even those that exist
  • [MBS-5250] – Non-specific dates create negative ages
  • [MBS-5264] – Track Parser results in edits changing durations on tracks in mediums with discids in some cases
  • [MBS-5295] – Webservice should distinguish between "" and NULL as a value for release->barcode.
  • [MBS-5298] – WikiDocs links are not scheme-independent
  • [MBS-5301] – gravatars do not get loaded over SSL when they should be
  • [MBS-5315] – ISE moving a discid
  • [MBS-5319] – QUnit testrunner is not compatible with current phantomjs versions.
  • [MBS-5335] – $entity is undefined in WebServiceStash.pm

Improvement

  • [MBS-1579] – Add Release editor: option to swap track title <> artist
  • [MBS-4066] – Simple RE parser speed boost
  • [MBS-4539] – Add h1 to release editor
  • [MBS-4831] – Exclude releases with pending art edits from the CAA report
  • [MBS-5152] – Allow removing a disc ID from the disc ID page
  • [MBS-5281] – Display relationships on release-group overview page
  • [MBS-5282] – language switcher should appear when number of languages > 1, not > 0
  • [MBS-5337] – Add more classes to entity pages and when displaying edits

Task

  • [MBS-2404] – Private mb dump files to be encrypted
  • [MBS-5169] – Add "Rock in China" to the "Other Databases" whitelist

The Git commit SHA for this release is ce2da4b67e39821348cd849ca1957b8e15c33f52, tag is v-2012-10-01.

Help us make some stickers!

MusicBrainz will be attending the Google Summer of Code mentor summit, and we’d love to have some stickers to hand out, so our fellow open-source friends can show off how cool we are. Sadly, we don’t have the skills to actually make the designs to send off to get printed… but that’s where you come in!

Do you do graphic design? Do you know someone who would be willing to help us? If so, please get in touch with us – leave a comment, tweet us or shoot an email to info@musicbrainz.org. Thanks!

Splunk supports MusicBrainz!

As part of Google’s Summer of Code program we accepted Dániel Bali to work on analyzing our web server logs to mine them for interesting information about MusicBrainz and people who are using MusicBrainz. (see a preview of this project)

To make that project a reality we had help from Splunk, the company that creates the fantastic data analysis tool by the same name. Splunk provided us with enterprise trial licenses during the summer and now going forward has accepted us into their Splunk for Good program. This program provides a free 10GB/day (it allows us to import 10GB of data into our Splunk server per day) license on a yearly basis.

We now count Splunk among our sponsors and we’re looking forward to rolling out Dániel’s work in October. Thank you Splunk and thank you to Joyce Morrell and Christy Wilson from Splunk for working with us to make this happen!

Search server release: 2012-09-18

We’ve just updated our search servers with a new release. This release adds support for a new improved json format for the search server and will be publicly available after the next mbserver release. We also now output the date (in the XML/json) the index was last updated so you know how old the results received are. This will be exposed to the end-user in the web search results in an upcoming release of musicbrainz-server.

Thanks to Paul Taylor and Aurélien Mino for making this release happen!

Improvement

  • [SEARCH-232] – Search server should return information about when the indexes were last updated

New Feature

  • [SEARCH-226] – Add Support for Json format as described in http://wiki.musicbrainz.org/User:kuno/Web_Service/JSON
  • [SEARCH-227] – Add prettyprint option for Json (and XML)