Server update, 2012-11-14

We’ve just finishing pushing out another update to the MusicBrainz web servers. Sadly, this one is both a little bit later and quite a bit smaller than our normal releases – blame the summit for that! Many thanks to Frederik “Freso” S. Olesen, Michael Wiencek, Wieland Hoffmann and the MusicBrainz team for their work in this release. Before we get to the release notes, a quick announcement regarding DBDefs:

DBDefs.pm

This release features a new way of working with DBDefs, and we recommend everyone who runs a MusicBrainz server to reset their DBDefs to the new system. However, this should only be necessary if you run a web server – if you only use replication it should be possible to continue with the old DBDefs. To use the new DBDefs:

  1. Copy lib/DBDefs.pm.sample to lib/DBDefs.pm.
  2. Edit lib/DBDefs.pm to match your configuration.

This will give you a DBDefs.pm configuration that will continue to work with future changes to DBDefs, rather than causing exceptions as it previously did.

Release notes

Here’s what we got fixed:

Bug

  • [MBS-4180] – Strange behaviour for artist credits when renaming artist
  • [MBS-5200] – Errors are not translatable
  • [MBS-5247] – Fix all the invalid HTML reported by MBS-5246
  • [MBS-5292] – /release/MBID/cover-art doesn’t list cover art if you have not loaded the edits dumps on your slave server
  • [MBS-5389] – Relationship editor doesn’t correctly handle relating a recording on the release to another recording on the release
  • [MBS-5431] – Moving entities to or merging them artists into Deleted Artist should not be allowed
  • [MBS-5438] – Text overlaps help bubble in relationship editor
  • [MBS-5454] – Broken link to `remove recording edit-type’ documentation on remove recording page
  • [MBS-5456] – Middle click paste does not trigger URL match/cleanup
  • [MBS-5464] – Blog posts on the front page are cached for too long
  • [MBS-5471] – No correct search results even with a recording ID (during Add Release)
  • [MBS-5476] – Pressing enter in empty fields selects instruments / vocals
  • [MBS-5480] – screen space with many artists in the add relationship widget
  • [MBS-5485] – Artist comments not always converted to use XML character references
  • [MBS-5509] – Interface language isn’t persistent enough
  • [MBS-5521] – Date placeholders in the relationship editor are not translated
  • [MBS-5529] – New relationship editor overheats
  • [MBS-5538] – Pressing enter in work add adds/removes ISWC instead of submitting

Improvement

  • [MBS-3353] – Accept XX:XX:XX track times
  • [MBS-4578] – Improve configuration of server
  • [MBS-5206] – Guess Case: Remove the Classical mode
  • [MBS-5246] – Implement html5 validation using validator.nu.
  • [MBS-5433] – Include interface language in stack traces
  • [MBS-5445] – All the widgets forms should submit when hitting <Enter> in a field
  • [MBS-5477] – Show country and gender in artist search results
  • [MBS-5537] – Display only ‘Add note’ on the note & vote submit button when the edit is already closed

Task

  • [MBS-5504] – Add "DHHU" to the "Other Databases" whitelist

Sub-task

  • [MBS-5488] – Amazon coverart over SSL
  • [MBS-5489] – Basic switch-scheme-only coverart over SSL

The Git tag for this release is v-2012-11-14.

MusicBrainz Summit 12 – a Brief Summary

Last weekend approximately 20 dedicated MusicBrainz fans and customers all got together at Universitat Pompeu Fabra, Barcelona to discuss all things metadata. Now that the weekend has passed, and everyone is back at their respective homes, I can fill you in on how this fantastic summit went. We talked about a lot of topics at the summit, and I encourage you to read the extensive notes that Ian took (a huge thanks!) if you’re interested in all the details.

Friday was a lazy day while people arrived, but we did spend a bit of time at the summit venue. We got an introduction from Music Kickup – a new Finnish startup which offers a cloud based record label for artists. Spotify presented their ingestion process, which was helpful as at MusicBrainz we’re beginning to plan the new Ingestr project.

On Saturday we got under way with the summit proper. Brewster Kahle from the Internet Archive spoke on how the Archive works, and its plans going forward to try and archive even more about music. The general motivation is to provide listeners with more ways to explore and discover content, and the Archive are looking for ways that MusicBrainz can help with its linking its metadata to the Archive’s content.

Next, we moved on to discuss the Artist Image Archive – or more generally, adding more images to MusicBrainz. The conclusion was here that this is certainly a wanted feature, and we would like to try and use Wikimedia Commons to store public domain/CC-licensed artwork and fall back to the Archive for other content. We also lightly talked about adding label images, and the Archive are again happy to host this content if that’s needed.

While waiting for the full CompMusic team to be present, we outlined Ingestr, a forthcoming MusicBrainz tool to work with dumps of metadata. CompMusic then presented what they are working on, and how it ties in with MusicBrainz. They are currently storing some data outside the MusicBrainz schema, but would love to store as much as possible inside our database (and we’d love that too!). They are interested in adding more information to works, such as ragas and talas for Indian classical music.

The final topic on Saturday was to try and get some ideas going on how we want to store events, locations and venues inside MusicBrainz. No conclusions were reached here, but there was a lot of excitement and considerations and what we want to store, with the suggestion that some of this work might make a fantastic Summer of Code project next year.

We picked up on Sunday, and dived straight in with a discussion on bringing multiple release countries/release dates back to single releases, with a consensus that this should be done. Work here will probably resume in February, with the next schema change.

Dynamic work attributes were next on the agenda, and we sketched out a plan on how we can add new attributes to works that don’t depend on schema changes. This should allow us to add ragas and talas for the CompMusic folks, but also many other interesting properties. Again, work on this one is looking likely to begin again for the next schema change.

Much like Summit 11, the instrument tree came up again. People were in agreement that we want richer data about instruments, and making them entities that can be used in relationships (including between instruments) is probably the way to go.

nikki outlined her proposal on data quality, and we all spent a bit of time discussing what we’re trying to achieve with data quality, how to overcome the social problems with the word ‘quality’, and so on.

Warp bought up the problem of capturing series of releases in MusicBrainz, and we discussed how to solve this problem (for example, consider the ‘Dubstep Allstars’ series of releases). We agreed that the best way to move forward is to introduce the idea of ordered series, and deal with unordered series later.

The difficult topic of box sets came up, and while we didn’t make a clear decision on how to solve this problem, people had the chance to explain solutions as they see them, and everyone has a good understanding of both the challenges of box sets and the information that we’re trying to capture.

Like in Summit 11, we once more discussed the hard problem of genres, as these have become something that customers are increasingly requesting. We outlined all the various solutions that other projects use (such as SoundCloud, Music Kickup, and SoundUnwound). This topic didn’t reach any type of conclusion, but it was good to have some cross-pollination of ideas.

We wrapped up the day with some social topics, and ocharles started a discussion on how we can better promote ourselves and communicate our new features. There was a lot of energy in this discussion, but some of the key ideas were – more community interaction on blog posts, a clear ‘checklist’ of what to do when we release big features. kepstin also volunteered to look at rewriting the landing page with something more interesting than the static page we currently have.

The summit has proven once again to be a fantastic experience, not just for the quality of discussions, but also the level of interaction between participants. With the Saturday group meal, socialising at the apartment and continued discussions during breaks at the summit itself, it was great to see people chatting, laughing and generally having a great time.

Thanks to everyone who came for making the summit what it was. We hope to see you all again next year!

Server update 2012-10-29

The MusicBrainz team are all mostly back after participating in Google Summer of Code and having a small face-to-face meetup, so we’ve finally got round to getting the latest release out. Because of so much travel, this is a tiny release, but here’s what’s changed:

Bug

  • [MBS-4727] – Regression : Can’t add ISWC when adding work
  • [MBS-5463] – Release editor fails on releases without disambiguation comment
  • [MBS-5478] – Can’t unselect all recordings / works

Improvement

  • [MBS-5422] – Show close time in open auto-editor elections

Virtual Machine for 2012-10-15 release now available

If you’ve been waiting for the VM for the 2012-10-15 version, your wait is over! I’ve updated the Server Setup page with the links to the latest VM. As usual it contains a replicated database that can easily catch up to the latest data.

Grab yours now and start replicating!

UPDATE: This virtual machine is broken — we’ll work to get a new one up soon.

Update on postgresql 8.4 issues and missing functions in upgrade

We’ve been seeing a number of issues with the upgrade scripts for this release reported here on the blog, in IRC, as well as the ticket tracker. We’ve got a couple bits of news for those upgrading from codebases older than mid-August and those using versions of PostgreSQL older than 9.0:

  1. MBS-5473: This is the ERROR: function controlled_for_whitespace(character varying) does not exist migration problem. Slaves with imports done before mid-August are missing a few stored procedures that are necessary for the upgrade. This has been fixed in the master branch as of commit 082e608c074641e2e8df2c0809d35421b02c7899 — to run the upgrade, run git fetch origin, then git checkout 082e608c074641e2e8df2c0809d35421b02c7899, then continue with step #6 of the original instructions. If you already started the upgrade but failed on the ‘merge-duplicate-credits’ script (as is likely, with this bug), don’t worry: at that point in the upgrade, restarting is not an issue.
  2. MBS-5475: This is the ERROR: syntax error at or near “ORDER” migration problem. Unfortunately, this one’s more serious: on PostgreSQL 8.4, our upgrade scripts don’t work, as one uses a feature introduced in 9.0 (as previously mentioned) After some hours of investigation, it appears that this issue can’t be resolved reasonably; therefore, for 8.4 users the upgrade path is a data re-import using a recent (post-schema-change) codebase and a data dump from October 17th or later. We’d also recommend updating to PostgreSQL 9.x at the same time if that is possible.

Sorry again for the inconvenience.

2012-10-15 migration on postgres 8.4 is broken

Our users have reported a bug where migrating a slave DB on postgres 8.4 results in this error:

ERROR: syntax error at or near “ORDER”
LINE 6: SELECT array_agg(artist_credit ORDER BY artist_credit AS…

If you are 8.4 please wait on your migration — we’re hoping to have a workaround ready today. If you need to get this fixed sooner, moving to postgres 9.x or importing the 2012-10-17 data dump would solve this problem.

Sorry for the troubles!

Updating MusicBrainz slave instances for 2012-10-15

If you have a replicated instance of MusicBrainz, please follow these instructions to get your server running on the new schema:

  1. Take down the web server running MusicBrainz, if you’re running a web server.
  2. Turn off cron jobs if you are automatically updating the database via cron jobs.
  3. Make sure your REPLICATION_TYPE setting is RT_SLAVE
  4. Switch to the new code with git fetch origin followed by git checkout v-2012-10-15-schema-change
  5. Run carton install --deployment to install any new perl modules.
  6. Run carton exec -- ./upgrade.sh from the top of the source directory.
  7. Set DB_SCHEMA_SEQUENCE to 16 in lib/DBDefs.pm
  8. Turn cron jobs back on, if needed.
  9. Restart the MusicBrainz web server, if needed.

This upgrade requires quite a bit of disk-space to execute; your slave may run into trouble if there is less than 10Gb of disk space free. If you’re on a disk space constrained machine, you may want to consider re-importing the data rather than upgrading in place. The next data dump should be available in about 14-16 hours from now.

Schema change release 2012-10-15

We just finished the latest schema change release. Please stay tuned for instructions on how to upgrade your MusicBrainz slave instances.

This release features work from Nikki, Navap, Bitmap, Freso and the MusicBrainz developers. Thanks for your work everyone! The following changes were included in this release:

Bug

  • [MBS-240] – Attached files not downloading properly in transcluded page
  • [MBS-3610] – Ratings : 502 Bad Gateway
  • [MBS-3853] – Annotation history page allows the same version to be selected for both Old and New
  • [MBS-3963] – Edit search "artist in my subscription" should hide the artist input field
  • [MBS-4033] – Offered and installed search plugin name mismatch
  • [MBS-4206] – Whitespace isn’t removed from the last join phrase
  • [MBS-4220] – Artists shown to last 0 days if begin year = end year
  • [MBS-4268] – "{age} ago" does not work for all languages
  • [MBS-4747] – Wrong diff or wrong edit allowed for an URL
  • [MBS-4812] – Empty "Remove release label" edit was entered
  • [MBS-4929] – Some fields on user’s "Open edits" page are duplicating while merging works
  • [MBS-4978] – Release merges with strategy "Merge mediums and recordings" don’t show up on the affected recordings’ open edits list.
  • [MBS-5047] – Changing secondary RG types is an autoedit for all users
  • [MBS-5051] – Empty artists with pending edits still get the « will be removed automatically » warning
  • [MBS-5090] – Some entities are marked as having pending edits when they don’t
  • [MBS-5177] – Editors can edit after removing their email address
  • [MBS-5253] – Table release_group_secondary_type_join misses a created field
  • [MBS-5264] – Track Parser results in edits changing durations on tracks in mediums with discids in some cases
  • [MBS-5306] – Internal server error when searching for edits by artist country and relationship type
  • [MBS-5321] – Some relationship editor tests are failing.
  • [MBS-5329] – Relationship editor breaks when using translations
  • [MBS-5330] – Medium formats untranslated on release pages and in release editor
  • [MBS-5331] – Edit note label is cut off when translated
  • [MBS-5332] – Last updated timestamp is missing on some details pages
  • [MBS-5333] – cover art types translate incorrectly
  • [MBS-5336] – Relationships get double-translated
  • [MBS-5342] – .pot files are generated with the ‘perl-format’ po flag when they are not being passed to sprintf
  • [MBS-5344] – Edits display page, for an add cover art edit, for a removed release, has invalid link
  • [MBS-5346] – TuneCore blocking blocks valid ISRCs
  • [MBS-5357] – Translations creeping into edit info
  • [MBS-5368] – Changing the link type in the relationship editor causes the dialog to jump
  • [MBS-5372] – AR editor: paste should detect entity type
  • [MBS-5377] – MediaWiki upgrade broke WikiDoc redirects
  • [MBS-5382] – AR editor: "part of" ARs are displayed backwards
  • [MBS-5384] – Relationship editor: track artists only shown when AC has different number of artists
  • [MBS-5385] – Relationship editor shouldn’t merge relationships with different dates
  • [MBS-5390] – Relationship editor isn’t loading related works for some recordings
  • [MBS-5391] – Relationship editor doesn’t support the "ended" checkbox
  • [MBS-5392] – Relationship editor should display dates consistent with the rest of the site
  • [MBS-5396] – relationship editor changes to MB.utility.keys cause it to die on null
  • [MBS-5406] – "Subscriptions" editor tab shows 3 empty <li> inside <ul> between « jesus2099 is subscribed to: » and p.pageselector
  • [MBS-5412] – Some statistics tabs have partially-untranslated headers
  • [MBS-5413] – e.g. "Artist-Artist" and "Artist-Artist relationships" not translating correctly on /statistics/relationships
  • [MBS-5414] – Javascript tests no longer run in older versions of Opera
  • [MBS-5415] – "Types:" (under cover art) should often be in singular
  • [MBS-5417] – WikiDocs: Prevent access to File and Image namespace
  • [MBS-5419] – Chrome frame wasn’t removed completely/properly
  • [MBS-5425] – Some ModBot edit notes end up stored translated

Improvement

  • [MBS-1226] – Address verification email should include the client’s IP address
  • [MBS-1530] – More space/sections on the add disc id page
  • [MBS-1789] – No way to compare discid track length against tracklist length
  • [MBS-2449] – Search page should say when it was last updated
  • [MBS-2492] – Don’t show quality levels in edit documentation where they don’t apply
  • [MBS-2543] – Make the dash at the end of /cdtoc/ and /cdstub/ URLs optional
  • [MBS-2650] – After adding a relationship, go to the overview tab
  • [MBS-2717] – User’s ratings page should be paginated
  • [MBS-3606] – Matched CD TOC page should have a link to the disc id
  • [MBS-3791] – Drop Description from URLs
  • [MBS-4104] – Show release merges under all track/recording artists
  • [MBS-4405] – Report: Artists that may be persons shoudn’t show obvious groups
  • [MBS-4690] – Include links to all image sizes
  • [MBS-5099] – Make release/work language clickable
  • [MBS-5236] – Add stats for work types
  • [MBS-5279] – Try to avoid duplicating links in the sidebar
  • [MBS-5285] – Add NOT NULL constraints to disambiguation & join phrase column
  • [MBS-5290] – CAA replication
  • [MBS-5350] – Add a way to check # of open edits and edits entered in the last day
  • [MBS-5352] – /doc/Edit_Types/38 and similar should link to the documentation.
  • [MBS-5362] – Add cleanup and autoselect for iTunes links
  • [MBS-5364] – Ask the user to confirm navigation away from the relationship editor (mainly for backspace)
  • [MBS-5365] – In the relationship editor, pressing enter while focused in the entity lookup should add the relationship
  • [MBS-5366] – Make it easier to reuse artists in the relationship editor
  • [MBS-5371] – Don’t show ModBot as a limited user
  • [MBS-5375] – Remove whitespace above doctype declaration
  • [MBS-5376] – Add artist credit description
  • [MBS-5404] – Improve the relationship editor’s multiselect plugin

New Feature

  • [MBS-5320] – All data dumps should be signed by MusicBrainz
  • [MBS-5324] – Relationship editor should allow using auto-editor privileges
  • [MBS-5363] – Allow adding/editing release group relationships in the relationship editor

Task

  • [MBS-5182] – Add final controlled_for_whitespace checks
  • [MBS-5289] – Finish/merge log statistic features
  • [MBS-5291] – Move existing timeline stats into its own schema
  • [MBS-5316] – Remove PGOPTIONS/search_path from ./admin/psql
  • [MBS-5354] – Remove drastically out-of-sequence cover art images
  • [MBS-5355] – Remove depedency on Function::Parameters
  • [MBS-5374] – Commit MB logo into repo

The Git commit SHA for this release is 5bd71d211dceb25a7a0d545a14cb3273e3e89f19, tag is v-2012-10-15-schema-change.