Category: Development

Server update, 2013-02-11 and an important notice regarding edits

We’ve just finished pushing out another two weeks changes to the MusicBrainz web site. While this release is predominantly a bug fix release with a few small improvements, we’ve made a fairly substantial change to the way edits are applied.

As of this release, all subsequent edits entered will have an expiration period of 7 days – a reduction from the previous 14 days. We’ve made this change in order to reduce the time that editors have to wait for changes to be applied, which should lead to an improved user experience; and we’ve also made the change to hopefully try and make the edit queue a little bit more managable. This change is exploratory, so if you find it counterproductive, we’d love to hear your thoughts. IRC, the forums and the mailing lists are all good channels to voice your feedback.

Also, we have finally made the switch to GitHub. While the existing repository URLs will continue to work, they will no longer be updated. If you want to stay up to date with the latest code, make sure to update your checkout information.

Many thanks to Frederik “Freso” S. Olesen, Michael Wiencek, Nicolás Tamargo and the rest of the MusicBrainz team for their work on this release. Here’s what’s new:

Bug

  • [MBS-3457] – Direct search results incorrectly reports a recording as "standalone"
  • [MBS-3962] – Edit search doesn’t really exclude artists with "is not"
  • [MBS-4522] – Empty annotations are not merged correctly
  • [MBS-5144] – The MusicBrainz logo in the top-left corner has a different background color than the rest of the header
  • [MBS-5395] – "Actions" column on alias page is too narrow for translations
  • [MBS-5432] – Internal server error when editing cover art
  • [MBS-5434] – Inconsistent terminology: primary/secondary types and type/extra types
  • [MBS-5506] – Edit search for "My vote is not" does not work as expected
  • [MBS-5525] – Blank edit relationship type edits
  • [MBS-5566] – No more autoedit mark ?
  • [MBS-5567] – Internal server error entering remove cover art edit
  • [MBS-5617] – "new image goes here" is not translatable
  • [MBS-5650] – ISE when attempting to approve already-closed release group edit
  • [MBS-5655] – It’s possible to "Remove [a] release label" although there’s none
  • [MBS-5696] – ModBot can fail to close ‘edit artist’ edits that violate uniqueness on (name, comment).
  • [MBS-5698] – Some move disc ID edits display nothing for the old release
  • [MBS-5700] – Subscribers aren’t removed when a user deletes their account
  • [MBS-5705] – Rows in the appearances section of the artist relationships tab are sometimes one column too short
  • [MBS-5719] – The track number of the last track on some 2-disc releases is missing
  • [MBS-5724] – Sort/copy name missing for work aliases
  • [MBS-5728] – Possible to enter ‘edit release group’ edits that fail to apply due to spaces in artist credits
  • [MBS-5732] – Empty labels don’t warn for deletion pending
  • [MBS-5740] – Useless "select all" checkbox on Release Duplicates tab
  • [MBS-5742] – Editing a work without JS on will enter a silent remove ISWC edit
  • [MBS-5748] – Not possible to "Approve" an edit where yours is the only "No" vote
  • [MBS-5750] – Subscriptions report filtering ignores labels
  • [MBS-5754] – Editing recordings from tracklist does not work if "Release Duplicate" is chosen
  • [MBS-5762] – "Work Type" and "Work Language" incorrectly left blank for "Merge Works" edits when viewing predefined edit searches
  • [MBS-5763] – Edit release edits not properly translated
  • [MBS-5764] – Work language column in the stats links to releases
  • [MBS-5770] – There is no constraint on release_label that either (or both) the label or catalog number are not null.
  • [MBS-5771] – Add utamap.com （うたまっぷ） in the lyrics URL white list
  • [MBS-5774] – "Vote on all edits" is not translatable
  • [MBS-5776] – Cover art types are not being translated on the cover art tab
  • [MBS-5777] – Users’ work ratings page has no title
  • [MBS-5786] – i18n: secondary RG type labels on overview are untranslatable
  • [MBS-5788] – No edit is created when trying to submit/attach a DiscID
  • [MBS-5791] – ISRCs can’t be removed
  • [MBS-5797] – Internal server error if the Wikipedia link is not really a Wikipedia link

Improvement

  • [MBS-1413] – Make profile bios support WikiFormat
  • [MBS-1774] – Webservice: expose UUID for AR type
  • [MBS-3535] – Stack traces should mention which server was handling the request
  • [MBS-4880] – Add Help/Info about How "Merge mediums and recordings" works
  • [MBS-4902] – Add Google+ links to the sidebar
  • [MBS-5505] – Add j-lyric.net to the lyrics whitelist
  • [MBS-5605] – Ratings on collection pages
  • [MBS-5707] – Add autoselect for IMDb URL relationships for labels
  • [MBS-5727] – Disc [medium] title missing from edit relationships page
  • [MBS-5767] – Add an index on (editor, id DESC)
  • [MBS-5775] – Update Facebook URL cleanup to use https
  • [MBS-5779] – Remove useless nbsp on common-macros.tt
  • [MBS-5798] – Normalise wikisource URLs to http
  • [MBS-5803] – Add autoselect for SecondHandSongs artist URL

Task

  • [MBS-5757] – Consider diminishing time edits stay open
  • [MBS-5766] – Open up /release/ and /work/ in robots.txt
  • [MBS-5810] – Add Open Library to the other databases whitelist

The Git tag for this release is v-2013-02-11.

Preparing for the May 15th schema change release

It it time for us to start the process towards the next schema change release. Starting today and for the next two weeks, we’re going to seek people to be the champion (sponsor) of a ticket. If you feel strongly about a schema change ticket getting taken care of, you should consider championing this ticket. Once you’ve decided to do adopt a ticket, you should assign the ticket to yourself.

Then, over the next two weeks it will be up to you to do the following:

  1. Drive consensus around the core concept of the ticket. If you go through the process of working up a ticket, but no one agrees with what you’re proposing, you’ve wasted your time. Make sure that you get buy in from others in the community. For instance, if Nikki doesn’t like it, chances are its not going to fly. 🙂
  2. Each schema change feature requires two tickets: 1) An SQL ticket that implements the actual changes to the database and defines the queries used to fetch the data. 2) A UI change ticket that implements the UI portions of the schema change ticket.
  3. Ensure that the ticket clearly states what needs to be done to implement the ticket. The ticket should essentially become or link to a requirements document. This requirements document should explain what the new feature should do. It should not explain how it should be done — we should leave the how to our developers who are going to implement the feature.
  4. Provide as much supporting documentation as you can. Mock-ups for UIs are deeply appreciated (even if they delve into the how realm of things) and very useful for meaningfully discussing these tickets.
  5. Have the ticket reviewed by a developer for clarity and completeness, then address any issues said developer may raise.

On 15 February, we’re going to look at the list of tickets that people have taken on and choose the ones that are clear enough to move forward. If you’ve done all the work outlined above, the chances are good that your ticket will be chosen to move forward. If your ticket is chosen to move forward, there will be more questions that the developers will raise — hopefully those can be tackled in the space of a week. After that we will take all of the well defined tickets and schedule them for implementation. All the other tickets that are not clear to implement will be rejected and will have to make another pass though this process in the autumn.

If you’re still interested, here is the list of schema change tickets that should be considered for this.

We’re going to follow the this schedule:

  • 1 Feb: Schema change ticket selection starts
  • 15 Feb: Select schema change tickets for implementation, start making tickets fully actionable
  • 1 March: Tickets must be fully actionable. Tickets that are not actionable will be dropped from the 15 May release.
  • 15 March: SQL tickets must be fully implemented.
  • 1 May: UI tickets must be fully implemented, start final ticket testing phase
  • 15 May: Release day

All of these dates have been added to our new community calendar.

IMPORTANT: Proposed changes to the data returned by our web service

Our current web service at the /ws/2 endpoint returns too much data in a lot of cases and in many cases we suspect that the programs making the calls to the service don’t actually consume all of that data. We’d like to reduce the amount of unused data our web service returns, in order to reduce our bandwidth costs. We propose that:

  • The web service will no longer includes aliases and tags in relation elements. Regardless of what entity you may request, if the results of your request includes a relation element, any alias or tag elements that are currently returned will no longer be returned.
  • The web service no longer includes aliases and tags in for the Various Artists artist anywhere, unless you specifically request the Various Artist from the /ws/2/artist endpoint.

We’ve mocked up these changes in the following XML files:

We think that this will have a minimal impact on our web service users. If you use our web service, please tell us what you think about this. If you know someone who is using our web service, but may not read this blog, please forward a link to this post to them.

For more background on our research into this topic, please take a look at this document.

Server Update, 2012-01-28

We’ve just finished releasing the last release for January, 2013. This release is a combination of bug fixes and improvements to existing features, and was worked on by Frederik “Freso” S. Olesen, Michael Wiencek, Nicolás Tamargo, Pavan Chander and the rest of the MusicBrainz team. Thanks for your hard work, everyone! Here’s what’s changed:

Bug

  • [MBS-5022] – Release Editor: Missing entities page doesn’t handle 0 or only 1 result
  • [MBS-5453] – Bad redirect(s) https→http
  • [MBS-5540] – Release editor eats tracklists
  • [MBS-5586] – ISE when trying to submit an “Edit artist” edit with no changes
  • [MBS-5609] – JSON webservice doesn’t send relationship attributes
  • [MBS-5614] – Links to edit notes to other edit notes always use http
  • [MBS-5691] – Artists’ recording lists show ISRCs but don’t link to them
  • [MBS-5699] – CD stubs use “disambiguation” instead of “comment”
  • [MBS-5711] – IPIs missing from IPI element in JSON webservice
  • [MBS-5712] – JSON webservice handles missing values inconsistently
  • [MBS-5713] – Alias types missing from the JSON webservice

Improvement

  • [MBS-4536] – Include the presence of cover art in the webservice
  • [MBS-5304] – “Other Databases” whitelist: “Trove” ammendment
  • [MBS-5447] – Bandwidth performance: revert MBS-4764 now that thumbnail quality has improved
  • [MBS-5606] – Implement STYLE-160 (The Session) in code
  • [MBS-5651] – Clicking link for packaging types loses all entered information
  • [MBS-5652] – Release Label sorting makes no sense on Release Group page
  • [MBS-5702] – Change the text on the “Read more…” link under wikipedia extracts
  • [MBS-5739] – Disconnect database handle on page timeout
  • [MBS-5741] – Time out edit searches in 45 seconds, rather than 30.
  • [MBS-5749] – 401 error: Add link to profile edit page

Task

  • [MBS-5758] – Remove schwag/store links from musicbrainz.org pages

The Git tag for this release is v-2013-01-28.

All Our Git Repositories Are Moving To GitHub

On February 11th 2013, we will be shutting down the MusicBrainz hosted Git repository, in favour of using GitHub for hosting. While we will continue to take backups, these will be internal and thus git.musicbrainz.org will cease to operate as it currently does.

There are a few reasons for this move, but the predominent reason is so MusicBrainz can host less. We currently host a lot of our own software and this is straining the already-limited resources we have. If someone else can do something just as well or better than we can – then let them! Using GitHub also means we get a slightly better repository browser, which can make communicating and discussing code easier. However, we are not moving to pull requests, though we may in the future.

We already mirror git.musicbrainz.org onto GitHub, so if you are only interested in read access, we recommend changing your local clone now, to save confusion in the future. To do this:

  1. cd to your musicbrainz-server clone.
  2. Run git remote set-url origin git://github.com/metabrainz/musicbrainz-server.git

That’s it! You should now be able to git pull just as you did before.

GitHub has a few other possible protocols for cloning, so if HTTP or HTTPS is better for you, then check out the GitHub repository for all the possible links.

If you’re a developer currently developing against our repositories, then you should either fork the repository on GitHub, or contact us to talk about getting your keys added.

Sorry for the disruption!

Server update, 2013-01-14

After delaying this release by a week due to Christmas/New Year getting in the way, we have pushed out a new update of the MusicBrainz web site. This release consists mostly of bug fixes and a few outstanding tasks that we still needed to complete. Thanks to Frederik “Freso” S. Olesen, Michael Wiencek, Johannes Weißl and the MusicBrainz team for their work on this release! Here’s what’s changed:

Bug

  • [MBS-5288] – Use API when checking for WikiDoc changes
  • [MBS-5577] – Relationship Editor displays "vocal vocals"
  • [MBS-5663] – Leap year in "Edit Relationships"
  • [MBS-5716] – Reorder mediums edit is missing a space between medium title and old position
  • [MBS-5717] – Reorder mediums edit for mediums without titles shouldn’t include colons

Improvement

  • [MBS-5723] – Allow sorting collections by all columns, not just a select few.

Task

  • [MBS-3856] – Revert MBS-1052 "Amazon should lookup cover art by barcode too"
  • [MBS-5729] – Consolidate Metabrainz:Sponsors and About/Sponsors

Sub-task

  • [MBS-3974] – Filter by release country

The Git tag for this release is v-2013-01-14.

Replication issues, and packet 64833 is large

  1. We’ve ushered in the new year by discovering, then solving some issues with replication; packet number 64831 (from yesterday, 1AM UTC) didn’t build correctly and needed a bit of manual prodding. However, it’s now been pushed out and replication should be back to normal.
  2. As part of our fix process, we turned off production of replication packets for the bulk of today. As a result, packet number 64833 covers what would otherwise have been about 20 packets, and thus is somewhat large. The import process for this packet will accordingly take somewhat longer than usual.

Sorry for any inconvenience, and happy new year!

Server update, 2012-12-24

We don’t rest for seasonal holidays at MusicBrainz, and we’ve just finished pushing out another release! This one is quite a small one release – thanks to Frederik “Freso” S. Olesen, Nicolás Tamargo, Pavan Chander and the MusicBrainz team for their work on this release. Here’s what’s changed:

Bug

  • [MBS-5288] – Use API when checking for WikiDoc changes
  • [MBS-5497] – CAA uploads are setting x-archive-meta-mediatype to ‘images’, but it should be ‘image’
  • [MBS-5664] – musicbrainz_unaccent serverlog error message

Improvement

  • [MBS-2966] – Give artist credits a special mark
  • [MBS-3412] – Recording name in tooltip when hovering cursor over track title
  • [MBS-4605] – S3 settings in DBDefs.pm are badly named

Task

  • [MBS-5440] – Add lieder.net to the lyrics whitelist
  • [MBS-5684] – Add Yate Tagger to Products menu and home page
  • [MBS-5685] – Add SongKong to Products menu and home page
  • [MBS-5686] – Update to new Amazon Affiliate Tags

Server update, 2012-12-10

We’ve just finished pushing out an update to the musicbrainz.org servers. While work continues on the new edit system and an ingestion solution, this is a fairly small release of mostly bug fixes. Thanks to Pavan Chander and the MusicBrainz team for their work on this release!

Bug

  • [MBS-5035] – Removing release leaves orphaned recordings
  • [MBS-5126] – Orphaned recordings do not get removed (regression)
  • [MBS-5548] – Git commit data output isn’t HTML escaped
  • [MBS-5576] – Release group bar when adding disc IDs is one column too short
  • [MBS-5581] – Some stuff untranslated that wasn’t before
  • [MBS-5618] – Wikipedia abstract logic doesn’t properly encode certain characters
  • [MBS-5622] – MB registration captcha does not work over SSL

Improvement

  • [MBS-4228] – Improve artist alias description
  • [MBS-4568] – Add <bdi> tags to help with rendering of RTL text
  • [MBS-5260] – Cache /text.js somehow
  • [MBS-5317] – Webservice needs to support fmt=json switch to fmt=jsonnew for search queries
  • [MBS-5477] – Show country and gender in artist search results

New Feature

  • [MBS-4108] – Display Wikipedia abstract inline

The Git tag for this release is v-2012-12-10.

Yet another fixed VM uploaded

Though the last version of the 2012-10-15 VM fixed the .ovf to remove an accidentally-included extra disk, this time an incorrect version of a different disk was added. This has been corrected and the finished product tested a bit, so hopefully this time we won’t need another followup blog post!

This time it’s MusicBrainz 2012-10-15.ova, in the usual places.