Author: ianmcorvidae

Server update, 2015-06-15 (a day late)

This release includes a variety of bugfixes, listed below, plus a few more entities are now available for collections.

The git tag for this release is v-2015-06-15.

Bug

  • [MBS-6065] – Internal server error when double submitting an add artist edit
  • [MBS-8408] – Existing external links can be edited to have deprecated types
  • [MBS-8409] – Track parser won’t parse times for data tracks if there’s a disc ID
  • [MBS-8410] – Track parser can change the data track boundary
  • [MBS-8421] – Recording list on a work’s page has misaligned data
  • [MBS-8425] – Wikipedia extract not shown for artist with Commons link in Wikidata
  • [MBS-8431] – Statistics page sorting is broken

Improvement

  • [MBS-6737] – Improve the warning banner
  • [MBS-8031] – make mirror server’s “return to musicbrainz.org” link go to the right place

Server update, 2015-06-01

We’re back with another fortnightly release of musicbrainz-server. A small set of bugs this release, as we’re still primarily working on the bits of UI that were not finished for last fortnight’s schema change release. MBS-7489, relationship artist credits, is now finished, and (not listed below), work collections are done and ready to be used. Still remaining are collections for a variety of additional entities (eventually, everything but URLs, which would just be silly) and tag upvoting/downvoting. Thanks to all contributors and we hope you enjoy these improvements!

The git tag for this release is v-2015-06-01.

Full list of bugs:

Bug

  • [MBS-8187] – URL with open edits not removed when no relationships are left
  • [MBS-8341] – Can’t seed work attributes to /work/create: HFH: param clash for edit-work.attributes value
  • [MBS-8384] – Series-series relationships aren’t displayed
  • [MBS-8401] – “send email” broken for me (only?)

Improvement

  • [MBS-7972] – Make edit relationship edits auto-edits when the endpoints don’t change

New Feature

  • [MBS-7489] – Artist Credits for Relationships

Task

  • [MBS-8375] – Add a bunch of new sites to otherdbs
  • [MBS-8378] – Fix redirect on /tags link to docs

Schema change release, 2015-05-18 (including upgrade instructions)

Our previously mentioned schema change release is finished! Below will be upgrade instructions, including configuration updates for replication access tokens.

This release does not include UI for several of the schema change patches, which will (hopefully) happen for next release on June 1. The incomplete patches are MBS-7489 (credits for artists in relationships), MBS-4145 (tag upvote/downvote), and MBS-8004 (collections for additional entity types). These patches have had their schema change components finished, but the UI was incomplete or needed more work.

Schema Change Upgrade Instructions

These are largely as previous upgrade instructions, using the tag v-2015-05-18-schema-change. The primary difference is the inclusion of configuring an access token for replication.

  1. Make sure your REPLICATION_TYPE setting is RT_SLAVE and your DB_SCHEMA_SEQUENCE is set to 21 in lib/DBDefs.pm. If you’re running a standalone server, you can run the upgrade, but it may be easier to just import a new data dump!
  2. Ensure you’ve replicated up to the most recent replication packet available with the old schema. (if you’re not sure, run ./admin/replication/LoadReplicationChanges and see what it tells you; if you’re ready to update, it should say “Mismatched schema sequence, 21 (database) vs 22 (replication packet)”).
  3. Take down the web server running MusicBrainz, if you’re running a web server.
  4. Turn off cron jobs if you are automatically updating the database via cron jobs.
  5. Switch to the new code with git fetch origin followed by git checkout v-2015-05-18-schema-change
  6. Run ./upgrade.sh (or carton exec -Ilib -- ./upgrade.sh if you’re using carton, with very old setups).
  7. Run cpanm --installdeps --notest . to ensure your perl-based dependencies are up to date. This release adds a dependency on LWP::Protocol::https, for fetching replication packets from the new server; many systems may already have this installed, but it should be verified.
  8. Set DB_SCHEMA_SEQUENCE to 22 in lib/DBDefs.pm as instructed by the output of ./upgrade.sh
  9. Assuming you have been updating your server with replication, it will now be necessary to configure an access token:
    1. Go to https://metabrainz.org/supporters/account-type and choose your account type as applicable. If you’re an individual, non-commercial user of the data, choose “non-commercial”; if not, choose an applicable tier in the “commercial” section. If you’re not sure of the appropriate tier, make your best guess; it can be adjusted if necessary.
    2. Then, from https://metabrainz.org/profile, create an access token, which should be a 40-character random alphanumeric string provided by the site.
    3. Finally, add this token to lib/DBDefs.pm under the REPLICATION_ACCESS_TOKEN configuration option. The final configuration section should look something like sub REPLICATION_ACCESS_TOKEN { "ck3UpgwgOXhWC6SpFcd99rZOTjzfrei3gQlgZZ9z" }.
    4. Don’t reveal your access token! If you do, inadvertently, you can use the MetaBrainz site to generate a new token, invalidating the old one. (The one in the example above is one I created for myself and then invalidated — don’t get any ideas, it won’t work!)
  10. Turn cron jobs back on, if applicable.
  11. Restart the MusicBrainz web server, if applicable. It’s also recommended you restart memcached.

Finally, the list of bugs closed this release:

Bug

  • [MBS-4436] – Medium titles cannot be longer than 255 charaters

Improvement

  • [MBS-1347] – Implement aliases for release groups, releases and recordings
  • [MBS-7906] – maybe don’t show “”≠null diff. in edit pages
  • [MBS-8279] – Remove empty_artists etc. database functions

New Feature

  • [MBS-8302] – Add Live Data Feed access token support

Task

  • [MBS-8266] – Make medium titles VARCHAR NOT NULL
  • [MBS-8278] – Update DB_SCHEMA_SEQUENCE in DbDefs.pm.sample
  • [MBS-8283] – Remove DB constraint that disallows empty event names

Not included in this list but also relevant is MBS-8349, which while fixed for a previous release, in this release is also applied to old slave servers, which may help performance for some queries.

Downstream Wikipedia link usage and migration to Wikidata

MusicBrainz has linked to Wikipedia for many years and we now have links to Wikidata as well. Wikidata, however, acts as a central repository for Wikipedia links, so it does not make sense for MusicBrainz to maintain its own separate set of Wikipedia links, especially since Wikipedia URLs are not very stable (because of page moves and deletions) and require a lot of maintenance. Most of our data with Wikipedia links is now also linked to Wikidata, so we plan to start removing Wikipedia links where we have a Wikidata link which has the same Wikipedia link.

What this means for downstream data users:

If you use Wikipedia links, we will provide Wikidata links but you will need to fetch the Wikipedia links you want from Wikidata separately. Wikidata has information on ways to access their data at https://www.wikidata.org/wiki/Wikidata:Data_access

We plan to start removing the links after the schema change this month, starting with the less common languages and entity types. It will take a while to work through the existing links, so we don’t expect to start removing English links from artists until after the Autumn schema change.

We recognise that some people may have code which depends on these links – if you’re using these links and the above sounds problematic, please let us know how you’re using the data (which languages and entity types) and how much time you would need to support Wikidata.

Server update, 2015-02-09

We’re back with another release! We’ve got a good amount in this release, with patches from the MetaBrainz team as well as chirlu, nikki, reosarevok, Freso, JesseW, and Johan Hattne. A short preview of changes:

  • Releases with many mediums (more than 10) now don’t load medium data by default — with JavaScript on, they can be expanded in place, and without javascript there are newly-minted pages per medium. This also removes the regrettable requirement for JavaScript being on to display release data, which we’d had as a temporary improvement here.
  • Table of Contents offsets for CDs are now returned when the discid endpoints are used.
  • The subscription emails were running into an issue with memory usage, which means some users were not getting emails (or only getting emails sporadically). The script has been reworked to have predictable and lower memory usage so that it’s not killed. Good job on our users for providing it a hard enough task it finally fell over though!
  • Importing and replication should now work with the newest version of DBD::Pg, which made a change that broke our code.
  • A variety of new improvements (many style-related) and bugfixes.

The git tag for this release is v-2015-02-09.

The full list of tickets fixed:

Bug

  • [MBS-2948] – Set track lengths from disc id should indicate the mediums that use the tracklist
  • [MBS-3452] – Merged release annotation is in wrong order
  • [MBS-3841] – Loading releases or release groups with lots of discs often gives 502
  • [MBS-5692] – Internal server error when trying to create a new user with a blank database
  • [MBS-7238] – span.name-variation disappeared from release pages
  • [MBS-7402] – release page regression. inline external URL relationships link to MB only
  • [MBS-7442] – User asked to confirm unchanged “strange” barcode
  • [MBS-7900] – Duplicated tags in tag-list element from ws/2 XML request
  • [MBS-7920] – Nonfunctional rating on release group merge page
  • [MBS-7968] – Inline search triggers searches when the field only contains whitespace
  • [MBS-7985] – Attributes help link does nothing
  • [MBS-8010] – Entity tabs are broken when adding a release group to an existing artist
  • [MBS-8019] – Release tabs broken on the remove disc ID page
  • [MBS-8056] – Loading large-ish releases is very slow in Opera 12
  • [MBS-8058] – Documentation says viewing data doesn’t require JS, which is wrong for releases
  • [MBS-8084] – lightbox html escaping
  • [MBS-8095] – “Edits for your subscriptions” emails have not been sent for at least 3 days
  • [MBS-8104] – Tracklist times entered without colon doesn’t work for times that are not three digits
  • [MBS-8117] – 504 Gateway Time-out when trying to view edits for a Release Collection
  • [MBS-8118] – “Editor flag” edit search criterion fails if given multiple values
  • [MBS-8126] – Import scripts fail with newest DBD::Pg
  • [MBS-8156] – Timeline graph checkboxes are broken in Opera
  • [MBS-8157] – “Script too large” error prevents releases from rendering
  • [MBS-8159] – No secondary release group types in recording search results
  • [MBS-8185] – Series and Events’ “other db” URL rels are not currently being validated against the whitelist
  • [MBS-8191] – Collection edit lists are not made sufficiently private

Improvement

  • [MBS-2656] – Sorting in subscription emails should use collation
  • [MBS-2737] – Limit size of release page
  • [MBS-5752] – Make sure Trove URLs are autocorrected depending on the entity to which they’re linked
  • [MBS-7943] – Add support for TheSession.org events
  • [MBS-8040] – Add admin interface for editing attribute tables
  • [MBS-8164] – Add tests for event collections to Data::Collection

New Feature

  • [MBS-8180] – Return TOC offsets with disc ID in web service

Task

  • [MBS-8106] – Add Instagram URLs to the sidebar
  • [MBS-8138] – Add a bunch of sites to the whitelist for DBs (II)
  • [MBS-8189] – Move last.fm autoselect to the new last.fm relationship

Server update, 2015-01-26

A small release this fortnight as our various developers have been working on bigger projects. However, a few fixes made it in, including making edit searches which time out a lot nicer to use/reword, some better documentation, and slightly better coordinate-parsing.

Thanks to bitmap, chirlu, and the rest of the usual team for this release!

Bug

  • [MBS-7396] – Move Disc ID box out of place
  • [MBS-7984] – Edit searches which time out prevent the user from improving their search
  • [MBS-8002] – Width missing on checkbox column of events list
  • [MBS-8081] – “Remove selected releases from collection” no longer works
  • [MBS-8097] – “Add relationship type” edits don’t store the ID of the new type
  • [MBS-8113] – Strip whitespace when parsing coordinates

Improvement

  • [MBS-6439] – Pasting MBIDs into search fields should not be a hidden feature
  • [MBS-7565] – manual→automatic series edit leads to potentially great data loss
  • [MBS-8101] – Don’t immediately start the edit search when following a “Refine” link
  • [MBS-8122] – Allow switching to direct search when inline (indexed) search errors
  • [MBS-8131] – Remove the “CD in hand” suggestion from the no edit note message

Server update, 2014-12-01

A day late, but hopefully no dollars short, we’re back with another release. This release is mostly bug fixes, as you’d expect right after a schema change release. nikki has done some work with our CSS, however, and chirlu did some work updating and clarifying our INSTALL.md. Thanks to them, mineo, and the MetaBrainz team for their work this release!

The git tag for this release is v-2014-12-01.

The usual list of bugs fixed:

Bug

  • [MBS-4232] – Edit artist shows artists credits section when it doesn’t apply
  • [MBS-4622] – Improve handling of cover art when JS is off
  • [MBS-6971] – No controls for uploading cover artwork in IE8
  • [MBS-7497] – cpanm install fails with Can’t locate File/Copy/Recursive.pm in @INC
  • [MBS-7952] – Release editor does not handle removed mediums correctly when reordering mediums
  • [MBS-7988] – Adding a pregap track to an existing tracklist and it shows as a data track
  • [MBS-7992] – Data track option doesn’t behave properly with disc IDs
  • [MBS-7993] – Edit medium edits for pre-gap tracks claim to change data tracks
  • [MBS-7995] – DBDefs.pm.sample is still at schema version 20
  • [MBS-8011] – Last.fm event URLs don’t get matched
  • [MBS-8013] – Cannot add data track to release
  • [MBS-8018] – “TypeError: MB.typeInfoByID[i] is undefined” when creating an event series
  • [MBS-8022] – Sorting event collections doesn’t work
  • [MBS-8024] – FixTrackLength script is setting incorrect lengths on pregap tracks
  • [MBS-8029] – Collections overview uses “Releases” even for events
  • [MBS-8033] – Encoding issue in some “explanation balloons”
  • [MBS-8034] – Duplicate relationships with attributes cause internal server error in release relationship editor

Improvement

  • [MBS-6332] – Allow sorting releases in collections by release date
  • [MBS-6333] – Allow sorting releases in collections by release country
  • [MBS-7824] – Add Barcode column to Collections releases view
  • [MBS-7997] – Show area in event lists
  • [MBS-8001] – Use larger Gravatar images on higher pixel density displays
  • [MBS-8009] – URL Cleanup should clean FB event links better
  • [MBS-8036] – Add URL matching for setlist.fm

Task

  • [MBS-6682] – Add eu.ftp.musicbrainz.org to INSTALL.md

Schema change upgrade instructions, schema 21

This upgrade shouldn’t be substantially different than past upgrades, now that we’ve fixed a few bugs with the process. To upgrade:

  1. Make sure your REPLICATION_TYPE setting is RT_SLAVE and your DB_SCHEMA_SEQUENCE is set to 20 in lib/DBDefs.pm.
  2. Ensure you’ve replicated up to the most recent replication packet available with the old schema. (if you’re not sure, run ./admin/replication/LoadReplicationChanges and see what it tells you).
  3. Take down the web server running MusicBrainz, if you’re running a web server.
  4. Turn off cron jobs if you are automatically updating the database via cron jobs.
  5. Switch to the new code with git fetch origin followed by git checkout schema-change-20-to-21
  6. Run ./upgrade.sh (or carton exec -Ilib -- ./upgrade.sh if you’re using carton, with very old setups).
  7. Set DB_SCHEMA_SEQUENCE to 21 in lib/DBDefs.pm
  8. Turn cron jobs back on, if needed.
  9. Restart the MusicBrainz web server, if applicable. It’s also recommended you restart memcached.

That’s it! The only real difference from the past is the specific tag to be used: schema-change-20-to-21, which is a couple of fix-up commits past the regular release tag.

Schema change update, 2014-11-17

We’re back with the schema change release, as promised! We only have a small collection of tickets, but several big things:

  • Pre-gap tracks and data tracks for CDs (where neither contribute to the discid, and pre-gap tracks have position of 0)
  • Collections can now be marked with types such as “owned” and “wishlist”, plus some special new types mentioned below.
  • CDStub data is now replicated.
  • All entities (except URLs) should now support tagging, as areas, instruments, and series were made taggable.
  • Events! And, additionally, event collections. All (non-deleted) users should have had an “Attending” and a “Maybe Attending” collection created, with the corresponding collection types.

Upgrade instructions will come in another blog post, though they should be substantially unchanged from past releases. Specifically, we’d like to confirm everything’s working correctly with a specific git commit, and make a new tag, before we post a recommendation, since there’s already been some problems discovered. Some slacker must not have tested this carefully enough (author whistles in an innocent-sounding fashion).

The git commit for this release (sans small fixes that have happened since release earlier today and any others that may need fixing) is v-2014-11-17-schema-change.

Bug

  • [MBS-7638] – CreateIndexes for instruments wrongly looks at label tables

Improvement

  • [MBS-967] – Support for hidden pre-gap tracks
  • [MBS-1059] – Types of list/collection
  • [MBS-7551] – Add folksonomy tag support to areas, instruments, and series
  • [MBS-7784] – Support for data tracks in tracklists

New Feature

Task

  • [MBS-7883] – Make sure delete_unused_url doesn’t remove URLs used in edits

Server update, 2014-10-13 (a day late)

I’m slacking again and it’s a day late, but here’s another server release, almost kinda on schedule!

We’ve included a few nice improvements this release:

  • Better ways to use Track MBIDs (added in May 2013!), by both linking to their URLs on the site and by allowing them to be used as a linked entity for release lookups (this work by ianmcorvidae, with a bit added by chirlu).
  • Place coordinates can now be selected using a map on the edit page and feedback on proper coordinate formatting needn’t wait for a full form submission (thanks nikki!). This new interface can be seen in the image accompanying this post!
  • When the Internet Archive is likely to reject Cover Art Archive uploads, a warning should now be shown on the site (thanks to the IA for implementing an endpoint to use for this, and nikki for integrating it).
  • Data entered on the site should now be normalized according to Unicode Normalization Form C (or NFC), which means that, for example, a character entered with a separate combining acute accent will be normalized to a combined character where one is available (thanks to chirlu for this work).
  • A variety of edit listing pages, including edit search, should be faster due to an overhaul in our query generation and database indexing (thanks to chirlu for this!).
  • A variety of other bug fixes and smaller improvements, as usual (thanks to bitmap, ianmcorvidae, nikki, and chirlu for various pieces).

The git tag for this release is v-2014-10-13.

The full list of issues, as usual:

Bug

  • [MBS-7750] – Video attribute displays confusingly as “(true)”
  • [MBS-7875] – “Recording of” relationship inherits instrument from artist
  • [MBS-7886] – Internal server error when trying to include ratings for places in the webservice
  • [MBS-7888] – Searching in a new tab from relationship editor disables search button
  • [MBS-7892] – Instrument merge process has not been updated for creditable attributes
  • [MBS-7893] – Disambiguation not displayed in area search results
  • [MBS-7905] – ISE when including place-rels and fmt=json
  • [MBS-7915] – Tracklist missing on “Set track durations” page
  • [MBS-7916] – “Set track durations” preview is broken for single-track mediums

Improvement

  • [MBS-3725] – webservice discid lookup should accept query with only a toc
  • [MBS-5553] – Tell the user before uploading when the Cover Art Archive is down
  • [MBS-6010] – Normalise text to NFC
  • [MBS-6829] – Select place location using a map
  • [MBS-6972] – Provide immediate feedback when editing coordinates
  • [MBS-7634] – Standardise lightbox image links from Commons
  • [MBS-7693] – Release Editor’s “edit the release group” does not open in a new tab
  • [MBS-7831] – Votes by Editor is ordered by edit time, but would be more useful ordered by vote time.
  • [MBS-7869] – Make edit search faster
  • [MBS-7895] – Make reordering mediums an auto-edit
  • [MBS-7896] – Deal better with track MBIDs when merging releases
  • [MBS-7897] – Provide a way to look up things by track MBID via the WS
  • [MBS-7898] – Link to track MBIDs from the tracklist