Schema change release: May 13, 2019

It’s been a while since our last schema change release in May 2017. To move forward on some features we’d like to add, we plan to have a Spring 2019 schema change release set for May 13th, 2019. This release should not be disruptive to downstream users, as we plan only to augment the schema with some new tables and columns, and not break any of the existing schema.

Here’s our list of tickets for the Spring 2019 schema change, with descriptions of what’s being changed:

MBS-1658: Add a free text field to collection items. This change will allow users of collections to annotate each item with free text, to hold personalized info about the item (for example, that a release is of a specific pressing, was purchased in 1999 at the Village Discount, or was a gift from your mom). This ticket will add a new TEXT column to editor_collection_release and all other entity-specific editor_collection_* tables. It will not add any new tables or modify any existing columns.

MBS-5387: Mark artist credits as having pending edits when they’re being edited. On an artist’s aliases tab, we provide the ability to directly edit artist credits associated with that artist. However, doing so doesn’t indicate anywhere that the artist credits are being changed (we typically highlight entities with pending edits). To resolve this, we’ll be adding an INTEGER edits_pending column to the artist_credit table. This ticket will not change any existing columns of the artist_credit or artist_credit_name tables.

MBS-5818: Make it possible to have ordered collections. Editor’s collections of entities are automatically ordered by field, for example, releases in collection are ordered by ascending release date. Sometimes, one might want to order collector’s items by other criteria, for example, most wanted releases. This change will enable editors to order collection items by hand if they want to. To do so, an INTEGER position column will be added to the editor_collection_* tables.

MBS-7480: Store cover art image file sizes. Knowing file sizes for cover images and their thumbnails would allow us to better detect when a thumbnail isn’t available, and also allow us to display file sizes to the user before they download an image. To do this, we’ll add four INTEGER columns to the cover_art_archive.cover_art table to store the file sizes in bytes: filesize, thumb_250_filesize, thumb_500_filesize, thumb_1200_filesize.

MBS-9428: Allow multiple users to share one collection. In some cases (like a collection of cleaned up entities several people want to keep an eye on, or a radio station’s collection of vinyl the station owns), it would be quite useful if multiple editors could add (and remove) entries from a collection. This would make cooperation easier and hopefully make community projects (such as the Composer Diversity Database project) easier to start and work on. We’ll add an editor_collection_collaborator table linking collections to the editors allowed to make changes in them (only the collection owner will be able to make changes to the list of allowed collaborators).

MBS-9491: Move hard-coded genres to a database table. We recently added genres to MusicBrainz, but they’re currently stored as an object in the server code. This change will move them to a new table genre (id, gid, name, comment).

MBS-10062: Add aliases for genres. Connected to the previous issue, we need a way to be able to specify “hiphop”, “hip hop” and “hip-hop” are all the same thing, and eventually to store translated versions of genres. This will add a table genre_alias (id, genre, name, locale, edits_pending, last_updated, primary_for_locale).

MBS-9973: Add a date added column for collection items. Editor’s collections have no editing history, thus it doesn’t allow to sort items by date of addition to the collection they belongs to. To allow this, editor_collection_* tables will get a TIMESTAMP added column.

MBS-10052: Add new tables for the event poster archive. This will move us another step toward CAA-84, giving us a place to store event posters, logos, and other related images. The schema change here will be to add a new event_art_archive schema, detailed in the comments in MBS-10052. There will be no change to the existing cover_art_archive schema.

The following tickets will also be included, but only involve adding some missing foreign keys, triggers, and constraints to standalone mirrors; these will not be created and have no effect on replicated mirrors.

MBS-9365: Adds a missing foreign key between the event_meta and event tables.

MBS-9462: Adds some missing l_event_url triggers to delete unused URLs.

MBS-9664: Adds constraints to prevent an entity from linking to itself in a relationship.

If you have any questions, please do leave a comment below or on the linked JIRA tickets!

Server update, 2017-09-18

This release enables relationship credits for areas and places, contains more URL cleanup & auto-select fixes, and changes the behavior of the artist-merge option to rename credits to only rename those that match the original artist names.

The git tag is v-2017-09-18.

Bug

  • [MBS-9380] – ReverbNation URLs are not correctly cleaned up
  • [MBS-9394] – Upper case CD Baby URLs are blocked as “not allowed or incorrectly formatted”
  • [MBS-9446] – PartialDate is missing a TO_JSON method
  • [MBS-9455] – “Performance of” attributes not showing if rel is orderable
  • [MBS-9471] – Relationship editor breaks if two link attribute types have the same name

New Feature

  • [MBS-9469] – Add autoselect for VIAF for places

Task

  • [MBS-9316] – Add Operabase to the otherDBs whitelist
  • [MBS-9353] – Do not show collections or user profiles to people not logged in

Improvement

  • [MBS-9019] – Extend relationship credits to places
  • [MBS-9303] – Add support for Facebook’s fb.com domain
  • [MBS-9334] – Enable URL cleanup for Recochoku artist URLs
  • [MBS-9435] – Update Rock.com.ar URL format
  • [MBS-9444] – Update the YouTube logo used in the sidebar
  • [MBS-9457] – Update the Patreon logo used in the sidebar
  • [MBS-9458] – “Rename artist and relationship credits” should only rename matching credits
  • [MBS-9459] – Enable relationship credits for Areas
  • [MBS-9467] – Add support for fallback Patreon user URLs with numeric ID

 

Server update, 2017-09-04

We’ve been focusing on finishing up some overdue schema change features (alternative tracklists, for one) and doing various refactorings/making some under-the-hood changes, which is why there hasn’t been a release in a while. But here’s a few small fixes that have accumulated.

The git tag is v-2017-09-04.

Bug

  • [MBS-9409] – Error merging mediums with and without pregap tracks

Task

  • [MBS-9416] – wikidata url format redirect addition

Improvement

  • [MBS-9434] – “stream for free” YouTube URL linked to a release should show as “stream on YouTube” instead of “watch on YouTube”

Server update, 2017-07-17

In order to deter spammers, bios and homepages can no longer be set by limited users (accounts less than 2 weeks old, or with less than 10 accepted edits).

Thanks to ferbncode, Sophist, and yvanzo for their work on the tickets below.

The git tag is v-2017-07-17.

Bug

  • [MBS-9342] – InitDb.pl fails to import data with DBD::Pg 3.6.0
  • [MBS-9372] – Revoking editing/voting editor rights also renders them unable to write edit notes
  • [MBS-9392] – JSON-LD schema.org structured data broken
  • [MBS-9401] – SQL generation fails for release groups (script/dump-entities-sql.pl)
  • [MBS-9403] – position attribute is missing in the json api output for media > track

Task

  • [MBS-9355] – Do not allow an editor to set bio/link if the user is limited
  • [MBS-9391] – Retry downloading dumps in docker container (for creating and importing dumps)

Improvement

  • [MBS-9323] – Artist web page with long wikipedia entry looks bad

Broken replication packet fix (#104949)

If your replicated slave threw an error trying to apply packet #104949 (showing the message ERROR: duplicate key value violates unique constraint "artist_alias_idx_primary"), then you can un-break things by doing the following:

    1. Get the latest code from the master branch:
      git checkout master && git pull origin master (OR, if you don’t want to update your code, clear the dbmirror_pending tables instead:
      echo 'TRUNCATE dbmirror_pending CASCADE; TRUNCATE dbmirror_pendingdata CASCADE;' | ./admin/psql READWRITE)
    2. Proceed with replication as normal, either via cron, or by running ./admin/replication/LoadReplicationChanges

You can also re-import from dump 20170605-031203 or later, and replicate from there. We’re very sorry for the inconvenience.

The issue here was caused by a bug in our alias merge code that interacted strangely with dbmirror. Since that code went untouched for years, the trigger for this issue must have been extremely rare. I’ve put in place a fix for the merge logic to ensure it doesn’t happen there again, and am investigating dbmirror’s behavior to see why it didn’t sequence the updates correctly.

Schema change release, 2017-05-15 (including upgrade instructions)

We’re happy to announce the release of our May 2017 schema change today! Thanks to all who were patient during today’s downtime as we released everything to our production servers.

This is a fairly minor release as far as schema changes go, but please do report any issues that you come across.

Currently, the only visible change for editors is the ability to add multiple lyrics languages to works. We’ve also modified the schema to support dynamic attributes for entities other than works, but the UI for that won’t be complete for another release or two.

Now, on to the instructions.

Schema Change Upgrade Instructions

Note: Importing the latest data dump is always a valid alternative to running ./upgrade.sh on an existing database, if you’d prefer to also get new data in one go. Just follow the relevant instructions in INSTALL.md. The rest of the instructions here assume an in-place upgrade.

  1. Make sure DB_SCHEMA_SEQUENCE is set to 23 in lib/DBDefs.pm.
  2. If you’re using the live data feed (your REPLICATION_TYPE is set to RT_SLAVE), ensure you’ve replicated up to the most recent replication packet available with the old schema. If you’re not sure, run ./admin/replication/LoadReplicationChanges and see what it tells you; if you’re ready to upgrade, it should say “This replication packet matches schema sequence #24, but the database is currently at #23.”
  3. Take down the web server running MusicBrainz, if you’re running a web server.
  4. Turn off cron jobs if you’re automatically updating the database via cron jobs.
  5. Switch to the new code with git fetch origin followed by git checkout v-2017-05-15-schema-change.
  6. Run cpanm --installdeps --notest . (note the dot at the end) to ensure your perl-based dependencies are up to date.
  7. Downgrade DBD::Pg by running cpanm TURNSTEP/DBD-Pg-3.5.3.tar.gz (version 3.6.0 breaks things currently).
  8. Run ./upgrade.sh (it may take a while to vacuum at the end).
  9. Set DB_SCHEMA_SEQUENCE to 24 in lib/DBDefs.pm as instructed by the output of ./upgrade.sh.
  10. Turn cron jobs back on, if applicable.
  11. Restart the MusicBrainz web server, if applicable. It’s also recommended you restart redis. If you’re accessing your MusicBrainz server in a web browser, run npm install followed by ./script/compile_resources.sh.

For those curious, here’s the list of resolved tickets (excluding MBS-8393):

Bug

New Feature

  • [MBS-9271] – Prevent usernames from being reused

Task

  • [MBS-9273] – Fix the a_ins_edit_note function in older setups to not populate edit_note_recipient for own notes
  • [MBS-9274] – Fix the edit_note_idx_post_time_edit index in older setups to handle NULL post_time

Improvement

  • [MBS-5452] – Support multiple lyric language values for works