MusicBrainz schema change release, 2021-05-17 (with upgrade instructions)

We’re happy to announce the release of our May 2021 schema change today! Thanks to all who were patient during today’s downtime as we released everything to our production servers.

This is a fairly minor release as far as schema changes go, but please do report any issues that you come across, especially related to the display of recordings, releases and release groups on artist and release group pages.

New, user-facing changes with this release are limited to the new ability to merge collections (MBS-10208) and the addition of ratings for places (MBS-11451). Additionally, MBS-11463 adds a new view that is used to fix a couple small requests related to disc IDs (MBS-11268) and release length calculation (MBS-11349). Two other changes – adding a first-release-date field to recordings (MBS-1424) and support for PKCE in OAuth (MBS-11097) are more or less end-user affecting but were already released on the main MusicBrainz servers a while ago. All other changes are under the hood only.

We ran into a few complications while working on this schema change update, so we decided to postpone two changes to our October schema change to ensure only stuff we are more confident on is released. Those are MBS-11457, which involves dropping the ordering_attribute column for series and would have had no direct effect on user experience, and MBS-11456, which would have added MBIDs for artist credits.

A few of the released new features and improvements — namely the first-release-date field for recordings, and the performance improvements to artist pages — make use of new materialized tables. These tables aren’t dumped, nor are they replicated, since they’re derived entirely from primary table data. Rather, we’ve added a new script to build them (admin/BuildMaterializedTables, included in the upgrade instructions below), and triggers to keep them up-to-date once they’re built. These triggers are created on replicated servers, too. If you use the web interface or web service at all, just note the extra step of running BuildMaterializedTables after upgrade.sh below!

A new release of MusicBrainz Docker is also available that solves an issue for live indexing and matches this update of MusicBrainz Server. See the release notes for update instructions.

Now, on to the instructions.

Schema Change Upgrade Instructions

Note: Importing the latest data dump is always a valid alternative to running ./upgrade.sh on an existing database, if you’d prefer to also get new data in one go. Just follow the relevant instructions in INSTALL.md. The git tag is v-2021-05-19-hotfixes. The rest of the instructions here assume an in-place upgrade.

1. Make sure DB_SCHEMA_SEQUENCE is set to 25 in lib/DBDefs.pm.
2. If you’re using the live data feed (your REPLICATION_TYPE is set to RT_SLAVE), ensure you’ve replicated up to the most recent replication packet available with the old schema. If you’re not sure, run ./admin/replication/LoadReplicationChanges and see what it tells you; if you’re ready to upgrade, it should say “This replication packet matches schema sequence #26, but the database is currently at #25.”
3. Take down the web server running MusicBrainz, if you’re running a web server.
4. Turn off cron jobs if you’re automatically updating the database via cron jobs.
5. If you’re using the live search indexing, stop it and, assuming sir is under the same directory as musicbrainz-server, run cd ../sir && python2.7 -m sir triggers && cd - && ./admin/psql < ../sir/sql/DropTriggers.sql && ./admin/psql < ../sir/sql/DropFunctions.sql
6. Switch to the new code with git fetch origin followed by git checkout v-2021-05-19-hotfixes.
7. Install newer dependencies Perl 5.30 or later and NodeJS 16 according to install prerequisites.
8. Run cpanm --installdeps --notest . (note the dot at the end) to ensure your perl-based dependencies are up to date.
9. Run ./upgrade.sh (it may take a while to vacuum at the end).
10. Set DB_SCHEMA_SEQUENCE to 26 in lib/DBDefs.pm as instructed by the output of ./upgrade.sh.
11. If you’re using the web interface or web service, run ./admin/BuildMaterializedTables --database=MAINTENANCE all to build new materialized tables. These will take several additional gigabytes of spaces and be kept up-to-date automatically via triggers. For more information, see INSTALL.md.
12. If you’re using the live search indexing, assuming sir is under the same directory as musicbrainz-server, run cd ../sir && git fetch origin && git checkout v2.1.0 && python2.7 -m sir triggers && cd - && ./admin/psql < ../sir/sql/CreateFunctions.sql && ./admin/psql < ../sir/sql/CreateTriggers.sql and rebuild indexes (by running cd ../sir && python2.7 -m sir reindex && cd -) then start it in watch mode (with cd ../sir && git fetch origin && git checkout v2.1.0 && python2.7 -m sir amqp_watch)
13. Turn cron jobs back on, if applicable.
14. Restart the MusicBrainz web server, if applicable. It’s also recommended you restart Redis. If you’re accessing your MusicBrainz server in a web browser, run ./script/compile_resources.sh.

Here’s the list of resolved tickets:

New Feature

• [MBS-10208] – Allow merging collections
• [MBS-11451] – Support ratings for places
• [MBS-11463] – Add view to easily access medium track lengths
• [MBS-11652] – Add support for artist series (hotfixed)

Improvement

• [MBS-10962] – Speed up listing artist’s releases
• [MBS-11268] – Show “Set track durations” on release/discids page
• [MBS-11460] – Add materialized tables to fetch release groups by artist or track artist

Database Schema Change Task

• [MBS-10647] – Add [no label] to b_del_label_special trigger for labels
• [MBS-11453] – Change entity0_cardinality, entity1_cardinality to SMALLINT
• [MBS-11459] – Create the edit_data_type_info function on mirrors
• [MBS-11464] – Drop table statistics.log_statistic
• [MBS-11466] – Change language.frequency and script.frequency to SMALLINT

Previously Released Changes

• [MBS-1424] – Add a ‘First release date’ field to recordings
• [MBS-10821] – Edit changing medium tracklist and format is stuck
• [MBS-11097] – Support PKCE (Proof Key for Code Exchange) by OAuth clients
• [MBS-11431] – Speed up /ws/js/check_duplicates