MusicBrainz Server update, 2022-06-20

This release includes the first genre improvements made possible by the May schema change, plus a bunch of small fixes and improvements. Annotation pages have been converted to React; if you notice anything strange with annotations, it might be a new regression so make sure to report it!

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to anshuman73, chaban, Lotheric, selflessself and yindesu for having reported bugs and suggested improvements. Thanks to mfmeulenbelt and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2022-06-20.

Continue reading “MusicBrainz Server update, 2022-06-20”

MusicBrainz Server update, 2022-06-07

We’re back after a bit of post-schema change rest! Because we were recovering from the crazy schema change times, there’s mostly small fixes here. Two changes (the update to React 18 and the change to allow some very long words to wrap) caused a few issues that were found and fixed in beta, but please do let us know if you find any that slipped through! The implementation of some remaining schema change features (more genre options, moods) will hopefully be released in two weeks.

Thanks to adamselene, CatQuest, chaban, darwinx0r, draconx, Jeluang, sheamuspatt, vzell and yindesu for having reported bugs and suggested improvements. Thanks to mfmeulenbelt and salo.rock for updating the translations. And thanks to all others who tested the beta version!

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

The git tag is v-2022-06-07.

Continue reading “MusicBrainz Server update, 2022-06-07”

Picard 2.8.1 released

Picard 2.8.1 is a maintenance release for the recently release Picard 2.8 with fixes for reported issues and updated translations.

The latest release is available for download on the Picard download page.

The detailed changes for this maintenance release are below. For an overview of the new features since Picard 2.7 please see our detailed release announcement for Picard 2.8.

Thanks a lot to everyone who gave feedback and reported issues.

Continue reading “Picard 2.8.1 released”

MusicBrainz schema change release, 2022-05-16 (with upgrade instructions)

We’re happy to announce the release of our May 2022 schema change today! Thanks to all who were patient during today’s downtime as we released everything to our production servers, and thanks to ikerm2003, mfmeulenbelt rinmon and salo.rock for updating the translations.

This is once again a fairly minor release as far as schema changes go, but please do report any issues that you come across, especially related to the propagation of ratings and tags.

New, user-facing changes with this release include withdrawn-only release groups showing in the official overview again (MBS-12208) and the final disappearance of Amazon cover art (MBS-12200). To this regard, the report of releases that have Amazon cover art but no Cover Art Archive front cover will stay available for editors to check their subscribed entities.

Additionally, several small changes will allow, in the next couple of releases, to store more information about genres (including URL relationships) and to recognize and special-case mood tags (MBS-12190). Another new feature that will start to be used in the API and artist credit pages is the addition of MusicBrainz IDs for artist credits, which allow referring to them with a unique and more persistent ID (MBS-11456). Finally, a few more under-the-hood only changes are made, which should ensure better performance for finding artists, events, etc. for all areas contained in a given one, and less bugs when adding and changing tags and ratings.

The area containment changes make use of a new materialized table. Like the ones we added last year, this table isn’t dumped nor replicated, since it is derived entirely from primary table data. Rather, it will be created during migration (or, in a new install, by running the admin/BuildMaterializedTables script) and triggers will be added to keep it up-to-date once it has been built. These triggers are created on replicated servers, too.

The accompanying new version of the search index rebuilder brings performance improvements for both the main server and mirrors, and simplifies maintenance. See the release notes for details.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Now, on to the instructions.

Schema Change Upgrade Instructions

Note: Importing the latest data dump is always a valid alternative to running ./upgrade.sh on an existing database, if you’d prefer to also get new data in one go. Just follow the relevant instructions in INSTALL.md. The git tag is v-2022-05-16.1-schema-change. The rest of the instructions here assume an in-place upgrade.

  1. Make sure DB_SCHEMA_SEQUENCE is set to 26 in lib/DBDefs.pm.
  2. If you’re using the live data feed (your REPLICATION_TYPE is set to RT_SLAVE), ensure you’ve replicated up to the most recent replication packet available with the old schema. If you’re not sure, run ./admin/replication/LoadReplicationChanges and see what it tells you; if you’re ready to upgrade, it should say “This replication packet matches schema sequence #27, but the database is currently at #26.”
  3. Take down the web server running MusicBrainz, if you’re running a web server.
  4. Turn off cron jobs if you’re automatically updating the database via cron jobs.
  5. If you’re using the live search indexing, stop it and, assuming sir is under the same directory as musicbrainz-server, run cd ../sir && python2.7 -m sir triggers && cd - && ./admin/psql < ../sir/sql/DropTriggers.sql && ./admin/psql < ../sir/sql/DropFunctions.sql
  6. Switch to the new code with git fetch origin followed by git checkout v-2022-05-16.1-schema-change.
  7. Run cpanm --installdeps --notest . (note the dot at the end) to ensure your perl-based dependencies are up to date.
  8. Run ./upgrade.sh (it may take a while to vacuum at the end).
  9. Set DB_SCHEMA_SEQUENCE to 27 in lib/DBDefs.pm as instructed by the output of ./upgrade.sh.
  10. If your REPLICATION_TYPE is set to RT_SLAVE, change it to RT_MIRROR. (The previous terminology will work for the time being, but is now deprecated.)
  11. If you’re using the live search indexing, assuming again that sir is under the same directory as musicbrainz-server, run cd ../sir && git fetch origin && git checkout v3.0.1 && python2.7 -m sir triggers && cd - && ./admin/psql < ../sir/sql/CreateFunctions.sql && ./admin/psql < ../sir/sql/CreateTriggers.sql and rebuild indexes which takes hours (by running cd ../sir && python2.7 -m sir reindex && cd -) then start it in watch mode (with cd ../sir && git fetch origin && git checkout v3.0.1 && python2.7 -m sir amqp_watch)
  12. Turn cron jobs back on, if applicable.
  13. Restart the MusicBrainz web server, if applicable. It’s also recommended you restart Redis. If you’re accessing your MusicBrainz server in a web browser, run ./script/compile_resources.sh.

Here’s the list of resolved tickets:

Fixed Bug

  • [MBS-5359] – *_tag tables are corrupt and need to be regenerated
  • [MBS-11760] – Removing the last use of a tag does not always remove the tag
  • [MBS-12369] – Standalone databases may be missing foreign keys for the documentation schema

New Feature

  • [MBS-12190] – Add Mood support in the database

Improvement

  • [MBS-11456] – Add MBIDs to artist credits in the database with merge
  • [MBS-12141] – Block tag names that are empty or have uncontrolled whitespace with database constraints
  • [MBS-12224] – Keep tags’ ref_count and aggregate vote counts updated with triggers
  • [MBS-12249] – Add a materialized area_containment table kept up-to-date with triggers
  • [MBS-12256] – Keep rating and rating_count column on *_meta tables up-to-date with triggers
  • [MBS-12313] – Clarify item naming in the Search drop down menu

Database Schema Change Task

  • [MBS-11457] – Drop the series ordering_attribute column
  • [MBS-11755] – Remove unused tags
  • [MBS-12157] – Remove support for Amazon cover art
  • [MBS-12200] – Drop schema objects related to Amazon cover art support
  • [MBS-12225] – Rename “slave” to “mirror” (inclusive language update)
  • [MBS-12250] – Create dbmirror2 schema on production and mirror servers
  • [MBS-12252] – Add edit_genre table
  • [MBS-12253] – Add relationship tables for genres
  • [MBS-12254] – Add genre_annotation table
  • [MBS-12255] – Add genre_alias_type table and make genre_alias consistent

MusicBrainz Server update, 2022-04-18

We’re back with another smallish release, mostly fixing minor bugs. This is our last release before the schema change on May 16; we are taking a one release break to have more time to concentrate on that. Of course you should still let us know if you find we have introduced any new bugs (er… totally intentional easter eggs that is!), but unless they are very serious, we will probably only fix them during the second half of May.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to arcresu, chaban and Cyberskull, for having reported bugs and suggested improvements. Thanks to mfmeulenbelt and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2022-04-18.

Continue reading “MusicBrainz Server update, 2022-04-18”

MusicBrainz Server update, 2022-03-28

This release makes a bunch of small changes, mostly to URLs. One change that might be particularly worth noting: while earlier the series creation form defaulted to “Release group series” as type, now there’s no default and the user needs to actively pick the kind of series they want. We expect this to help with a relatively common issue where editors would try to add a new series for releases and accidentally create a release group series, leading to confusion about why they couldn’t add their release to it.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to CatQuest, chaban, Cyberskull, jesus2099, mfmeulenbelt and mr_maxis for having reported bugs and suggested improvements. Thanks to mfmeulenbelt and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2022-03-28.

Continue reading “MusicBrainz Server update, 2022-03-28”

Schema change release: May 16, 2022

Today we’re announcing a MusicBrainz database schema change release planned for May 16, 2022. The majority of these changes follow the theme of improving data integrity and consistency, performance, or just cleaning up old cruft. Others relate to new features for genres and artist credits. We’re also introducing a new entity based on tags, like genres that came before it: Mood. See below for more details, including information on how these changes affect the schema or existing data. We expect people will encounter zero breaking changes, but it doesn’t hurt to double check, especially if you have a specific or non-standard use of the database!

Here’s our list of tickets for the Spring 2022 schema change:

Schema changes

  • MBS-12256: Keep rating and rating_count column on *_meta tables up-to-date with triggers. An internal-only change to help us keep aggregate rating information (i.e. an average rating and count of ratings for each entity) up-to-date more easily, and help keep these values accurate. This change affects master/standalone databases, but should have no impact on mirror servers, where such triggers are not created. It’s possible that some existing aggregate ratings data was out of sync and will be updated with this change.
  • MBS-12224: Keep tags’ ref_count and aggregate vote counts updated with triggers. Like the change above for ratings, this is primarily internal-only and intended to help us keep tag counts in sync, though an adjacent goal is to make features like MBS-960 easier to implement. This also revives the tag.ref_count column, which hasn’t been updated for years, in order to provide a faster way of sorting tags/genres by usage count. Like above, this change should have no impact on mirror servers schema-wise, but will fix some existing corrupt tag counts.
  • MBS-12249: Add a materialized area_containment table kept up-to-date with triggers. Pages that make use of area containments, e.g. the list of artists from an area, which are expected to account for sub-areas, are currently quite slow and we’d like to improve upon this. The slowness is related to the recursive queries we use to get contained sub-areas – these queries are uncached and calculated on-the-fly. This ticket addresses these performance issues by caching area containment information in a new aptly-named area_containment table. Consistent with the tag and rating tickets above, this table will also be kept up-to-date with triggers. This change should have no impact on mirror servers except to make certain area requests faster; it does not affect existing data.
  • MBS-12250: Create dbmirror2 schema on production and mirror servers. The dbmirror extension we use to generate our replication packets a.k.a Live Data Feed is a 20 year old tool. It has issues and limitations that are difficult to fix, and we aim to replace it with something more maintainable. We wrote dbmirror2 to do that, but still have the task of getting it deployed to mirrors seamlessly. This will happen invisibly without any changes needed on mirrors! The action for this ticket is to simply create the schema for dbmirror2; it’s not actually used for replication yet. We’ll first have a testing phase and make sure external projects like mbdata work with the new replication packet format.
  • MBS-12200: Drop schema objects related to Amazon cover art support. For a long while, releases with Amazon URLs would be checked for cover art on Amazon, and if found, a link to the image would be cached for display. Unfortunately Amazon’s API to do this changed, and we haven’t synced artwork from them in years. We still have many old images cached and we still display those, but they aren’t guaranteed to be in sync. Last year we decided to drop support for displaying these, while giving time for users to upload any correct images to the Cover Art Archive. To help with this, we have a report of releases with Amazon cover art but no Cover Art Archive front cover.

    The schema change here involves dropping the release_coverart table (which was private and non-replicated) and the release_meta.amazon_store column (which was completely empty and unused). This change should have no impact on mirror servers, unless you were using this table or column for your own purposes, because they should be otherwise empty.
  • MBS-12141: Block tag names that are empty or have uncontrolled whitespace with database constraints. Recently we discovered a bug where empty or blank tag names could be submitted in the web service. This has since been fixed, but we’d also like to prevent such empty tags by adding a database constraint. This change has no effect on mirror servers schema-wise, where such constraints are not created. A few blank tags have already been deleted from the production database, but otherwise existing data is not affected. If any such tags exist in your standalone database, they’ll be deleted.
  • MBS-12252: Add edit_genre table. A requirement to start storing edit history for genres (right now changes leave no trail). This will add the empty table to mirror servers as well, but will not affect any existing data.
  • MBS-12253: Add relationship tables for genres. A requirement to be able to relate genres to other entities (such as URLs for the equivalent Wikipedia or Rate Your Music pages). This will add the empty tables (and accompanying example tables in the documentation namespace) to mirror servers as well, but will not affect any existing data.
  • MBS-12254: Add genre_annotation table. A requirement to make it possible to eventually add annotations to genres. This will add the empty table to mirror servers as well, but will not affect any existing data.
  • MBS-12255: Add genre_alias_type table and make genre_alias consistent. Originally the (as yet unused) genre_alias table was designed as a heavily simplified version of the alias tables for other entities. In retrospect, this was not a good decision, since it would make it harder to just use the generic implementation of our alias code for genres. As such, we’re adding a genre_alias_type table (originally genre aliases had no types) and replacing the genre_alias table with one having the extra columns matching other entities’ alias tables. These changes will also happen to mirror servers, but since the genre_alias table was completely unused it should not cause any issues. In case any standalone (not mirrored) servers were using genre_alias, we will ensure any existing data is transferred to the new version of it.
  • MBS-12241: Drop the whitespace_collapsed database constraint. We’ve had a constraint for years that tries to ensure that columns like entity names do not contain multiple consecutive spacing characters (disallowing names such as “This    Title”). In retrospect, this was overreaching, since there are several cases in which a specific number of spaces in a title can be shown to be artist intent. Additionally, we recently discovered that we had some very old data in the database that actually violated this constraint (causing issues when importing data to a standalone server). The data seemed to actually be correct (i.e. some of the aforementioned edge cases) so rather than amending it, we’re removing the constraint. This will have no effect on mirrors since they don’t run constraints.
  • MBS-12225: Rename “slave” to “mirror” (inclusive language update). We recently got a request from a long-time supporting organization to pick a different term for what we’ve historically called “slave server”. Since we were already sometimes using “mirror server” to mean the exact same thing, we are just changing the official name and will use “mirror server” in the future. RT_SLAVE will still work in DBDefs.pm, at least for now, but we’d suggest changing to the new (and equivalent) RT_MIRROR in your mirror servers’ DBDefs.pm. We’re likely to eventually drop support for RT_SLAVE in a future schema change, so we’ll remind you about changing it in future upgrade instructions.
  • MBS-12190: Add Mood support. Music mood was originally meant to be automatically calculated by the now-discontinued AcousticBrainz project. That’s obviously no longer in the cards, but this data would still be quite useful for ListenBrainz. As such, we’re planning to add basic support for mood tags in MusicBrainz in the same way we currently do genres. This can then be leveraged by ListenBrainz to collect the information directly from users playing music through their BrainzPlayer, and it can also of course be entered directly from MusicBrainz in the same way other tags (including genre tags) can. This will add new mood tables to mirrors and will detect some previously generic tags as mood tags, but it won’t cause any changes to the underlying data.
  • MBS-11760: Expand the database triggers which remove empty tags to all entities. When the last use of a tag (up- or downvote) is removed, we use triggers to completely remove the tag from the database. Some of the relevant triggers (for events, places, recordings and releases) were never created though, so if the last use was on an entity of one of those types the empty tag would not get purged. This doesn’t affect mirrors directly since the triggers don’t run on mirrors (but they should no longer get unused tags coming through replication).
  • MBS-11457: Drop series ordering_attribute. This column was added back when we were expecting to have different types of ordering attributes for series, but we have never used it. We planned to remove it last year already, but that required equivalent changes on the search server that couldn’t be made at the time. We are planning to just drop the column.
  • MBS-11456: Add MBIDs and redirect tables for artist credits. Adds a gid column to the artist_credit table, and a new artist_credit_gid_redirect table. It generates MBIDs for existing artist credits that will be replicated to mirrors.

    Artist credit MBIDs will be mainly exposed through the web service at first. The MBIDs will allow public identification of artist credits outside of MusicBrainz, and open the possibility of some more features in the future.
  • MBS-12208: Show withdrawn release groups in the official artist overview. The Withdrawn release status was added recently. Release groups containing only releases with this status were meant to be shown in the main (official) artist overview, but the way this is implemented means a small edit to the get_artist_release_group_rows function was needed. Mirrors using the materialized tables will need to update the data; more info will be provided as part of the upgrade instructions.

We’ll post upgrade instructions for standalone/mirror servers on the day of the release. If you have any questions, feel free to comment below, or on the linked JIRA tickets if relevant there!

NB: This post was updated on 21 March to include a ticket (MBS-12208) that we forgot to list originally

MusicBrainz Server update, 2022-03-14

This release, aside from fixing a few more small bugs, adds a lot of new features to the filtering options available on artist pages. You can now filter works (including by whether the artist was a writer or a performer), filter release groups by secondary type, specify you want only cases with no types set at all. Additionally, pages with lists of edits have just been converted to React, so do let us know if you see any strange behaviour there.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to rinsuki for updating the cleanup for niconico external links. Thanks to chaban and jesus2099 for having reported bugs and suggested improvements. Thanks to Besnik, mfmeulenbelt and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2022-03-14.

Continue reading “MusicBrainz Server update, 2022-03-14”

MusicBrainz Server update, 2022-02-28

This release includes a fair amount of small improvements and bugfixes, including many changes to the edit list headers. It also allows editors to set their gender to the recently-added “non-binary” option.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to atj, CatQuest, chaban, HibiscusKazeneko, jesus2099, Lotheric, ocharles, PierPiero for having reported bugs and suggested improvements. Thanks to Besnik, grafi_tt, mfmeulenbelt, outsidecontext and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2022-02-28.

Continue reading “MusicBrainz Server update, 2022-02-28”

MusicBrainz Server update, 2022-02-14

This is a small release of bugfixes and improvements, since a lot of the team’s focus recently has been on improving code documentation and tests and other behind the scenes tasks. It does fix a couple bugs that had been reported multiple times though (the release editor entering bogus empty annotation edits, and the area selection dropdown sometimes failing to work properly) so hopefully it will still make some of your editing a bit better!

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to yyoung for updating the cleanup for Lantis external links. Thanks to bsammon, jesus2099, kellnerd, Lotheric, mr_maxis, nikki, yindesu and yyoung for having reported bugs and suggested improvements. Thanks to mfmeulenbelt and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2022-02-14.

Continue reading “MusicBrainz Server update, 2022-02-14”