Month: May 2020

PostgreSQL 12 Upgrade Instructions for MusicBrainz Server

Thanks to everyone for your patience during our downtime today. As promised, here are steps to follow to upgrade your own PG instance to v12. (Confused? See the previous blog post on this subject.)

If you’re already running v12, there are still some instructions you must follow!

For MusicBrainz Docker

If you’re running the new MusicBrainz Docker setup, an upgrade script exists for you to use. See the release notes for specific – hopefully brief – instructions.

For a Manual Setup (INSTALL.md Based)

If you aren’t using Docker but rather set up musicbrainz-server by hand following INSTALL.md, see the steps below.

Know that as an alternative, you can always import new data dumps from scratch (again following the steps in INSTALL.md) into a new PG 12 cluster. Just make sure you’re on the v-2020-05-18-postgres12 tag of musicbrainz-server while doing so.

If on the other hand you don’t mind getting your hands a bit dirty, you can use the quicker method below. Like INSTALL.md, this assumes you’re using Ubuntu/Debian and their postgresql-common cluster management tools.

If you’re already running v12, you should still follow these steps; however, you can skip the ones involving apt-get, pg_dropcluster, and pg_upgradecluster. The main steps you need to follow in this case are running the 20200518-pg12-before-upgrade.sql and 20200518-pg12-after-upgrade.sql scripts in that order.

On distros other than Debian/Ubuntu where the postgresql-common tools aren’t available, you’ll have to manage with initdb and pg_upgrade on your own.

  1. First take down the web server running MusicBrainz (stop plackup) to prevent database access.
  2. Turn off any cron jobs updating or accessing the database (e.g. for the live data feed/replication packets).
  3. Switch to the latest musicbrainz-server code with:
    git fetch origin && \
        git checkout v-2020-05-18-postgres12
  4. With PG 9.5 (or whatever version you’re using) still running, run the following “pre-upgrade” script:
    psql -U postgres -d musicbrainz_db \
         -f admin/sql/updates/20200518-pg12-before-upgrade.sql

    This assumes that “postgres” is the name of your PG superuser, and “musicbrainz_db” is the name of your database. If you see a few messages about things not existing, that’s normal.

  5. Install packages for PostgreSQL 12. On Ubuntu/Debian you can obtain them from the PGDG apt repo.
    apt-get update && \
        apt-get install postgresql-12 postgresql-server-dev-12

    If you’re installing postgresql-12 for the first time, this will automatically create a new cluster at /var/lib/postgresql/12/main. Remove that empty cluster. Don’t run this if you already had v12 installed and have data there!

    pg_dropcluster --stop 12 main
    If you did already have v12 installed with musicbrainz_db running there, leave the cluster alone and skip the next step involving pg_upgradecluster.

    In the unlikely event that you already have a v12 cluster, but also have musicbrainz_db running in a separate, older cluster, these instructions won’t work for you. We recommend importing fresh data dumps into the v12 cluster and dropping the old one.

  6. Upgrade the old cluster. This assumes it’s version 9.5; if you’re using version 10 or 11, make sure to replace 9.5 below with 10 or 11. If you have other databases in your old cluster besides musicbrainz_db, be aware that this will upgrade all of them to PG 12.
     pg_upgradecluster -v 12 9.5 main
  7. If all goes well, the new cluster should be up and running. (You can drop the old one if you like; the output of the pg_upgradecluster command will tell you how.) Now run the following “post-upgrade” script on the database:
    psql -U postgres -d musicbrainz_db -f \
         admin/sql/updates/20200518-pg12-after-upgrade.sql
    This may take a bit, as it has to recreate some indexes.
  8. The upgrade is complete. You can turn cron jobs back on, if applicable.
  9. Restart the MusicBrainz web server / plackup, if applicable. If you’re accessing the server in a web browser, the usual release upgrade steps apply, like running ./script/compile_resources.sh again.

If you run into any trouble following the above, please let us know and we’ll try to help resolve your issue as soon as possible!

MusicBrainz Docker composes with Solr 7

The MusicBrainz virtual machine is dead, long live the MusicBrainz Docker Compose project. In fact, the virtual machine has been running it for years. Mostly because the data loaded with the virtual machine was too soon obsolete, it doesn’t seem worth it anymore. Plus, new search indexes are much larger than before, and using Docker Compose directly is much more versatile.

The MusicBrainz Docker Compose project has been deeply revamped since two years ago and now ships the new search server based on Solr 7. It can be used for mirroring the MusicBrainz website and database, testing your own app with a local MusicBrainz web service, or developing the MusicBrainz Server itself. Check out the release notes!

Thanks to everyone who reported issues and contributed patches for two years!

MusicBrainz Server update, 2020-05-11

This is the last update before upgrading to Postgres 12. It is mainly focused on React conversion but also carries ten small bugfixes and improvements.

Thanks to navap for hacking the user interface. Thanks to admiy, chaban, fabe56, Freso, jesus2099, zas for having reported bugs and suggested improvements. Thanks to kellnerd, mfmeulenbelt, and salorock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2020-05-11.

Bug

Improvement

  • [MBS-10737] – Allow thesession.org URLs for Places
  • [MBS-10761] – Disallow YouTube links at wrong levels
  • [MBS-10804] – Remove redundant user header from edit page
  • [MBS-10805] – Support for Amazon.AE/NL/SG/TR ASINs

React Conversion Task

  • [MBS-9910] – Convert wikidocs transclusion admin templates to React/JSX
  • [MBS-10748] – Convert the relationship doc page to React
  • [MBS-10760] – Convert Remove Track edit to React
  • [MBS-10762] – Convert historic Remove Release/Releases edits to React
  • [MBS-10764] – Convert historic Remove Label Alias edits to React
  • [MBS-10765] – Convert historic Add, Move and Remove DiscID edits to React
  • [MBS-10773] – Convert historic Change Release Quality edit to React
  • [MBS-10775] – Convert historic Add/Remove Relationship edits to React
  • [MBS-10790] – Convert historic MAC/SAC edits to React
  • [MBS-10791] – Convert historic Change RG edit to React
  • [MBS-10792] – Convert historic Change Artist Quality edit to React
  • [MBS-10811] – Convert historic Edit Relationship edit to React

Picard 2.3.2 released

Picard 2.3.2 has been released and is available for download. This is mainly a bug-fix release with few minor improvements, see below for a full list of changes.

This release contains code changes by zas, rdswift, outsidecontext, RaysDev and new contributors Undearius and Gabrielcarvfer, who also is working on some interesting changes regarding UI responsiveness.
Thanks a lot to everybody who contributed to this release with code, translations, bug reports and general feedback.

Download

Picard 2.3.2 is available for download from the download page.

Changes

Below is a list of all changes since Picard 2.3.1. If you are upgrading from an older version you might want to read previous release post.

Breaking change:

$find script function now returns “” rather than -1 on not found, for the sake of consistency. You may want to review your scripts.

Notable improvements:

Picard was trying to guess the file format based on file content. This was very IO expensive and in many cases not needed. So we decided to change the logic to:

  • Try guessing the format by file extension first
  • If no extension or no matching file extension is available, guess by file content
  • If format was guessed by extension and loading the file with mutagen fails with an exception, retry with guessing by file content. Retry loading if the newly guessed format is different from the format guessed by extension.

Also notable, On Windows 10, support for fractional scaling was implemented.

The complete list of changes is the following:

Bug

  • [PICARD-1775] – $firstwords function doesn’t catch IndexError
  • [PICARD-1776] – $datetime crashes when invalid format specified
  • [PICARD-1781] – Have $find return “” rather than “-1” on not found
  • [PICARD-1783] – Deleting performer, comment or lyrics tag fails for some cases in ID3, Vorbis, MP4 and Apev2 tags
  • [PICARD-1784] – Host not found error when downloading PDF artwork
  • [PICARD-1785] – $delete(_id3:TXXX:foo) does not delete the ID3 frames
  • [PICARD-1786] – Deleting tag stored in ID3 TIPL frame only removes first occurrence
  • [PICARD-1787] – Deleting case-insensitive TXXX frames does not delete anything
  • [PICARD-1788] – Saving ID3 tags marked as case-insensitive causes duplicated TXXX frames
  • [PICARD-1790] – Crash when selecting plugin that can be updated
  • [PICARD-1791] – Network access is disabled error under VPN service
  • [PICARD-1795] – iTunes tags not removable (reappear after being deleted)
  • [PICARD-1801] – List index out of range when saving Vorbis file
  • [PICARD-1803] – Instrument EWI is rewritten “e w i”

Improvement

  • [PICARD-1777] – Support fractional scaling on Windows 10
  • [PICARD-1800] – Simplify Unicode ‘Bullet’ to ASCII ‘Hyphen-minus’
  • [PICARD-1809] – Optimize format detection logic

Reminder: Upgrading to PostgreSQL 12 on May 18, 2020

As we announced in February, in two weeks time (May 18, 2020) we’ll be upgrading our production database server to PostgreSQL v12 (from v9.5). At the same time, v12 will become the minimum supported version for MusicBrainz Server, so we ask that you upgrade afterwards as soon as possible! If you’re still unsure, a Q&A is below.

When do I need to upgrade my postgres by?

As soon as possible after May 18 if you’d like to keep your musicbrainz-server code up to date.

How do I perform the upgrade?

We’ll provide instructions closer to May 18. It’s recommended that you don’t upgrade until then, since we’ll be providing scripts to resolve some issues.

Will the live data feed (replication packets) stop working right away if I don’t upgrade?

No, as long as you keep your musicbrainz-server code checkout on the v-2020-05-11 tag (which will be the final release before May 18) or earlier. Future releases may work for a while too.

This is not a schema change release, so replication will continue to work smoothly until you upgrade. No tables or views will change.

However, to make the upgrade process smoother we’ll be dropping the musicbrainz-collate and musicbrainz-unaccent extensions, instead using PG’s builtin collation support for the former and replacing the latter with the unaccent extension from postgresql-contrib. A few SQL functions are being added to enable this, and some indexes need to be rebuilt. This will all happen as part of upgrade scripts we provide (or you can import from scratch). Some features of musicbrainz-server that use these old extensions may cease to work if you don’t apply them.

The extension changes above don’t actually make use of any new PG 12 features. We’ll avoid using such features for at least 1 month.

If I’m already running PostgreSQL 12, do I need to do anything?

Yes, but things will be easier for you. As mentioned in the previous answer, we’ll be dropping the musicbrainz-collate and musicbrainz-unaccent extensions to make the upgrade process smoother for pre-v12 instances. So you’ll only have to run some upgrade scripts we provide to replace those extensions and rebuild some indexes.

My host/distribution doesn’t have PostgreSQL 12 yet!

If you’re running Debian or Ubuntu, the PGDG maintains an APT repository with the latest versions. These are the same packages MetaBrainz uses in production.

Amazon RDS supports PostgreSQL 12 since March 31.

I absolutely cannot upgrade yet! What should I do?

You can stay on the v-2020-05-11 release of musicbrainz-server or earlier until then. Replication packets (i.e. the live data feed) will continue to work until the next schema change on that tag, but you’ll have upgraded to v12 by then, right?

Instead of performing a pg_upgrade and running these upgrade scripts you mentioned, can I just import fresh data dumps into a new v12 cluster?

Of course. Just make sure your musicbrainz-server git checkout is on the v-2020-05-18 tag (once that’s released) or later before performing the import. And keep in mind it may be slower than a direct upgrade.