Reminder: Upgrading to PostgreSQL 12 on May 18, 2020

As we announced in February, in two weeks time (May 18, 2020) we’ll be upgrading our production database server to PostgreSQL v12 (from v9.5). At the same time, v12 will become the minimum supported version for MusicBrainz Server, so we ask that you upgrade afterwards as soon as possible! If you’re still unsure, a Q&A is below.

When do I need to upgrade my postgres by?

As soon as possible after May 18 if you’d like to keep your musicbrainz-server code up to date.

How do I perform the upgrade?

We’ll provide instructions closer to May 18. It’s recommended that you don’t upgrade until then, since we’ll be providing scripts to resolve some issues.

Will the live data feed (replication packets) stop working right away if I don’t upgrade?

No, as long as you keep your musicbrainz-server code checkout on the v-2020-05-11 tag (which will be the final release before May 18) or earlier. Future releases may work for a while too.

This is not a schema change release, so replication will continue to work smoothly until you upgrade. No tables or views will change.

However, to make the upgrade process smoother we’ll be dropping the musicbrainz-collate and musicbrainz-unaccent extensions, instead using PG’s builtin collation support for the former and replacing the latter with the unaccent extension from postgresql-contrib. A few SQL functions are being added to enable this, and some indexes need to be rebuilt. This will all happen as part of upgrade scripts we provide (or you can import from scratch). Some features of musicbrainz-server that use these old extensions may cease to work if you don’t apply them.

The extension changes above don’t actually make use of any new PG 12 features. We’ll avoid using such features for at least 1 month.

If I’m already running PostgreSQL 12, do I need to do anything?

Yes, but things will be easier for you. As mentioned in the previous answer, we’ll be dropping the musicbrainz-collate and musicbrainz-unaccent extensions to make the upgrade process smoother for pre-v12 instances. So you’ll only have to run some upgrade scripts we provide to replace those extensions and rebuild some indexes.

My host/distribution doesn’t have PostgreSQL 12 yet!

If you’re running Debian or Ubuntu, the PGDG maintains an APT repository with the latest versions. These are the same packages MetaBrainz uses in production.

Amazon RDS supports PostgreSQL 12 since March 31.

I absolutely cannot upgrade yet! What should I do?

You can stay on the v-2020-05-11 release of musicbrainz-server or earlier until then. Replication packets (i.e. the live data feed) will continue to work until the next schema change on that tag, but you’ll have upgraded to v12 by then, right?

Instead of performing a pg_upgrade and running these upgrade scripts you mentioned, can I just import fresh data dumps into a new v12 cluster?

Of course. Just make sure your musicbrainz-server git checkout is on the v-2020-05-18 tag (once that’s released) or later before performing the import. And keep in mind it may be slower than a direct upgrade.

MusicBrainz Server update, 2020-04-27

A large variety of issue types have been addressed in today’s release!

As a new feature, search indexes are now dumped and made available along with database dumps under the FTP directory search-indexes. They are mainly intended to be loaded on a MusicBrainz slave server to start a mirror with search.

Among improvements, a noticeable one is to lighten the area’s overview page that was heavily crowded with all sorts of relationships that have been scattered over more specific tabs.

As for bugfixes, the major one secures user/admin forms against CSRF attacks.

Thanks to atj for contributing code to support Traxsource URLs. Thanks to alex_s7, chaban, danbloo, Lotheric, murdos, Skeebadoo for reporting issues. Thanks to kellnerd, Jormangeud, mfmeulenbelt, salorock for updating the translations in German, Finnish, Dutch, Italian, respectively. And thanks to all others who tested the beta version!

The git tag is v-2020-04-27.

Bug

  • [MBS-10359] – Guess feat. artists from track titles do not give expected result
  • [MBS-10677] – Place type shown as null on WS event place rels
  • [MBS-10717] – Cookie attributes must be adjusted to work with with new behavior in browsers
  • [MBS-10719] – “remove PUID” edit doesn’t load
  • [MBS-10742] – “Show more” country miscount
  • [MBS-10756] – Inconsistent default sort order for recordings on Works page
  • [MBS-10778] – User/admin forms are prone to CSRF attacks

New Feature

  • [MBS-10546] – Dump MB Solr data along with MB DB full export

Improvement

  • [MBS-1921] – Display edit link under annotations
  • [MBS-9086] – Move most relationships away from area overview
  • [MBS-10666] – Collapse work artists when there are too many on merge pages
  • [MBS-10741] – Make “relationship [attribute] in use” pages consistent
  • [MBS-10755] – Add entity type restrictions for musik-sammler.de URLs
  • [MBS-10781] – Add support for Traxsource URLs

React Conversion Task

  • [MBS-10740] – Convert /relationship static pages to React
  • [MBS-10751] – Convert Remove PUID edits to React

MusicBrainz Server update, 2020-04-13

No Easter egg in today’s update but rather a dozen or so of small bugfixes and convenient improvements.

Thanks Rotab for the pair of bugfixes he submitted, to CatQuest, chaban, chirlu, FSpy, HibiscusKazeneko, JesseW, KRSCuan, MichelV, wcw1966 for issues they reported, to salorock for the Italian translation he updated, and to all others who tested the beta version!

The git tag is v-2020-04-13.

Bug

  • [MBS-7465] – Tag cloud isn’t updated
  • [MBS-9169] – Inconsistent locale identifiers
  • [MBS-9728] – Recently-added Unicode emojis can’t be used in titles
  • [MBS-9894] – Timeline shows future date
  • [MBS-10360] – Whitelist User-Agent header in CORS Preflight requests
  • [MBS-10640] – Incorrect donation status in “Donation Check” tab
  • [MBS-10688] – Attaching a CDTOC that already exists on the medium gives a cryptic error
  • [MBS-10718] – Duplicate series “part of” relationships which got grouped are harder to detect
  • [MBS-10730] – Recording is displayed twice in artist overview when credited multiple times

Task

  • [MBS-10735] – Remove (discontinued) CD Baby links from the sidebar

Improvement

  • [MBS-5641] – Show release language/script in reports ReleasesWithUnlikelyLanguageScript, NoLanguage and NoScript
  • [MBS-10679] – Link to the JSON WS from the Details tab
  • [MBS-10680] – Link to WS docs from the Details tab
  • [MBS-10724] – Make sorting options (area, date, artist) consistent
  • [MBS-10747] – Change wording/phrasing of status description for open edits
  • [MBS-10753] – Use artist sort names for artist collection ordering

MusicBrainz Server update, 2020-03-31

This time we have a couple bugfixes, a couple of small improvements, and continued changes to simplify the React table code.

Thanks to chaban, kellnerd, Kid Devine, and all others who submitted patches, reported issues, tested beta version, or contributed to translations!

The git tag is v-2020-03-31.

Bug

  • [MBS-10704] – Relationships header shown for works with no rels to be displayed
  • [MBS-10707] – Open Add DiscID edit already adds discID, cancelling does not remove it
  • [MBS-10712] – ISE when trying to load “merge release” edit

Improvement

  • [MBS-10669] – Restrict “Metal Archives” album links to release
  • [MBS-10699] – Remove “medley” from Guess Case when at the end of a title

Task

  • [MBS-10365] – Refactor display of entity list/table
  • [MBS-0401] – Replace handshaking protocol with bit mask

MusicBrainz Server update, 2020-03-16

Mostly bugfixes today, along with a new data report, plus usual small improvements, React conversion and refactoring. We also blocked ratings for unverified users, since a specific artist was getting hundreds of ratings by people creating many accounts per person just for that (people, please be nice!).

Thanks to andrybo6, chaban, cyna, dimpole, dseomn, gioele, hibiscuskazeneko, jesus2099, kellnerd, mfmeulenbelt, nikki, outsidecontext, panos, sophist, sothotalker, yindesu, and all others who submitted patches, reported issues, tested beta version, or contributed to translations!

The git tag is v-2020-03-16.

Bug

  • [MBS-7835] – [No lyrics] works showing up as “No linguistic content” in old edit summaries
  • [MBS-9359] – Web service shows “mul” languages, but all single languages are duplicates
  • [MBS-9651] – Top voters overall includes more edits than editor profile’s yes+no+approve
  • [MBS-10628] – [no label] should not be editable
  • [MBS-10633] – Can’t change capitalization on username
  • [MBS-10661] – Recording-work rel not showing pending edits on release page
  • [MBS-10683] – Map doesn’t focus on new marker when creating a new Place
  • [MBS-10684] – Message over Place’s map is misleading
  • [MBS-10685] – Place maps: Zooming by clicking doesn’t follow the cursor
  • [MBS-10693] – Artist merge submission page doesn’t show any error message if you attempt to merge a special purpose artist

Improvement

  • [MBS-4090] – Report for recordings with a single track but with different spellings
  • [MBS-10607] – Give context to translations of “unknown” / “(unknown)”
  • [MBS-10676] – Add MD5/SHA256 checksum files for JSON data dumps
  • [MBS-10695] – Show full language names on work lists

React Conversion Task

  • [MBS-10618] – Convert Edit work edit to React

Other Task

  • [MBS-10671] – Block ratings for unverified users
  • [MBS-10689] – Update MapBox API calls
  • [MBS-10692] – Change React imports to “import * as React”

MusicBrainz Server update, 2020-03-02

It’s now possible to find genres through the web service with a new /ws/2/genre lookup endpoint. Additionally, this update fixes some bugs and refactors a lot of stuff behind the curtains, including the continued conversion to React of edit display pages.

Thanks to chaban, cyna, jesus2099, kellnerd, and speardog who contributed through issue reporting, code changes, translation updates, and beta testing.

The git tag is v-2020-03-02.

Bug

  • [MBS-10515] – Weird display of recording info on release merge pages
  • [MBS-10593] – “Edit release label” edit fails to load if country name is undef
  • [MBS-10594] – “Add annotation” edits are stuck
  • [MBS-10649] – Release label edit shows [removed] if label is deselected
  • [MBS-10659] – “Edit relationship” edit not showing credits if they don’t change

New Feature

  • [MBS-10166] – Add genre lookup to the web service

Improvement

  • [MBS-9654] – Show expanded artist credits in “Merge recordings” edits
  • [MBS-10643] – Autoselect relationship type for CPDL artist URLs

React Conversion Task

  • [MBS-10058] – Convert entity merge edits to React
  • [MBS-10617] – Convert “Add work” edit to React

MusicBrainz Server update, 2020-02-18

Today’s release brings a fair amount of small bug fixes and improvements to the website. Notably, it changes the way we display relationships to group ones that are similar (such as different instruments by the same performers) and makes editing history (but not editors and edit notes!) available to visitors without needing to log in.

Thanks to chaban, chirlu, cyberskull, cyna, dimpole, hibiscuskazeneko, jeandeaual, kellnerd, lotheric, mfmeulenbelt, murdos, nikki, rdswift, salorock, supersandro2000, v6lur, and all others who submitted patches, reported issues, tested beta version, or contributed to translations!

The git tag is v-2020-02-18.

Bug

  • [MBS-2436] – “Edit medium” edits don’t link to new recordings
  • [MBS-10209] – Error parsing URL when entering Instagram link on artist’s page
  • [MBS-10223] – ArtistRoles lists don’t use relationship credits
  • [MBS-10329] – URL cleanup breaks Amazon Music URLs
  • [MBS-10576] – Medium merge error messages with links are not properly formatted
  • [MBS-10583] – Event dates displayed twice in event lists
  • [MBS-10586] – Release duplicates tab shows [missing media] when medium format is simply unknown
  • [MBS-10587] – Medium counts broken in Release Duplicates list
  • [MBS-10588] – Medium formats untranslated in Release Duplicates list

New Feature

  • [MBS-3873] – New Report: Release groups with name not matching release’s name
  • [MBS-10331] – Add new URL cleanup/autoselect/sidebar for Amazon Music URLs
  • [MBS-10620] – Add the Discography of American Historical Recordings to the otherdbs whitelist

Improvement

  • [MBS-4326] – Add “Add discid” edits in release artist(s) editing history
  • [MBS-7678] – Combine display of multiple relationships to the same entity
  • [MBS-9164] – Show edit history (but not edit notes) when not logged in
  • [MBS-10308] – Cleanup maniadb URLs
  • [MBS-10584] – Collapse work artists when there are too many
  • [MBS-10623] – Automatically add US store country-code to iTunes URLs missing it
  • [MBS-10625] – Block even more smart links

React Conversion Task

  • [MBS-9917] – Convert the instrument public pages to React
  • [MBS-10401] – Convert Edit Series edit to React

MusicBrainz Server update, 2020-02-04

This release includes a new feature: if you find a mystery MBID (such as “69755ab1-409e-3ad7-902f-3a839042799c”) and you don’t know what it might be for, you can use https://musicbrainz.org/mbid/69755ab1-409e-3ad7-902f-3a839042799c to find out without having to specifically select the entity type. Keep in mind some MBIDs for non-entities (such as packaging types) won’t work with this method!

If you’re a user of our JSON web service and use begin or end area information on artists, please be aware of MBS-10072! Starting with this release, we’re resolving a long-standing bug whereby the mentioned properties have been incorrectly outputted as begin_area and end_area, while search requests correctly outputted them as begin-area and end-area (using hyphens in line with the rest of the format). The interim fix released today is to output both sets of properties for at least 6 months, and then remove the underscored variants from the response. Please update any code you have that accesses begin_area or end_area on artists to use begin-area and end-area instead!

Today’s chunk of React conversion has been made by Cyna, so thanks to him for his dedication! Thanks also to SuperSandro2000 for updating our PostgreSQL collate extension with ICU configuration changes. And finally, thanks to chaban, jesus2099, mfmeulenbelt, roboron, and all others who reported issues or tested beta version or translated today’s release!

The git tag is v-2020-02-04.

Bug

  • [MBS-10072] – In the JSON WS, artist nodes use begin_area, end_area except search responses which use begin-area, end-area
  • [MBS-10366] – Medium edits removing pregap when disc ID is attached are not rejected
  • [MBS-10579] – Regression: The es-ES and el-GR translations are broken

New Feature

  • [MBS-6539] – Redirect musicbrainz.org/mbid/$mbid to the appropriate entity page

Improvement

  • [MBS-5460] – Add shift-click multi-(de)select to the release editor’s Recordings tab
  • [MBS-5572] – Add shift-click multi-(de)select to the edit artist ACs
  • [MBS-9477] – Disable Discid tab for releases that can’t have a discid
  • [MBS-10568] – Show number of discIDs on discID tab
  • [MBS-10569] – If no mediums on a release can have discIDs, say so on its discID tab

React Conversion Task

  • [MBS-10390] – Convert Add Place edit to React
  • [MBS-10392] – Convert Add Series edit to React
  • [MBS-10398] – Convert Edit Place edit to React

Other Task

  • [MBS-10322] – Convert fabfile (Fabric 1) to shell scripts (Bash 3)

Upgrading Postgres instead of schema change: 18 May, 2020

Hello!

We’ve long procrastinated upgrading our production Postgres installation and we’ve decided to forego a schema change upgrade and instead upgrade Postgres to version 12.x. (We will migrate to whatever the latest stable version in the 12.x series will be).

This means that on 18 May we will not make any changes to the MusicBrainz schema, but  we will have some amount of down-time and/or read-only time while we upgrade Postgres on our production servers. We haven’t sorted out all of the exact details of how we will carry out this database upgrade, but the date is now confirmed.

If you operate a replicated instance of the MusicBrainz database we STRONGLY urge you to upgrade your installation shortly after we upgrade the production servers. After this release our team may start using Postgres features not available in Postgres 9.5.x, which is our current production version.

As usual for our releases that impact our downstream users, we will post many more details closer to the date and once the migration is complete, we will post detailed instructions on how you can upgrade your own installation.

Please post any questions you may have!

Thanks!

MusicBrainz Server update, 2020-01-20

This release mostly fixes small bugs. Please note that the display code for release lists (for area, artist, collection, instrument, label and series) has been reworked too.

Thanks to chaban for continuously reporting issues, hibiscuskazeneko for paying attention to external links, rotab who fixed a couple of bugs, and all others who reported issues or helped test or translate today’s release!

The git tag is v-2020-01-20.

Bug

  • [MBS-10492] – Regression: When minute component starts with 0 (zero) it’s omitted
  • [MBS-10501] – Collaborator avatars missing from collection page
  • [MBS-10522] – Subscribers not transferred after entity merge
  • [MBS-10531] – Invalid requests are sent to maps service when access token is not set
  • [MBS-10536] – Release group link “see all versions of this release” has span.name-variation
  • [MBS-10553] – User report reason is sent to admins translated
  • [MBS-10560] – Regression: release edits display abbreviated rather than full country names in their release events
  • [MBS-10565] – Can’t add a new type for series
  • [MBS-10567] – Only show allowed series entity types when creating series types
  • [MBS-10571] – Localized ModBot notes are not properly formatted when sent via email
  • [MBS-10572] – Pages that display release events trigger an error when a non-English UI language is selected: “Domain `countries` was not found.”

Improvement

  • [MBS-10552] – Add Deezer links to the sidebar