MusicBrainz Server update, 2021-04-05

This time we have a lot of small bug fixes and improvements, and two changes that are likely to be more noticeable. The first one: the list of tags for a user is now paginated, and it shows the upvoted and downvoted tags separately. This is probably good news all around. The second one: long lists of relationships are also paginated. For example, the relationships page for the London Symphony Orchestra now only shows some of the orchestra relationships directly, with a link to “See all 17,303 relationships”. This change was needed, since these pages had become so big for some artists (and areas) that they were timing out since they took more than 30 seconds to load. This reduces the time for the largest pages to a few seconds – still a lot, but much more manageable. We understand that not having all the relationships shown at once might cause some issues with the workflow of some of our hardcore users. Sorry about that, but hopefully it won’t be too bad! If some use cases are very problematic, let us know and we will think about how to make it better.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to CatQuest, chaban, EndofLine, HibiscusKazeneko, jesus2099, kellnerd, Kid Devine, navap, Sophist and yindesu for having reported bugs and suggested improvements. Thanks to mfmeulenbelt and salo.rock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2021-04-05.

Fixed Bug

  • [MBS-8477] – /user/<user>/tag/<tag> pages don’t distinguish between upvotes and downvotes
  • [MBS-8807] – Artist, Label and Series subscribers sub-page wrongly shows “You have no collections!”
  • [MBS-9543] – Seeding a string of “0” results in the field being blanked out
  • [MBS-11320] – When the “I don’t know the tracklist for this medium.” checkbox is unchecked on an existing release, the “Add medium” window pops up
  • [MBS-11321] – Strange things happen when adding the tracklist to a release with exactly 1 unknown medium
  • [MBS-11373] – Relationship merging in relationship editor no longer replaces empty dates
  • [MBS-11386] – Area containment missing from indexed area search results
  • [MBS-11442] – Entering a tag from the Add Cover Art page disables tagging and cover art submission
  • [MBS-11452] – Private user subscriptions tab shown on collection and subscribers page
  • [MBS-11454] – Can’t edit release with format/date clash even if error already existed
  • [MBS-11462] – Private subscriptions/tags/ratings user tabs uselessly shown to admins
  • [MBS-11473] – ISE when loading edit/open while logged out
  • [MBS-11508] – Qobuz favicon is missing
  • [MBS-11510] – Labels for arrows to move tracks up/down are not showing
  • [MBS-11531] – Filtering reports gives ISE
  • [MBS-11538] – Collections data missing in some sidebars

Improvement

  • [MBS-1178] – Relationship view needs paging.
  • [MBS-1870] – Showing a users tags should be paginated
  • [MBS-9310] – Link to created relationship type in Add Relationship Type edits
  • [MBS-9349] – Add Pinterest links to the sidebar
  • [MBS-10691] – Allow seeing tags the user has downvoted in user/tags
  • [MBS-10726] – Several release editor buttons say “disc” but really mean “medium”
  • [MBS-11429] – Don’t block editing if existing URL fails foul of later-added blocking rule
  • [MBS-11432] – Allow also linking places and works to Operabase
  • [MBS-11455] – Make linking of user statistics more consistent
  • [MBS-11467] – Use sort name to order lists of artists for tags
  • [MBS-11476] – Don’t show own open edits in /edit/open
  • [MBS-11485] – Allow irc(s):// links on expand2react
  • [MBS-11503] – Block smart links: trac.co
  • [MBS-11506] – Point home page ‘data’ link to ‘MB Database’

New Feature

  • [MBS-9851] – Add “is not edit author” option when searching for edits by “Edit Note Author”
  • [MBS-11222] – Add a report for releases with media without a tracklist

React Conversion Task

  • [MBS-10690] – Convert user tag list page to React

Our servers are on the move again!

Our servers are on the move again!

You might remember that a few years ago we moved all our servers from the US to Europe. Sadly, that has proven too costly* and prone to fires, so we have decided our servers will continue their travel east and move to India, where hosting them will be significantly cheaper and cooling can be provided by sun-covering swarms of locusts. Most of the non-slacking members of our dev team are based in India by now anyway, so it feels like the right place!

As part of our cost-cutting process, and since air travel is not environmentally friendly anyway, we sent our servers East by boat. Unluckily, that means they had to wait their turn for quite a while in the only traffic jam you can see from space. Luckily that is starting to get resolved now, but it will still take quite a while for the servers to clear the traffic jam, reach their destination and then be located among the twenty thousand containers in the ship carrying them, especially with all other jammed ships arriving at the same time! We’re hoping to be done with it all in a couple months though, after which you can hopefully expect a return to our usual (not-particularly-)fast service.

In the meantime, our whole infrastructure is being hosted on some old servers that our friends at the Universitat Pompeu Fabra in Barcelona donated to us. They claimed they were getting too old for use in a rigorous academic environment, that as we all know in Spain requires systems to work continuously for a few hours every weekday with only a short break for a midday nap. These donated servers are now running from our executive director’s basement. Any issues you have noticed recently are solely caused by this (and have nothing to do at all with the skill and attention span of our development team).

* It could be argued that we should be able to cover the costs given we have had several years of good financial stability by now. While that is true, when we realised that the alternative is to cut costs and spend the savings on fancy ham and beers, the right choice became obvious to us.

Schema change release: May 17, 2021

We’re having a schema change release on May 17, mostly to make small changes that will make our queries more efficient, ensure better constraints, and make some hardcoded options editable without schema changes in the future. We are also upping the required versions of both Perl and Node.js to 5.30 and 16.0 respectively (see the “Minimum version requirements” section below.)

Here’s our list of tickets for the Spring 2021 schema change, with descriptions of what’s being changed:

Schema changes

  • MBS-1424: Add a “first release date” field to recordings. A very popular request for years, this allows requesting the date of the first ever release a recording appeared on. This adds materialized tables recording_first_release_date and release_first_release_date which are updated via triggers whenever the earliest date changes. The change was released as an optional extension to the main MusicBrainz server schema on Dec 16, 2020, but it will be added to the main schema during this schema change.
  • MBS-10208: Allow merging collections. Users who decide two of their collections should be joined into a larger one should be able to do so without having to move all the entities in the collection manually. This requires adding a editor_collection_gid_redirect table (equivalent to other existing x_gid_redirect tables) to ensure the old collection links redirect to the one they have been merged into.
  • MBS-10566: Convert allowed_series_entity_type and allowed_collection_entity_type to tables to allow for additions without schema changes. The constraints allowed_series_entity_type and allowed_collection_entity_type specify which types of entity can be used in series and collections, respectively. As such, if we want to add the possibility to create series of artists, we need to modify the constraint during a schema change. For ease of use, we are moving the constraints to be their own tables instead, allowing us to update them as needed in the future outside of a schema change release.
  • MBS-10647: Add [no label] to b_del_label_special trigger for labels. The b_del_label_special trigger ensures that any attempt to remove a special purpose label fails. Currently it only checks the special case “Deleted label”, but since “[no label]” is also a special purpose label that should never be deleted, we will add its ID to the trigger check.
  • MBS-10821: Remove orphaned recordings from collections for deletion. Replaces a single function, delete_orphaned_recordings(), to add a new clause that makes it so that recordings referenced only in collections (but not linked to anything else in the database) can be deleted as orphans. This was released on the main MusicBrainz servers on June 15, 2020, but it will be added to the main schema during this schema change.
  • MBS-10962 / MBS-11460Add materialized tables and indexes to fetch releases and release groups by artist or track artist. These tickets will address performance issues on our current artist pages. They do not modify any existing tables, but as mentioned, add some new tables (to be updated via triggers) and indexes.
  • MBS-11097: Support PKCE (Proof Key for Code Exchange) by OAuth clients. Adds two new columns to the editor_oauth_token table. This feature is opt-in, but allows public OAuth2 clients to mitigate auth code interception attacks. The change was released as an optional extension to the main MusicBrainz server schema on Sep 21, 2020, but it will be added to the main schema during this schema change.
  • MBS-11431: Speed up /ws/js/check_duplicates. Adds new indexes only (on the artist, label, place, and series tables, plus their respective alias tables). Improves some slow queries in the editing interface related to duplicate checking, i.e. finding other entities with the same name. Since this is a non-breaking change, it was released on the main MusicBrainz servers on Mar 15, 2021, but it will be added to the main schema during this schema change.
  • MBS-11451: Support ratings for places. Places can be reviewed in CritiqueBrainz yet cannot be rated in MusicBrainz. This is a strange state of affairs: clearly, if they are worth reviewing in depth, they also deserve the option to rate them. As such, we will add place_rating_raw and place_meta tables, in the same way we have for other ratable entities.
  • MBS-11453: Change entity0_cardinality, entity1_cardinality to SMALLINT. The cardinality columns of the link_type table are used to indicate whether the entity on each side of the relationship is expected to have only a few uses of the relationship type in question, or many (too many to comfortably display/edit). This is what stops every single recording of a work showing up in an edit work page, for example. At the moment we allow only two values for cardinality, 0/1. While it is possible that we will want to allow a few other values in between what is effectively “do not show anywhere ever” and “show all the time”, it is clear we will not need more than 32.000 values. As such, we are moving these columns from INTEGER to SMALLINT to reduce table size.
  • MBS-11456: Add MBIDs and redirect tables for artist credits. Adds a gid column to the artist_credit table, and a new artist_credit_gid_redirect table. The MBIDs will allow public identification of artist credits outside of MusicBrainz, and open the door to useful features in the future.
  • MBS-11457: Drop series ordering_attribute. This column was added back when we were expecting to have different types of ordering attributes for series, but we have never used it. We are planning to just drop the column.
  • MBS-11459: Add a script to create edit_data_type_info. edit_data_type_info is a function used for development that we added in 2020 and we will now add to mirrors as well, both for consistency and for anyone who uses their mirror for development and just wants to use it.
  • MBS-11463: Add view to easily access medium track lengths. Our current way of finding the length of a medium is to either load all its tracks, then sum the durations, or to use the duration of its disc ID when present. The first of these requires a lot more processing than just getting the durations straight from the database, while the second ignores any data tracks not on the CD TOC. This adds a medium_track_durations view that allows easy access to the duration of the track lengths for any medium.
  • MBS-11464: Drop statistics.log_statistic. We added the statistics.log_statistic table for a Google Summer of Code project back during a 2012 schema change, but the work never got finished and implemented. We are not planning to implement it anymore, so this table is useless and we will be dropping it.
  • MBS-11466: Change language.frequency and script.frequency to SMALLINT. The frequency column of a language or script indicates how often it’s used and lets us sort the most frequently used entries at the top of our lists. But we don’t store an exact count, just a number from 0-4 indicating “frequently used,” “hidden,” or “other.” Like MBS-11453 above, we don’t expect to need a full INTEGER type to store these columns at any point in the future, so can safely move them to SMALLINT.

Minimum version requirements

We’re raising the minimum required version for Node.js to v16 (which is the next LTS release coming in April 2021). Our current required version, v10, is hitting the end of its life cycle in April, and given there shouldn’t be a particular difficulty installing the current Node.js LTS on any system, it makes sense to just upgrade to the most recent LTS by the time of the schema change day.

We’re also raising the minimum required version for Perl to 5.30. The latest version is 5.32, but the current Ubuntu LTS (20.04), which is likely to be the next base image for our Docker containers, only provides 5.30. Our current required version, 5.18, was released all the way back in 2013, so moving to 5.30 is already a fairly significant improvement.

We’ll post final details about this release just prior to the release and shortly after we complete it, including instructions on how you can update your own copy. If you have any questions, please do leave a comment below or on the linked JIRA tickets!

MusicBrainz Server update, 2021-03-15

This week we have a lot of small fixes and improvements, including a patch that should fix to the elusive bug where some recording pages would sometimes fail to load with a type error. The fix for that issue involved a large change in how we move data around behind the scenes, and in turn caused a few new issues that were reported and fixed during the beta testing period (even more thanks than usual to all testers!). If you happen to find any small errors that might still have remained undetected during the beta testing period, please do let us know.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to babelfisch, CatQuest, chaban, Cyberskull, Cyna, Freso, HibiscusKazeneko, jacobbrett, jesus2099, kellnerd, Kid Devine, salo.rock, uLuGaBi and Yurim for having reported bugs and suggested improvements. Thanks to Besnik, Jesús Heriberto López Cisneros, mfmeulenbelt, salo.rock, SpearDog, stevemarsan and SustReal for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2021-03-19-hotfixes.

Fixed Bug

  • [MBS-8232] – Guess case lowercases after U+201C (left double quotation mark)
  • [MBS-8621] – Inconsistent result between track and release guess case after apostrophe
  • [MBS-9581] – Event locations (areas/places) don’t use relationship credits in lists
  • [MBS-10004] – JSON-LD MBID URIs should be HTTP
  • [MBS-10866] – TypeError was shown for a recording page
  • [MBS-10976] – Unicode Private Use Area characters are stripped during edit process
  • [MBS-11384] – Some (release) countries are untranslated
  • [MBS-11385] – Country names untranslated in indexed search results
  • [MBS-11407] – Errors that go directly through Controller->error expect .tt templates
  • [MBS-11409] – Recurring warning in server logs for recording’s page
  • [MBS-11410] – Edit could not be created while at the same time approving another one had a time-out
  • [MBS-11420] – “Paste Credits” function doesn’t apply artist credit if it matched existing name
  • [MBS-11423] – Beginner editors report doesn’t show any data
  • [MBS-11424] – Release seeding landing page joins multiple values into one
  • [MBS-11426] – Track URL links open wrong medium when medium position is not aligned with CD numbers

Improvement

  • [MBS-9836] – Guess Case: stop uppercasing “the” in artist names
  • [MBS-9866] – Add some info of what IPI and ISNI are on add forms
  • [MBS-9871] – Display all non-part-of series relationships for a series on pages for entities in the series
  • [MBS-9884] – Use “Created” as label for the begin date of character-type artist
  • [MBS-9987] – JSON-LD: Use “Concept URI” for Wikidata IRIs in the sameAs relation
  • [MBS-10012] – Treat HTTP & HTTPS versions of a link as identical in the entity editor
  • [MBS-10076] – Give a friendlier error when trying to delete an attribute that has children
  • [MBS-10416] – Prevent entering control character in annotation
  • [MBS-10487] – Allow adding instruments and areas from inline search for users with the right privileges
  • [MBS-11398] – Add Qobuz links to sidebar
  • [MBS-11402] – Block smart links: many.link
  • [MBS-11405] – Don’t show ended URL rels in “deprecated relationships” reports
  • [MBS-11408] – Clarify Edit Note Author edit search options
  • [MBS-11419] – Show YouTube Music links on the sidebar
  • [MBS-11422] – Block smart links: gate.fm
  • [MBS-11427] – Add new fields to release editor seeding testing page (seed-love-bug)
  • [MBS-11431] – Speed up /ws/js/check_duplicates
  • [MBS-11433] – Add Mp3tag to the Products header menu and main page tagger list

React Conversion Task

  • [MBS-11403] – Convert attribute in use page to React/JSX

MusicBrainz Server update, 2021-03-01

Welcome to another MusicBrainz release! This time we have a fair amount of small bugs and improvements both newly requested and recovered from the ancient times of… well, a few years ago, because we spent quite a bit of time looking at older tickets.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to CatQuest, chaban, Freso, HibiscusKazeneko, kellnerd,  Lucas Werkmeister, pmepepnoute, Psychoadept, salorock, Sotho Tal Ker and Trevor for having reported bugs and suggested improvements. Thanks to mfmeulenbelt, Remko de Keijzer, salorock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2021-03-01.

Fixed Bug

  • [MBS-10117] – Password length error message does not indicate correct password length
  • [MBS-10744] – Gender of editors is not translatable
  • [MBS-11378] – Artist gender in artist overview is untranslated
  • [MBS-11387] – “instrument:” attribute name is untranslated when editing relationships
  • [MBS-11390] – Editor profile added entities: Incorrect count when there are pre-NGS release adds
  • [MBS-11395] – muziekweb.nl links not shown on sidebar
  • [MBS-11396] – CollectionType/SeriesType admin page shows entity_type rather than type of allowed entities
  • [MBS-11397] – Place-Series relationship does not show on series page

Improvement

  • [MBS-9095] – Add Recochoku and mora links to the sidebar
  • [MBS-9706] – Normalize OCRemix URLs (including HTTPS)
  • [MBS-10195] – Block Google Amp links & prompt user to get canonical link instead
  • [MBS-10227] – Display entity count on collection page
  • [MBS-10343] – Display a placeholder for darkened images
  • [MBS-10452] – Implement tagger links for releases on recording pages
  • [MBS-10470] – Rename “IRC” link in footer to “Chat (IRC)”
  • [MBS-10557] – Normalize Recochoku URLs to use HTTPS
  • [MBS-11287] – Add Apple Touch icon to the MusicBrainz website
  • [MBS-11389] – Top-align edit medium Tracklist tables
  • [MBS-11394] – Don’t try to load cover art for darkened releases

MusicBrainz Server update, 2021-02-15

This is a fairly small release that brings a few bug fixes and some small improvements, while the team works on some background tasks.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to david-russo and jesus2099 for submitting patches. Thanks to hibiscuskazaneko, kellnerd and Trevor for having reported bugs and suggested improvements. Thanks to mfmeulenbelt and salorock for updating translations. And thanks to all others who tested the beta version!

The git tag is v-2021-02-15.

Fixed Bug

  • [MBS-9573] – Search for edits loses track of type criteria
  • [MBS-11354] – Recording edit view shows up/down arrows for recordings of a single work
  • [MBS-11366] – Harmful links not greyed out once they have been removed
  • [MBS-11368] – Regression: track title span.name-variation disappeared from Edit Medium edit

Improvement

  • [MBS-11329] – Return privileges as a number in internal editor JSON
  • [MBS-11352] – Add Starzik to the “malware domains” blacklist and isGreyedOut
  • [MBS-11359] – Add GeoNames to the sidebar
  • [MBS-11365] – Allow Resident Advisor /podcast URLs for releases
  • [MBS-11369] – Don’t use top vertical align data in tables for details
  • [MBS-11370] – Historic edit track length: display track lengths of 0 ms or -1 ms as unknown

Leaked email address incident: 2020-11-23

We’re saddened to write that we’ve let some of our users down by accidentally leaking their email addresses and birth dates via a bug in the web pages of musicbrainz.org. This caused some users to receive unwanted spam emails.

However, we would like to emphasize that no passwords, passwords hashes or any other bits of private user information other than email addresses and birth dates were leaked.

If you have never added or edited an annotation on MusicBrainz, then your email address and birth date were never leaked and you can ignore this — your data has not leaked.

What happened

About two weeks ago a MusicBrainz editor contacted us to say that their email address that was in use only at MusicBrainz had received spam. The user changed the email address to a very distinct email address in order to rule out a spammer guessing the updated email address. But it happened again, and the user received email to the unguessable email address. 

At this point we began an audit of the MusicBrainz server codebase in an attempt to find out where the leak was, patch it as soon as possible, and discover who was affected by it.

What we found

On 2019-04-26 we released a new version of the MusicBrainz server and in this version we added email addresses to the list of editor data we pass to our server to build MusicBrainz pages. The goal of this was to display them in admin-facing pages to, ironically, be able to fight spammers who were using MusicBrainz as a spamming tool. We also added the editor’s birth date, to be able to congratulate them on their birthday. Neither of these cases should have ever been a problem, since the private data should only be used on pages built and sent from our own server (where the data cannot be seen by anyone else), and any editor info sent to the users’ browser goes through a “sanitizing” process eliminating all this private information.

After some digging, we discovered that due to a bug we had overlooked in the code that stripped this data, the addresses and dates had started being sent to the browser whenever an entity page with an annotation was requested. The email address and birth date of the last person to have edited an annotation in MusicBrainz (any annotations, attached to any of our entities) was leaked on the page for the entities in question. This data was contained in a massive block of JSON data in the page source and was never shown on the web page for humans to see, which is why this issue went undetected for so long.

Who was affected

We looked at all editors who wrote any annotations that were displayed between the date the problematic code was released and the date the bug was fixed. This can mean either the annotation was written during this time period, or it was written before that but (being the latest version of the annotation for the entity) it was still displayed during this time period. This gave us a total of 17,644 editors whose data was at some point visible from the JSON block in at least one entity’s source code. We sadly do not have a way to know for sure how many of the affected were actually ever found and stored by spammers, since we attempt to block botnets as much as possible. As such, we simply have no way of knowing who was really affected by this leak — only who might have been.

What we’ve done

Once we detected the issue on November 22, we immediately put out a hotfix to all production (and beta) servers plugging the leak. The hotfix acted to sanitize the editor data by removing email addresses and birth dates from the JSON. We also deployed two additional changes that should help prevent similar issues from occurring, by avoiding sending sensitive editor data to our template renderer altogether. See all changes from the git tag v-2020-11-22-hotfix.

We are planning to improve our testing infrastructure to detect exposure of editor data — this will become a routine part of our continuous integration process. We are also going to ensure that any pull request dealing with editor data goes through a strict testing checklist.

How did spammers get these email addresses?

You might be wondering how such an obscure leak in a web page can end up in spammers finding and using your email — you’re not alone. 

Our sites are under near constant traffic from seemingly random internet bots fetching thousands of our pages in a day, with no apparent goal. All of our metadata is available for download, so why would someone download pages from us at random?

Well, we now know — web pages can contain a whole host of random data that shouldn’t be there. Email addresses, birth dates and such are just the starting point — there have been websites that have leaked credit card numbers and even login passwords, possibly compromising the integrity of user accounts.

In this case it appears that a botnet kept downloading pages from musicbrainz.org and driving the load on our servers up. We’ve been trying to block botnets ever since they’ve come into existence, but this is a laborious task that is never complete.

It appears that spammers used the botnet to scour the internet for private data such as emails to then send out lovely spam emails to all compromised users.

Summary

We would like to wholeheartedly apologize for this data leak. We take data privacy seriously and we aim to have high standards about privacy and data security. We find ourselves frustrated by the endless data leaks that happen on the Internet on a seemingly continuous basis and work hard to avoid committing these mistakes in our domain. However, we’re also human and we do make mistakes periodically. As explained above, we’re working to improve our systems and processes in order to prevent this from happening again.

We hope that you accept our most sincere apologies for this leak.

Robert Kaye, Michael Wiencek, Nicolás Tamargo and Yvan Rivierre

MusicBrainz Server update, 2019-09-12

It’s time for another server update! This release mostly includes small improvements to make the MusicBrainz site show data in places where it was missing and have more clear messages for the users. We have a lot of other small improvements in the pipeline which we hope to release in the next couple of updates, so if this doesn’t help with any of your pet peeves hopefully those will!

Thanks to CatQuest, chaban, danbloo, demosdemon, eey0re, ianmcorvidae, ijabz, jesus2099, Lotheric, murdos, PeterCodar, $nake, SothoTalker for having reported issues, and to every single one of you who tested the beta version and updated website localizations.

The git tag is v-2019-09-12.

Bug

  • [MBS-4478] – Misleading messages when adding new entities through an edit
  • [MBS-10273] – Huge and weird spacing in front of the release year column on artist pages in beta
  • [MBS-10320] – Don’t wrongly nag local users of MB slave server
  • [MBS-10337] – ISRCs and ratings not shown when artist overview consists of recordings only

Improvement

  • [MBS-975] – Permanently (301) redirect from track/ to recording/
  • [MBS-4161] – List blog relationship type under the External links section
  • [MBS-4787] – Permanently (301) redirect http://musicbrainz.org/ns/mmd-2.0# to web service documentation
  • [MBS-5049] – Show edit note syntax help in edit page too
  • [MBS-10269] – Release editor: Open artist credits preview in a new window
  • [MBS-10280] – When deleting user, cancel open edits from newest to oldest
  • [MBS-10291] – Consider “、” as a delimiter when splitting featured artists
  • [MBS-10324] – Lowercase “Takes” with guess case
  • [MBS-10336] – Clarify disc.track on recording pages
  • [MBS-10338] – Show ratings on artists’ recording page

MusicBrainz schema change release, 2019-05-13 (with upgrade instructions)

We’re happy to announce the release of our May 2019 schema change today! Thanks to all who were patient during today’s downtime as we released everything to our production servers.

This is a fairly minor release as far as schema changes go, but please do report any issues that you come across, especially any related to genres and collections.

Visible changes with this release are limited to an indication if a specific artist credit is being edited (MBS-5387). Work on some of the changes to collections and genres is quite advanced, and we’re hoping to release some of the new features onto beta already in a week or so from now, while others might take a while longer.

Now, on to the instructions.

Schema Change Upgrade Instructions

Note: Importing the latest data dump is always a valid alternative to running ./upgrade.sh on an existing database, if you’d prefer to also get new data in one go. Just follow the relevant instructions in INSTALL.md. The git tag is v-2019-05-13-schema-change. The rest of the instructions here assume an in-place upgrade.

  1. Make sure DB_SCHEMA_SEQUENCE is set to 24 in lib/DBDefs.pm.
  2. If you’re using the live data feed (your REPLICATION_TYPE is set to RT_SLAVE), ensure you’ve replicated up to the most recent replication packet available with the old schema. If you’re not sure, run ./admin/replication/LoadReplicationChanges and see what it tells you; if you’re ready to upgrade, it should say “This replication packet matches schema sequence #25, but the database is currently at #24.”
  3. Take down the web server running MusicBrainz, if you’re running a web server.
  4. Turn off cron jobs if you’re automatically updating the database via cron jobs.
  5. Switch to the new code with git fetch origin followed by git checkout v-2019-05-13-schema-change.
  6. Install newer dependencies Yarn and NodeJS 8 or later according to install prerequisites.
  7. Run cpanm --installdeps --notest . (note the dot at the end) to ensure your perl-based dependencies are up to date.
  8. Run ./upgrade.sh (it may take a while to vacuum at the end).
  9. Set DB_SCHEMA_SEQUENCE to 25 in lib/DBDefs.pm as instructed by the output of ./upgrade.sh.
  10. Turn cron jobs back on, if applicable.
  11. Restart the MusicBrainz web server, if applicable. It’s also recommended you restart redis. If you’re accessing your MusicBrainz server in a web browser, run ./script/compile_resources.sh.

Here’s the list of resolved tickets:

Bug

  • [MBS-5387] – ACs being edited aren’t marked as having pending edits on the aliases tab
  • [MBS-9365] – event_meta_fk_id was never created as part of any upgrade script
  • [MBS-9462] – Standalone databases created before schema 21 are missing some l_event_url triggers
  • [MBS-10146] – Regression: ISE on Remove DiscID page
  • [MBS-10149] – Swap track titles with artist credits fails to update both fields properly
  • [MBS-10150] – Regression: The link to the release group reviews in the release page is broken

Improvement

  • [MBS-9664] – Add database constraints to disallow loop relationship
  • [MBS-10044] – Add place area to place lists

Database Schema Change Task

  • [MBS-10052] – Add new schema for the event art archive
  • [MBS-10173] – Create a genre table in the DB and populate it with existing genres
  • [MBS-10174] – Create an addition timestamp in the DB for new collection items
  • [MBS-10175] – Create a position integer in the DB for collection items
  • [MBS-10176] – Create a comment text field in the DB for collection items
  • [MBS-10177] – Create an editor_collection_collaborator table for collaborative collections
  • [MBS-10178] – Create a genre_alias table
  • [MBS-10181] – Create filesize for cover art and each thumb in the DB

React Conversion Task

  • [MBS-9925] – Convert collection pages to React
  • [MBS-10179] – Convert all entity list components to React

Classical Clean Up #4: Hyperion

Hi all!

We almost got Dvořák fully cleaned, with only a page and a half of hard-to-fix recordings from compilations left. Which honestly is a great result (especially since most people won’t care that much about those compilations).

That said, I thought we could do something different this time, and hopefully avoid albums with no info available at all 🙂 The best way of doing that is to focus on a label instead of a composer: ideally, a label that offers (almost) all of their booklets for free so that everything can be checked without needing to have a copy of the release. One of the best examples of this is hyperion (official site), a British label that puts out all sorts of interesting stuff from medieval music to contemporary classical. So this February we’ll clean up hyperion (Hyperion? the logo is lowercase, anyway!) releases 🙂

Tools

  • Our user loujin has made a nice dashboard that shows our current hyperion (and related label) releases, and the ones from their complete catalog list that we seem to be missing. It matches by barcode, so if we’re missing the barcode, the release will still appear on the “we’re missing this” list – make sure we really are missing it before adding a new one! 🙂
  • A hyperion website importer has been written by loujin specifically for this cleanup.
  • My own Classical Editor’s Toolbox, especially if you’re a relatively new editor. You’ll definitely want to install most of the userscripts mentioned there.
  • The label website, of course.
  • Discogs pages for hyperion and helios. Usually, the label page will be better than these, but some old releases (especially vinyls) might be on Discogs and not in the official catalog.

How to use the Hyperion website

The website has a lot of info! Here’s an introduction, I’m sure I’m missing some stuff.

  • Choose the right label! In general, you can look at the catalog number: CDA = hyperion, CDH = helios (other sublabels and distributed labels are more obviously different).
  • For full booklets, click “Digital booklet (PDF)” under the cover art. It might not be always there but I can remember almost no cases where it wasn’t 🙂 All the booklets include a request not to upload them elsewhere, so let’s respect that: please do not upload the full booklets to the Cover Art Archive. Keep in mind when something has been re-released on Helios, the Helios booklet will also be linked from the old Hyperion version. It’s generally safe to follow this booklet, but of course if you know something was printed differently on the old tracklist you should keep it like it actually was 🙂
  • For a big cover image, click on the cover and then right-click + open image in new tab. These are ok to add to the Cover Art Archive: please do upload them! 🙂
  • For the release date (up to the month) see the box on the top right side of the page.
  • Barcodes are often not available on the release page itself for some reason, but you can get them from loujin’s list, from the full catalogue itself or by searching Amazon for the catalog number and looking at the back cover.
  • Hyperion often re-releases stuff on the budget sublabel Helios, or as part of collections. If you see “Superseded by CDH12345” (or whatever the catalog number) to the right of the cover under the title area, you’re in luck! You can fix two releases instead of one with just a bit more effort 😉 (if one of the two is missing, just fix the existing one, then create the missing one based on the now-fixed version). Remember, CDA = hyperion, CDH = helios.

Other hints

  • Remember you have the full liner notes. This is very helpful when trying to identify works! If in doubt, check what the liner notes say. If that still doesn’t help (say, you have one of Dowland’s “A Fancy” and no idea which one that is) just leave it unlinked, don’t guess the work.
  • The website will sometimes be more specific than the booklet about which performers perform on which works or work parts. If the booklet is not too clear, see if specific performers are printed under the track title on the website’s tracklist.
  • Recording dates are usually more exact on the booklet than the right-side box. Even if you see “Recording details” there, check the booklet first 🙂 Old booklets might have only recording dates but no locations – recent ones seem to include both pieces of info almost always.
  • When choosing release artists, I’d suggest following the cover, not the website (if the website says Johannes Brahms and the cover Brahms, use just “Brahms”).
  • The hyperion website entries can be linked either with “purchase for X” or “discography page” relationships. I’d suggest at least “discography page” (with the purchase ones on top if desired), but just linking it is already good – that’s the quickest route to booklets, after all! 🙂

What to work on

  • Take either loujin’s dashboard or the actual label pages in MusicBrainz, look at releases and see what seems to need work. An easy start is releases that still have the performers on the title rather than the artist field 🙂 You can also look at the data quality column: anything with “unset”, “low” or “normal” should be missing stuff (if not, go and change data quality to high!).
  • Add missing releases: in loujin’s dashboard you can see releases that haven’t matched to MB by barcode or catalog number. Make a quick check in case they’re in MusicBrainz but missing the info, but most of them are simply missing and need to be added!
  • If you’ve added all the info from the booklet (including engineers, copyright info and whatnot) and added the covers please set the release data quality to high from the sidebar. That way, other people can see that and not check the data again 🙂 If something is terribly entered and you don’t have time to fix it, feel free to set it to low quality to point the mess to others!

As always, if you have any doubts or questions or you just want to ask the community to help with something, you can post under this 🙂

PS: Thanks to Chhavi Shrivastava for the banner!