MusicBrainz Server update, 2020-05-11

This is the last update before upgrading to Postgres 12. It is mainly focused on React conversion but also carries ten small bugfixes and improvements.

Thanks to navap for hacking the user interface. Thanks to admiy, chaban, fabe56, Freso, jesus2099, zas for having reported bugs and suggested improvements. Thanks to kellnerd, mfmeulenbelt, and salorock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2020-05-11.

Bug

Improvement

  • [MBS-10737] – Allow thesession.org URLs for Places
  • [MBS-10761] – Disallow YouTube links at wrong levels
  • [MBS-10804] – Remove redundant user header from edit page
  • [MBS-10805] – Support for Amazon.AE/NL/SG/TR ASINs

React Conversion Task

  • [MBS-9910] – Convert wikidocs transclusion admin templates to React/JSX
  • [MBS-10748] – Convert the relationship doc page to React
  • [MBS-10760] – Convert Remove Track edit to React
  • [MBS-10762] – Convert historic Remove Release/Releases edits to React
  • [MBS-10764] – Convert historic Remove Label Alias edits to React
  • [MBS-10765] – Convert historic Add, Move and Remove DiscID edits to React
  • [MBS-10773] – Convert historic Change Release Quality edit to React
  • [MBS-10775] – Convert historic Add/Remove Relationship edits to React
  • [MBS-10790] – Convert historic MAC/SAC edits to React
  • [MBS-10791] – Convert historic Change RG edit to React
  • [MBS-10792] – Convert historic Change Artist Quality edit to React
  • [MBS-10811] – Convert historic Edit Relationship edit to React

Picard 2.3.2 released

Picard 2.3.2 has been released and is available for download. This is mainly a bug-fix release with few minor improvements, see below for a full list of changes.

This release contains code changes by zas, rdswift, outsidecontext, RaysDev and new contributors Undearius and Gabrielcarvfer, who also is working on some interesting changes regarding UI responsiveness.
Thanks a lot to everybody who contributed to this release with code, translations, bug reports and general feedback.

Download

Picard 2.3.2 is available for download from the download page.

Changes

Below is a list of all changes since Picard 2.3.1. If you are upgrading from an older version you might want to read previous release post.

Breaking change:

$find script function now returns “” rather than -1 on not found, for the sake of consistency. You may want to review your scripts.

Notable improvements:

Picard was trying to guess the file format based on file content. This was very IO expensive and in many cases not needed. So we decided to change the logic to:

  • Try guessing the format by file extension first
  • If no extension or no matching file extension is available, guess by file content
  • If format was guessed by extension and loading the file with mutagen fails with an exception, retry with guessing by file content. Retry loading if the newly guessed format is different from the format guessed by extension.

Also notable, On Windows 10, support for fractional scaling was implemented.

The complete list of changes is the following:

Bug

  • [PICARD-1775] – $firstwords function doesn’t catch IndexError
  • [PICARD-1776] – $datetime crashes when invalid format specified
  • [PICARD-1781] – Have $find return “” rather than “-1” on not found
  • [PICARD-1783] – Deleting performer, comment or lyrics tag fails for some cases in ID3, Vorbis, MP4 and Apev2 tags
  • [PICARD-1784] – Host not found error when downloading PDF artwork
  • [PICARD-1785] – $delete(_id3:TXXX:foo) does not delete the ID3 frames
  • [PICARD-1786] – Deleting tag stored in ID3 TIPL frame only removes first occurrence
  • [PICARD-1787] – Deleting case-insensitive TXXX frames does not delete anything
  • [PICARD-1788] – Saving ID3 tags marked as case-insensitive causes duplicated TXXX frames
  • [PICARD-1790] – Crash when selecting plugin that can be updated
  • [PICARD-1791] – Network access is disabled error under VPN service
  • [PICARD-1795] – iTunes tags not removable (reappear after being deleted)
  • [PICARD-1801] – List index out of range when saving Vorbis file
  • [PICARD-1803] – Instrument EWI is rewritten “e w i”

Improvement

  • [PICARD-1777] – Support fractional scaling on Windows 10
  • [PICARD-1800] – Simplify Unicode ‘Bullet’ to ASCII ‘Hyphen-minus’
  • [PICARD-1809] – Optimize format detection logic

Reminder: Upgrading to PostgreSQL 12 on May 18, 2020

As we announced in February, in two weeks time (May 18, 2020) we’ll be upgrading our production database server to PostgreSQL v12 (from v9.5). At the same time, v12 will become the minimum supported version for MusicBrainz Server, so we ask that you upgrade afterwards as soon as possible! If you’re still unsure, a Q&A is below.

When do I need to upgrade my postgres by?

As soon as possible after May 18 if you’d like to keep your musicbrainz-server code up to date.

How do I perform the upgrade?

We’ll provide instructions closer to May 18. It’s recommended that you don’t upgrade until then, since we’ll be providing scripts to resolve some issues.

Will the live data feed (replication packets) stop working right away if I don’t upgrade?

No, as long as you keep your musicbrainz-server code checkout on the v-2020-05-11 tag (which will be the final release before May 18) or earlier. Future releases may work for a while too.

This is not a schema change release, so replication will continue to work smoothly until you upgrade. No tables or views will change.

However, to make the upgrade process smoother we’ll be dropping the musicbrainz-collate and musicbrainz-unaccent extensions, instead using PG’s builtin collation support for the former and replacing the latter with the unaccent extension from postgresql-contrib. A few SQL functions are being added to enable this, and some indexes need to be rebuilt. This will all happen as part of upgrade scripts we provide (or you can import from scratch). Some features of musicbrainz-server that use these old extensions may cease to work if you don’t apply them.

The extension changes above don’t actually make use of any new PG 12 features. We’ll avoid using such features for at least 1 month.

If I’m already running PostgreSQL 12, do I need to do anything?

Yes, but things will be easier for you. As mentioned in the previous answer, we’ll be dropping the musicbrainz-collate and musicbrainz-unaccent extensions to make the upgrade process smoother for pre-v12 instances. So you’ll only have to run some upgrade scripts we provide to replace those extensions and rebuild some indexes.

My host/distribution doesn’t have PostgreSQL 12 yet!

If you’re running Debian or Ubuntu, the PGDG maintains an APT repository with the latest versions. These are the same packages MetaBrainz uses in production.

Amazon RDS supports PostgreSQL 12 since March 31.

I absolutely cannot upgrade yet! What should I do?

You can stay on the v-2020-05-11 release of musicbrainz-server or earlier until then. Replication packets (i.e. the live data feed) will continue to work until the next schema change on that tag, but you’ll have upgraded to v12 by then, right?

Instead of performing a pg_upgrade and running these upgrade scripts you mentioned, can I just import fresh data dumps into a new v12 cluster?

Of course. Just make sure your musicbrainz-server git checkout is on the v-2020-05-18 tag (once that’s released) or later before performing the import. And keep in mind it may be slower than a direct upgrade.

The ODI publishes two reports on Sustainable Data Institutions

The Open Data Institute has just published two reports: Designing Sustainable Data Institutions and Designing Trustworthy Data Institutions which include insights provided by us regarding our MusicBrainz project.

When I was starting out MusicBrainz and was trying to work out how to make the project sustainable, I would’ve given just about anything to have access to these reports. I am proud that, nearly 20 years later, I was able to contribute to these reports so that others may benefit from our hard work.

I find the section Suggestions for those scoping, designing and running data institutions on page 40 of the PDF version of Designing Sustainable Data Institutions quite enlightening:

  1. Ensure your revenue model aligns with your organisational goals
  2. Understand how your revenue sources will change during your institution’s lifecycle
  3. Consider both financial and non-financial aspects of sustainability
  4. Identify and mitigate future risks
  5. Learn from others

Each of these points represent a whole collections of small lessons that I’ve learned by (often painful) experience of the past years. Also, I feel that these points are not strictly limited Data Institutions, but many also apply to making open source projects sustainable. If you’re in the business of running a data or open source organzation, I would strongly encourage you to read this paper!

Also very interesting is the second report about Designing Trustworthy Data Institutions:

For example, the representative from MusicBrainz said, “[A culture of honesty] builds trust, and this trust builds sustainability”

Compared to sustainability, the concepts of trust were much more clear to me from the beginning. However, that doesn’t make this report any less relevant — especially in current times, I welcome an emphasis on trust!

Thank you to the ODI for including MusicBrainz and doing all of the hard work on these reports!

 

MusicBrainz Server update, 2020-04-27

A large variety of issue types have been addressed in today’s release!

As a new feature, search indexes are now dumped and made available along with database dumps under the FTP directory search-indexes. They are mainly intended to be loaded on a MusicBrainz slave server to start a mirror with search.

Among improvements, a noticeable one is to lighten the area’s overview page that was heavily crowded with all sorts of relationships that have been scattered over more specific tabs.

As for bugfixes, the major one secures user/admin forms against CSRF attacks.

Thanks to atj for contributing code to support Traxsource URLs. Thanks to alex_s7, chaban, danbloo, Lotheric, murdos, Skeebadoo for reporting issues. Thanks to kellnerd, Jormangeud, mfmeulenbelt, salorock for updating the translations in German, Finnish, Dutch, Italian, respectively. And thanks to all others who tested the beta version!

The git tag is v-2020-04-27.

Bug

  • [MBS-10359] – Guess feat. artists from track titles do not give expected result
  • [MBS-10677] – Place type shown as null on WS event place rels
  • [MBS-10717] – Cookie attributes must be adjusted to work with with new behavior in browsers
  • [MBS-10719] – “remove PUID” edit doesn’t load
  • [MBS-10742] – “Show more” country miscount
  • [MBS-10756] – Inconsistent default sort order for recordings on Works page
  • [MBS-10778] – User/admin forms are prone to CSRF attacks

New Feature

  • [MBS-10546] – Dump MB Solr data along with MB DB full export

Improvement

  • [MBS-1921] – Display edit link under annotations
  • [MBS-9086] – Move most relationships away from area overview
  • [MBS-10666] – Collapse work artists when there are too many on merge pages
  • [MBS-10741] – Make “relationship [attribute] in use” pages consistent
  • [MBS-10755] – Add entity type restrictions for musik-sammler.de URLs
  • [MBS-10781] – Add support for Traxsource URLs

React Conversion Task

  • [MBS-10740] – Convert /relationship static pages to React
  • [MBS-10751] – Convert Remove PUID edits to React

MusicBrainz Server update, 2020-04-13

No Easter egg in today’s update but rather a dozen or so of small bugfixes and convenient improvements.

Thanks Rotab for the pair of bugfixes he submitted, to CatQuest, chaban, chirlu, FSpy, HibiscusKazeneko, JesseW, KRSCuan, MichelV, wcw1966 for issues they reported, to salorock for the Italian translation he updated, and to all others who tested the beta version!

The git tag is v-2020-04-13.

Bug

  • [MBS-7465] – Tag cloud isn’t updated
  • [MBS-9169] – Inconsistent locale identifiers
  • [MBS-9728] – Recently-added Unicode emojis can’t be used in titles
  • [MBS-9894] – Timeline shows future date
  • [MBS-10360] – Whitelist User-Agent header in CORS Preflight requests
  • [MBS-10640] – Incorrect donation status in “Donation Check” tab
  • [MBS-10688] – Attaching a CDTOC that already exists on the medium gives a cryptic error
  • [MBS-10718] – Duplicate series “part of” relationships which got grouped are harder to detect
  • [MBS-10730] – Recording is displayed twice in artist overview when credited multiple times

Task

  • [MBS-10735] – Remove (discontinued) CD Baby links from the sidebar

Improvement

  • [MBS-5641] – Show release language/script in reports ReleasesWithUnlikelyLanguageScript, NoLanguage and NoScript
  • [MBS-10679] – Link to the JSON WS from the Details tab
  • [MBS-10680] – Link to WS docs from the Details tab
  • [MBS-10724] – Make sorting options (area, date, artist) consistent
  • [MBS-10747] – Change wording/phrasing of status description for open edits
  • [MBS-10753] – Use artist sort names for artist collection ordering

A very important announcement

In these troubling times (New Roman) it has come to our Attention that our Esteemed program “Picard” (from now on known as “The Captain”) is written with Very Silly™ code. This will just not do!

The MetaBrainz Foundation is a very serious organisation! It will not stand for such Silly™ things as Moose, Tissues or Membranophones!

The program (from now on known as “Sir Patrick”) has in addition to pythonesque codings also several oblique references to: burlesque starships, electronic hair-colour, Monty (including “the full”), several references to technologically minded stick-figures, John Cage, and disused towels left by your mother!

For now the program’s (from now on known as St. Paddy) most recent new release (version 2.45!§🥷) vil quickly bee retracted (bzzt!) and replaced by Harder, Better, Faster, Stronger Picard (from now on known as (♫♪♬) (version 2.4√-1€DONTBLINK;)) – this to Prevent excesses such as: Rock climbing Monkies, Tobogganing Dates, Breakdancing Pastries and–

Right, that’s enough of that, we apologise for the previous part of this blog post, it is Very Silly™ and the MetaBrainz Foundation is a very serious organisation which does not abide with such silliness!

We will now return to our regular scheduled blog post! –

Now look here!

What? Stop this at once!

That’s it, I quit!

MusicBrainz Server update, 2020-03-31

This time we have a couple bugfixes, a couple of small improvements, and continued changes to simplify the React table code.

Thanks to chaban, kellnerd, Kid Devine, and all others who submitted patches, reported issues, tested beta version, or contributed to translations!

The git tag is v-2020-03-31.

Bug

  • [MBS-10704] – Relationships header shown for works with no rels to be displayed
  • [MBS-10707] – Open Add DiscID edit already adds discID, cancelling does not remove it
  • [MBS-10712] – ISE when trying to load “merge release” edit

Improvement

  • [MBS-10669] – Restrict “Metal Archives” album links to release
  • [MBS-10699] – Remove “medley” from Guess Case when at the end of a title

Task

  • [MBS-10365] – Refactor display of entity list/table
  • [MBS-0401] – Replace handshaking protocol with bit mask

ListenBrainz release 2020-03-28

We’ve just finished pushing a new release to the production server for ListenBrainz. We’ve spent quite a long time working on this because we needed to completely revamp how we were generating user statistics and that process is now finally complete and live. The other good news on user statistics it that we now have a generalized framework for creating them and that should make it much easier to create more user statistics going forward. We’ve triggered the stats engine to produce updated top artist statistics for everyone and those should update for users automatically sometime later today.

This release also includes an improved importer from last.fm, moving it to react and making it more friendly on a mobile device. This particular feature hasn’t been super well tested, so if you find a problem, please submit a bug report.

Next, if your listening history is screwed up for some reason, you can now delete all listens and start over, perhaps with a clean import from last.fm.

Finally, this release includes a pile of security updates to make the overall system more secure, but users shouldn’t notice anything different.

Thank you to iliekcomputers, Mr_Monkey, ishaanshah[m], shivam-kapila, pristine__ and everyone else who was involved in creating this update!

MusicBrainz Server update, 2020-03-16

Mostly bugfixes today, along with a new data report, plus usual small improvements, React conversion and refactoring. We also blocked ratings for unverified users, since a specific artist was getting hundreds of ratings by people creating many accounts per person just for that (people, please be nice!).

Thanks to andrybo6, chaban, cyna, dimpole, dseomn, gioele, hibiscuskazeneko, jesus2099, kellnerd, mfmeulenbelt, nikki, outsidecontext, panos, sophist, sothotalker, yindesu, and all others who submitted patches, reported issues, tested beta version, or contributed to translations!

The git tag is v-2020-03-16.

Bug

  • [MBS-7835] – [No lyrics] works showing up as “No linguistic content” in old edit summaries
  • [MBS-9359] – Web service shows “mul” languages, but all single languages are duplicates
  • [MBS-9651] – Top voters overall includes more edits than editor profile’s yes+no+approve
  • [MBS-10628] – [no label] should not be editable
  • [MBS-10633] – Can’t change capitalization on username
  • [MBS-10661] – Recording-work rel not showing pending edits on release page
  • [MBS-10683] – Map doesn’t focus on new marker when creating a new Place
  • [MBS-10684] – Message over Place’s map is misleading
  • [MBS-10685] – Place maps: Zooming by clicking doesn’t follow the cursor
  • [MBS-10693] – Artist merge submission page doesn’t show any error message if you attempt to merge a special purpose artist

Improvement

  • [MBS-4090] – Report for recordings with a single track but with different spellings
  • [MBS-10607] – Give context to translations of “unknown” / “(unknown)”
  • [MBS-10676] – Add MD5/SHA256 checksum files for JSON data dumps
  • [MBS-10695] – Show full language names on work lists

React Conversion Task

  • [MBS-10618] – Convert Edit work edit to React

Other Task

  • [MBS-10671] – Block ratings for unverified users
  • [MBS-10689] – Update MapBox API calls
  • [MBS-10692] – Change React imports to “import * as React”