MusicBrainz Server update, 2021-01-11

To welcome this new year, we present a new release of MusicBrainz Server, which is mostly about fixing bugs and making small usability improvements, while continuing the React conversion. Beyond the website, the two most noticeable changes to API users are a bug fix in our OAuth implementation and the addition of recordings’ first release date to release lookup results.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to atj, chaban, cyberskull, darwinx0r, dragonzeron, drsaunde, fabe56, hibiscuskazaneko, jesus2099, loujin, nikki, salorock for having reported bugs and suggested improvements. Thanks to francescoSardo, mfmeulenbelt, salorock, and Skyjaython for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2021-01-11.

Fixed Bug

  • [MBS-7752] – Clicking on Subscribe/Unsubscribe sends me to the MusicBrainz homepage
  • [MBS-10913] – Since-removed standalone recordings show as being created on “add” edit
  • [MBS-10954] – DiscID tab disabled on “remove disc ID” page
  • [MBS-11181] – Unable to add ended relationship when another relationship to the same target already exists
  • [MBS-11183] – Artist name not preserved as alias after merge
  • [MBS-11233] – Seeding capability was broken during lodash removal
  • [MBS-11237] – Wrong rel info loaded when seeding relationships with target MBID
  • [MBS-11240] – Wrong country detected for Amazon links
  • [MBS-11250] – JSON rating lookup returns the serialized entity instead of the rating
  • [MBS-11253] – JSON tag lookup returns the serialized entity instead of the tag
  • [MBS-11262] – Release sidebar status is untranslated
  • [MBS-11263] – Some valid beatport URLs are not allowed
  • [MBS-11264] – Guess Case > French mode > Chain of c combined with ‘ incorrectly convert it in uppercase
  • [MBS-11265] – Artist not always shown for reorder medium edits
  • [MBS-11275] – Recordings shown as deleted when moving disc ID
  • [MBS-11276] – Same barcode warning points to release being edited
  • [MBS-11278] – Ratings not listed despite rating average and count being present
  • [MBS-11281] – Relationship merge code ignores invalid date periods
  • [MBS-11285] – User rating style doesn’t stick in collections
  • [MBS-11291] – Wrong bootleg check in “ReleasesSameBarcode” report
  • [MBS-11313] – OAuth PKCE S256 verification implementation is not RFC compliant

Improvement

  • [MBS-6048] – Allow submitting edit note when adding ISRCs through the WS
  • [MBS-8141] – Link the disc ID on the remove disc ID page
  • [MBS-8169] – Add a link to the timeline for individual statistics
  • [MBS-11126] – Historic edits: display track lengths of 0 ms or -1 ms as unknown
  • [MBS-11221] – Normalize whosampled.com links to HTTPS
  • [MBS-11234] – Add validation for whosampled links
  • [MBS-11246] – Support for Amazon.SE ASINs
  • [MBS-11248] – Block smart links: li.sten.to
  • [MBS-11274] – Always show the timeline line if accessing via a direct stat link
  • [MBS-11304] – Add recordings’ first release date to API “release” lookup’s output
  • [MBS-11306] – Update CDJapan URL cleanup to use HTTPS + add them to the sidebar
  • [MBS-11309] – Block ToneDen smart links

React Conversion Task

  • [MBS-11228] – Convert Change Release Quality edit to React
  • [MBS-11229] – Convert cover art edits to React
  • [MBS-11235] – Convert Edit Instrument edit to React
  • [MBS-11239] – Convert historic Edit Release Events edit to React
  • [MBS-11243] – Convert first set of cover art forms and pages to React

Other Task

  • [MBS-11247] – Stop highlighting (discontinued) BBC Music relationships

Picard 2.5.6 released

Picard 2.5.6 is a maintenance release. This fixes issues with the context menu of the metadata view and long standing problems with the app signature on macOS Sierra and High Sierra.

The latest release is available for download on the Picard download page.

Bugfixes

  • [PICARD-1943] – App does not start on macOS 10.12 / 10.13, Gatekeeper reports it as damaged
  • [PICARD-2074] – Crash when trying to add new tags
  • [PICARD-2083] – Snap version: path to fpcalc gets invalid after update
  • [PICARD-2087] – Adding new tags crashes Picard with Qt < 5.10

MusicBrainz servers update, December 2020

Let’s end the year 2020 on a brighter note with new releases of both MusicBrainz Server and its companions for advanced indexed search.

The most urgent task was to lower the risk of leaking private data again (see previous incident) by reducing the scope of editor data passed to the renderer.

The most visible improvement of the website is the addition of instrument illustrations by IROM. Thanks to him for these very welcomed drawings. See sistrum for an example; we will keep adding illustrations to other instruments little by little.

The most useful improvements to the API are the addition of their first release date to recordings (both in lookup and search) and release groups (which were missing it in search results), and the addition of MBIDs for artists’ gender and releases’ status.

Advanced indexed search has also been improved to allow searching for releases by type of packaging, for recordings and release groups by first release date, and to simplify searching for releases by type of medium’s format.

Finally, a fair number of smaller bugs have also been fixed, and React conversion is being continued.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to amCap1712 for fixing a pair of bugs in outputting search API results, and jesus2099 for fixing a bug in seeding the release editor. Thanks to avilla, CatQuest, chaban, jgrmstr, jstranger, Lotheric, loujin, nikki, Psychoadept, and serg for having reported bugs and suggested improvements. Thanks to Besnik, mfmeulenbelt, and salorock for updating the translations. And thanks to all others who tested the beta version!

The git tags are v-2020-11-12 for MusicBrainz XML Metadata Schema, v-2020-12-14 for MusicBrainz Server, v-2020-12-24 for MusicBrainz Simple Solr Search Server Schema, v2.0.0 for Search Index Rebuilder, and v3.4.1 for MusicBrainz Solr Query/Response Writer.

Bug

  • [MBS-4555] – Seeding the release editor when logged out loses the seeded information
  • [MBS-8438] – The same event is displayed twice on an artist’s “Events” tab if they have multiple roles on it
  • [MBS-10664] – Regression: HTML tags are incorrectly displayed in editor’s bio
  • [MBS-11092] – Some issue with userscripts/plugins and beta site
  • [MBS-11169] – Artist credit doesn’t display properly in release pages when credits are at the bottom of the release instead of inline
  • [MBS-11207] – Error messages from DBI are badly encoded
  • [MBS-11212] – Incorrect quality attribute in “ws/2/release”
  • [MBS-11214] – Relationships with different link order not split on “see at bottom” medium rels display
  • [MBS-11220] – Release editor seeding is displayed in wrong language
  • [MBS-11223] – Releases with “unknown tracklist” on a medium failed to load in the “Edit Relationships” tab
  • [MBS-11227] – Unexpected URL used when removing release groups from merge queue
  • [SEARCH-623] – Gender ID is missing from JSON/XML artist search results
  • [SEARCH-624] – Status ID is missing from release in JSON/XML search results

Improvement

  • [MBS-1424] – Add a “First release date” field to recordings
  • [MBS-11188] – Block smart links: “album.link”
  • [MBS-11224] – Releases with no medium should show a clear message in the relationship editor
  • [MBS-11225] – Make “staticbrainz.org/irombook/” URLs autoselect for image relationship
  • [MBS-11271] – Reduce the scope of editor data passed to the template renderer
  • [SEARCH-319] – Add first release date to the results of indexed search for release groups
  • [SEARCH-574] – Make release’s format search field insensitive to spaces and separators

New Feature

  • [MBS-10590] – Allow displaying IROMBOOK instrument images on MusicBrainz
  • [MBS-11216] – Split report “InstrumentsWithoutWikidata” from “InstrumentsWithoutAnImage”
  • [SEARCH-218] – Add first release date to the fields and results of indexed search for recordings
  • [SEARCH-384] – Add first release date to the fields of indexed search for release-groups
  • [SEARCH-590] – Add packaging to the fields of indexed search for releases

React Conversion Task

  • [MBS-11018] – Convert Add Relationship Type edit to React
  • [MBS-11211] – Convert the login page to React

Other Task

  • [MBS-10634] – Rework the “Instruments without an image” report to only look at IROMBOOK images
  • [MBS-11210] – Allow more subpaths for “saisaibatake.ame-zaiku.com”

Playlists and personalised recommendations in ListenBrainz

Just in time for Christmas we are pleased to announce a new feature in our most recent release of ListenBrainz, the ability to create and share your own playlists! We created two playlists for each user who used ListenBrainz containing music that you listened to in 2020. Check out your lists at https://listenbrainz.org/my/recommendations. Read on for more details…

With our continuing work on using data in ListenBrainz to generate recommendations, we realised that we needed a place to store lists of music. That sounded like playlists to us, so we added them to ListenBrainz. As always, we did this work in the public ListenBrainz repository. You can now create your own playlists with the web interface or by using the API. Recordings in playlists map to MusicBrainz identifiers. If you’re trying to add something and can’t find it, make sure that it’s in MusicBrainz first.

Once you have a playlist, you can listen to it using our built-in BrainzPlayer, or export it to Spotify if you have an account there. If you have already linked your Spotify account to ListenBrainz you may have to re-authenticate and give us permission to create playlists on your behalf. Playlists can also be exported in the open JSPF format using the ListenBrainz API.

Over the last year we’ve started thinking about how to use data in MetaBrainz projects to generate recommendations of new music for people to listen to. For this reason, we started the Troi recommendation framework. This python package allows developers to build pipelines that take data from different sources and combine it in order to generate recommendations of music to listen to. We have already developed data sources using MusicBrainz, ListenBrainz, and AcousticBrainz. If you are a developer interested in working on recommendations in the context of ListenBrainz we encourage you to check it out.

Now that we can store playlists we needed some content to fill them with. Luckily we have some great projects worked on by students over the last few years as part of MetaBrainz’ participation in the Google Summer of Code project, including this year’s work on statistics and summary information by Ishaan. Using Troi and ListenBrainz statistics, we got to work. Every user who has been contributing data to ListenBrainz recently now has two brand new 2020 playlists based on the top recordings that you listened to in 2020 and the recordings that you first listened to in 2020. If you’re interested in the code behind these playlists, you can see the code for each (top tracks, first tracks) in the troi repository.

If you’re a long-time user of ListenBrainz you may be familiar with the problem of matching your listens to content in MusicBrainz to be able to do things with it. We’ve been working hard on a solution to this problem and have built a new tool using typesense to provide a quick and easy way to search for items in the MusicBrainz database. You are using this tool when you create a playlists using the web interface and search for a recording to add. This is still a tech preview, but in our experience it works really well. Thanks to the team at typesense for helping us with our questions over the last few weeks!

This work is still in its early days. We thought that this was such a great feature that we wanted to get it out in front of you now. We’re happy to take your feedback, or hear if you are having any problems. Open a ticket on our bug tracker, come and talk to us on IRC, or @ us. Did we give you a bad jam? Sorry about that! We’d love to have a conversation about what went well and what didn’t in order to improve our systems. In 2021 we will start generating weekly and daily playlists for users based on your recent listens using our collaborative filtering recommendations system.

Merry Christmas from the whole MetaBrainz team!

Picard 2.5.4 hotfix for Windows startup issues

We had many reports of Windows users not being able to launch the just released Picard 2.5.3. This is a hotfix release to address this issue. There are no changes for other platforms.

The updated version is available from the Picard download page.

Thanks a lot to everyone reporting on this issue and helping to get this resolved quickly and sorry for the trouble.

Changes

Picard 2.5.3 released

The Picard team is happy to announce the release of Picard 2.5.3. This release fixes a performance regression introduced in Picard 2.5.2 and brings many more bug fixes and improvements to existing functionality.

The latest release is available for download on the Picard download page.

What’s new?

Bugfixes

  • [PICARD-2016] – AcoustID API Key is not stripped
  • [PICARD-2017] – Picard crashes when removing entries on the right side while loading
  • [PICARD-2019] – Saving tracks to SMB share on Windows 10 results in ever more nested folders
  • [PICARD-2020] – Multi-value album or recording ID tags prevent Picard from loading the proper albums
  • [PICARD-2021] – SameFileError when moving files between network path and local path on Windows
  • [PICARD-2022] – Crash accessing network share without access rights on Windows
  • [PICARD-2023] – Appdata file not generated on non-Linux platforms
  • [PICARD-2028] – Deleting albums and saving files is extremely slow
  • [PICARD-2031] – Scripting documentation link 404
  • [PICARD-2036] – MultiMetadataProxy::pop is not flagged as a WRITE_METHOD; this breaks the “keep” plugin
  • [PICARD-2037] – Improve Info/Error tab readability
  • [PICARD-2045] – After fingerprint, unsaved tracks have green tick
  • [PICARD-2050] – File selector pane jumps around horizontally instead of expanding / collapsing the folder
  • [PICARD-2056] – Interface color changes are not saved
  • [PICARD-2058] – Add File dialog does not show files with uppercase extension on case-sensitive file systems
  • [PICARD-2059] – Scripting Documentation shows extra line for each function
  • [PICARD-2062] – Searching for similar tracks can remove current album even if there are unmatched tracks
  • [PICARD-2064] – Cluster shows empty album column

Improvements

  • [PICARD-2034] – Add context menu entry for copy and paste to metadata view
  • [PICARD-2035] – More verbose tooltip for album error icon
  • [PICARD-2038] – Integrate metadata box clipboard with system clipboard
  • [PICARD-2039] – Unify error handling for albums, non-album tracks and files, show errors in info dialog
  • [PICARD-2044] – Add date and originaldate fields to the choice of columns in the list views
  • [PICARD-2046] – AcoustID submission can fail due to body size limit of AcoustID server
  • [PICARD-2047] – Improve contrast for console text in dark mode
  • [PICARD-2057] – Allow showing all files in Add Files dialog
  • [PICARD-2063] – Add an option to disable automatic horizontal scrolling in file browser

The complete list of changes of this and previous releases is available in the changelog. You can also discuss new features or usage on our forums.

Acknowledgements

This release contains code contributions by Sophist, mineo, BSDKaffee, zas, and outsidecontext. Many thanks also to all translators and everybody who suggested features and reported bugs in our community forums or on the issue tracker.

Leaked email address incident: 2020-11-23

We’re saddened to write that we’ve let some of our users down by accidentally leaking their email addresses and birth dates via a bug in the web pages of musicbrainz.org. This caused some users to receive unwanted spam emails.

However, we would like to emphasize that no passwords, passwords hashes or any other bits of private user information other than email addresses and birth dates were leaked.

If you have never added or edited an annotation on MusicBrainz, then your email address and birth date were never leaked and you can ignore this — your data has not leaked.

What happened

About two weeks ago a MusicBrainz editor contacted us to say that their email address that was in use only at MusicBrainz had received spam. The user changed the email address to a very distinct email address in order to rule out a spammer guessing the updated email address. But it happened again, and the user received email to the unguessable email address. 

At this point we began an audit of the MusicBrainz server codebase in an attempt to find out where the leak was, patch it as soon as possible, and discover who was affected by it.

What we found

On 2019-04-26 we released a new version of the MusicBrainz server and in this version we added email addresses to the list of editor data we pass to our server to build MusicBrainz pages. The goal of this was to display them in admin-facing pages to, ironically, be able to fight spammers who were using MusicBrainz as a spamming tool. We also added the editor’s birth date, to be able to congratulate them on their birthday. Neither of these cases should have ever been a problem, since the private data should only be used on pages built and sent from our own server (where the data cannot be seen by anyone else), and any editor info sent to the users’ browser goes through a “sanitizing” process eliminating all this private information.

After some digging, we discovered that due to a bug we had overlooked in the code that stripped this data, the addresses and dates had started being sent to the browser whenever an entity page with an annotation was requested. The email address and birth date of the last person to have edited an annotation in MusicBrainz (any annotations, attached to any of our entities) was leaked on the page for the entities in question. This data was contained in a massive block of JSON data in the page source and was never shown on the web page for humans to see, which is why this issue went undetected for so long.

Who was affected

We looked at all editors who wrote any annotations that were displayed between the date the problematic code was released and the date the bug was fixed. This can mean either the annotation was written during this time period, or it was written before that but (being the latest version of the annotation for the entity) it was still displayed during this time period. This gave us a total of 17,644 editors whose data was at some point visible from the JSON block in at least one entity’s source code. We sadly do not have a way to know for sure how many of the affected were actually ever found and stored by spammers, since we attempt to block botnets as much as possible. As such, we simply have no way of knowing who was really affected by this leak — only who might have been.

What we’ve done

Once we detected the issue on November 22, we immediately put out a hotfix to all production (and beta) servers plugging the leak. The hotfix acted to sanitize the editor data by removing email addresses and birth dates from the JSON. We also deployed two additional changes that should help prevent similar issues from occurring, by avoiding sending sensitive editor data to our template renderer altogether. See all changes from the git tag v-2020-11-22-hotfix.

We are planning to improve our testing infrastructure to detect exposure of editor data — this will become a routine part of our continuous integration process. We are also going to ensure that any pull request dealing with editor data goes through a strict testing checklist.

How did spammers get these email addresses?

You might be wondering how such an obscure leak in a web page can end up in spammers finding and using your email — you’re not alone. 

Our sites are under near constant traffic from seemingly random internet bots fetching thousands of our pages in a day, with no apparent goal. All of our metadata is available for download, so why would someone download pages from us at random?

Well, we now know — web pages can contain a whole host of random data that shouldn’t be there. Email addresses, birth dates and such are just the starting point — there have been websites that have leaked credit card numbers and even login passwords, possibly compromising the integrity of user accounts.

In this case it appears that a botnet kept downloading pages from musicbrainz.org and driving the load on our servers up. We’ve been trying to block botnets ever since they’ve come into existence, but this is a laborious task that is never complete.

It appears that spammers used the botnet to scour the internet for private data such as emails to then send out lovely spam emails to all compromised users.

Summary

We would like to wholeheartedly apologize for this data leak. We take data privacy seriously and we aim to have high standards about privacy and data security. We find ourselves frustrated by the endless data leaks that happen on the Internet on a seemingly continuous basis and work hard to avoid committing these mistakes in our domain. However, we’re also human and we do make mistakes periodically. As explained above, we’re working to improve our systems and processes in order to prevent this from happening again.

We hope that you accept our most sincere apologies for this leak.

Robert Kaye, Michael Wiencek, Nicolás Tamargo and Yvan Rivierre

Picard 2.5.2 released

Picard 2.5.2 is a maintenance release, fixing some bugs and providing minor improvements to the recent 2.5.1 release. Thanks a lot to everyone who gave feedback and reported issues.

The latest release is available for download on the Picard download page.

What’s new?

Bug

  • [PICARD-1948] – ScaleFactorRoundPolicy breaks text rendering on Linux
  • [PICARD-1991] – Case-only changes to file names are not applied on case insensitive file systems on Linux
  • [PICARD-1992] – Case-only changes to file names are not applied on FAT32 and exFAT file systems on Windows 10
  • [PICARD-2001] – Directory drag & drop from file browser to cluster area broken
  • [PICARD-2004] – Metadata changes loaded asynchronously by plugins are reset if file gets matched to track
  • [PICARD-2005] – Modified fields are sometimes not correctly marked as changed when multiple files are selected
  • [PICARD-2006] – “Local files” cover provider does not detect cover files for files already present at release loading time
  • [PICARD-2012] – Loaded files not shown in UI if release MBID is a redirect
  • [PICARD-2014] – Config upgrade from Picard < 1.3.0 to version 2.4 or later fails

Improvement

  • [PICARD-1828] – Allow assigning cover art to multiple selected files
  • [PICARD-1999] – Provide binary distributions for Windows and macOS on PyPI
  • [PICARD-2007] – Disable analyze / audio fingerprinting for MIDI files

The complete list of changes of this and previous releases is available in the changelog. You can also discuss new features or usage on our forums.

MusicBrainz Server update, 2020-11-02

Right after Halloween, this new release of MusicBrainz Server tricks some bugs and treats some improvements, plus some work on the usually terrifying React conversion and updates to handle external links.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to chaban, darwinx0r, kellnerd, hibiscuskazaneko, jesus2099, lotheric, snartal, and tularion for having reported bugs and suggested improvements. Thanks to grafi_tt, mfmeulenbelt, salorock, and shepard for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2020-11-02.

Bug

  • [MBS-6666] – Artist credits not renamed from artist edit page unless the artist name is changed
  • [MBS-10281] – Improper encoding of ISE pages
  • [MBS-10829] – Indexed recording search fails to find recording with no length
  • [MBS-11160] – Internal server error pages display empty stack traces
  • [MBS-11161] – Internal server error page sometimes not returned when an error occurs
  • [MBS-11186] – Inconsistent username font-weight for edit owner
  • [MBS-11194] – TypeError: Cannot read property ‘linkTypeID’ of undefined (part 2)
  • [MBS-11204] – ISE: Validation failed for \’Int\’ with value undef

Improvement

  • [MBS-7219] – Only display “Show only standalone recordings instead” when there are standalone recordings to display
  • [MBS-11158] – Document URL link_type integers for release editor seeding
  • [MBS-11177] – Do not show useless “Description:” label in entity type doc boxes
  • [MBS-11185] – Add “is not” operator for relationship type in edit search
  • [MBS-11192] – Add voting-icon for Approved
  • [MBS-11197] – Add validation for Mainly Norfolk links
  • [MBS-11199] – Update 7digital.com URL cleanup

React Conversion Task

  • [MBS-11195] – Convert the artist credit renamer to React

Other Task

  • [MBS-11182] – Remove LyricWiki links from the sidebar
  • [MBS-11189] – Remove PureVolume links from sidebar
  • [MBS-11196] – Add saisaibatake.ame-zaiku.com to “other databases” for instruments
  • [MBS-11200] – Add works to VGMdb autocleanup