Category: Search Server

End-of-life for external access to MusicBrainz search server

For MusicBrainz mirrors, MusicBrainz server development, and custom applications, there used to be a direct access to search.musicbrainz.org which was actually inherited from the previous search infrastructure until 2018.

Keeping it available in the current infrastructure uses a compatibility layer which has been unintentionally broken last year. This issue has been reported only once. The alternative and much more sensible solution which is to install a local search server has improved in reliability and simplicity in the meantime. Polling the community shown that only one person would be interested which isn’t enough to justify the maintenance overhead for this replaceable service.

Therefore it has been decided to close this access for good.

Thanks to everyone who made this transitioning service to work and handle about 40k requests a day on average!

MusicBrainz schema change release, 2022-05-16 (with upgrade instructions)

We’re happy to announce the release of our May 2022 schema change today! Thanks to all who were patient during today’s downtime as we released everything to our production servers, and thanks to ikerm2003, mfmeulenbelt rinmon and salo.rock for updating the translations.

This is once again a fairly minor release as far as schema changes go, but please do report any issues that you come across, especially related to the propagation of ratings and tags.

New, user-facing changes with this release include withdrawn-only release groups showing in the official overview again (MBS-12208) and the final disappearance of Amazon cover art (MBS-12200). To this regard, the report of releases that have Amazon cover art but no Cover Art Archive front cover will stay available for editors to check their subscribed entities.

Additionally, several small changes will allow, in the next couple of releases, to store more information about genres (including URL relationships) and to recognize and special-case mood tags (MBS-12190). Another new feature that will start to be used in the API and artist credit pages is the addition of MusicBrainz IDs for artist credits, which allow referring to them with a unique and more persistent ID (MBS-11456). Finally, a few more under-the-hood only changes are made, which should ensure better performance for finding artists, events, etc. for all areas contained in a given one, and less bugs when adding and changing tags and ratings.

The area containment changes make use of a new materialized table. Like the ones we added last year, this table isn’t dumped nor replicated, since it is derived entirely from primary table data. Rather, it will be created during migration (or, in a new install, by running the admin/BuildMaterializedTables script) and triggers will be added to keep it up-to-date once it has been built. These triggers are created on replicated servers, too.

The accompanying new version of the search index rebuilder brings performance improvements for both the main server and mirrors, and simplifies maintenance. See the release notes for details.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Now, on to the instructions.

Schema Change Upgrade Instructions

Note: Importing the latest data dump is always a valid alternative to running ./upgrade.sh on an existing database, if you’d prefer to also get new data in one go. Just follow the relevant instructions in INSTALL.md. The git tag is v-2022-05-16.1-schema-change. The rest of the instructions here assume an in-place upgrade.

  1. Make sure DB_SCHEMA_SEQUENCE is set to 26 in lib/DBDefs.pm.
  2. If you’re using the live data feed (your REPLICATION_TYPE is set to RT_SLAVE), ensure you’ve replicated up to the most recent replication packet available with the old schema. If you’re not sure, run ./admin/replication/LoadReplicationChanges and see what it tells you; if you’re ready to upgrade, it should say “This replication packet matches schema sequence #27, but the database is currently at #26.”
  3. Take down the web server running MusicBrainz, if you’re running a web server.
  4. Turn off cron jobs if you’re automatically updating the database via cron jobs.
  5. If you’re using the live search indexing, stop it and, assuming sir is under the same directory as musicbrainz-server, run cd ../sir && python2.7 -m sir triggers && cd - && ./admin/psql < ../sir/sql/DropTriggers.sql && ./admin/psql < ../sir/sql/DropFunctions.sql
  6. Switch to the new code with git fetch origin followed by git checkout v-2022-05-16.1-schema-change.
  7. Run cpanm --installdeps --notest . (note the dot at the end) to ensure your perl-based dependencies are up to date.
  8. Run ./upgrade.sh (it may take a while to vacuum at the end).
  9. Set DB_SCHEMA_SEQUENCE to 27 in lib/DBDefs.pm as instructed by the output of ./upgrade.sh.
  10. If your REPLICATION_TYPE is set to RT_SLAVE, change it to RT_MIRROR. (The previous terminology will work for the time being, but is now deprecated.)
  11. If you’re using the live search indexing, assuming again that sir is under the same directory as musicbrainz-server, run cd ../sir && git fetch origin && git checkout v3.0.1 && python2.7 -m sir triggers && cd - && ./admin/psql < ../sir/sql/CreateFunctions.sql && ./admin/psql < ../sir/sql/CreateTriggers.sql and rebuild indexes which takes hours (by running cd ../sir && python2.7 -m sir reindex && cd -) then start it in watch mode (with cd ../sir && git fetch origin && git checkout v3.0.1 && python2.7 -m sir amqp_watch)
  12. Turn cron jobs back on, if applicable.
  13. Restart the MusicBrainz web server, if applicable. It’s also recommended you restart Redis. If you’re accessing your MusicBrainz server in a web browser, run ./script/compile_resources.sh.

Here’s the list of resolved tickets:

Fixed Bug

  • [MBS-5359] – *_tag tables are corrupt and need to be regenerated
  • [MBS-11760] – Removing the last use of a tag does not always remove the tag
  • [MBS-12369] – Standalone databases may be missing foreign keys for the documentation schema

New Feature

  • [MBS-12190] – Add Mood support in the database

Improvement

  • [MBS-11456] – Add MBIDs to artist credits in the database with merge
  • [MBS-12141] – Block tag names that are empty or have uncontrolled whitespace with database constraints
  • [MBS-12224] – Keep tags’ ref_count and aggregate vote counts updated with triggers
  • [MBS-12249] – Add a materialized area_containment table kept up-to-date with triggers
  • [MBS-12256] – Keep rating and rating_count column on *_meta tables up-to-date with triggers
  • [MBS-12313] – Clarify item naming in the Search drop down menu

Database Schema Change Task

  • [MBS-11457] – Drop the series ordering_attribute column
  • [MBS-11755] – Remove unused tags
  • [MBS-12157] – Remove support for Amazon cover art
  • [MBS-12200] – Drop schema objects related to Amazon cover art support
  • [MBS-12225] – Rename “slave” to “mirror” (inclusive language update)
  • [MBS-12250] – Create dbmirror2 schema on production and mirror servers
  • [MBS-12252] – Add edit_genre table
  • [MBS-12253] – Add relationship tables for genres
  • [MBS-12254] – Add genre_annotation table
  • [MBS-12255] – Add genre_alias_type table and make genre_alias consistent

MusicBrainz servers update, December 2020

Let’s end the year 2020 on a brighter note with new releases of both MusicBrainz Server and its companions for advanced indexed search.

The most urgent task was to lower the risk of leaking private data again (see previous incident) by reducing the scope of editor data passed to the renderer.

The most visible improvement of the website is the addition of instrument illustrations by IROM. Thanks to him for these very welcomed drawings. See sistrum for an example; we will keep adding illustrations to other instruments little by little.

The most useful improvements to the API are the addition of their first release date to recordings (both in lookup and search) and release groups (which were missing it in search results), and the addition of MBIDs for artists’ gender and releases’ status.

Advanced indexed search has also been improved to allow searching for releases by type of packaging, for recordings and release groups by first release date, and to simplify searching for releases by type of medium’s format.

Finally, a fair number of smaller bugs have also been fixed, and React conversion is being continued.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to amCap1712 for fixing a pair of bugs in outputting search API results, and jesus2099 for fixing a bug in seeding the release editor. Thanks to avilla, CatQuest, chaban, jgrmstr, jstranger, Lotheric, loujin, nikki, Psychoadept, and serg for having reported bugs and suggested improvements. Thanks to Besnik, mfmeulenbelt, and salorock for updating the translations. And thanks to all others who tested the beta version!

The git tags are v-2020-11-12 for MusicBrainz XML Metadata Schema, v-2020-12-14 for MusicBrainz Server, v-2020-12-24 for MusicBrainz Simple Solr Search Server Schema, v2.0.0 for Search Index Rebuilder, and v3.4.1 for MusicBrainz Solr Query/Response Writer.

Bug

  • [MBS-4555] – Seeding the release editor when logged out loses the seeded information
  • [MBS-8438] – The same event is displayed twice on an artist’s “Events” tab if they have multiple roles on it
  • [MBS-10664] – Regression: HTML tags are incorrectly displayed in editor’s bio
  • [MBS-11092] – Some issue with userscripts/plugins and beta site
  • [MBS-11169] – Artist credit doesn’t display properly in release pages when credits are at the bottom of the release instead of inline
  • [MBS-11207] – Error messages from DBI are badly encoded
  • [MBS-11212] – Incorrect quality attribute in “ws/2/release”
  • [MBS-11214] – Relationships with different link order not split on “see at bottom” medium rels display
  • [MBS-11220] – Release editor seeding is displayed in wrong language
  • [MBS-11223] – Releases with “unknown tracklist” on a medium failed to load in the “Edit Relationships” tab
  • [MBS-11227] – Unexpected URL used when removing release groups from merge queue
  • [SEARCH-623] – Gender ID is missing from JSON/XML artist search results
  • [SEARCH-624] – Status ID is missing from release in JSON/XML search results

Improvement

  • [MBS-1424] – Add a “First release date” field to recordings
  • [MBS-11188] – Block smart links: “album.link”
  • [MBS-11224] – Releases with no medium should show a clear message in the relationship editor
  • [MBS-11225] – Make “staticbrainz.org/irombook/” URLs autoselect for image relationship
  • [MBS-11271] – Reduce the scope of editor data passed to the template renderer
  • [SEARCH-319] – Add first release date to the results of indexed search for release groups
  • [SEARCH-574] – Make release’s format search field insensitive to spaces and separators

New Feature

  • [MBS-10590] – Allow displaying IROMBOOK instrument images on MusicBrainz
  • [MBS-11216] – Split report “InstrumentsWithoutWikidata” from “InstrumentsWithoutAnImage”
  • [SEARCH-218] – Add first release date to the fields and results of indexed search for recordings
  • [SEARCH-384] – Add first release date to the fields of indexed search for release-groups
  • [SEARCH-590] – Add packaging to the fields of indexed search for releases

React Conversion Task

  • [MBS-11018] – Convert Add Relationship Type edit to React
  • [MBS-11211] – Convert the login page to React

Other Task

  • [MBS-10634] – Rework the “Instruments without an image” report to only look at IROMBOOK images
  • [MBS-11210] – Allow more subpaths for “saisaibatake.ame-zaiku.com”

MusicBrainz Server update, 2020-10-19

Today’s MusicBrainz Server brings a new data report, a continued conversion to React, some bugfixes and small improvements, but also tests refactoring.

Meanwhile, the search server has been updated twice in a row to fix bugs in JSON output mostly with MB Solr 3.2 (release notes) and MB Solr 3.3 (release notes), including the MusicBrainz API breaking change announced last month.

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to amCap1712 for fixing bugs in MB Solr, and to loujine for contributing code with yet a new data report. Thanks to Avamander, bonchiver_, chaban, draconx, eloise_freya, GTF1982, hawke, hibiscuskazeneko, jesus2099, jrv, kellnerd, Kid Devine, Psychoadept, selflessself, and wcw1966 for having reported bugs and suggested improvements. Thanks to kellnerd, mfmeulenbelt, and salorock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2020-10-19.

Bug

  • [MBS-10221] – Track Parser not filling in artists
  • [MBS-11149] – Misspelling of the word “misspellings”
  • [MBS-11150] – Add CD-TOC wrongly defaults to last listed artist when none selected
  • [MBS-11156] – Track parser unsets “Various Artists” track artist credits
  • [MBS-11162] – Work type description bubble starts as default even if type is selected
  • [MBS-11174] – Editor profile added entities: Missing Add release edit type 216
  • [MBS-11176] – Heading of the release group section in the external links sidebar has disappeared

Improvement

  • [MBS-5225] – Allow showing tracklists everywhere when attaching/viewing discIDs
  • [MBS-7256] – Add “Expand all mediums” option to the release page
  • [MBS-8725] – Allow mediums to have an unknown tracklist
  • [MBS-11115] – Show detailed information when attaching disc IDs
  • [MBS-11139] – Use HTTPS for display on Library of Congress links
  • [MBS-11163] – Show type descriptions when editing entities
  • [MBS-11165] – Update the VK logo used in the sidebar
  • [MBS-11167] – Normalize vk.com links to HTTPS
  • [MBS-11173] – When deleting users, change their No and Yes votes on pending edits to Abstain

New Feature

  • [MBS-11117] – Report for mediums with very long durations from discID

React Conversion Task

  • [MBS-11141] – Convert Edit Relationship edit to React
  • [MBS-11152] – Convert entity/ratings page to React

Other Task

  • [MBS-11148] – Remove Google Play links from the sidebar

MusicBrainz Server update, 2020-09-21

React conversion tasks are conspicuously absent from today’s release, but that’s just because we needed to take some time to get it all working with the recent refactoring. This new server update mainly brings strong security improvements for the OAuth service. It also comes with a fair amount of smaller bugfixes and improvements. The most noticeable of these probably are the added details to the merge recordings’ form and the statistics by entity type on editors’ profile pages.

Announcement for MusicBrainz API users: A small but breaking change will be deployed on October 19th (in one month from now), to fix the JSON formatting of release packaging in search results (SEARCH-579).

A new release of MusicBrainz Docker is also available that matches this update of MusicBrainz Server. See the release notes for update instructions.

Thanks to kellnerd and loujine for contributing code. Thanks to calculator.ftvb, chaban, hibiscuskazeneko, jesus2099, kellnerd, lalinksy, psychoadept, rdswift, and spitzwegerich for having reported bugs and suggested improvements. Thanks to jesus2099, kellnerd, mfmeulenbelt, outsidecontext, and salorock for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2020-09-21.

Bug

  • [MBS-10880] – Series automatic ordering (without numbers) fails for new release group
  • [MBS-11065] – Smart link blocks affecting legitimate links
  • [MBS-11069] – Diff highlighting not visible for certain display resolutions
  • [MBS-11098] – Big Cartel URLs are denied for labels
  • [MBS-11101] – Series relationships not showing for work series

Improvement

  • [MBS-2768] – Display AcoustIDs, Annotation and any other useful info when merging recordings
  • [MBS-7473] – Adding a new discid: Allow to specify the target by its releaseid
  • [MBS-11017] – Normalize IMSLP URLs to HTTPS and add validation
  • [MBS-11058] – Tighten security of OAuth service
  • [MBS-11061] – Don’t allow MusicBrainz URLs in relationships
  • [MBS-11062] – Link basic how-tos from the front page
  • [MBS-11086] – Add icon for tabs with errors in release editor
  • [MBS-11109] – Block further more smart links
  • [MBS-11119] – Set a Content-Security-Policy header on account/admin related forms

New Feature

  • [MBS-7485] – OAuth token revokation through API
  • [MBS-9769] – Show entities added statistics on editor profile page
  • [MBS-10835] – Disallow creating new accounts with an e-mail already in use
  • [MBS-11097] – Support PKCE (Proof Key for Code Exchange) by OAuth clients

Task

  • [MBS-10921] – Clear editing history of unrelated recording-of relationship edits

MusicBrainz Server update, 2020-06-02

Now that PostgreSQL has been upgraded to version 12 (see earlier instructions), regular improvements, bugfixes, and React/JSX template refactoring are back on the menu. The most noticeable improvement is probably that we are now able to display more specific error messages in the URL relationship editor when a link is not allowed, instead of always giving a generic “not good”-style message. There are even new features, for admins only, that will allow them to spot and delete sock-puppets and to temporarily disable edit notes from editors who continue to be disrespectful to others without having to delete their accounts outright.

In news not directly connected to the MusicBrainz website, the public search endpoint has been moved from the old search server to the Solr-based search server. This can be used by slave servers if they do not need, or can not afford, to host their own search indexes.

And while talking about slave servers, a new release of MusicBrainz Docker is also available. It follows this update of MusicBrainz Server and fixes a regression that affects servers with live indexing. See the release notes for update instructions.

Thanks to Cyna for converting more edits’ display to React, to KamranMackey and navap for updating external links’ icons. Thanks to alastairp, BestSteve, chaban, chirlu, cyberskull, Freso, hibiscuskazeneko, jesus2099, and Kid Devine for having reported bugs and suggested improvements. Thanks to eduardomariohs, kellnerd, mfmeulenbelt, salorock, and stich94 for updating the translations. And thanks to all others who tested the beta version!

The git tag is v-2020-06-02.

Bug

  • [MBS-9010] – Alerts about new edit notes are separate for production and beta
  • [MBS-10613] – Version info footer is shown on some pages in production
  • [MBS-10732] – Internal Server Error: Adding a collaborator to a collection without selecting from autocomplete dropdown
  • [MBS-10839] – “Add selected recordings for merging” missing in standalone-only overview
  • [MBS-10849] – Add release group preview on RE shows “This entity has been removed”
  • [MBS-10850] – Bad Amazon URLs report doesn’t ignore “streaming page” relationship
  • [MBS-10855] – Historic track edit involving since-removed entity doesn’t show recording
  • [MBS-10863] – “Edit barcode” edit doesn’t show barcode

New Feature

  • [MBS-10834] – New account flag for disabling ability to write edit notes
  • [MBS-10845] – Tool to allow account admins to look up accounts by e-mail

Improvement

  • [MBS #1516] – Update Facebook, Google Play, and Spotify icons
  • [MBS-7822] – Update OverClocked ReMix favicon
  • [MBS-8412] – Align number of edits per page
  • [MBS-9516] – Display specific error messages depending on URL validation rules
  • [MBS-9963] – Update Genius link format and logo
  • [MBS-10412] – Update URL cleanup for Niconico URLs, specifically channel links for artists
  • [MBS-10727] – Deny kasi-time.com URLs for lyrics
  • [MBS-10789] – Add validation for Genius links
  • [MBS-10813] – Update the Bandcamp logo used in the sidebar
  • [MBS-10831] – Allow niconico channel links for other entities than artist
  • [MBS-10840] – Capitalise “in key” info correctly in English guess case
  • [MBS-10841] – Add “Guess case” per-medium
  • [MBS-10842] – Remove report user link from deleted editor profiles
  • [MBS-10853] – Link to overview page for edits by subscribed editors in subscription email

React Conversion Task

  • [MBS-10397] – Convert Edit Event edit to React
  • [MBS-10399] – Convert Edit Recording edit to React
  • [MBS-10793] – Convert historic Move Release edit to React
  • [MBS-10799] – Convert historic Move Release to RG edit to React
  • [MBS-10817] – Convert Edit Label edit to React

Other Task

  • [MBS-7781] – Merge duplicate artist credits
  • [MBS-10785] – Remove link to FreeDB Gateway documentation
  • [MBS-10822] – Change tableColumns tables to use named parameters
  • [MBS-10860] – Merge the production and beta Redis stores
  • [MBS-10878] – Convey search queries from slave servers to Solr

MusicBrainz Docker composes with Solr 7

The MusicBrainz virtual machine is dead, long live the MusicBrainz Docker Compose project. In fact, the virtual machine has been running it for years. Mostly because the data loaded with the virtual machine was too soon obsolete, it doesn’t seem worth it anymore. Plus, new search indexes are much larger than before, and using Docker Compose directly is much more versatile.

The MusicBrainz Docker Compose project has been deeply revamped since two years ago and now ships the new search server based on Solr 7. It can be used for mirroring the MusicBrainz website and database, testing your own app with a local MusicBrainz web service, or developing the MusicBrainz Server itself. Check out the release notes!

Thanks to everyone who reported issues and contributed patches for two years!

We’ve finally released our new Solr search (Server update, 2018-06-30)

The new search is live on MusicBrainz with this server update, as announced in previous blog post. This release also continues the rewrite to React, improves and fixes the handling of external URLs. The git tag is v-2018-06-30.

Sub-task

  • [MBS-9736] – Convert the artist search results page to React

Bug

  • [MBS-8334] – Digest auth with username containing non-ascii characters fails
  • [MBS-9730] – Cannot link to a Bandcamp Daily review page in release group relationships
  • [MBS-9734] – Inconsistency between the JSON search API and the lookup/browse one in ws/2/
  • [MBS-9742] – Some Library of Congress URLs are not recognised
  • [MBS-9743] – Beatport URL cleanup fails for names starting with digit

Improvement

  • [MBS-9408] – Add Juno Download links to the sidebar
  • [MBS-9439] – Make changing a URL between HTTP and HTTPS an autoedit
  • [MBS-9740] – Update Facebook URL cleanup

Many people thought this day would never come. 🙂

Releasing our new Solr search infrastructure

Hey folks, samj1912 here again o/

As you might know, we recently did a massive upgrade of our search infrastructure. If you have not been following our Solr updates, definitely check out our other blog detailing our search server journey and the improvements and changes that come with the new search.

We have had a beta run with Solr this last week and fixed most of the show-stopping bugs. As such, we have been stress testing our Solr search by replaying our production logs on it, live.

Solr search seems to solvr almost all our qualms with search and as such, we have made the decision to use Solr for our production search servers.

The purpose of this blog post, as nicely worded by our BDFL Rob is –

Speak now or forever hold your pickle. In a week, the ole search servers gets it.

And it’s basically that, if you haven’t experimented with Solr search, please read our earlier blog to know what’s what. If you find any bugs, please report them on our ticket tracker. In case there are no new show-stoppers reported, that must absolutely be fixed before we switch to Solr on the main website, we will be killing the old search servers and replacing them with our brand new Solr ones in a week.

Apart from that, we have made a discourse thread to report any minor improvements in the search results.

Another thing, I’d like to remind everyone is that, with our switch to this new Solr infrastructure, the version 1 web service (ws/1)will soon be discontinued. As announced earlier, we will keep it alive till 31st July 2018 but it will get the axe on 1st August 2018, 12 pm GMT.