Server update, 2019-01-22

More bugs fixed, that’s our wish for 2019! Additionally, this release comes with usual update of URL handlers and a heap of code being rewritten behind the scenes. Thanks to Jaykin for the patch he wrote during GCI. Thanks to culinko, drsaunde, jesus2099, johnlinp, jrv, outsidecontext, silentbird, thomasboehm, zastai for having reported issues. The git tag is v-2019-01-22.

Sub-task

  • [MBS-9954] – Convert the tag cloud and tag entity list pages to React

Bug

  • [MBS-4469] – Annotation syntax : lists syntax only partly working
  • [MBS-4709] – Can’t use square brackets in annotations
  • [MBS-8978] – Schema does not include id attribute for <packaging> element of a release
  • [MBS-8979] – Schema does not include <cdstub> as top-level subelement of <metadata>
  • [MBS-8980] – Schema says a <cdstub> has a <comment> element, but the webservice returns a <disambiguation> instead.
  • [MBS-9089] – “change all artists on this release that match…” does nothing if you hit “Next” instead of “Done”
  • [MBS-9605] – Tracklist editor badly converts track length over 1 h
  • [MBS-9865] – Release editor’s artist credit recently broken
  • [MBS-9867] – Slave server cannot display page with annotation
  • [MBS-9941] – Some edit pages are missing JS gettext domains in non-English translations
  • [MBS-9957] – Cover art image types not updated properly
  • [MBS-9966] – Event creation on place page does not seed
  • [MBS-9967] – Regression: Subscribers tab in user profile is broken
  • [MBS-9971] – Regression: Sidebar ISNI code is not a link anymore
  • [MBS-9980] – Regression: “Edit User” tab link is shown to own non-admin user

Improvement

  • [MBS-8803] – Document [<entity>:<mbid>|<name>] links in annotation editor page
  • [MBS-9470] – Expose UUID of relationship attribute type
  • [MBS-9838] – Update 22 external link formats from http to https
  • [MBS-9969] – Display external links to lyrics in the sidebar
  • [MBS-9970] – Display links to Anison Generation, B.R.A.H.M.S. (Ircam), and Classical Archives in the sidebar

Server update, 2018-12-18

This release features four new reports of data that might need fixing. Pages with the MBID of entities that have been merged are now redirected to use the MBID of the merged entity. This release also includes a significant amount of bug fixes and updates genres list and external link handler. Thanks to coders G30rg3, TheFaR8, PROTechThor (all GCI students!) and dominikh for their contributed patches, and thanks to issue reporters bgstewart, cyberskull, jesus2099, hibiscuskazeneko, michelv, paulakreuzer, salo.rock, suriv, and wcw1966 for their input. The git tag is v-2018-12-18.

Bug

  • [MBS-8311] – Annotation field in the release editor does not mention licenses
  • [MBS-9402] – Empty “Lyrics languages” column in “Merge works” edits
  • [MBS-9859] – Amazon cover art source stats don’t count https URLs
  • [MBS-9863] – License links are ignored if https
  • [MBS-9844] – Cannot cancel removing/revoking application’s access
  • [MBS-9872] – “Ended” flag change not shown in edit relationship page
  • [MBS-9889] – Release page “Found in N user collections” links to incorrect URL
  • [MBS-9892] – Unable to add Australian ASIN
  • [MBS-9895] – Regression: Approving link removals is now harder (auto editors)
  • [MBS-9896] – Release pages throw an internal server error when CritiqueBrainz is down
  • [MBS-9905] – Tipeee language subdomains are not recognised
  • [MBS-9906] – Link to profile when preferences have been saved is broken

New Feature

  • [MBS-3685] – New Report: Releases with label and artist of the same name
  • [MBS-6232] – New Report: Releases with cover art archive images but no front image
  • [MBS-8178] – New Report: Events with sequential numbering that are neither part of a series nor part of a larger event
  • [MBS-9907] – New report: Places without coordinates

Task

  • [MBS-9901] – Normalize Dailymotion URLs to HTTPS
  • [MBS-9930] – Disable CAA images on the homepage for Chrome-based browsers
  • [MBS-9934] – Update blog and JIRA links to metabrainz.org

Improvement

  • [MBS-7192] – Show work types in inline search
  • [MBS-9277] – Handle Mixcloud social network URLs
  • [MBS-9748] – Handle Ko-fi patronage URLs
  • [MBS-9765] – Do not display event dates twice in search results
  • [MBS-9827] – Redirect merged entity MBID pages to new MBID
  • [MBS-9831] – Handle Bandcamp lyrics URLs
  • [MBS-9847] – Show event collections in reverse chronological order by default
  • [MBS-9873] – Handle Discogs composition URLs
  • [MBS-9883] – Show Wikidata: QID rather than just QID on sidebar
  • [MBS-9900] – Add information on requesting instruments/genres to their lists
  • [MBS-9944] – Handle niconico video URLs
  • [MBS-9947] – Allow linking to podcast episode on Spotify and Deezer

Server update, 2018-11-01

This release includes a first implementation of genres – expect more information as a blog post in the following days. The search results page has been converted to React for every type of search except the edit search, which is implemented separately. The homepage, the ISWC page and the sidebars have been converted too. Additionally, the password hashes have been strengthened, external URL handlers have been updated as usual, and ten bugs have been fixed. Thanks to issue reporters chirlu, darwinx0r, jesus2099, hibiscuskazeneko, ravenworks, spellew, yeeeargh, and zastai for their input.  The git tag is v-2018-11-01.

Sub-task

  • [MBS-9604] – Convert ISWC index page to React
  • [MBS-9813] – Convert the homepage to React
  • [MBS-9830] – Convert taglookup templates to React
  • [MBS-9850] – Convert sidebars to React
  • [MBS-9877] – Convert all search results pages to React

Bug

  • [MBS-8062] – Event indexed search doesn’t show artists/location/date
  • [MBS-9422] – Attach CD TOC page shows all tracks in tracklists as deleted
  • [MBS-9438] – Event dates and times display as “(Object object)” in drop-down search results
  • [MBS-9641] – Label-Series relationship does not show on series page
  • [MBS-9788] – Series edit history indicates wrong entity type
  • [MBS-9804] – Label dates do not appear on direct search
  • [MBS-9816] – Cannot change password if logged in with a different letter-cased username
  • [MBS-9817] – Regression: Client-side rendered components are not localized
  • [MBS-9846] – Beatport URL with numeric slug cannot be linked to
  • [MBS-9856] – Regression: Buggy formatting on some web search results

New Feature

  • [MBS-9492] – Add genres as a subset of tags

Task

  • [MBS-9208] – Increase bcrypt cost parameter
  • [MBS-9721] – Add GDPR links
  • [MBS-9763] – Normalize setlist.fm URLs to HTTPS
  • [MBS-9766] – Normalize worldcat.org URLs to HTTPS

Improvement

  • [MBS-9210] – Re-hash passwords on login
  • [MBS-9503] – Add a link to AcousticBrainz on the Details tab of recordings
  • [MBS-9764] – Allow adding events from places and artists

Server update, 2018-08-14, with new virtual machine

A new virtual machine for running your own mirror of MusicBrainz is now available. Thanks to InvisibleMan78 for the continuous feedback and to Jeff Sturgis for his work with Docker Compose! This is the last virtual machine to include the soon defunct search server. Links:

Main changes:

  • Added helper script set-web-server-name to allow connecting to the web site from a remote host;
  • Added helper script turn-port to allow turning service ports on/off for db (postgresql), musicbrainz, redis, and search;
  • Fixed docker containers startup on VM boot;
  • Fixed potential access token loss due to helper scripts reindex;
  • Fixed and improved documentation (now copied inside the VM);
  • Improved error handling in helper scripts;
  • Updated MusicBrainz server, search server, base image and dependencies.

This server release converts more search results pages to React and fixes seven bugs. Thanks to @riipah for fixing @TouhouDB handler. Search scores are no longer displayed on search results page for clarity as they misleadingly weighed down the the importance of results appearing on the first page but not in the first place. The git tag is v-2018-08-14.

Sub-task

  • [MBS-9760] – Convert the event search results page to React
  • [MBS-9761] – Convert the series search results page to React

Bug

  • [MBS-9687] – Field error messages are not always translated
  • [MBS-9694] – TouhouDB URLs are not recognized
  • [MBS-9705] – URL overview header link is now appended with /show
  • [MBS-9744] – OAuth userinfo endpoint returns latin1 encoded JSON
  • [MBS-9755] – Fix unknown search in statistics for scripts
  • [MBS-9757] – Search page form is sometimes in the wrong language
  • [MBS-9784] – IMDb verification too strict (# of digits)

Task

  • [MBS-9780] – Normalize Creative Commons URLs to HTTPS

Improvement

  • [MBS-9767] – Remove search score display from the UI

We’ve finally released our new Solr search (Server update, 2018-06-30)

The new search is live on MusicBrainz with this server update, as announced in previous blog post. This release also continues the rewrite to React, improves and fixes the handling of external URLs. The git tag is v-2018-06-30.

Sub-task

  • [MBS-9736] – Convert the artist search results page to React

Bug

  • [MBS-8334] – Digest auth with username containing non-ascii characters fails
  • [MBS-9730] – Cannot link to a Bandcamp Daily review page in release group relationships
  • [MBS-9734] – Inconsistency between the JSON search API and the lookup/browse one in ws/2/
  • [MBS-9742] – Some Library of Congress URLs are not recognised
  • [MBS-9743] – Beatport URL cleanup fails for names starting with digit

Improvement

  • [MBS-9408] – Add Juno Download links to the sidebar
  • [MBS-9439] – Make changing a URL between HTTP and HTTPS an autoedit
  • [MBS-9740] – Update Facebook URL cleanup

Many people thought this day would never come. 🙂

Releasing our new Solr search infrastructure

Hey folks, samj1912 here again o/

As you might know, we recently did a massive upgrade of our search infrastructure. If you have not been following our Solr updates, definitely check out our other blog detailing our search server journey and the improvements and changes that come with the new search.

We have had a beta run with Solr this last week and fixed most of the show-stopping bugs. As such, we have been stress testing our Solr search by replaying our production logs on it, live.

Solr search seems to solvr almost all our qualms with search and as such, we have made the decision to use Solr for our production search servers.

The purpose of this blog post, as nicely worded by our BDFL Rob is –

Speak now or forever hold your pickle. In a week, the ole search servers gets it.

And it’s basically that, if you haven’t experimented with Solr search, please read our earlier blog to know what’s what. If you find any bugs, please report them on our ticket tracker. In case there are no new show-stoppers reported, that must absolutely be fixed before we switch to Solr on the main website, we will be killing the old search servers and replacing them with our brand new Solr ones in a week.

Apart from that, we have made a discourse thread to report any minor improvements in the search results.

Another thing, I’d like to remind everyone is that, with our switch to this new Solr infrastructure, the version 1 web service (ws/1)will soon be discontinued. As announced earlier, we will keep it alive till 31st July 2018 but it will get the axe on 1st August 2018, 12 pm GMT.

MusicBrainz Search Overhaul

Hello people o/, samj1912 here.

I am extremely glad to announce that we are finally launching our Solr search on the MusicBrainz beta server!

Just a little history before I announce the new features and toys you get to play with:

Solr started as something that could replace our existing search infrastructure. If you have been a MusicBrainz user for a while, you might know that our search has quite an indexing latency and it takes as much as 3 hours for new edits to show up in the search results. In part because updating the search index involved doing an entire re-index of the database. With the high latency and the resources it took, the current search server left much to be desired.

Another area that our current search lacked in, was showing popular results and search ranking. Searching for a famous artist or place returned results that contained a lot of noise, and more often than not, contained results that weren’t relevant to what the user had in mind when they searched for it.

These were the two major problems that motivated us to shift to a better infrastructure for our search needs.

Thus, MB-Solr was born.

It has been in development for quite some time now. The coding for the project started with Mineo back in 2014 and was carried forward by Jeff Weeksio in GSoC 2015. But due to lack of development resources and other, more pressing needs, the project was put on a hold for a while, until Roman started working on it. However, he left MetaBrainz before he could finish this work, so when I joined the MetaBrainz team, the first and foremost task that was assigned to me was getting Solr working and ready for production.

After struggling with multiple moving parts and services, tons of issues with maintaining compatibility with our existing web-service API, rowing up and down multi-threading/processing hell, learning just enough about information retrieval to get our search relevance on point and countless hours sifting through Solr documentation to get our Solr cluster fine-tuned and running fast enough to keep up with our web traffic… we are finally here.

I am pretty sure I would’ve rage-quit dozens of times during this last year if I was doing this all alone.

As such, we have our trusty sysadmin Zas to thank for taking care of all the deployment needs and making sure Solr was well-tested (believe me we toyed with Solr like little kids in a sandbox) and wasn’t going to fail and wake him up 3 AM in the morning with red alerts all over. Mineo, Bitmap and Yvanzo were there, with much-needed code reviews and help with all things Solr and MusicBrainz. Our style leader Reosarevok, and CatQuest helped us test our new search relevance configuration. And of course, we had our BDFL, Rob over-seeing things and whipping them into shape (with chocolate and mismatched socks of course).

Anyway, here’s what you are here for:

New features/improvements

  • (Almost) Instantaneous search-index updates – Edit something and immediately see it in the search results. Say goodbye to that note you used to see below the search telling you that you have to wait. Who likes waiting anymore – seriously, it’s 2018.
  • Better search results – We wanted to make sure you were getting the right Queen and London as the top result. You can finally link your favorite artist to London, UK as opposed to London, Arkansas. Don’t believe me? Go try it out.
  • Less load on our servers – Meaning we can serve more of your requests, faster. Getting tired of waiting for tagging your bajillion songs in Picard? Well, you still gotta wait, but less so, now that we are better equipped to handle your requests.

What has stayed the same

  • WS/2 Search API – We know you devs hate doing that extra work to maintain your applications’ compatibility with that one site that changes its API on a whim. Well, we wouldn’t want you to spend those hours following that one int to float change that broke everything ever. As such we have worked hard to make sure that Solr doesn’t change any of our WS/2 search schema.

What’s gone

  • WS/1 Search API – We deprecated WS/1 back in 2011. With the new search servers in place, there are only 3 words for those still using it after WS/1 being deprecated 7 years – ‘poof, it’s gone’. The service still works on our main website, but its search functionality will be phased out soon, while the entire service will be discontinued in August 2018 as announced earlier.

Now, you must be thinking there is some catch, some slip. Well so do I, which is why we are releasing this beta for you to test the heck out of our new search over at the MusicBrainz beta site. If you haven’t used it before, worry not – it has all your personalizations and all our cool music metadata from our main site. You should feel at home. (Note: The MusicBrainz beta site works on the live data. Any edits you make on the MusicBrainz beta site will also be reflected on the main site.)

So please! Go check it out!

If you feel you aren’t getting what we promised you or you want more of those shiny new features or that this blog was too long or like a TV commercial, feel free to complain at our Ticket Tracker for Solr. You get your promised features bug-free and our devs get to earn their living. It’s a win-win.

Happy testing!