Server update, 2016-11-21

This is our first release since the NewHost move, so it’s a mishmash of changes required for that and whatever else people submitted in between.

First, some dependency changes: memcached has been dropped as a dependency. We now use Redis as a cache, which had already been used for storing login sessions. In production we run two separate Redis instances: one as a cache (note: configure the DBDefs settings CACHE_MANAGER_OPTIONS and PLUGIN_CACHE_OPTIONS) and one as a persistent session store (DATASTORE_REDIS_ARGS). But it’s also safe (if less flexible) to use a single Redis instance without a maxmemory setting for both, because cached entries will expire by default. So if you already had Redis running locally, you may not need to make any changes to your configuration.

This release also requires the Perl module DBD::Pg to be version 3, which you may already have. If not, you can upgrade only that module with this command:

cpanm TURNSTEP/DBD-Pg-3.5.3.tar.gz

Thanks also to chirlu for upgrading reCAPTCHA on our registration page (which has hopefully reduced new spam accounts), Zastai for fixing some oversights in our JSON webservice, and reosarevok for removing the restriction on score URLs.

The git tag is v-2016-11-21.

Bug

  • [MBS-8762] – ISNI doesn’t appear on JSON WS
  • [MBS-9035] – Coro dependency causes build failure on Perl 5.22 and newer
  • [MBS-9091] – JSON WS: Aliases do not include begin/end date and ended flag
  • [MBS-9127] – NewHost: UI language does not match “lang” in cookie
  • [MBS-9132] – cannot add an alias (area)

Improvement

  • [MBS-9031] – Replace all usages of memcached with redis
  • [MBS-9130] – Upgrade to new reCAPTCHA (version 2)

Task

  • [MBS-9096] – Upgrade DBD::Pg to version 3
  • [MBS-9110] – Disable the score whitelist

The end of the replication nightmare!

I’m pleased to report that our nightmare of finding/reconstructing the missing replication packets is finally over!

Through many heroic hours of work, Bitmap and Chirlu have reconstructed the missing replication packets. All clients should now be on their way to being up to date. We’ve learned a number of lessons (some good, some bad — that’s life, right?) in this ordeal and we hope to avoid these issues in the future.

An integral part of this recovery process were a number of people from our community who helped us: Users mbcz, rembo10 and xeam sent us their complete DB dumps! Bitmap used these to sanity check and diff several other database to finally extract the missing packets. Thank you for dropping what you were doing and sending us a few GB of data over blazingly fast connections. Without you this would not have been possible; and this is not an exaggeration. Thank you!

After some more rest we’re going to continue to put out smaller fires that remain from the move to NewHost, but for now, the big fires are put out. Just in time for the weekend!

In the 11 year history of the replication stream we’ve had to have users restart their stream about 3-4 times because of problems on our end. Zero would’ve been nicer, but I’m proud that we’ve been able to make this system work for so long. On a daily basis we seem to have about 400 replicated copies of MusicBrainz running all over the world. Clearly this part of our service is well used and I sleep a little better at night knowing that our most critical data is backed up across the globe.

Just for fun, here is a graph of the replication API usage over the last 6 months:

hourly-api-usage.png

Towards the end the graph shows the week plus long break, then a small blip as some of our replicas got unstuck yesterday and the much larger spike shows the rest of the replicas getting unstuck. Now, as to what caused the blip in mid-October — I have no idea.

Anyways, please accept my apologies for the replication stream outage and keep replicating!

Thanks!

Server update, 2016-10-24

Today we have more URL cleanup and general link fixes from chirlu & yvanz. Thanks to those two again. 🙂 This’ll be the last musicbrainz-server release before we move to our new hosting facility. But that move should be done within two weeks, so hopefully the next release won’t be delayed. Thanks again for your patience.

The git tag is v-2016-10-24.

Bug

  • [MBS-7164] – URL cleanup doesn’t allow iTunes “Song” download links
  • [MBS-9044] – iTunes: cleanup “geo.itunes” and block/cleanup “linkmaker.itunes” links

Improvement

  • [MBS-6314] – iTunes sidebar links should specify country
  • [MBS-8802] – Clean up iTunes audiobook and podcast URLs
  • [MBS-9079] – Clean up Wikidata URLs with “/entity”

Task

  • [MBS-8269] – Disallow Commons gallery pages and categories
  • [MBS-9078] – Improve URL cleanup for Wikimedia Commons
  • [MBS-9087] – Link to current VIAF URL format

Server update, 2016-10-10

With a looming deadline for switching to our new hosting facilities (early November), we didn’t have any release last month, and this one is several days late. But we’ve got some good contributions from chirlu and yvanz, especially in the area of URL validation and cleanup, which we’re finally releasing. Much of yvanz’s work has been to refactor our URL cleanup code so that it’s stricter and more robust. Thanks, chirlu & yvanz! You can view the complete changelog below.

The git tag is v-2016-10-10.

Bug

  • [MBS-8744] – Block Generasia URLs from Releases
  • [MBS-9069] – Submitting cover art with a long comment fails silently

Improvement

  • [MBS-8796] – Set a referrer policy for HTTPS pages
  • [MBS-9012] – New Discogs image URLs need cleaning up into release URLs
  • [MBS-9053] – Clean up commons.m.wikimedia.org into the non-mobile site
  • [MBS-9072] – Verify that tport is an integer

Task

  • [MBS-5733] – Allow “otherdatabases” to be validated in URLCleanup
  • [MBS-5736] – Test that the JavaScript catches bad URL entries
  • [MBS-6378] – Add tests for validationRules constraints in URLCleanup.js

Server update, 2016-08-15

This release features some more URL cleanup improvements from Yvanz, as well as a couple UI fixes, and some new events in our timeline graph from Mineo. Thanks, Yvanz and Mineo!

The git tag for today’s release is v-2016-08-15. (Note: This blog post was published a day late.)

Bug

  • [MBS-8301] – Relationship tree misrepresented in the relationship editor
  • [MBS-9042] – Misaligned rows in editor profile general information using some languages

Improvement

  • [MBS-4864] – Add more events to the timeline

Task

  • [MBS-9038] – Add ChangeTip, Flattr, Patreon, PayPal.Me and Tipeee to the sidebar and autoselect
  • [MBS-9039] – Add Kickstarter and Indiegogo to the sidebar and autoselect

Server update, 2016-08-01

We have one URL cleanup fix this week for ResidentAdvisor links — thanks to Yvanz for working on that! We’re still working very hard on moving to a more reliable hosting platform, which is why this week’s release is a small one again. Most of our services have been ported to Docker at this point, and we’re planning to start testing the deployment of these soon. We appreciate everyone’s patience!

The git tag is v-2016-08-01.

Bug

  • [MBS-8148] – ResidentAdvisor reviews get selected as Other Database

Server update, 2016-07-04

This is a small bug-fix release. Server development has slowed down while we work on writing Docker containers for our new hosting infrastructure. But code contributions are always welcome. Thanks to Ulrich Klauer (chirlu) for his work on MBS-8806 this release.

The git tag is v-2016-07-04.

Bug

  • [MBS-8806] – Artist credits disappearing from tracklist & recordings after editing
  • [MBS-8987] – Untouched URLs are automatically/suddenly cleaned up
  • [MBS-8996] – Error message when trying to merge releases displays as “ARRAY(stuff)”
  • [MBS-8997] – Merging releases with “(Disc 1)”, “(Disc 2)” etc in titles selects all as disc 1
  • [MBS-8999] – Track lengths are in the wrong table column in the release editor

New Feature

  • [MBS-8994] – Allow passing &dismax=true through to the search server so Picard can show the same results as on the website

Server update, 2016-06-20

Our release today contains the usual lot of bug fixes and improvements. Thanks to yvanz, chirlu, and reosarevok for their contributions. The git tag is v-2016-06-20.

Bug

  • [MBS-7358] – Edit types not translated on Release editor / edit note tab
  • [MBS-7963] – Release editor doesn’t prevent too long disambiguation comments
  • [MBS-8809] – ASIN cleanup can be defective
  • [MBS-8951] – Recent notes is broken
  • [MBS-8968] – JSON version of artist includes gender-id value, which has no xml equivalent
  • [MBS-8977] – Edit searches with relationship type conditions cause an internal server error
  • [MBS-8983] – « Show only results that are in my subscribed entities. » checkbox broken
  • [MBS-8985] – Fix CPDL URLs matching and clean it up
  • [MBS-9000] – HTML special characters are not escaped on aliases page

Improvement

  • [MBS-7111] – Improve URL matching for musik-sammler.de links

Task

  • [MBS-8982] – Stop “fixing” piano with Guess Case

Server update, 2016-06-06

This server release contains fixes for bugs introduced in the recent schema change, and some other small improvements listed below. Thanks to reosarevok, chirlu, yvanz, and gcilou for their contributions this time around. The git tag is v-2016-06-06.

Bug

  • [MBS-7986] – Misleading error message for event setlists
  • [MBS-8941] – Beta: Last AC in a release sometimes doesn’t change with the rest when changing release artist
  • [MBS-8942] – Beta: cursor moves to the end of AC search field when removing a letter in the middle of the name
  • [MBS-8950] – iTunes favicon is not displayed on the sidebar
  • [MBS-8959] – Can’t add new packaging types – ISE

Improvement

  • [MBS-8824] – Update the rateyourmusic logo used in the sidebar
  • [MBS-8925] – Limit BookBrainz relationship to BookBrainz URLs

Task

  • [MBS-8957] – Update SecondHandSongs icon and add BBC logo in the sidebar