Hosting issues & downtime tonight

For the past week we’ve been battling a variety of hosting issues from search servers acting up, gateways dropping packets and now our database server freaking out for no particular reason. We managed to fix/mitigate the issues in the first hiccups, which is good. And for three days everything looked peachy. Then, out of the blue our database server did this:

DB server load

You don’t have to understand much about hosting computers to understand that this is bad. Out of the blue our server started having to work much harder than before. Normally that means that something using the server has changed. We’ve looked for a traffic increase — we can’t find one. We’ve examine someone being abusive to us — we found a couple users, but blocking them didn’t change anything. We tried turning off non-essential services that make use of the database server, but nothing ever changes. We’ve restarted the database server. We’ve slapped this, we’ve poked that, we’ve prodded, undid and tested just about everything we can think of. But, the load comes in spikes and recedes again; over and over.

We’ve had amazing help from a number of people, but several skilled computer geeks with a support from lots of others haven’t managed to make a dent in things. We’re exhausted and we need a bit of a break. So, that’s what we’re doing for the next 7 or so hours.

Then at 15h PDT, 18h EDT, 22h UK, 23h CET we’re going to start to upgrade to the latest version of Postgres 9.1. We hope to be down for less than half an hour — but you never know. We’ll tweet about the downtime and put up a banner on the MusicBrainz site to let people know when exactly we’ll take the site down.

Sorry for the hassle — we’re all amazingly frustrated right now — please bear with us.

Server update, 2014-08-18

A day late with putting this out, due to some sleep mis-scheduling on the part of yours truly, but we’re back with another release! This release includes a variety of small bug fixes and improvements, as well as a reworking of the track parser, which now supports a more complete set of options to control exactly which information is parsed and which is updated in the tracklist. Thanks to bitmap, chirlu, ianmcorvidae, nikki, and reosarevok for their work on this release!

The git tag for this release is v-2014-08-18.

Full list of issues in this release:

Bug

  • [MBS-7753] – Username displays wrongly on area pages
  • [MBS-7799] – unable to add artist as database entry
  • [MBS-7800] – Last.fm URL cleanup doesn’t work for .br and .com
  • [MBS-7804] – URL cleanup for places doesn’t validate links to Discogs or Other Databases
  • [MBS-7805] – series-rels inc parameter doesn’t work since WebServiceInc was not updated

Improvement

  • [MBS-5925] – Show whether you’re subscribed to a collection on the collection overview
  • [MBS-6705] – Move “No linguistic content” lyrics language from “Other” to “Frequently used”
  • [MBS-7270] – The option to parse vinyl track numbers can be confusing
  • [MBS-7749] – Rap Genius is now Genius
  • [MBS-7797] – When editors are not referenced in the DB, delete their rows entirely rather than renaming to Deleted Editor #N
  • [MBS-7798] – Release relationship editor needs a loading indicator
  • [MBS-7803] – Add reports for entities with annotations

New Feature

  • [MBS-3730] – Feature parity: Please re-add the ability to parse only times
  • [MBS-3732] – Feature parity: Please re-add the ability to parse only titles
  • [MBS-4921] – Add a [No lyrics] option to works language

Task

  • [MBS-7785] – Update ESTER to account for the merger of its branches

Brief downtime this Friday/Saturday, at 2 UTC (19 PDT/22 EDT/4 CEST)

As some of you with slave databases may have noticed, we made a slight error and excluded some of the new tables from our schema change in replication (tracked by MBS-7603).

Luckily, the tables in question are presently quite small, meaning that if we act now we can add them without needing a formal schema change release (by simply replicating out the changes). Less luckily, this process requires taking some very heavy locks on the ‘series’ database table, so we’ve chosen to have a small amount of downtime to run the script, at (or shortly after) the time mentioned in the title, which also gives us a chance to restart our database server to incorporate some extensions we’d intended but failed to add during the schema change release last week.

We don’t foresee this process taking more than a few minutes, and no action should be necessary for downstream data users.

Server update, 2014-04-14

A bit later in the day than I’d like, but we’re back with another release! Primarily bug fixes, but this release also includes the long-in-progress work to derive recording durations from track durations; thanks to ocharles and bitmap for dragging that through the various parts of the process! Thanks also to d4rkie, reosarevok, mavit, and chirlu for providing patches on top of the work of the MetaBrainz team.

The git tag for this release is v-2014-04-14.

Full notes:

Bug

  • [MBS-4021] – /ws/2/release doesn’t include artist aliases for recordings on Various Artists releases
  • [MBS-7064] – Internal server error displaying relationship type edit
  • [MBS-7068] – Need Email::Sender 0.120002 or newer
  • [MBS-7297] – Internal server error when searching for edits by ID with large numbers
  • [MBS-7315] – Removing URL entity via Edit Release page results in SQL error
  • [MBS-7386] – Regression: Attach TOC to new release – TOC/DiscId is lost when based on existing release/tracklist
  • [MBS-7390] – ArtistsThatMayBeGroups report needs to ignore orchestras and choirs
  • [MBS-7392] – Place duplicate checks don’t work
  • [MBS-7393] – mora.jp parser makes wrong links
  • [MBS-7395] – Works that still have work attributes can’t be deleted
  • [MBS-7410] – Merge queue with more than one entry shows the wrong help text
  • [MBS-7416] – Edit-medium diff shows recording changes that aren’t actually changes
  • [MBS-7420] – Can’t submit artist edit form after adding and removing a URL
  • [MBS-7426] – Tiny distracting flaw in Edit Recording page ISRC help text
  • [MBS-7429] – Ambiguous field ‘id’ in search query involving edit ID BETWEEN and “Artist in subscribed editors”
  • [MBS-7431] – URL encoding problem with inline URL editing

Improvement

  • [MBS-2021] – Set recording times automatically
  • [MBS-2302] – Set recording length when adding a Disc ID
  • [MBS-7150] – Indicate artists on the release group and recording parts of the ratings page
  • [MBS-7153] – Indicate artists on the release group and recording parts of the tags page
  • [MBS-7291] – Detect and mark big duration differences on recording merge edits
  • [MBS-7391] – Remove verbiage from /entity/create
  • [MBS-7417] – Add a visual indicator when times are very different in release merge edits

New Feature

  • [MBS-3987] – Disable recording duration editing

Task

  • [MBS-7425] – Add maniadb.com to the other dbs whitelist