Upgrading Postgres for MusicBrainz Live Data Feed users

We’re slowly approaching that time of year: Schema change release time. After skipping our fall update to focus on some internal tasks, we’re ready to have another schema change release in the spring: May 16, 2016

We have started the process to collect features we wish to release for this schema change release and we’ll be publishing that list in the coming weeks. However, we’re contemplating the impact of one more change we’d like to make: Upgrading to a more recent version of Postgres.

Internally we are going upgrade to Postgres 9.5, which was recently released, so we expect that the Postgres team will have worked out the most significant kinks before we’re ready to move to it. However, even though we are moving to 9.5, we are considering the impact on our downstream users/customers who need to make the same or similar change.

While we are moving to version 9.5 of Postgres, we have the option of only adopting features from Postgres 9.4, which means that our downstream users may continue to use Postgres 9.4. However, Postgres 9.5 has some nice features we’d like to use (e.g. UPSERT), so we’re pondering if it is possible for us to require Postgres 9.5 from all of ours Live Data Feed users starting on May 16, 2016. 

We have already informally queried a few of ours users and so far it seems that requiring Postgres 9.5 is feasible. If you are a Live Data Feed user and feel that this requirement of Postgres 9.5 is too much for your and your organization by May 16, 2016, please leave a comment to this blog post!

Notifications and messaging in MetaBrainz projects

During the last MusicBrainz summit in Barcelona we decided to start working on finding possible ways to implement two features that have been requested for a long time:

  1. Messaging between users
  2. Notifications about various actions in MetaBrainz projects

Since MetaBrainz is more than just MusicBrainz these days, we also want to integrate these features into other projects. That, for example, means when a user is reading reviews on CritiqueBrainz they can see notifications about comments on their edits on MusicBrainz. Same applies to messaging. These features are intended to encourage our communities to communicate more easily with each other.

Messaging

http://tickets.musicbrainz.org/browse/MBS-8721

The only ways of communication we have right now are two IRC channels, forums that we plan to replace with Discourse, and comments on individual edits. Sometimes we end up sending private emails to editors for one reason or another. Perhaps it is better to have our own messaging system for this purpose? I imagine it being similar to messaging systems on forums, reddit, etc. We would like to know what you think potential uses are for this and how it might look like to be useful.

Notifications

http://tickets.musicbrainz.org/browse/MBS-1801

Site-based notifications are another thing that people have been asking for a long time. For example, these notifications can be related to edits on MusicBrainz, reviews on CritiqueBrainz, datasets in AcousticBrainz, etc. It can be an addition or replacement for email notifications that we currently have in MusicBrainz. Maybe something similar to the inbox feature that the Stack Exchange network has. People should be able to choose if they want to keep receiving email notifications or only use the new site-based notifications.

Progress so far

We looked at a couple of ways to implement this functionality.

First suggestion was to use the Layer toolkit. The problem with it is that we don’t want to be dependent on closed software and another company’s infrastructure, especially in case of such important features.

Second was to use the XMPP protocol to handle communication and notifications. We tried to implement a proof of concept using this protocol and encountered several issues at the start:

  • It’s unclear how to store messages and process them later;
  • It can be problematic to reuse the same connection in different browser;
  • There are plenty of things that we’ll need to implement on top of this protocol ourselves (like authentication, storage, notifications).

Repository with everything that was implemented so far is at https://github.com/metabrainz/xmpp-messaging-server. Given these problems we started considering implementing our own server(s) for this purpose.

You can take a look at the document where we collect most information about current progress.

Feedback

There’s plenty of feedback on the site-based notifications feature request, and we have a pretty good understanding of what’s needed. This is not the case with the messaging feature. We explored several options for implementing this kind of functionality and decided that it’s time to refresh the list of requirements to get an idea of what needs to be done.

The goal of this blog post is to encourage discussion and gather ideas. If you are interested in these features, please share your thoughts and suggestions.

Schema change release, 2011-07-11

Today we released our first schema change update since NGS. This change is quite a radical one, as it merges both of our databases (“READWRITE” and “RAWDATA”) into a single database. For most users of the database, this probably won’t affect you, but you’re encouraged to run the upgrade process anyway. Here’s what you need to do:

  1. Take down the web server running MusicBrainz, if you’re running a web server.
  2. Turn off cron jobs if you are automatically updating the database via cron jobs.
  3. Set DB_SCHEMA_SEQUENCE to 13 in lib/DBDefs.pm
  4. Make sure your REPLICATION_TYPE setting is RT_SLAVE
  5. Switch to the new code with git fetch origin followed by git checkout v-20110711-schema-change
  6. Run ./upgrade.sh from the top of the source directory.
  7. Install the perl modules Algorithm::Merge and Algorithm::Diff
  8. Turn cron jobs back on, if needed.
  9. Restart the MusicBrainz web server, if needed.

This process may take a while, as it has to dump one database into another, and download a few extra changes to ensure slaves aren’t missing any data. The RAWDATA database should no longer be in use and you should be able to drop it, but waiting to see that everything is working well might be a good idea.

This schema change does not introduce any new data. For everyone else, here’s a list of what got fixed since the last release!

Bug

  • [MBS-1977] – ModBot is unable to close some edits
  • [MBS-1979] – Unable to edit a “later translated versions” relationship with change direction
  • [MBS-2026] – Subscribed artists open edits won’t load for editors with large amounts of subscriptions
  • [MBS-2442] – MB postgres unaccent extension overwrites unaccent.so library shipped with postgres-contrib
  • [MBS-2689] – Timeline isn’t working in Opera 10
  • [MBS-2698] – Some tables are not replicated through Live data feed
  • [MBS-2812] – Adding track times has caused a failed dependency for an edit which changed the track titles.
  • [MBS-2826] – Web service returns malformed XML (not escaped properly)
  • [MBS-2831] – “Edit medium” edit display: Artist credits changes are incorrect
  • [MBS-2974] – I don’t receive e-mail when someone votes no to my edit any more
  • [MBS-2981] – Changing the case of recording comments should be auto-edits
  • [MBS-2986] – tracklist_index was not populated during NGS
  • [MBS-2995] – Caught exception in MusicBrainz::Server::Controller::WS::2::ReleaseGroup->release_group_browse “Can’t call method “format”

Improvement

  • [MBS-1500] – Remove tracklist_index.tracks
  • [MBS-1707] – Advanced tracklist in RE: “Title” to “Disc title”
  • [MBS-2242] – Disable editing of Medium title when there’s only one medium
  • [MBS-2434] – Combine READWRITE and RAWDATA
  • [MBS-2462] – Other edit types that don’t highlight what has changed between old and new values
  • [MBS-2583] – “Edit Medium” should show you which medium is being edited with respect to the overall release
  • [MBS-2767] – Release-group XML result should include first release date