Month: September 2008

Next generation schema: Where we are today

Sorry for the delay in continuing my MusicBrainz Server Roadmap updates — we’ve been wrestling with some server configuration nightmares…

I’ll start by giving some background on the Next Generation Schema (NGS). People have been calling for an improved schema that can intelligently handle classical music, proper artist attribution and support for packages of releases (among many other things). In 2005 we held a summit in Germany where we laid down some groundwork for the requirements for a new schema and created a first rough draft schema. Holes were quickly poked into that schema, but this let us find the weak points and let us do a better job of designing the schema for the next attempt. The next attempt was an invite only summit held in London in 2007, where we created a schema that should be pretty close to what we’re actually going to put in place.

Since then we’ve been debating how to make NGS a reality. And to say the least, its been a real pain so far. The change from the existing schema to NGS is a very large project and will be far from trivial. For instance, do we migrate to NGS in one step or take a few steps? Can we keep our existing edit system or do we need to rewrite it from scratch? Now that we have the schema done, what should the user interface look like? How can we make it simple for those users who want to do simple data changes? How can we allow more expert users to make all sorts of detailed changes while keeping the user interface simple? What would the best tools for building a new UI be?

To answer these questions, Lukáš has been working on the NGS playground in his spare time. The NGS playground is Lukáš’ attempt to answer these questions. So far, he has answered two questions:

  1. We should move to NGS in one step. Moving to NGS in multiple step steps will cause too many headaches and too much lost work. Each step would require extensive work to glue the old portions to the new portions and these glue layers would later be discarded. Overall, not a very efficient means of moving forward.
  2. We need to dump the edit system and start over. We’ve learned a lot about how to do an edit system right — and the existing system won’t cut it moving forward. Its time to start over.

In the process, Lukáš is trying out a bunch of new tools to see how they fare. He’s also working in Python to see if that is the proper language to move forward with. While I haven’t run the NGS playground yet, Lukáš has the schema finished except for the Works concept, which is pretty straightforward. Also, I believe that the script to convert an existing database to the new format is also done. However, Lukáš’ work is only a playground — its an attempt to see how we can pull off the user interface for NGS. That’s NGS in a nutshell today.

In my next post I will continue this thread with thoughts on how we go from today’s mb_server and the Template Toolkit work and move towards the NGS work that Lukáš is doing. For now, if you’re a computer geek who is interested in looking at NGS, please take a look at the NGS playground. There isn’t much documentation yet — so consider this a “some assembly required” project. We’re interested to hear your comments on the playground — please feel free to post them here.

Our bug tracker is broken

Our bug tracker is broken at the moment. After a fussy afternoon of futzing with our server configuration we’ve not managed to bring back our bug tracker.

We’ll work to fix that first thing tomorrow morning. Sorry for the hassle.

UPDATE: Dave fixed the bug tracker and moved it to use MySQL as its backend database, which should make the setup less brittle than using sqlite. Thanks Dave!

Next server release: Collections, improved voting, ratings, last update/dashboard, improved tag user interface

The next server release will contain the following new features:

  1. Collections — this is Niklas’ SoC project. Collections will allow MusicBrainz users to indicate which releases are in their collection and have MusicBrainz show which releases for a given artist are missing. This feature will also allow you to watch a list of artists and when a new release for your watched artist is entered into MusicBrainz, you will receive an email alerting you of the new release. This new feature will go on public test hopefully sometime this week after we resolve issues with our test server. For details visit the in-progress documentation page or visit Niklas’ blog.
  2. Improved voting — Murdos has been working on a number of improvements to make voting easier. I haven’t had the chance to look at his work yet, but I will try to make this new feature available on our old test server. Stay tuned for details and a chance to test.
  3. Ratings — Murdos has also picked up where Sharon from last year’s Summer of Code left off. Rating music will give us a number of benefits and gives us fuel to create more advanced features like collaborative filtering for artists. Once this work is complete we’ll upload it to a test server and let you play.
  4. Last update/Dashboard — I’m working on a feature that keeps track of when data in MusicBrainz was changed. This then makes it possible to create a Dashboard page that shows recently changed data in MusicBrainz. The Dashboard will also feature hot edits (edits that receive a lot of votes/comments) and edit statistics to give our users a better view into what data is currently changing inside MusicBrainz. For details, see the documentation page.
  5. Improved tag interface — After last year’s new tag feature, we’ve collected a lot of feedback on how to improve the tags. I will work to improve the tag features to make them easier to use and more visible the MusicBrainz interface.
  6. Bug fixes — We have already fixed a number of bugs and will tackle many more bugs before the release.

There will probably be other features in the next release, but these are the major features we have planned right now. Unfortunately we do not have a release date set yet. However, the next server release is my highest priority right now.

Stay tuned for more blog posts about new features going into testing!

UPDATE: Voiceinsideyou adds that you can see the bugs that are currently slated to be fixed for the upcoming release here in our bug tracker.

Dump PayPal and go to Google Checkout?

The MetaBrainz PayPal account has become a testing ground for people to test stolen credit cards. We were constantly getting tons of donations (dozens per DAY!) of tiny donations at first. Then they turned to $1 donations. And now they are $30-$40 per piece. And each and every one of these gets challenged and the money turned back over to PayPal. In the process I get about 3-5 emails in total about for each of these donations. Plus it clutters our books with tons of useless entries that make oversight of the books a real pain. I’m really sick of this!

Google offers their checkout services and even offers special deals for non-profits — I would very much like to switch over to Google Checkout. Can you think of any reason to not dump PayPal and go with Google Checkout?

General MusicBrainz update: We've hired Oliver Charles!

Its been a busy summer behind the scenes at MusicBrainz! All this week I want to post a series of blog entries to catch you up on what we’ve been up to and how I expect us to move ahead with development of new MusicBrainz features. Over the summer we had three students working on various Google Summer of Code projects, making all sorts of interesting progress. In today’s blog post I’ll cover Oliver Charles’ (Acid2) work on his Template Toolkit branch. Later this week I will cover other SoC projects and current work on NGS in an effort to lay out a roadmap for future MusicBrainz development.

For lots of details on Oliver’s work, please check out his blog. Without delving into details here, I’ll say that his work is an important stepping stone in getting to the fabled NGS. Many people have commented that the current mb_server codebase is too confusing and too hard to setup. This has the effect of keeping developers away from hacking on the server. The general consensus was that we needed to clean up the codebase to make it easier to follow, separate business logic from the user interface logic and to use some more modern tools.

Oliver worked to address all of these points over the summer. We were clear on the fact that this project was too much to finish in the time that was available, so that the project would remain unfinished at the end of summer. Fortunately, MetaBrainz has been signing up more customers, so we finally have enough money in the bank to hire a coder to keep working on new server features. To that end, we’ve just hired Oliver part time to continue working on his project! Currently, the goal is to finish his Template Toolkit work by the end of the year and to start a beta test phase at the beginning of next year.

Hiring a developer to continue hacking on new features for the server is really important. My time in the last year has been taken up by keeping MusicBrainz running and keeping our search servers from melting. Unfortunately we’re coming up on a year without a server update, which is tragic. Having money in the bank should allow us to finally speed up the pace at which we’re updating our servers. Ideally I would like to have 4 updates per year — we may not reach that just yet, but I can see three releases happening in 2009!

Tomorrow I will post an update on Niklas’ collection work and the upcoming server release. Stay tuned.

For now, congratulations to Oliver Charles and welcome as an independent contractor to the MetaBrainz Foundation!

Planned downtime: Friday Sept 12, 14:00 PDT, 21:00 GMT, 22:00 BST

Tomorrow, Friday Sept 12 MusicBrainz will be down intermittently for about 30 minutes from 14:00 PDT/21:00 GMT/22:00 BST. We’re going to migrate to our new switch, which means that various portions of MusicBrainz will be unavailable for brief periods of time. Hopefully the overall unavailability of the service will be a lot closer to 5 minutes than 30 minutes. As usual, we will do our best to keep the interruptions brief.

Sorry for the inconvenience.

Update at 22:49 BST by djce: we’re done – it all went according to plan.  The site was down for just under 15 minutes.  Thank you for your patience.

Amazon opens "SoundUnwound" to public beta, and becomes a MetaBrainz customer!

Robert Kaye writes:

I’m pleased to announce that Amazon became a customer of the MetaBrainz Foundation last year! They’ve been quietly working on a site to promote and let users discover new music: SoundUnwound.

Amazon is now ready to let the world play with their new toy — but please be aware that the site is only in beta and may still contain some snags. I had the pleasure of previewing their work when I visited them in May and I was quite impressed with their methods of presenting information about music. I think MusicBrainz can learn a few things from their work.

However, I would like to stress that Amazon is currently only using the MusicBrainz data for SoundUnwound. I know there are a few people in our community (myself included) who would like to see Amazon use MusicBrainz data in their main store. That is currently not a point of discussion — the cooperation between Amazon and MusicBrainz is limited to SoundUnwound.

I’ll continue to work to convince Amazon that they should use our data in more of their operations, but for now, let’s take one step at a time!

Welcome on board, Amazon!