Category: Development

Call for testing: November 23 server release

Murdos, Niklas, Luks, Dave Evans and I have all been working on the upcoming server release and we would like to invite you to help us test this massive release. Most of the features for this release have been debugged, but a few bugs are still outstanding.

The test server has been updated with all of the latest features and is ready for you to log in with your normal user account with the password ‘mb’. The list of features in this release is described in this blog post and this post.

Submit new bug reports to our bug tracker. Also see, the milestone for this release to see which bugs have been fixed and which bugs are still open. Please check the open bug list before reporting a new bug.

Thanks to everyone who helped take part in the development for this release!

One more feature from good measure: CD Stubs

The final feature to be included in the upcoming release are CD Stubs. From the documentation page:

Many times people would like to contribute CD data to MusicBrainz, but would prefer to not learn to be part of MusicBrainz. These people would rather just toss some data into a pile and go on with their lives to listen to their music or rip their CDs. CD Stubs enable users to do just that.

With this release people who come to look up a CD, but MusicBrainz does not know about that CD, will be given a choice to either enter the CD as a CD Stub or to join MusicBrainz and to enter the CD into MusicBrainz proper. (This is the red pill/blue pill decision point). Note that users who are logged in to MusicBrainz when looking up a CD will not be given the option to enter a CD Stub.

To submit a CD Stub to the test server, use any MusicBrainz enabled CD lookup application to submit a CD to MusicBrainz. You must not be logged into the test server, otherwise you will never be given the option to submit a CD Stub. Then follow the instructions for how to submit a CD Stub.

To retrieve the CD Stub you must use the XML Webservice. CD Stubs are not available via the old RDF web service. Use this URL to fetch data for CDs that will also return CD Stubs if no CDs are available in the main DB:

http://[test.]musicbrainz.org/ws/1/release?type=xml&discid=[discid]&toc=[toc]

This URL is also available via libdiscid and its example program discid.

Discography, ratings, enhanced voting, dashboard, timeline and related tags now on test server

Murdos has been busy merging the various development branches into trunk — thanks for your work. I’ve updated the test server with the latest codebase. Come check out the latest new features:

(to log in use your normal log in name and password ‘mb‘)

We’ve got more bug fixes coming in the next couple of weeks. Also, the next server release has been scheduled for 2008-11-23. As usual, please report bugs to our bug tracker.

UPDATE: I had to clear everyone’s collections because of a bug. That’s fixed now — please start over again.
UPDATE2: Due to complaints by stodgy brits, Music Newz is now called Music News. 🙂

Running mb_server on Apache2

A number of people have expressed interest in running mb_server on Apache 2, rather than Apache 1 as mb_server currently requires. Our current roadmap calls for the next mb_server release to stay on Apache 1 and the following Template Toolkit branch to move to Apache2. There will be a lot of changes to the codebase going to TT, so it makes sense to make the switch to Apache 2 at that time since we have to test everything anyway.

LAADHARI Saber has created a “quick and dirty” port of mb_server to apache2. The patch uses the Apache2 compat module and still has some problems with Auth.pm in the Web Service.

Given our roadmap, we’re not going to accept this patch into our codebase, but I wanted to offer up this patch to anyone who wanted to use it.

Thanks LAADHARI Saber!

Server roadmap: SVN branches

If you’re not interested in following server development as it pertains to our SVN repository, then skip this post.

Today, Lukáš, Oliver (acid2) and I had a bit of a chat, discussing how to proceed with branches in our SVN repository. The chat ambles along for a while, so I’ll recap our plans here:

  1. Oliver will revert some of the refactoring changes he has made to the TemplateToolkit branch, which throw a bit of a monkey wrench into starting new branches.
  2. Lukáš will merge the old 20071014 (current live) branch back to trunk and the start a new branch for my work on the next release.
  3. Lukáš will then port bits of the TT branch to trunk so that he can base his work for Release Groups on the object model work in the TT branch.
  4. Lukáš the proceeds to work on Release Groups.
  5. Oliver continues to work on TT, sans the refactorings.
  6. I continue to work the next release in my branch. Ratings, discography and enhanced voting will be merged into the next release branch, not into trunk.

This gives us the flexibility for each one of us to work into their direction and it doesn’t commit us to the TT branch. While all things point to TT being the basis for our next release, its good to keep our options open for now. I’ll post a follow up post once these branches have been properly put in place.

Thanks Lukáš and Oliver!

NGS: From here to there

[ Before reading this post, make sure to read the previous NGS related post ]

The question that is on my mind right now is how to build a coherent roadmap that gets us from the mb_server codebase that we’re running today to the NGS codebase, complete with new edit system. The factors that play into this are:

  1. mb_server codebase: This is the codebase that we’re running today. We’re updating it one more time this year and then early next year we hope to move to the Template Toolkit work.
  2. Template Toolkit: This is Oliver Charles’ work to clean up our codebase. Template Toolkit is available for perl and looks like it will be available for Python soon. Our hope is the clean up the codebase so that we’re ready to take on more developers to help with the development — especially as we move closer to NGS.
  3. NGS playground: See the previous post for details on this.
  4. NGS proper: This is the finished NGS that we roll out onto the MusicBrainz servers.

Finally, the BBC has been keen on getting what they are calling Cultural Identifiers. This name is a bit of a misnomer — essentially it would be the release related portions of NGS. Release groupings that allows us a more product centric approach to managing releases. Right now we list and identify releases with different track layouts as totally separate releases, even though they ought to be properly related. The BBC wishes this work to happen sooner than later and have indicated that they would be willing to sponsor this work.

That’s awesome, right??

Well, yes. But there is one problem. In the last post we concluded that we should move to NGS in one fell swoop. And now the BBC would like us to take an intermediate step? As much as we agreed that moving to NGS in one step, I think we must work with our most visible partner. Since we are severely resource constrained (we have just enough money to hire a part time University student right now) I feel compelled to find a way to get the BBC what they want as soon as possible while accepting money from them to boost our development funds. Taking money from the BBC may allow us to accelerate our development schedule towards NGS. But at the same time, it may slow us down getting to NGS.

I’m very much looking for feedback on how to best make this happen and how to best accomplish all these goals. Do you think that adding an intermediary step in exchange for funds from the BBC is an acceptable compromise?

Next generation schema: Where we are today

Sorry for the delay in continuing my MusicBrainz Server Roadmap updates — we’ve been wrestling with some server configuration nightmares…

I’ll start by giving some background on the Next Generation Schema (NGS). People have been calling for an improved schema that can intelligently handle classical music, proper artist attribution and support for packages of releases (among many other things). In 2005 we held a summit in Germany where we laid down some groundwork for the requirements for a new schema and created a first rough draft schema. Holes were quickly poked into that schema, but this let us find the weak points and let us do a better job of designing the schema for the next attempt. The next attempt was an invite only summit held in London in 2007, where we created a schema that should be pretty close to what we’re actually going to put in place.

Since then we’ve been debating how to make NGS a reality. And to say the least, its been a real pain so far. The change from the existing schema to NGS is a very large project and will be far from trivial. For instance, do we migrate to NGS in one step or take a few steps? Can we keep our existing edit system or do we need to rewrite it from scratch? Now that we have the schema done, what should the user interface look like? How can we make it simple for those users who want to do simple data changes? How can we allow more expert users to make all sorts of detailed changes while keeping the user interface simple? What would the best tools for building a new UI be?

To answer these questions, Lukáš has been working on the NGS playground in his spare time. The NGS playground is Lukáš’ attempt to answer these questions. So far, he has answered two questions:

  1. We should move to NGS in one step. Moving to NGS in multiple step steps will cause too many headaches and too much lost work. Each step would require extensive work to glue the old portions to the new portions and these glue layers would later be discarded. Overall, not a very efficient means of moving forward.
  2. We need to dump the edit system and start over. We’ve learned a lot about how to do an edit system right — and the existing system won’t cut it moving forward. Its time to start over.

In the process, Lukáš is trying out a bunch of new tools to see how they fare. He’s also working in Python to see if that is the proper language to move forward with. While I haven’t run the NGS playground yet, Lukáš has the schema finished except for the Works concept, which is pretty straightforward. Also, I believe that the script to convert an existing database to the new format is also done. However, Lukáš’ work is only a playground — its an attempt to see how we can pull off the user interface for NGS. That’s NGS in a nutshell today.

In my next post I will continue this thread with thoughts on how we go from today’s mb_server and the Template Toolkit work and move towards the NGS work that Lukáš is doing. For now, if you’re a computer geek who is interested in looking at NGS, please take a look at the NGS playground. There isn’t much documentation yet — so consider this a “some assembly required” project. We’re interested to hear your comments on the playground — please feel free to post them here.

Next server release: Collections, improved voting, ratings, last update/dashboard, improved tag user interface

The next server release will contain the following new features:

  1. Collections — this is Niklas’ SoC project. Collections will allow MusicBrainz users to indicate which releases are in their collection and have MusicBrainz show which releases for a given artist are missing. This feature will also allow you to watch a list of artists and when a new release for your watched artist is entered into MusicBrainz, you will receive an email alerting you of the new release. This new feature will go on public test hopefully sometime this week after we resolve issues with our test server. For details visit the in-progress documentation page or visit Niklas’ blog.
  2. Improved voting — Murdos has been working on a number of improvements to make voting easier. I haven’t had the chance to look at his work yet, but I will try to make this new feature available on our old test server. Stay tuned for details and a chance to test.
  3. Ratings — Murdos has also picked up where Sharon from last year’s Summer of Code left off. Rating music will give us a number of benefits and gives us fuel to create more advanced features like collaborative filtering for artists. Once this work is complete we’ll upload it to a test server and let you play.
  4. Last update/Dashboard — I’m working on a feature that keeps track of when data in MusicBrainz was changed. This then makes it possible to create a Dashboard page that shows recently changed data in MusicBrainz. The Dashboard will also feature hot edits (edits that receive a lot of votes/comments) and edit statistics to give our users a better view into what data is currently changing inside MusicBrainz. For details, see the documentation page.
  5. Improved tag interface — After last year’s new tag feature, we’ve collected a lot of feedback on how to improve the tags. I will work to improve the tag features to make them easier to use and more visible the MusicBrainz interface.
  6. Bug fixes — We have already fixed a number of bugs and will tackle many more bugs before the release.

There will probably be other features in the next release, but these are the major features we have planned right now. Unfortunately we do not have a release date set yet. However, the next server release is my highest priority right now.

Stay tuned for more blog posts about new features going into testing!

UPDATE: Voiceinsideyou adds that you can see the bugs that are currently slated to be fixed for the upcoming release here in our bug tracker.

Unplanned Downtime

It looks like the main web site has dropped off the ‘net – most likely the server has crashed. I’ve asked the good people at Digital West to reboot the server. Please bear with us… hopefully we’ll be back up soon!

Update: the server is back (so it was down for just over an hour).

Update 2: We have two new servers on order to bring much needed redundancy to the site. So, the next time our flaky web server crashes the site should keep running. We hope to have those machines in rotation early in September. (read: after Burning Man. 🙂 )