Category: Development

Next server release: Collections, improved voting, ratings, last update/dashboard, improved tag user interface

The next server release will contain the following new features:

  1. Collections — this is Niklas’ SoC project. Collections will allow MusicBrainz users to indicate which releases are in their collection and have MusicBrainz show which releases for a given artist are missing. This feature will also allow you to watch a list of artists and when a new release for your watched artist is entered into MusicBrainz, you will receive an email alerting you of the new release. This new feature will go on public test hopefully sometime this week after we resolve issues with our test server. For details visit the in-progress documentation page or visit Niklas’ blog.
  2. Improved voting — Murdos has been working on a number of improvements to make voting easier. I haven’t had the chance to look at his work yet, but I will try to make this new feature available on our old test server. Stay tuned for details and a chance to test.
  3. Ratings — Murdos has also picked up where Sharon from last year’s Summer of Code left off. Rating music will give us a number of benefits and gives us fuel to create more advanced features like collaborative filtering for artists. Once this work is complete we’ll upload it to a test server and let you play.
  4. Last update/Dashboard — I’m working on a feature that keeps track of when data in MusicBrainz was changed. This then makes it possible to create a Dashboard page that shows recently changed data in MusicBrainz. The Dashboard will also feature hot edits (edits that receive a lot of votes/comments) and edit statistics to give our users a better view into what data is currently changing inside MusicBrainz. For details, see the documentation page.
  5. Improved tag interface — After last year’s new tag feature, we’ve collected a lot of feedback on how to improve the tags. I will work to improve the tag features to make them easier to use and more visible the MusicBrainz interface.
  6. Bug fixes — We have already fixed a number of bugs and will tackle many more bugs before the release.

There will probably be other features in the next release, but these are the major features we have planned right now. Unfortunately we do not have a release date set yet. However, the next server release is my highest priority right now.

Stay tuned for more blog posts about new features going into testing!

UPDATE: Voiceinsideyou adds that you can see the bugs that are currently slated to be fixed for the upcoming release here in our bug tracker.

Unplanned Downtime

It looks like the main web site has dropped off the ‘net – most likely the server has crashed. I’ve asked the good people at Digital West to reboot the server. Please bear with us… hopefully we’ll be back up soon!

Update: the server is back (so it was down for just over an hour).

Update 2: We have two new servers on order to bring much needed redundancy to the site. So, the next time our flaky web server crashes the site should keep running. We hope to have those machines in rotation early in September. (read: after Burning Man. 🙂 )

New search servers

I’m happy to say that we’ve now got extra search servers – a failover pair, in fact, so we’re no longer reliant on a single search server. In fact shortly we hope to bring a third server into the pool too. Not bad considering that, so far, we’ve only ever had one (or zero) search servers.

What this means for you is that searches should be faster. It also means that the future performance of the web site is now more assured than it was before – we’re in a substantially better position to handle extra traffic.

Most of the work was done by Robert Kaye; I only helped to polish off the edges 🙂

Squashing the rise of the sock puppets

We’ve recently seen a rise in Sock Puppets here at MusicBrainz. We’ve observed editors creating separate sock puppet accounts who vote through the edits of the editor in order to get changes through MusicBrainz faster. This practice obviously side-steps our peer-review system, and up until now we’ve had to have other editors go through and follow the trails of naughty editors to clean up after them.

To avoid this from happening continually, we’ve update the main server with a minor patch that requires people to have more than 10 approved edits in order to vote on other people’s edits. This makes creating a sock-puppet account much harder — each sock puppet account created will need to have a lot of work invested in it before it can be useful. We’re hoping that this simple tweak will discourage sock puppeteers.

Call for search server testing

After I gave some history in the last post, I’d like to put out a call for testing for the new search server. In moving from Lucene to Xapian I’ve fixed a number of bugs, some of which have been lingering for a while. Also see the list of bugs we still have open and plan to fix before the release.

If you have a pet-peeve bug that’s been annoying you, please check to see how our new Xapian test server is handling things now. (Please be patient with our the dev server, the box needs an upgrade soon!)

If you are a fluent speaker in Chinese, Japanese, Korean or Thai, please take a moment to look up some artists! We had some problems with searching Chinese text, but I think I fixed it, but I am not proficient in any of the applicable languages, so please help sanity check me!

Unless I find more bugs, this new search server will go into production sometime next week. If you find a bug, please report it to the usual place.

Search: Why is it so important?

After many days of tinkering, the new search server has passed its tests and is nearly ready for deployment next week. After my last post on the search services, there were lots of questions, so I’ll give some more history on why I’m working on this now:

  1. The old Lucene based search services worked well, but installing them was a major pain. Installing compilers by hand, sacrificing chickens and hoping that things would work wasn’t my idea of fun.
  2. Lucene has a philosophy of working out of the box without significant tweaks. That’s great if you’re indexing a bunch of text, but indexing music metadata from an SQL database is a bit of a different beast. The usual Lucene tricks didn’t work so well for us, so we couldn’t tweak it to work better for us. Xapian requires a little more tuning out of the box, but our search results are much better now than they were before.
  3. Sending metadata lookup traffic to a service like Xapian is generally a good idea, as a single Xapian server can handle lookup traffic more elegantly than a Postgres database. And adding more search servers is easier than adding more database servers.
  4. Our traffic is growing — I expect us to handle twice as much traffic in July as we did the July before. A lot of this traffic growth is coming from people using our web-service to look up music. If the web-service slows down, the rest of the site slows down as well. So I’m trying to stay ahead of the curve an anticipate when we reach capacity and be able to add more machines as necessary

As of next week, MusicBrainz will have twice as much rack-space (20U’s of space!) and we can finally rack the two new servers that were donated a few months ago. Fortunately due to dropping bandwidth costs, this new space doesn’t really come at a greater expense to us — I expect our hosting costs to stay nearly the same as they are now. (about $1000/mo, btw)

This will allow us to have 3 times the search capacity we have now, which should keep the site working for a while longer. In fall I hope to start moving our web-service to Amazon’s EC2 service, which should allow us to get as much capacity as we need.

As soon as I get the new search services deployed I’m putting my head down and coding the next server update. So, keep your fingers crossed that this process goes smoothly.

Discographies database schema review

If you’re interested in Niklas’ Summer of Code project to implement Discography support in MusicBrainz, I would suggest that you follow his blog and read his latest entry: “Database design and a question to users“. Niklas and I have been working on the design of the database tables that will enable his SoC project. We think we’re collecting the right information, but I’m nearly always wrong. So, if you have database design experience, please take a look at this latest post has tell us just how wrong we are. 🙂

Mac OS X Developer for Picard releases wanted

Its clear that I won’t find the time to package up Picard for OS X anytime soon. I’ve put out one Intel based DMG, but haven’t found the time to create a Universal Binary package of Picard. 🙁

If you have the following:

  • Knowledge of building Mac OS X Application Bundles
  • Python knowledge
  • Love for Picard
  • Access to Intel and PPC Macs

We would very much like to talk to you. The last item isn’t crucial — I suppose we can get people in the community to test your builds for platforms you have no access to. Please leave a comment if you’re interested in helping out.

Come play with a new search engine

Do you have a search bug that really annoys you? If so, please come help me test a new search engine!

I’ve ported our search services to a new text search engine called Xapian. While the indexes are bigger on disk, it is easier to install, much faster to index and probably also faster to search. And, over Lucene it has vastly fewer problems. And you can perform stop word searches!

Come play with it on my dev server! (Never mind the connection being slow and indexes being a few weeks old) Report issues to the usual place please!