Category: Server

Call for search server testing

After I gave some history in the last post, I’d like to put out a call for testing for the new search server. In moving from Lucene to Xapian I’ve fixed a number of bugs, some of which have been lingering for a while. Also see the list of bugs we still have open and plan to fix before the release.

If you have a pet-peeve bug that’s been annoying you, please check to see how our new Xapian test server is handling things now. (Please be patient with our the dev server, the box needs an upgrade soon!)

If you are a fluent speaker in Chinese, Japanese, Korean or Thai, please take a moment to look up some artists! We had some problems with searching Chinese text, but I think I fixed it, but I am not proficient in any of the applicable languages, so please help sanity check me!

Unless I find more bugs, this new search server will go into production sometime next week. If you find a bug, please report it to the usual place.

Search: Why is it so important?

After many days of tinkering, the new search server has passed its tests and is nearly ready for deployment next week. After my last post on the search services, there were lots of questions, so I’ll give some more history on why I’m working on this now:

  1. The old Lucene based search services worked well, but installing them was a major pain. Installing compilers by hand, sacrificing chickens and hoping that things would work wasn’t my idea of fun.
  2. Lucene has a philosophy of working out of the box without significant tweaks. That’s great if you’re indexing a bunch of text, but indexing music metadata from an SQL database is a bit of a different beast. The usual Lucene tricks didn’t work so well for us, so we couldn’t tweak it to work better for us. Xapian requires a little more tuning out of the box, but our search results are much better now than they were before.
  3. Sending metadata lookup traffic to a service like Xapian is generally a good idea, as a single Xapian server can handle lookup traffic more elegantly than a Postgres database. And adding more search servers is easier than adding more database servers.
  4. Our traffic is growing — I expect us to handle twice as much traffic in July as we did the July before. A lot of this traffic growth is coming from people using our web-service to look up music. If the web-service slows down, the rest of the site slows down as well. So I’m trying to stay ahead of the curve an anticipate when we reach capacity and be able to add more machines as necessary

As of next week, MusicBrainz will have twice as much rack-space (20U’s of space!) and we can finally rack the two new servers that were donated a few months ago. Fortunately due to dropping bandwidth costs, this new space doesn’t really come at a greater expense to us — I expect our hosting costs to stay nearly the same as they are now. (about $1000/mo, btw)

This will allow us to have 3 times the search capacity we have now, which should keep the site working for a while longer. In fall I hope to start moving our web-service to Amazon’s EC2 service, which should allow us to get as much capacity as we need.

As soon as I get the new search services deployed I’m putting my head down and coding the next server update. So, keep your fingers crossed that this process goes smoothly.

Amazon ASIN cleanup

Back in prehistoric times, MusicBrainz used to use an automated script for matching releases to Amazon’s ASINs. But quickly people interjected and demanded to be able to use ARs to associate ASINs to releases. So, we added support for using ARs, but we never got rid of the old system and that has caused a … Continue reading “Amazon ASIN cleanup”

Back in prehistoric times, MusicBrainz used to use an automated script for matching releases to Amazon’s ASINs. But quickly people interjected and demanded to be able to use ARs to associate ASINs to releases. So, we added support for using ARs, but we never got rid of the old system and that has caused a few bugs over time.

I’ve written a script that takes the first step in removing the old amazon ASINs and converts them to AR links. I’ve run this script on my test server musicbrainz.homeip.net — please go to that server and pick your favorite ASIN screw up and let me know if its working.

If this script turns out to work well, I can run it on the main server later this week in order to remove these pesky ASIN issues.

Thanks!

UPDATE: This script has been run on the main server and all Amazon ASINs should now be user editable. No new links have been generated — we’ve only converted old style ASIN matches to ARs so that our editors can make changes.

Unplanned downtime

The primary web server has just gone offline unexpectedly – we’re looking into it. Meanwhile you can still the other parts of MusicBrainz (read on for details). Sorry for the inconvenience – please bear with us, and check back here for updates. Thanks!

The primary web server has just gone offline unexpectedly – we’re looking into it. Meanwhile you can still the other parts of MusicBrainz (read on for details).

Sorry for the inconvenience – please bear with us, and check back here for updates. Thanks!

Continue reading “Unplanned downtime”

Search server fixed.

UPDATE: The main search functions are back! FreeDB search will be offline until sometime tomorrow. Our search server finally gave up the ghost and its dead. We’re bringing up a back-up server to take over for it, but it will take another 8 hours for the indexes to be complete. Sorry for the inconvenience! In … Continue reading “Search server fixed.”

UPDATE: The main search functions are back! FreeDB search will be offline until sometime tomorrow.

Our search server finally gave up the ghost and its dead. We’re bringing up a back-up server to take over for it, but it will take another 8 hours for the indexes to be complete.

Sorry for the inconvenience!

In the meantime, you can donate some money!

UPDATE: by Dave at 1730 UTC

Until about 1715 UTC the search server was only available to the master site – this is now fixed. Sorry for the additional inconvenience, and thanks for bearing with us.

Previously Dave said:

The server server seems to be a little sick again today – the load average was over 100 just now. As a result I’m trying to reboot the server now, and I’ve put request rate-limiting in place on the main server, so you may find that various things aren’t working at the moment. 🙁

Watch this space for updates.

Upgrade instructions for mb_server release 20071014

If you have an existing installation of the MusicBrainz server, follow the instructions below to upgrade your installation to the latest and greatest. You can skip steps #0 and #7 if you’re not running a slave server.

If you have an existing installation of the MusicBrainz server, follow the instructions below to upgrade your installation to the latest and greatest. You can skip steps #0 and #7 if you’re not running a slave server.

Continue reading “Upgrade instructions for mb_server release 20071014”

Main server updated!

We just finished the update to the main server! The final set of changes that are now live are documented in our server release notes. Now its time to go an apply tags to your favorite MusicBrainz data! Thanks to Dave, Lukáš, Murdos, Acid2 and Jugdish for contributing towards this server release! Its been a … Continue reading “Main server updated!”

We just finished the update to the main server!

The final set of changes that are now live are documented in our server release notes. Now its time to go an apply tags to your favorite MusicBrainz data!

Thanks to Dave, Lukáš, Murdos, Acid2 and Jugdish for contributing towards this server release! Its been a long time in the coming, but we managed to squeeze in a number of significant improvements in this release.

UPDATE: One of the changes to the database will cause a few Edit Relationship edits to fail. (e.g. #76493) — sorry for the inconvenience!

Server release post-poned to Oct 14th

After finding a couple of serious issues upon doing a final code review, I’ve decided to post-pone the server release until Oct 14th. I’ve had to make invasive changes that require more testing, so I do not feel comfortable doing the update today. Sorry if you’ve been looking forward to this release, but I want … Continue reading “Server release post-poned to Oct 14th”

After finding a couple of serious issues upon doing a final code review, I’ve decided to post-pone the server release until Oct 14th. I’ve had to make invasive changes that require more testing, so I do not feel comfortable doing the update today.

Sorry if you’ve been looking forward to this release, but I want to ensure that things go smooth when we do make the switch. And it also seems that we have record traffic right now, so there are too many things telling me to wait with the upgrade.

Please send chocolate. 🙂

Server release post-poned

Due to a number of loose ends that we still need to tighten up (and lack of sleep on part of the people doing the work) we’re going to rest, then fix issues and aim to complete the server update tomorrow. Sorry for the delay!

Due to a number of loose ends that we still need to tighten up (and lack of sleep on part of the people doing the work) we’re going to rest, then fix issues and aim to complete the server update tomorrow.

Sorry for the delay!