Category: Server

The attempt to switch the database to the new Sun server has failed and we’re still running with the old set of hardware. For some reason Postgres refuses to run with any kind of acceptable level of performance on this new machine. Thus, we’re setting aside this server for the time being and instead working to bring a replacement server online as soon as possible.

We’re also working with Dell to determine what the problem on the server is, and luckily the servers still have an extended warranty covering them with 4 hour on-site service. However, it appears that we need to diagnose the problem before Dell is willing to send someone out. I’ve run a memory test on the server and it passed all of its tests.

I’m out of ideas. I’m frustrated with Sun and Dell and will no longer waste time on their hardware trying to solve this problem. Instead I will work with whatever hardware I can scrounge up and get the service moved over to our spare server and then we’ll sort out the issues with our glorious servers.

It looks like it will at least be Friday before we can make the next attempt to fix this. Again, sorry for the inconvenience — we’re doing our best to fix this problem ASAP.

Tomorrow January 17th MusicBrainz will be unavailable starting at Noon PST, 15:00 EST, 20:00 UK, 21:00 CET, for at about 1 hour.

The recent problems with certain artists not being found are being caused by a failing motherboard (specific guess: failing DMA controller). Fortunately the database server is still working well and the database has not been affected by this hardware malfunction.

Tomorrow Dave Evans and myself will bring moose, our newly donated Sun Server, online and swap it out for the current database server. Once the new database server is up and running, we will want to monitor it for a while to make sure its doing its job properly. Once we’re convinced that it is doing its job, we will take the site down again and swap the drives in our then-to-be obsoleted database server with our ailing web server and bring the site back up.

After that we can remove the failing server and purchase a replacement motherboard — if you’d like to make a donation to help us cover the cost of the new motherboard (est: $800), we would greatly appreciate your support!

Thanks to Sun for the spiffy server that will get us out of a this bind and a big thanks to Dave Evans for his hard work getting our servers stable again.

I apologize for the inconvenience all this may have caused you.

We spent some quality time with the servers today and updated them with the latest release. After some minor tweaking things appear to be behaving normally now. For details on what we’ve changed, please see our official release notes. For a complete list of bugs fixed, see the milestone for this release.

Thanks to everyone who helped give feedback, develop and test this server release!

The test server has been updated and is running the latest version of the next server source code. All of the lucene/searching bugs in the milestone for the next release have been fixed and seem to be running well now. Just yesterday I finished off the bugs by adding paging to the search results and redirecting if the search results in only one hit. This should pretty much close out the outstanding searching issues.

That leaves a host of Web Service related issues that I will tackle after the Thanksgiving break (more on that in the next post).

Finally, there have been some changes with the Amazon cover-art servers. The test server is now running with the latest mappings to these servers, but we’re not entirely sure that these are all correct. If you care about Amazon cover-art working well, please take a minute and see if the cover art on the test server is working well. For this and any other issues, please report any bugs.

Currently the next release is scheduled for December 10th. If too many bugs crop up or other issues arise, we’ll hold off with the release until December 17th.

Technorati Tags: development, musicbrainz, service

The TRM server hardware has been giving us some issues in the last week. We’re having to reboot the TRM server a lot more frequently than I care for, so I will take the TRM server down tomorrow and have a look to see what the issue might be.

The TRM server will be offline for about 1-2 hours tomorrow Wednesday August 23rd, at Noon PST/2000 BST/2100 CEST.

UPDATE: The outage is now over — I could not find any problems with the server, so the random lock-ups go unexplained. 🙁

Technorati Tags: outage, service

The main server was just updated with the latest changes.

Technorati Tags: release, server

After passing on the release last weekend, I think we’re ready to try again. This is the last call for bugs for this release! If you find a serious bug, please enter it into trac and I’ll see about getting it fixed. Please don’t enter any enhancements requests for this release — I’m only fixing bugs at this point and will do no enhancements right now.

The bug fix update release is scheduled for this Sunday. Go test on the staging server and report bugs or look at the list of closed bugs.

Technorati Tags: musicbrainz, release

Too many bugs have rolled into allow us to to a bug fix update this weekend. Given the update fiasco from last time, we’re going to wait until the issues settle and these bugs are addressed.

I know a lot of people are anxious to get this bug fix release out. If you are one of those people, make sure you get your bug reports in ASAP. Unloading a wad of bug reports that should’ve been raised weeks ago within 48 hours of a planned release will only hamstring the release process. If you’re one of the people who is anxious about getting bug fixes out, please enter your concerns now and not 2 days before the next attempted release date.

We’ll shoot for next Sunday as a the next update time.

Sorry folks!

Technorati Tags: musicbrainz

I’m pleased to announce that MusicIP has just open sourced a small extension to the MusicBrainz server!

MusicIP contracted me to write a set of SQL scripts that would take their mirror of the MB database and create an extra table that stores the first release date for each track. As you may know we have this for albums, but we haven’t had (or needed) this for the track level.

If you’d like to check out this extension, you can find it here. Take a look at the README file to see how this should be used.

Please note that this code is checked into the RELEASE_20060712 branch — once we’re finished with dev work on this branch we will merge it back into the trunk.

Big thanks to Wendell Hicken and Matthew Dunn of MusicIP!

Technorati Tags: musicbrainz, release

In ticket #1731 we’re currently discussing the merits of having the artist aliases searched by default. Compare these two searches:

1. Search without aliases for “Jennifer”
2. Search with aliases for “Jennifer”

The perfect match for Jennifer is half-way down the page of results if aliases are included. Jennifer is all the way at the top (where it should be) when the aliases are not searched.

So, why does this happen?

Take for instance the top hit of the with aliases search: Jennifer Paige. She has an alias for “Jennifer Page“, so when Lucene ranks the search results, the word Jennifer appears twice, which is a better match than when the word appears only once. This disturbed our users and it plain feels wrong to me.

Then I tried to play with Lucene’s term boosting functions. Take this query:

aritst:jennifer^10 sortname:jennifer^10 alias:jennifer^0.0000001

In English, it says to search for Jennifer in artist names and sortnames and to make these hits 10 times more “important” than normal hits. It also says to search aliases and make hits from this vastly less important than normal hits. The result, Jennifer is at the top as we want. But, what happens when we search for Bjork (not Björk)?

We get this mess where Björk is the last search hit with a score of 0. (this is not the best example since the next version of search will automatically find Björk when searching for Bjork, but it still illustrates the problem)

As you can see, tweaking the searches to make things better one way, will make other searches worse. Do you think it its more important to search aliases by default or to have better search results by default?

Technorati Tags: musicbrainz, search