Server Troubles

Recently the server has been hit by patches of instability – large load spikes, running out of memory, and processes getting killed here and there. When the most recent out-of-memory condition occurred (last night) the SSH server was one of the processes which got killed, which is why the server had to be rebooted a … Continue reading “Server Troubles”

Recently the server has been hit by patches of instability – large load spikes, running out of memory, and processes getting killed here and there. When the most recent out-of-memory condition occurred (last night) the SSH server was one of the processes which got killed, which is why the server had to be rebooted a little while ago.

I’m fairly sure I more or less know what’s been causing the problems, and have made a few changes to try to reduce the chance of it happening again.

One of the worst causes of the problem is looking up a TRM with a large number of tracks. The worst TRM by far for this is the “silence” TRM, with (currently) over 900 tracks. As a result I’ve had to, for now at least, disallow lookups on this TRM – doing so will now simply return an error. Sorry 😦 Maybe it can be made to do something more helpful in future.

The other change is that if you do a lookup on any TRM which has more than 100 tracks then only 100 of those tracks will be returned. However so far there are no TRMs (except “silence”) with over 100 tracks, so this won’t affect anyone, yet. As the data grows, it will though.

Sorry for any inconvenience caused (hey, I’m apologising again. This is getting to be a habit). But I’m sure you’d rather have a server which doesn’t keep crashing and locking us all out. Hey ho.

Server Updates

“Add Disc ID” moderations, and Annotations.

Changes mainly of interest to MusicBrainz Users

“Add Disc ID” Moderations

Whenever a disc ID is added to an existing album, it is now tracked
via an “Add Disc ID” moderation.  This applies both to disc IDs added
via the “CD lookup” interface (in which case the moderation is credited to
whoever performed the lookup), and also to those added as a result of a
FreeDB lookup (which fall under the “FreeDB” moderator). 
“Add Disc ID” moderations are not used in the case where
an album and a disc ID are added at the same time.

Annotations

Annotations allow you to add notes to artists and albums. 
See How Annotations Work
and the Annotations FAQ
Thanks to Matthias Friedrich for building the foundations of this feature.

Bugs and RFEs Closed

Dave Evans

Try out the latest server features

I’ve opened my development box up to the world – you can see what I’m working on at http://dev-mb.djce.org.uk/.  The mb_server I have running at the moment, for example, implements “Add Disc ID” moderations.  Play nicely!

I’ve opened my development box up to the world – you can see what I’m working on at http://dev-mb.djce.org.uk/.  The mb_server I have running at the moment, for example, implements “Add Disc ID” moderations.  Play nicely!

Lucene web service

In the last two weeks I managed to combine working on MusicBrainz, creating a new open source project and earning money to pay the bills! This is quite rare these days, so I am pleased all around. As some of you may know, I have been doing contract work for CD Baby. When Derek, the … Continue reading “Lucene web service”

In the last two weeks I managed to combine working on MusicBrainz, creating a new open source project and earning money to pay the bills! This is quite rare these days, so I am pleased all around.

As some of you may know, I have been doing contract work for CD Baby. When Derek, the owner and lead geek at CD Baby, asked me what MusicBrainz does for searching, I launched into a long cheerleading rant about Lucene. I managed to convince Derek that Lucene is the way to go, and to convince him to sponsor the open source development of the new Lucene Web Service. Luckily Derek agreed that as long as the project was going to be available under the BSD license that he would agree to open source the work.

Triple cheers for Derek and CD Baby please!

So, the web service is now done and I’ve applied for a new project on SourceForge — once that is approved, I will release the source code for everyone to check out. I’ll post another message here when that is complete.

If you’d like to check out the working web service, try this link.

TRM Database Pruned

The TRM database has been pruned again, making the system much faster and more reliable again.

At about 1930hrs UTC on November 4th the TRM database
was “pruned” again (see the previous time
for more information about this). 
This time we removed all TRMs apart from the ones attached to
MusicBrainz tracks, where the TRM had been looked up at least once. 
This is a slightly more aggressive prune compared to the previous time. 

The TRM database is now about
40% of the size it was before, which (like last time) means that
it now fits easily into the server’s memory, so the server as a whole
runs quickly
and reliably.

Annotations – final testing?

I’ve made what I hope will be the final series of changes to the annotations work (at least, for its first release) – please see this posting on mb-users for details. Please test it, and let me know what you think. Thanks!

I’ve made what I hope will be the final series of changes to the annotations work (at least, for its first release) – please see this posting on mb-users for details. Please test it, and let me know what you think. Thanks!

Duplicate Artist/Album Reports Updated (expect delays)

The raw data feeding into the duplicate artist/album reports has been updated. These are available under Edit The Data/Suggestions as the last two entries: “Albums that might need merging” and “More possibly duplicate artists”. The last time this report was generated (July) we had 1703 possible album duplicates and 1872 possible artist duplicates. We now … Continue reading “Duplicate Artist/Album Reports Updated (expect delays)”

The raw data feeding into the duplicate artist/album reports has been updated. These are available under Edit The Data/Suggestions as the last two entries: “Albums that might need merging” and
“More possibly duplicate artists”. The last time this report was generated (July) we had 1703 possible album duplicates and 1872 possible artist duplicates. We now have 2714 possible album duplicates and 2424 possible artist duplicates.

Note that there is a delay between when I upload the raw data and it is reflected on the server – I think this was set up to happen once a day, but it may be only once a week.

As always, if anyone is looking at these, and there is a confirmed false positive, let me know and I will (a) make sure it doesn’t show up in the next report, and (b) see if I can improve the overall reporting. So far very few people have submitted false positives.

Annotations – please try it out!

The annotations work I’ve been talking about over the last few days is now live on the test server. Please have a play and let me know what you think. Is it too limited? Does the page layout need tweaking? Maybe the way the moderations work isn’t quite right? All feedback gratefully received.

The annotations work I’ve been talking about over the last few days is now live on the test server. Please have a play and let me know what you think. Is it too limited? Does the page layout need tweaking? Maybe the way the moderations work isn’t quite right? All feedback gratefully received.

Continue reading “Annotations – please try it out!”

I did mention that Lucene rocks, right?

I decided that I wanted to put together a comprehensive test of Lucene so I could show how powerful, fast and accurate Lucene is. This is just a simple Python script that is not integrated to the rest of MusicBrainz — it doesn’t even touch the Postgres DB! My little test is hosted in the … Continue reading “I did mention that Lucene rocks, right?”

I decided that I wanted to put together a comprehensive test of Lucene so I could show how powerful, fast and accurate Lucene is. This is just a simple Python script that is not integrated to the rest of MusicBrainz — it doesn’t even touch the Postgres DB!

My little test is hosted in the staging server — the DNS should’ve propagated by now. Come check it out:

http://search.musicbrainz.org

Annotations, part II

Following on the Annotations work, I’ve now got it automatically handling artist / album merges and deletes, which was one of my main concerns before. I think this makes it just about ready to test. It’s a very simple feature, which is possibly one reason why I think it will work – I’m not sure … Continue reading “Annotations, part II”

Following on the Annotations work, I’ve now got it automatically handling artist / album merges and deletes, which was one of my main concerns before. I think this makes it just about ready to test. It’s a very simple feature, which is possibly one reason why I think it will work – I’m not sure it needs to be any more complex – not for the first release anyway.

Update: see the Wiki for documentation.

Continue reading “Annotations, part II”