Ailing server replaced!

After 9 days of frustrating work to replace our failing main web server, we’ve implemented plan K (plans A – J failed!). Plan K involved making a previously unreliable server (lingling) reliable again by installing software RAID and then moving all of the services onto this machine. We just finished with that and the new … Continue reading “Ailing server replaced!”

After 9 days of frustrating work to replace our failing main web server, we’ve implemented plan K (plans A – J failed!). Plan K involved making a previously unreliable server (lingling) reliable again by installing software RAID and then moving all of the services onto this machine. We just finished with that and the new web server seems to be handling the load well so far.

Now its up to Dell to fix our server — fortunately, much to my surprise, we still have a service contract on that machine. So, its up to Dell to diagnose and fix the box and then we’ll put it back into rotation. Also, Steve Wyles (inhouseuk) has agreed to lend a hand in diagnosing the issues on moose, the new Sun Server — we’re suspecting a copy of Solaris that is too old to handle this new architecture.

So, expect more downtime in the future as we shuffle fixed machines back into rotation. Hopefully we’ll be able to proceed with a more sane pace from here on out.

Once again, giant kudos to Dave Evans for all his hard work! Thanks Dave!

MusicBrainz downtime redux

The attempt to switch the database to the new Sun server has failed and we’re still running with the old set of hardware. For some reason Postgres refuses to run with any kind of acceptable level of performance on this new machine. Thus, we’re setting aside this server for the time being and instead working … Continue reading “MusicBrainz downtime redux”

The attempt to switch the database to the new Sun server has failed and we’re still running with the old set of hardware. For some reason Postgres refuses to run with any kind of acceptable level of performance on this new machine. Thus, we’re setting aside this server for the time being and instead working to bring a replacement server online as soon as possible.

We’re also working with Dell to determine what the problem on the server is, and luckily the servers still have an extended warranty covering them with 4 hour on-site service. However, it appears that we need to diagnose the problem before Dell is willing to send someone out. I’ve run a memory test on the server and it passed all of its tests.

I’m out of ideas. I’m frustrated with Sun and Dell and will no longer waste time on their hardware trying to solve this problem. Instead I will work with whatever hardware I can scrounge up and get the service moved over to our spare server and then we’ll sort out the issues with our glorious servers.

It looks like it will at least be Friday before we can make the next attempt to fix this. Again, sorry for the inconvenience — we’re doing our best to fix this problem ASAP.

MusicBrainz downtime: January 17th, Noon PST

Tomorrow January 17th MusicBrainz will be unavailable starting at Noon PST, 15:00 EST, 20:00 UK, 21:00 CET, for at about 1 hour. The recent problems with certain artists not being found are being caused by a failing motherboard (specific guess: failing DMA controller). Fortunately the database server is still working well and the database has … Continue reading “MusicBrainz downtime: January 17th, Noon PST”

Tomorrow January 17th MusicBrainz will be unavailable starting at Noon PST, 15:00 EST, 20:00 UK, 21:00 CET, for at about 1 hour.

The recent problems with certain artists not being found are being caused by a failing motherboard (specific guess: failing DMA controller). Fortunately the database server is still working well and the database has not been affected by this hardware malfunction.

Tomorrow Dave Evans and myself will bring moose, our newly donated Sun Server, online and swap it out for the current database server. Once the new database server is up and running, we will want to monitor it for a while to make sure its doing its job properly. Once we’re convinced that it is doing its job, we will take the site down again and swap the drives in our then-to-be obsoleted database server with our ailing web server and bring the site back up.

After that we can remove the failing server and purchase a replacement motherboard — if you’d like to make a donation to help us cover the cost of the new motherboard (est: $800), we would greatly appreciate your support!

Thanks to Sun for the spiffy server that will get us out of a this bind and a big thanks to Dave Evans for his hard work getting our servers stable again.

I apologize for the inconvenience all this may have caused you.

Trying Google Analytics

I am currently in the middle of writing the year-end report for the MetaBrainz Foundation and its clear that our ancient web statistics system using webalizer is hopelessly outdated. Following a glowing recommendation from Wendell at MuiscIP, I’ve turned on Google Analytics for the MusicBrainz site so we can get a better idea as to … Continue reading “Trying Google Analytics”

I am currently in the middle of writing the year-end report for the MetaBrainz Foundation and its clear that our ancient web statistics system using webalizer is hopelessly outdated. Following a glowing recommendation from Wendell at MuiscIP, I’ve turned on Google Analytics for the MusicBrainz site so we can get a better idea as to what our traffic is and where it comes from.

Google Analytics works by having the client browser send a tiny ping request to Google — this tiny request is enough to track all sorts of things about the MusicBrainz traffic. If for some reason this should cause some issues, please post a comment here.

If it does work out and gives us good results, we’ll keep it. If it causes problems we’ll dump it. Let’s wait and see!

Technorati Tags: ,

The old RDF based web service has been deprecated

The subject pretty much says it all — the new XML Web Service has been debugged and is now stable. This new web service was designed to replace the old web service since its better designed, more concise and a lot simpler to use.

If you are using the old web service directly or have been using libmusicbrainz (1.x/2.x) or libtunepimp (all versions) then you are using the old web service and you should make an effort to migrate your application to use the new XML web service. The old web service will stay in service for all of 2007, but we may get rid of it as early as 2008.

Technorati Tags: ,

Call for early testing: Labels

Lukáš Lalinský has recently finished coding support for Labels inside of MusicBrainz! This new set of features allows MusicBrainz editors to enter information about Music labels (recording companies), create Advanced Relationship links to labels and also supports bar codes for releases. A lot of these features have been eagerly anticipated by the community for some … Continue reading “Call for early testing: Labels”

Lukáš Lalinský has recently finished coding support for Labels inside of MusicBrainz!

This new set of features allows MusicBrainz editors to enter information about Music labels (recording companies), create Advanced Relationship links to labels and also supports bar codes for releases. A lot of these features have been eagerly anticipated by the community for some time, and now you can head over to the test server to try them out yourselves.

Before heading over to the test server, you should probably read the documentation for the labels feature. As usual, if you find any bugs, please enter the into our bug tracker.

Thanks for your hard work on this exciting new features Lukáš!

Technorati Tags: , ,

John Buckman of Magnatune joins the MetaBrainz board of directors

[ I apologize for the lateness of this post — the holidays kept me busy ] I’m pleased to announce that John Buckman of Magnatune and Bookmooch joined the MetaBrainz Board of Directors on December 13th. John is both a geek and a music lover and he embodies both of these passions with his online … Continue reading “John Buckman of Magnatune joins the MetaBrainz board of directors”

[ I apologize for the lateness of this post — the holidays kept me busy ]

I’m pleased to announce that John Buckman of Magnatune and Bookmooch joined the MetaBrainz Board of Directors on December 13th. John is both a geek and a music lover and he embodies both of these passions with his online record label Magnatune. Given his position in the cutting edge space of online record labels and his technical background make him perfectly suited for helping out the MetaBrainz Foundation as a board member.

John replaces Joichi Ito, who has recently taken on the role of chairman of the Creative Commons. Joi is busier than ever, making it difficult for him to participate in the day-to-day operations of the MetaBrainz Foundation. Joi, just as Larry Lessig, stays on as board member emeritus in an advisory role.

Thank you for all of your help Joi and welcome aboard John!

Technorati Tags: ,

Sun Microsystems donates a beefy database server!

I’m very pleased to announce that Sun Microsystems has donated at beefy database server to the MetaBrainz Foundation! This 2U rackmount server comes with 12GB RAM, two dual-core AMD processors (2216) @2.4Ghz, 4 SAS 73GB Drives, dual power supplies and enough fans to make a complete racket. And, to top it all off its really … Continue reading “Sun Microsystems donates a beefy database server!”

I’m very pleased to announce that Sun Microsystems has donated at beefy database server to the MetaBrainz Foundation!

This 2U rackmount server comes with 12GB RAM, two dual-core AMD processors (2216) @2.4Ghz, 4 SAS 73GB Drives, dual power supplies and enough fans to make a complete racket. And, to top it all off its really sweet looking and made by Sun, the legendary makers of reliable hardware!

I bet some of you are wondering why Sun would simply donate a $11k server to the MetaBrainz foundation, right? Well, during a session break at the Future of Music Policy Summit in Montréal Paul Lamere from Sun Research came up to me and introduced himself: “Hi! I’m Paul Lamere from Sun Research and we love the MusicBrainz data — is there anything we can do to help?”

Our hardware wish-list still had a powerful database server on it and knowing how I want to grow MusicBrainz this new server was becoming more important all the time. So I expressed my desire for a new database server and Paul and I continued to talk quite a bit about this during our stay in Montréal. In the coming weeks Paul made a concerted effort to stay on point and continuously follow up on the donation process. From the time we first spoke about the possible donation until we actually had hardware in hand was only about 10 weeks time — an amazingly fast process — I was expecting a 6 month lead time for something this large.

This new server will allow us to free up the existing database server to be another web server front end. Stimpy our front-end web server has been showing signs of being overloaded (hence the recent couple of downtimes. 🙁 ) and bringing online another front end for all of our web service queries will spread the load over more machines. The only downside is that we need to purchase more power since we’re at our max for our current contract.

I’m quite excited by this development. This database server gives us room to grow features and the number of users we can handle at the same time. We have more hard drives to spread the load across and room to easily grow the memory to 16GB (or even 32GB, but at greater cost). I think this machine is going to be our workhorse server for the next few years.

Thank you Paul Lamere, Sun Research, Sun Microsystems — we deeply appreciate your support of MusicBrainz!

P.S. It has been suggested that a dedicated MusicBrainz editor be give then chance to name the new machine. If you have someone in mind, please post a comment below.

Technorati Tags: ,

Server updated

We spent some quality time with the servers today and updated them with the latest release. After some minor tweaking things appear to be behaving normally now. For details on what we’ve changed, please see our official release notes. For a complete list of bugs fixed, see the milestone for this release. Thanks to everyone … Continue reading “Server updated”

We spent some quality time with the servers today and updated them with the latest release. After some minor tweaking things appear to be behaving normally now. For details on what we’ve changed, please see our official release notes. For a complete list of bugs fixed, see the milestone for this release.

Thanks to everyone who helped give feedback, develop and test this server release!

Donations to MetaBrainz (UPDATED!)

Over the last few weeks I’ve been thinking more about how to fund a developer to work on MusicBrainz server features full time. Part of that thinking was taking a critical look at our traffic and our donations. Here is a graph of our overall traffic to musicbrainz.org:

usage.png

Aside from a traffic spike in Dec 2005/Jan 2006, our traffic is steadily increasing, which means our bandwidth/hosting bill is also increasing. It used to be that donations covered our bandwidth/hosting costs, but that hasn’t been the case since we switched to the new hosting facility in February of this year. Our donations per month (excluding larger “special occasion” donations) are shown below:

donations-200612.png

Compared to our traffic, there is much more variability here and aside the “special occasion” big ticket donation (from Google, MusicIP, Cory Doctorow, etc) our donations from end users have shrunk, if anything. I would like to increase our rate of donations so that we can have donations cover our traffic costs again and have our licensing income go towards paying people to work on MusicBrainz.

I know that we’re not trying hard enough to solicit donations from our tagger users. Back in the day before Dave implemented auto pruning on the TRM server, the TRM server would slow down and tagger users would get a “we need more hardware, donate please” message. This would result in a wave of donations from users. To me, this shows that tagger users will respond if forced to. Our “please donate” nag dialog on the tagger is not effective in getting enough people to dig around in their wallets and make a donation to MusicBrainz.

My first approach to nagging users more to donate is to add a nag screen to the taglookup.html page that looks up a file in case the file has no PUID or TRM fingerprint. This page is most likely in use by people tagging their music collections and less so by people focusing on editing MusicBrainz.

Currently, people who donate $10 will receive no nag screen in Picard for six months and I think I’d like to apply this rule here as well. Also, other prolific contributors to MusicBrainz will also not receive the nag screen (via the seekrit NoNag flag on user’s accounts). Before I tromp off to implement this, I’d like to ask you for some feedback:

  1. What do you think of this idea in general?
  2. Do you think that six months no nag is a fair deal for $10?
  3. How do you think we should implement this nag screen?

My first idea is to have a nag panel appear over (and deliberately obscure) the taglookup results every 5 lookups. The nag panel will require people to click on the panel to fully be able to view the results. If the user has JavaScript turned off, the nag panel will be shown above the results and lots of white space will be shown below the nag panel, requiring the user to scroll down to view the results.

What do you think?

UPDATE: Thanks for all your feedback! I’ve starting coding the nag dialog and also started the process of getting another credit card processing company. So far I’ve gotten a quote that should be reasonably competitive to PayPal — if this goes through we’ll be offering people the choice to donate with their credit card via this new company or via PayPal. Stay tuned!

Technorati Tags: ,