Open source projects: Do you have servers that need RAM?

As part of our recent server donation, we’ve got piles of 20 1GB ECC server modules kicking around that we won’t put to use. Rather than them go to waste, I would much rather send them to *your* open source project and have you use them. Before too long we will have some servers to donate as well. If you need ECC ram or a server for your open source project, please leave a comment with the following information:

  • The name and URL of the project
  • The exact type of ram you need or what kind of server you need and what you plan to use it for.

I’ll make a list of people interested in servers and I will attempt to match them up with servers as they become available.

Last.fm donates a 48-port gig switch!

Our friends at Last.fm just donated a 48 port gig switch to MusicBrainz! This new (to us) switch will allow us to shorten the update cycle for our indexed searches. Currently it takes about 3 hours to generate the indexes and to push them out to the search servers. With this new switch, we should be able to push the indexes quite a bit faster, which should shave off 60-90 minutes off our update cycle.

Thanks so much for the donation! Thanks to Adrian for making this happen in next to no time!

Server donation and finances for 2011

I’m pleased to announce that an anonymous company decided to donate a pile of 20 Supermicro servers to us!! I’ve tallied up and estimated a value for all of these servers and it comes to $49,480!

Out of that pile of servers, I’ve built 10 servers that are very close to the servers that we purchased during our fundraiser last year. Two servers are nearly ready for use and a bunch of other servers decorate (read: fill up) my tiny office. Thanks so much to our anonymous donors — you’re going to help us grow over the next couple of years! With the rate at which our traffic is growing, the timing of this donation couldn’t be better!

If anyone wants to send dark chocolate to our donors as a thank you, please do! Just send them to MetaBrainz and I’ll pass them on our friends.

The timing of this donation was especially spot-on since it put us into the black for 2011. With a scant excess revenue (retained earnings) of $4,166.91 on revenues of $239,756.07, we barely made it. Phew. See our financial reports for all of 2011, if you care to get more details!

That closes an exciting year for MusicBrainz/MetaBrainz! We hope to have the annual report published before the month is out.

Thanks for an amazing 2011 everyone!

MetaBrainz Foundation Annual Report for 2010

Introduction

2010 was a big development year for us with an amazing amount of tangible progress on the Next Generation Schema (NGS). NGS would prove to be our number one task in 2010, with no server releases happening at all because of our focus on NGS.

Early in the year we increased our engineering capacity with Kuno Woudt joining us as a full time developer in February; Oliver Charles moved from a part time position into a full time position in June 2010. Their sole focus for the entire year was to finish NGS, and we got within a few months of finishing it.

We transitioned from Subversion to Git as our version control system in the process of writing NGS. From Nov 2009 to Feb 2010 the MusicBrainz Server codebase went from ~224,000 lines of code down to ~72,000 (the lowest since 2003!). By the end of 2010 we reached ~130,000 lines of code. NGS does a lot more, but with a lot less code. NGS was a much needed cleanup and overhaul of our aging codebase.

Financially we started 2010 off pretty weak, but got an early boost with a $50,000 donation from Richard Jones (one of the founders of last.fm) and later a $40,000 donation from Google. These generous donations allowed us to continue focusing on NGS — thank you Google and Richard Jones!

MetaBrainz took on The GuardianmusiXmatch, and ZeeZee as new data customers in 2010.

Google Summer of Code 2010 resulted in an exciting new addition to our product line – the MusicBrainz Android App. Developed by Jamie McDonald, the Android app allows anyone to carry the MusicBrainz Database around in their pocket wherever they go. It is a very handy app to settle music debates at parties!

Jess Hemerly conducted a study and wrote a paper on MusicBrainz as part of her Master’s program at the UC Berkeley School of Information. Among many other topics, she answered questions such as ‘Why do people contribute?’, ‘What characterizes editors’ participation?’, and ‘What is the role of metadata in music technology?’.

Profit & Loss

In 2010 the foundation took in $177,740.94 and spent $172,904.94 for a total excess income of $4,836.00.

Income

Direct Donations $90,278.89
PayPal Donations $6,867.99
Consulting $2,669.75
Live Data Feed Licenses $52,682.76
CC Data Licenses $5,100.00
Amazon Associates $1,299.94
Tagger Affiliates $18,147.62
CD Baby Affiliate $12.00
Bank Credits $0.10
Bank Interest $681.89
Total Income: $177,740.94

Expenses

Bank Fees $995.30
PayPal Fees $1,263.63
Rent $2,856.00
Hardware $5,455.15
Travel $4,806.26
Internet $184.56
Development $109,991.19
Gifts $458.63
Events $270.98
Hosting $16,900.00
Filing Fees $60.00
Software $99.00
Entertainment $331.53
Books $20.99
Insurance $2,025.00
Accounting $1,200.00
Shipping $87.51
Payroll Taxes $25,639.39
Advertising $259.82
Total Expenses: $172,904.94

The Profit & Loss shows:

  • In 2010 the foundation spent $22,355.15 on hosting and hardware costs and served out 3.2 billion web hits and 1.5 billon web service hits. Calculating a cost per hit, we find that we spent $6.93 per one million web hits and $14.50 per one million web service hits. Compared to our 2009 figures of $8.66 per one million web hits and $14.37 per one million web service hits, these numbers didn’t change much.
  • End-user donations via PayPal came to $6,867.99 which is roughly 10% less than last year. End-user donations came to less than 4% of our overall income due to the much larger role of sponsors such as Google and Richard Jones.
  • Development costs in the form of salaries paid to Robert Kaye, Oliver Charles, and Kuno Woudt came to $109,991.19. It is amazing what we were able to accomplish with such a limited budget for paid engineers.
  • In 2010 we earned $52,682.76 from live data feed licenses and $5,100.00 from Creative Commons licensed data for a total of $57,782.76. This is up 28.8% from the total of $44,878.50 in 2009.

Balance Sheet

The balance sheet for the end of 2010 showed the MetaBrainz Foundation with $77,011.94 retained earnings, a net income of $4,836.00, and cash assets totalling $81,847.94.

Traffic

The following chart shows our overall web traffic to musicbrainz.org for 2006 – 2010:

Musicbrainz Traffic 2006-2010

The blue line represents the overall number of hits to musicbrainz.org and the red line shows how many of the overall hits were web service (API) hits. As in previous years, our web service hits represent about 85% – 90% of our overall traffic. Please note that in September of 2009 we switched to a more accurate method for keeping track of our overall web services hits. Prior to this, the graph shows the sum of the artist/release/track counts, rather than the total web service traffic.

Our traffic grew considerably in the first half of 2010, but then leveled off for the second half of 2010. We don’t know what accounted for this leveling off, but we suspect that the lack of new features for the MusicBrainz server generally decreased interest in the project.

Top contributors

Top Editors
1. drsaunde 73982
2. brianfreud 72121
3. gswanjord 58933
4. murdos 54517
5. HumHumXX 43985
6. dimpole 39527
7. salo.rock 38644
8. nikki 37770
9. reosarevok 37702
10. refresh_daemon 34806
11. jesus2099 33515
12. Senax 30956
13. MeinDummy 25521
14. mr_maxis 24743
15. Billy Yank 20248
16. dinog 20215
17. crazee_canuck 18537
18. ojnkpjg 18469
19. Bitmap 17612
20. kepstin 16550
21. NAvAP 16095
22. Jeroen 16064
23. rswarbrick 15972
24. zos18 15204
25. fred576 14897
Top Voters
1. chabreyflint 49247
2. salo.rock 48651
3. murdos 33786
4. Locustus 29267
5. SuicideScrub 24312
6. bogdanb 22473
7. nikki 21085
8. brianfreud 20606
9. gswanjord 19276
10. Bitmap 17956
11. MClemo 17362
12. drsaunde 15082
13. KRSCuan 13922
14. dinog 13844
15. reosarevok 9650
16. MeinDummy 9262
17. HumHumXX 8534
18. mr_maxis 7617
19. PhantomOTO 7194
20. articpenguin 7092
21. ojnkpjg 6977
22. Plagueis 6877
23. alphaseven 6762
24. fatih 6611
25. alllysssa 5819

Server farm

At the end of 2010, MusicBrainz had 14 machines in service. From the top, going down:

  • moose: Our database server
  • scooby: Our aging catch all server: blog, forums, mailing lists, etc
  • catbus: Raw database server (raw tags, collections, etc)
  • bender: Former TRM server, now idle cold spare machine
  • blik: memcached
  • stimpy, dexter: web service servers
  • cartman: Search server, index builder
  • wiley: New catch all server: SVN, git, jira, wiki, trac, mail, backups
  • lenny/carl: Redundant network gateways
  • tails: Front end web server
  • asterisk: Search server
  • jem: Search server

MusicBrainz uses 6mbits of bandwidth per second and draws 21 Amps of current for a power consumption of about 2,310 Watts. MusicBrainz physically occupies 20Us of space (half of a rack) at Digital West in San Luis Obispo, CA.

Words of Appreciation

2010 was a challenging year for us, starting off with rocky finances, but support from Richard Jones and Google put us back on track. There were many people who thought that we could not ship NGS or that MusicBrainz would languish while we worked to complete NGS. Given that we had no server releases at all in 2010, we are pleased that the project remained relevant and that our community believed in us to finish NGS.

MusicBrainz would like to thank its community of stellar editors (see above), its core developers (Lukáš Lalinský, Oliver Charles, Kuno Woudt, Aurélien Mino), our hero of system administration, Dave Evans and our goddess of bug tracking, bug fixing, editing and all things unicode, Nikki. We thank Jamie McDonald for the awesome Android app he wrote and we’d also like to thank Pavan Chander for all of his contributions.

We’d like to thank Richard Jones, Google and every single donor who donated money to MetaBrainz in 2010. We’d also like to thank our board of directors (Cory Doctorow, Brian Zisk, Matt Wood, Rachel Segal/Carol Smith), our pro bono legal advisors Daniel Appelman and Ed Cavazos, our awesome hosting company Digital West and all of our customers. Finally, we would also like to thank the music teams at the BBC for their continued support and for motivating us to bring NGS to a close.

Looking for sponsors to purchase two development machines

The MusicBrainz community has two very active contributors (navap and reosarevok) who do not have and cannot afford to buy decent development machines/laptops. MusicBrainz is carefully watching its finances currently and can’t really afford to spend money on this right now, so I’m curious to see if the community would come together to support these two awesome contributors.

Navap has done a lot of editing and web site improvements (a series of 8 or so nitpick patches that greatly improved the look/feel of NGS) and countless other tasks to improve MusicBrainz. Reosarevok is a very active editor and has recently become our wiki god; he is also wishing to help out more with the development. Both of these two awesome contributors could be more productive with decent machines.

We’re looking to buy two $500 Linux capable machines that have a couple of cores and about 4GB of RAM each. Would you please consider chipping in to help our contributors? You can make a cash donation (please let us know your donation should go towards laptops) or even send a laptop to our contributors. If you’d like to help out with a donation of an actual machine, please leave a comment in the blog and I’ll get in touch with you.

Thanks!

Our hardware fundraiser is complete: We raised $15,527.50

After pestering people for a month, I’m pleased to announce that we’ve successfully closed our fundraiser. We’ve exceeded our goal by raising $15,527.50!

The following companies & individuals contributed:

$5000.00 Google
$3708.50 Individual user donations
$1389.00 SoundCloud
$2000.00 EchoNest
$1000.00 Matt Mullenweg (WordPress)
$1000.00 Decibel.net
$500.00 Magic MP3 Tagger
$500.00 Grooveshark
$300.00 Songkick
$130.00 Affinity Chiropractic (the people MetaBrainz rents an office from)
$15,527.50 in total!

I will send out thank you mails to our 83 individual donors later today! Thanks to everyone who contributed! I’ve already ordered the hardware we wanted and will start installing it this week. Hopefully next week we can start loading NGS on this new hardware.

Google now matching donations for NGS hardware fundraiser!

I’m pleased to announce that the Google Open Source Programs Office has pledged to match donations we receive for our NGS fundraiser! Google will match donations up to $5,000!

If you haven’t chipped in to support us purchasing hardware for our NGS roll-out, now is the time! Your donations count twice as much, because of Google’s matching pledge!

Thank you so much to Chris DiBona and the rest of the Open Source Programs Office at Google!

Network changes coming on Sunday Sept 13

We’re going to be changing how the traffic routes through our servers on Sunday September 13th near mid-day PDT, late evening UK time. In theory you should not notice any traffic interruptions, but theory doesn’t always match up with practice.

Before we start the maintenance, we’re going to announce the switch over to the new route in our IRC channel #musicbrainz on irc.freenode.net .

With a little luck no one will notice our tinkering.

Update by djce: The change is now complete and seems to be working just fine.  Thank you for your patience.

Planned downtime: Friday Sept 12, 14:00 PDT, 21:00 GMT, 22:00 BST

Tomorrow, Friday Sept 12 MusicBrainz will be down intermittently for about 30 minutes from 14:00 PDT/21:00 GMT/22:00 BST. We’re going to migrate to our new switch, which means that various portions of MusicBrainz will be unavailable for brief periods of time. Hopefully the overall unavailability of the service will be a lot closer to 5 minutes than 30 minutes. As usual, we will do our best to keep the interruptions brief.

Sorry for the inconvenience.

Update at 22:49 BST by djce: we’re done – it all went according to plan.  The site was down for just under 15 minutes.  Thank you for your patience.

Bandits at high noon!

The traffic to MusicBrainz is growing steadily! (steady growth is much better than exponential growth!)

We’ve recently added two more search servers, which are now humming along nicely. But now we’re seeing that our main web server is starting to get saturated, so we need to add more capacity. In order to use our existing machines effectively and to greatly increase the redundancy of our site, we’ve asked out board of directors to approve the purchase of two more servers. I expect this to be approved shortly.

In the meantime we’re getting more serious about people who are violating our very liberal terms of service that state that no single client instance is to make more than one Web Service call to MusicBrainz per second. We’ve been blocking mis-behaving clients more aggressively and have decided to block all clients that use this User-Agent id string:

Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5

If you’re using a MusicBrainz enabled application and all of the sudden it stopped working today, please leave us a comment below. We’d love to identify the offending application so we can contact the author of the application.

Also, the following IP address have been blocked from using MusicBrainz, either because of their client string or sending too many requests to MusicBrainz:

85.144.88.11, 69.181.45.228, 80.58.205.43, 205.209.78.232, 195.235.104.253, 85.240.173.204, 92.195.16.20, 89.241.123.24, 71.12.186.1, 204.17.31.126, 98.64.41.178, 77.188.36.140, 141.155.126.17, 217.225.157.236, 91.9.246.252, 66.91.250.27, 217.41.4.13, 83.76.160.150, 91.67.8.254, 88.191.58.11, 141.30.218.96, 88.88.124.26, 82.42.72.21, 91.9.230.72,
213.22.39.14, 86.0.180.87

In the last 16 hours, the following IPs have been over the limit:

Total requests IP address
3187 58.170.57.43
3344 71.76.7.125
3523 92.193.11.17
3789 61.216.59.7
4495 67.180.35.184
4670 218.66.36.76
6521 91.47.225.62
7993 67.109.84.5
8558 91.47.135.154
189763 88.191.58.11

If you find your IP address on this list, please make a donation to the MetaBrainz Foundation and we’ll consider unblocking you after you assure us that you will not be making more than 1 request per second.