Wanted: Wiki, Documentation, Trac and UserVoice guardians

In working on figuring out how to integrate UserVoice into our workflow its become painfully clear that we need to have some people take “ownership” over a few portions of MusicBrainz. Back in the day when Cristoph König (Don Redman) was our Wiki warden, our wiki was well groomed and worked smoothly. Alas Cristoph has been swallowed whole by a University in Germany and we may never see him again. 😦

Its time to find volunteers who are interested in taking on personal ownership of these aspects of MusicBrainz. I’m looking for 2-4 volunteers, one for each of the following areas described below. However, please note that these people are not expected to carry out the bulk of the work that needs to be done to get these projects into an improved state. I envision each of these people being motivators and leaders who can focus and encourage the efforts of other volunteers to help them achieve their goals — there is a big difference between a leader and a workhorse.

  1. Wiki: This person should be well versed in Wikis and understand their general nature. A “Wiki Warden” should be willing to help with the upcoming wiki migration to MediaWiki and have a vision for how our wiki should be organized and cleaned up. This person should be willing to do more work initiallycleaning up the wiki and then spend a few hours every week watching over activity in the wiki and gently steer the wiki into a direction of sense and overall usefulness.
  2. Documentation: This person should be working with the Wiki Warden (it could even be the same person) to coordinate the creation/maintanance of Wiki pages for use with our WikiDocs documentation system. This person should be have an overall vision for how to organize our documentation and to move from our current state to a more organized and useful documentation system for MusicBrainz.
  3. Trac: This person should be familiar with trac and work to remove the cruft that has accumulated in the bug tracker. We need someone to close old bugs that no longer apply, highlight bugs that are important and generally reduce the duplication present in the system. It would be best if this person could also take part of the weekly developer chats in order to stay in tune with the development process.
  4. UserVoice: While a lot of people have volunteered to take part in working with UserVoice, we need one person to take the lead and be in charge of the process. I would love it if this person could help us to settle on one of the proposed UserVoice workflows.

I’d like to stress once again that I am not looking for people to do a lot of work — I’m looking for leaders who can do a little work, but motivate others a lot of help out and create a thriving sub-communities that allow MusicBrainz to become more concise and cohesive. If you’re interested in one or more of these positions, please post to the comments.

Problem delivering mail to gmail / googlemail

This week MusicBrainz experienced problems while trying to deliver mail to gmail.com / googlemail.com. The problem is now fixed, but regrettably this means that some messages that MusicBrainz should have sent are now lost.

This week MusicBrainz experienced problems while trying to deliver mail to gmail.com / googlemail.com. The problems started on Tuesday morning (UK time).  On Friday morning the problem was identified as a broken DNS server, which was then fixed, thus resolving the problem.

Regrettably this means that some messages that MusicBrainz should have sent are now lost.  The number of lost messages is approximately:

  • 103 ‘subscriptions’ messages from Tuesday
  • 61 ‘subscriptions’ messages from Friday
  • 297 other messages (new user signup, edit notes, etc)

Please accept our apologies for this error.

MusicBrainz Server Roadmap

After considering all of the options and taking in tons of feedback from our developers, our community and our customers, I’ve finally settled on the following road map for the MusicBrainz server. The plan allows us to follow the release early, release often methodology and should hopefully make most people happy:

TemplateToolkit/Catalyst Release

  • Date due: Late March/April
  • Non schema change release
  • Deployed as beta.musicbrainz.org starting in early march
  • 100% features of 2008-11-23 release supported.
  • Full Guess Case supported, but no new features
  • Many UI improvements, including a generally speedier interface
  • Internationalization support: Support will be under the hood, but we may not immediately support new languages.
  • Ease of installing mb_server: Use modern tools to draw in more server developers.
  • Search improvements: CD Stubs. Possibly: Advanced Relationships, edits
  • Closing a whole host of old bugs, introducing many new ones. 🙂

This release doesn’t add many new features, but in general we feel that the user interface experience will improve so much that our end users will be happy with this new release. Also, since we’re keeping to a non-schema change release, we will be able to load this release and let people test with live data on beta.musicbrainz.org. I believe this well get more people to come help us test the new interface and hopefully have a smooth rollout of the new features.

NGS Release

  • Date due: To be determined — hopefully sometime this summer
  • Major schema change release
  • Full NGS object model under the hood.
  • Expose ReleaseGroups, if not other new NGS features
  • Keep old edit system in place

The major piece of work in this release is the new NGS schema/object model. How we graft the existing edit system on top of the new schema remains to be seen — this task contains many unknowns, which is why it was yanked from the upcoming release. We’ll end up doing more work overall by having the TT/Catalyst release first, but this approach allows us to release early, release often.

NGS Improvements releases (may be one or more releases)

  • Date due: weeks after NGS
  • Non-schema change releases
  • Bug fixes
  • Incremental improvements in NGS
  • Exposing new NGS features if the NGS release didn’t expose 100% of the new features

Depending on how we work out the NGS release, I can see smaller follow up releases that improve the NGS release or expose more portions of NGS that weren’t previously exposed. Exactly how this shapes up will not become clear until we near the NGS release.

Edit System Rewrite

  • Timeframe: To be determined
  • Schema change release
  • Improve may aspects of our edit system.

Our edit system has been straining under its current load for some time. This release will throw out the edit system entirely and build a new more flexible system that allows the user to change more data with fewer edits and allows the user to find edits easier.

If you have questions on the roadmap, please post them in the comments.

Improving the MusicBrainz user experience: Would you like to help?

A few people have been commenting on how MusicBrainz has a few bugs that bother them on a daily basis, yet these bugs are never fixed. From a developers perspective its really hard to see which bugs in our large bug list really matter to end users — its hard to figure out which bugs should be fixed first. Rather than fixing the bugs that affect most people first, developers tend to fix the bugs that have the most insistent users shouting for those fixes. Unfortunately fixing bugs for overly vocal users may not be the same as fixing bugs that will improve life for the most people.

Warp suggested that we check out Get Satisfaction, while others suggested using UserVoice. Both of these services provide an intermediate layer between the developers and the end users to provide feedback about the bugs that are important to them. End users can enter the bugs that bother them and everyone can vote on those bugs/issues. While we already have a bug tracking system, this system is geared to the less technical users out there — not everyone loves entering bugs into trac. Plus UserVoice’s voting mechanism allows the most important bugs to rise to the top of the list.

I’ve explored UserVoice a little since they offer a free plan for Open Source projects. (Not to mention that their site is in Orange, which is a big draw for me. 🙂 ) It seems that it would be easy to integrate this with MusicBrainz. But, it seems that this is one more chore for someone to manage and the last thing I need is to take on another chore. I’m willing to do the technical integration, but I would need some help from volunteers to manage the day-to-day operations of UserVoice. I envision volunteers to pay attention to the data that collects at UserVoice and maintain a mapping between UserVoice issues and actual bug/enhancement tickets inside the MusicBrainz bug tracker. If you already play with trac and help us maintain it, this might be another aspect in which you could help MusicBrainz.

I’d like to know:

  • Is using UserVoice a good idea?
  • Do you see any potential problems in using it?
  • Would you be interested in being an Admin on UserVoice to help manage MusicBrainz’ UserVoice site?

Please let me know in the comments!

New Picard builds for Mac available

Timothy Lee says:

After a month of trying universal builds through macports unsuccessfully, using a tool named ‘unify’ (h/t John) to lipo i386 and ppc arch bins together unsuccessfully, and bugging just about everyone I know to use their i386 tiger machine (I have an i386 leopard and a ppc tiger) unsuccessfully I’ve made the decision to halt my progress on trying to deliver a UB.

What I do have though is two builds. I have one PPC build that was built on a tiger machine and one i386 build that was built on a leopard machine. I have not had a chance to test my i386 leopard build on a tiger machine (read above failure to ‘borrow’ a tiger machine) but there is a chance it may work. If you feel like you have a better handle on the ‘lipo’ process, please, take my builds and smash them together and let me know! (It may be desirable NOT to deliver a UB as
each separate build of Picard is fairly large).

Timothy is looking for feedback on these builds. If you’ve been waiting for a complete version of Picard that includes working PUID generation, then please try these builds:

Picard for OS X Intel i386 (md5)
Picard for OS X PPC (md5)

If you have problems running these please enter a bug report and use the component “Picard Tagger (Mac OS X Packaging)“. Thanks very much Tim and everyone else who has helped along this somewhat frustrating process.

MetaBrainz Foundation Annual Report for 2008

Here is the MetaBrainz Foundation annual report for 2008:

Introduction

2008 was a busy year for MusicBrainz: Our traffic doubled from January to December, we grew from 8 to 13 servers and we earned $35,000 more than in 2007. Most importantly we started 2008 with only one full time employee and ended the year with 2 and a half full time employees. Between having more employees and having three students work on Google Summer of Code projects, we’ve had more code written in 2008 than we have ever had before. A lot of these projects were long-term projects that would take us well into 2009 before seeing benefit from this work. Consequently we had only one server update in all of 2008, but we’re expecting to have about 3 major releases this year.

Despite widespread upheaval in the world economy, MusicBrainz managed to grow and stay consistent all through 2008. We added Amazon.com, MetaWeb and Cloudspeakers as our major customers and Google sponsored MusicBrainz with another $30,000 donation. Finishing 2008 with $71,000 in the bank MusicBrainz starts 2009 strong and healthy.

Profit & Loss

In 2008 the foundation took in $126,442.26 and spent $94,717.79 for a total profit of $31,724.47. (non-profits can still earn a profit, but that profit must be reinvested into the company, not paid out to its officers and directors.) A detailed break down of where the income came from and where it went to is shown below:

Income

Direct donations $30,219.00
PayPal donations $13,271.25
Consulting (GSoC Mentor income) $1,500
Live Data Feed licenses $60,472.56
CC Data Licenses $2,400.00
Interest $1,211.92
Amazon Associates $1060.72
CD Baby affiliate $11.00
Tagger Affiliates $16,294.94
Bank Credits $0.87
Total income:   $126,442.26

Expense

Bank fees $240.90
PayPal fees $1,546.61
Rent $2,856.00
Hardware $9,107.98
Travel $2,456.61
Internet $275.41
Marketing $1,564.69
Development $60,415.16
Supplies $38.23
Gifts $100.00
Events $792.17
Hosting $11,904.50
Filing Fees $45.00
Software $239.92
Entertainment $75.15
Insurance $2008.00
Accounting $965.00
Shipping $86.46
Total expense:   $94,717.79

This Profit & Loss shows:

  • It cost $21,012.48 to host MusicBrainz in 2008 — this includes hardware and hosting costs.
  • PayPal donations, which represent end-user support for MusicBrainz, came to $13,271.25, about 10.5% of our total income. It has been our goal to have the end-users pay for hosting MusicBrainz, but this year we fell significantly short of that goal due to greatly increased hardware costs. However, if we take into account the income from the Tagger Affiliate program, then the picture improves quite a bit. Tagger Affiliate income is MusicBrainz’ 10% revenue share of end-user registrations of third party applications that use MusicBrainz. Counting these, our end user contributions total $29,566.19, which exceeds the cost to host MusicBrainz by over $9,000.
  • As expected, salaries continue to be the largest expense for The Foundation. In 2008 it paid $58,000 in salaries to Robert Kaye, which was still below a reasonable salary level for an Engineer/Executive Director in California. Its first non-founder employee Oliver Charles was paid $2,415.16 for just over three months of work.
  • $62,872.56 was earned licensing the MusicBrainz data for commercial use, which represents 49.7% of the total income. This is up over 9% from last year.

In 2008 the foundation spent, $21,012.48 on hosting and hardware costs and served out 1.3 billion web hits and 790 million web service hits. Calculating a cost per hit, we find that we spent $16.21 per one million web hits and $26.58 per one million web service hits. These values are down significantly from the 2007 values of $17.89 and $33.58 respectively.

Balance Sheet

The balance sheet for the end of 2008 showed the MetaBrainz Foundation with $39,525.20 retained earnings, a net income of $31,249.67 and total cash assets of $71,249,67.

Traffic

The following chart shows our traffic growth for 2007 and 2008:

Musicbrainz Traffic 2007-2008

The blue line represents the overall number of hits to musicbrainz.org. The red line shows how many of the overall hits were web service (API) hits; as you can see towards the end of 2007 half of our overall hits were web service hits. Towards the end of 2008 our web service hits accounted for 60.9% of our overall hits, up from 50% in 2007.

Top data contributors

Top editors:

drsaunde 141064
ojnkpjg 40796
murdos 33517
voiceinsideyou 32660
knakker 31598
brianfreud 30242
teleguise 29756
bgibbard 23676
MrH 22521
Isomer 21203
cooperaa 19808
mudcrow 19162
fred576 17634
chiark 17561
StoneyBoh 16637
zos18 16466
Jormangeud 16264
leivhe 15983
Mistoffeles 15710
Creap 15373
jesus2099 12990
Dr. Default 12099
montesquieu 11810
infofarmer 10990
nikki 10015

Top voters:

ojnkpjg 35834
drsaunde 35374
voiceinsideyou 28376
chiark 20704
mudcrow 19892
murdos 14793
leivhe 12864
bogdanb 12560
bgibbard 11898
cooperaa 10954
symphonick 10642
teleguise 9778
vincentrichter 8668
MClemo 8302
DrMuller 6847
bawjaws 6208
infofarmer 5861
PhantomOTO 5747
Kerensky97 5740
debris77 5611
lukz 5460
dmppanda 5163
krazykiwi 5097
dj_empirical 5035
Atedos 4970

A big thank you to all of the editors/voters who contributed! MusicBrainz would be nothing without your hard work!

Server farm

MusicBrainz has grown to 13 machines in service:

MusicBrainz Servers

From the top, going down:

  • moose: Our database server
  • scooby: Our aging catch all server: blog, wiki, forums, mailing lists, etc
  • catbus: Raw datavbase server (raw tags, collections, etc)
  • bender: Former TRM server, now idle cold spare machine
  • blik: Backups and memcached
  • stimpy, dexter: Web, web service servers
  • cartman: Search server
  • wiley: New catch all server: SVN, trac
  • lenny/carl: Redundant network gateways
  • misty/jem: Search servers

MusicBrainz currently requires somewhere between 3 and 4 mbits of bandwidth per second and draws 18 Amps of current for a power consumption of nearly 2,000 Watts. MusicBrainz physically occupies 20Us of space (half of a rack) at Digital West in San Luis Obispo, CA.

Words of Appreciation

2008 was an exciting year for MusicBrainz and it could not have happened without the countless volunteers who have spent inordinate amounts of time and effort working on MusicBrainz. It is not possible to thank everyone who has contributed — for starters see the list of top editors and top voters. There were so many people behind the scenes or at partner companies that are not readily apparent from the outside.

MetaBrainz would like to thank Matthew Wood and his entire team at the BBC; through Matthew’s efforts the relationship between the BBC and MusicBrainz has been flourishing. Wendell Hicken and Sergey Borisov at MusicIP are the stars at MuiscIP who look out for MusicBrainz every day. Lukáš Lalinský, Oliver Charles, Aurélien Mino and Brian Schweitzer — your contributions to the MusicBrainz server cannot be underestimated! Dave Evans, you’re MusicBrainz’ unsung hero for all of your efforts behind the scene — MusicBrainz keeps running because of you!

Thanks to everyone who helped MusicBrainz in 2008 — MusicBrainz could not exist without you!

Finally, in 2008 quite a few contributors donated chocolate to MusicBrainz as a form of appreciation. Last year working on MusicBrainz was helped along with chocolate from Germany, Switzerland, Finland, Norway, the United Kingdom. Thanks to everyone who sent in chocolate!

MetaBrainz Foundation Annual Report for 2007

(Yes, this report is a year late — I had spent a lot of time on the annual report for 2006 and got so little feedback on it that I assumed no one cared. But now in 2008 I’ve been getting many more requests, so I’ve gone back and with the help of Pavan compiled the stats for 2007.)

Profit & Loss

In 2007 the foundation took in $89,645.59 and spent $64,543.84 for a total profit of $25,101.75. (non-profits can still earn a profit, but that profit must be reinvested into the company, not paid out to its officers and directors.) A detailed break down of where the income came from and where it went to is shown below:

Income

Direct donations $30,205.00
PayPal donations $14,575.59
Consulting (GSoC Mentor income) $1,500
Live Data Feed licenses $30,340.47
CC Data Licenses $1,900.00
Interest $357.45
Amazon Associates $714.39
Tagger Affiliates $10,052,69
Total income:   89,645.59

Expense

Bank fees $64.65
PayPal fees $1,570.92
Rent $2,779.00
Hardware $732.06
Travel $2,933.49
Internet $89.01
Marketing $765.00
Development $39,500
Supplies $56.88
Events $400.92
Hosting $12,240.00
Filing Fees $25.00
Software $-41.45
Entertainment $27.32
Insurance $1,996.00
Accounting $1,200.00
Total expense:   $64,543.84

This Profit & Loss shows:

  • It cost $12,972.06 to host MusicBrainz in 2007 — this includes hardware and hosting costs.
  • PayPal donations, which represent end-user support for MusicBrainz, came to $14,575.59, 16.3% of our total income. The Foundation aims to have end-user donations pay for the actual cost of running the service and the end-users did in fact pay for slightly more than what it cost to host MusicBrainz in 2007!
  • As it is with a lot of companies, salaries accounted for the largest slice of expenses. The Foundation paid $39,500 in salaries to Robert Kaye in 2007, which was still far below a reasonable salary level for an Engineer/Executive Director in California.
  • $32,240.47 was earned licensing the MusicBrainz data for commercial use, which represents 40% of the total income.

In 2007 the foundation spent $12,972.06 on hosting and hardware costs and served out 725 million web hits and 386 million web service hits. Calculating a cost per hit, we find that we spent $17.89 per one million web hits and $33.58 per one million web service hits. These values are up slightly from the 2006 values of $17.56 and $30.58 respectively.

Balance Sheet

The balance sheet for the end of 2007 showed the MetaBrainz Foundation with $14,323.45 retained earnings, a net income of $25,101.75 and total cash assets of $39,525.20.

Traffic

The following chart shows our traffic growth for 2007 and 2008:

Musicbrainz Traffic 2007-2008
The blue line represents the overall number of hits to musicbrainz.org. The red line shows how many of the overall hits were web service (API) hits; as you can see towards the end of 2007 half of our overall hits were web service hits.

Top data contributors

Top editors:

drsaunde 125665
ojnkpjg 106211
brianfreud 102090
mudcrow 44423
murdos 32111
teleguise 26541
dmppanda 23264
cooperaa 16705
Schika 16420
zout 16160
voiceinsideyou 15268
artysmokes 13380
Kerensky97 12174
foolip 11403
lukz 11031
bench12345 10014
AnAlach 9323
gioele 8742
headlocker 8450
Pianissimo84 8364
SenRepus 8183
leivhe 7815
Liff 7748
lytron 7740
zos18 7533

Top voters:

drsaunde 47850
ojnkpjg 41365
brianfreud 27382
voiceinsideyou 19874
cooperaa 19348
headlocker 16183
lukz 13792
mudcrow 12861
gioele 11259
cadalach 10677
bogdanb 10496
CatCat 9214
symphonick 8767
teleguise 8571
Dr. Default 8313
BrianG 7936
crazee_canuck 7842
keschte 7838
cybercox 7802
Xaiver 7699
Pianissimo84 7175
murdos 7165
artysmokes 6495
helver 6349
debris77 6276

A big thank you to all of the editors/voters who contributed!