Please tell us what you think about our proposed "no waiting" access to our Web Service

We’re working to add a paid option to our Web Service for commercial users and for end users who would like to have faster access. We’re finally getting close to being able to offer this service and we would like to get some feedback from our users about this.

We are proposing to add “no waiting” access to our web service — this proposed service would:

  1. Allow continuous sequential access to our version 2 web service without delays between calls. You would not be required to have any delays between calls to the web service (our current service requires a 1 second delay between calls)
  2. Still have a global rate limit that may temporarily deny callers access to our web service (with 503 responses) if our service gets overloaded. We would work hard to ensure that our service would not reach this limit, since its a paid service, but we cannot guarantee that.
  3. Not allow concurrent (more than one call at a time) calls to our web service per user. We reserve the right to terminate your service if we find that you are making concurrent calls to our “no waiting” service.

Our existing web service will not be affected by this new service — the existing service will remain free and limited to one request per second as it is now. Initially the new service is intended for end-users who wish to have faster access to our web service. Once we’ve ironed out the kinks in this new service we will offer this service for commercial customers as well.

Finally, we’ve set up a very short survey (3 questions only!) to gather some feedback from you about this service. We’re mainly trying to establish a reasonable price for this service. Please take our survey and let us know what you think and how much you’d pay for our service.

All of your responses will be private and the survey does not ask for any information about you. Thanks, we appreciate your thoughts!

User agent based throttling is now live

Yesterday we talked about rolling out our throttling based on User-Agent strings. A few minutes ago we pushed this feature live on our servers so now the updated rules are in effect. python-musicbrainz/0.7.3 users are now allowed 500 requests every 10 seconds and every single one of these requests is constantly being used. No surprise here. 🙂

For the exact details on what is throttled and how to get around your application being throttled, see our rate limiting documentation.

Current web service rate limiting documentation

We’ve just added a page that documents what we’re currently blocking on our Web Service. We hope to lift the block on python-musicbrainz/0.7.3 tomorrow and instead throttle the number of requests it can make in a given period of time.

I’ll post another entry once we’re done with making those changes.

Dear python-musicbrainz/0.7.3 application, we need to talk!

An application that uses our python-musicbrainz/0.7.3 client library has been putting undue load on our servers all at once. This application looks up something at MusicBrainz at 03:00UTC causing our servers to be overloaded at that time each day.

To protect our servers from being overloaded we’re going to block this application from 3:00 UTC – 4:00 UTC. We’re hoping that this will alllow us to identify the application and start a dialog with the application authors. Once we have established communication with the authors and worked up a plan to fix this, we’re going to release the block.

We really dislike blocking applications, but if applications are being inconsiderate of our resources, we’re left with few options. We hope to hear from the application authors soon so we can resolve this issue. Also, we’re moving forward with our plans to require User-Agent strings that properly identify applications using our service to fix this problem going forward.

If you are the author of said application, please leave a comment with information on how we can get in touch with you.

Release editor service interruption

In an effort to mitigate/fix MBS-3379 we need to restart the service that keeps the session information for our release editors. We’re going to do that tomorrow Saturday October 28 at Noon PDT, 3PM EDT, 8PM London, 9PM Amsterdam. If at this time you have a release editor open, submitting your edits will fail and you will need to start your edits over again.

Sorry for the inconvenience.

Web service user-agent string blocking reminder

I would like to remind Web Service users that on 16 November we’re going to block generic User-Agent strings from accessing our web service. Earlier we said:

The User-Agent string needs to identify the application and the version of the application that is making the request; having a generic User-Agent string like “Java/1.6.0_24″ or “PHP/5.3.4″ does not allow us to properly identify the application making the requests.

IMPORTANT: 6 Months after we release NGS (Nov 16th) we’re going to start blocking common generic User-Agents strings, so please make sure that you send us a proper User-Agent header as part of your request.

You have been warned. 🙂

The FreeDB gateway has been updated to NGS!

I’m pleased to announce that the FreeDB gateway, which lets you fetch MusicBrainz data via FreeDB enabled applications, has been updated to use the NGS database. As of now, its updating with new data from the main MusicBrainz server and you should be able to look up new CDs.

To use the FreeDB gateway, set the FreeDB server in your application to freedb.musicbrainz.org on port 80. For more information, please take a look at our wiki page for the gateway.

Thanks to Lukas for porting this code to NGS!

Record traffic, new server in NGS rotation

We’re currently seeing record levels of traffic right now. Since the beginning of August our traffic has gone from 9M hits per day to 14M hits per day today, which is a significant increase in traffic. Fortunately our servers were able to handle the extra load, but we started getting near capacity.

Yesterday I started working on taking one of the servers from the classic site and moving it into the NGS cluster. We finished this a couple of hours ago and we now have a maximum limit of 227 queries per second (qps).

If you find any spurious troubles with the site in the next day or so, please open a ticket.

Google uses MusicBrainz data in some of its searches!

Earlier this week I met with Shawn Simister, who works on Google’s Freebase project (former from MetaWeb) to touch base about how MusicBrainz is being utilized inside of Google. MusicBrainz represents a large chunk of the music data in Freebase and in turn the Freebase data is used as one of the sources of data for Google’s search.

Shawn explains this in more detail:

You can actually see a couple areas where we’re using the Freebase music data publicly. First, in the structured refinements in search. If you search for lady gaga albums and scroll to the bottom to see “Album searches for Lady Gaga”. Also you can see videos clustered by topic in YouTube Topics and many of the topics are music-related.

It’s important to keep in mind that Musicbrainz is just part of the solution. It’s a pretty big part of Freebase music data and therefore its likely to be a pretty big component in these results but as you know the search results team at Google is pretty secretive about what all goes into the results page so even I can’t tell for certain when they’re using Freebase/Musicbrainz data for any given result.

I think it’s important that people don’t mistake this as a one-to-one relationship between Musicbrainz data and Google results because there are quite a few steps in between but there’s definitely a strong connection there and we really appreciate everything that the Musicbrainz community is doing and hope that Musicbrainz community continues to grow.

I find this tremendously exciting to hear, since I proposed a very similar thing to Google many years ago. While this idea was rejected back in the day, I’m excited to see that Google is now using our data for it searches. Every person who has ever contributed to MusicBrainz should be proud!

Thank you to everyone and thank you Shawn for shedding some light on this!