Tag: downtime

Current ListenBrainz server issues/high load, Part 2

This follow-up post covers how and why our servers are still being impacted, an explanation and examples of what we consider “botting” in ListenBrainz – and an apology, because our last post had some baaad comms!

Firstly, we are still experiencing massive server load from an increase in listen submissions related to the BTS fan exodus from last.fm (not all bad! more on this below) – and we are now being hit by another influx of AI scrapers.

Our apologies to our users, new and old, as we continue to work to mitigate the extra load. Remain assured that listens are not being lost – our processing queue is lagging behind, so listens may take (up to) hours to appear.

LB radio and the popularity endpoints have been temporarily disabled, to reduce server load.

Continue reading “Current ListenBrainz server issues/high load, Part 2”

Current ListenBrainz server issues/high load

Our apologies if you are experiencing outages or slowness on ListenBrainz currently. First it was AI scrapers, now it’s K-pop fans…

In short, we are experiencing massive server load from an unexpected and massive increase in listen submissions. Please bear with us as the small (but mighty) ListenBrainz team works on possible mitigations.

None of your listens are being lost, they are all sitting safely in our (massively overloaded) processing queue:

Try to spot when the trouble started!
Continue reading “Current ListenBrainz server issues/high load”

Downtime for PostgreSQL / MusicBrainz schema change upgrade: Monday, May 11, 15:00 UTC

On Monday, May 11, at 15:00 UTC (8am PT, 11am ET, 5:00pm CEST), we’ll be:

  • Upgrading our production database server to PostgreSQL v18.
  • Performing the MusicBrainz schema version 31 upgrade.

See the previous announcement for more information.

We’ll be working to restore services as quickly as possible, but expect MusicBrainz, ListenBrainz, the Cover Art Archive, and BookBrainz to be down for the hour. Thanks in advance for your patience!

Afterward, we’ll post instructions on the blog about how to upgrade your MusicBrainz mirror server.

Downtime today for PostgreSQL / MusicBrainz schema change upgrade: 17:00 UTC (10am PT, 1pm ET, 7pm CEST)

Today (Monday, May 13) at 17:00 UTC (10am PT, 1pm ET, 7:00pm CEST), we’ll be:

  • Upgrading our production database server to PostgreSQL v16.
  • Performing the MusicBrainz schema change upgrade.

See the previous announcement on this topic for more information.

Expect MusicBrainz and services that depend on its database (MetaBrainz, ListenBrainz, the Cover Art Archive, CritiqueBrainz, BookBrainz) to be down for the hour, but we’ll be working to restore services as quickly as possible.

Afterward, we’ll post instructions here on how to upgrade your MusicBrainz mirror server (whether using musicbrainz-docker or otherwise).

P.S. The initially announced upgrades for MusicBrainz search engine are just about to reach our beta website, and thus are postponed for mirrors too.

Schema change release: Today at 17h UTC

We’re going to start our schema change release process today at 17h UTC.

We anticipate having a short downtime of a few minutes as we”ll need to restart our database server. As usual, we’re not certain when we will start the downtime, but we’ll keep people posted about our progress in IRC and on Twitter.

Once we’re done with the release we will post instructions on this blog on how to upgrade any replicated instances of MusicBrainz you might be running.

Stay tuned!

Final sprint towards NewHost

So tomorrow is the day when the old servers at Digital West will be put to rest. Our developers and system administrator have been hard at work over the last few weeks (and especially the last few days!) getting everything ready for a hopefully smooth transition to hosting everything at Hetzner in Germany.

The vast majority of our sites and services are in fact already being served from Germany, but the largest one—musicbrainz.org—remains in the US for another night.

Our transition of the final services, such as musicbrainz.org, has already started, but the very last sprint of moving the remaining bits over will start tomorrow, November 8th, at around 6 AM EST / noon UTC / 13:00 CET and will follow the plan laid out in this Google document:
https://docs.google.com/document/d/1MgqZ4hiKC0MZJ400ZCOD8JlcjBjh4GNpakauf51YKQQ/edit?usp=sharing

Expect some downtime, plenty of read-only time, and general wonkiness as we get the last gears set in place to hopefully keep us running smoothly for the next several years to come!

Sandbox upgrade and downtime: Tuesday, 24 November, 2015 @ 17:00 UTC

Tomorrow, Tuesday, 24 November, 2015 at 17:00 UTC, we will be bringing down rika, our sandbox server, to update the operating system. We expect this to take approximately two and a half hours, during which time rika will be updated to Ubuntu 14.04 LTS. If you are running any services (such as a MusicBrainz sandbox) on rika, remember to back up your data (like the MOTD says!), and be aware that you will need to restart these services once the server is available. Thank you for understanding the brief inconvenience as we put shiny new things in place.

UPDATE: Unfortunately, something has gone wrong during the update process. rika is responding to pings, but no services have come up. We will do our best to resolve this quickly, but there is currently no ETA.

Schema upgrade downtime: Monday, 18 May, 2015 @ 17:00 UTC

Our next schema change version will be released on Monday, 18 May, 2015 around 10am PDT/1pm EDT/17:00 UTC/18:00 BST/19:00 CEST. We expect that MusicBrainz will be unavailable for 15 – 30 minutes during this time. We will put up the downtime notification on the site and tweet from @musicbrainz right before the release.

Since we’re total slackers, we still haven’t set up our backup database server since it suffered a hardware failure. This means that we won’t be able to put the site into read-only mode and will require us to do a full downtime. Hopefully for our next schema change we’ll have tackled our backlog of sysadmin duties and will have a backup DB server to make the release easier.

Sorry for any trouble this may cause you.

P.S. Look for another blog post on Sunday for details on where to get your access tokens for the Live Data Feed.

Service downtime to fix some database issues

This Friday we’re going to need to take a 15-20 minute downtime to fix a few leftover issues from our recent schema change. We tried to do this without downtime, but the service got progressively slower, so we’re electing to take some downtime.

We’ll be down shortly after Noon PST, 3PM EST, 20:00 UK, 21:00 CET for about 15-20 minutes.

Sorry for the hassles this causes.