Massive connectivity issues

As you are probably aware, we’ve been having lots of network connectivity issues with all services hosted at Digital West in California (all of our projects, except ListenBrainz and AcousticBrainz).

Today we spent all morning trying to replace what we thought to be a faulty switch. That process didn’t go very well at all – we hit every conceivable issue that we could’ve hit. And a few more.

But, in this process we connected our gateway machines directly to our uplink (not through our switch) and the network issues persisted! After testing this setup with both of our machines, we’ve now conclusively eliminated all of our equipment as the possible source of trouble.

At this point our troubles lie in the hands of Digital West to fix. Thankfully the day staff will return to work in a few hours and hopefully we will make some progress on this issue then.

Sorry for all of this hassle. 😦

20 thoughts on “Massive connectivity issues”

  1. For the time being, I am using http:/ (not HTTPS), it works. 🙂

  2. will this be causing timeouts for musicbrainz server replication? server is failing: 550 cant connect to

  3. After trying every trick online I could find to get Picard to load my albums today (as it always worked before), I stumbled across this. Hopefully you get the site up and running again soon. Strange that I feel lost now as I am unable to keep working on my mp3s – as I’ve been at it for several days 12+ hours daily till today as I have nearly 20000 that I have ripped from vinyl/8-track/cassette/cds and was hoping to finally “finish” fixing tags and getting rid of duplicates. Hurry back.

  4. Looking like the Digital West day-shift wasn’t enough.
    http:/musicbrainz not working here.

  5. @Dave: Yes, the same here, no more replication for a virtual machine slave server.

  6. The website is loading fine now and so is my albums in Picard – looks like you got it solved. Glad it’s back. 🙂

  7. As a regular user, allow me to say thank you for the service, thank you for the update, and been there done that on the outage. We get it, no criticism.

  8. Page Not Found … thumbs up that it get’s solved soon. Love your services.

  9. It seems the front page, entities and editing all work at the moment. However, documentation (i.e. the wiki) are not accessible. I recommend anyone needing to access the documentation to use the Internet Archive’s Wayback Machine, like so:

    Thanks for the hard work, sorry to hear about the amount of issues in the last few weeks, but great work on how transparent you are about it.

  10. I’m having issues with replication. I had set up a new VM server and replication had made it to about April I think. (I tried the fresh database load and it didn’t work, so I was doing replication cause it was working). Now I’m stuck at: Replication packet #95626 not available. Did something with replication go down? It was working after this connectivity problem, but did some or one of the replication packets get lost from all of this?

  11. Does your replication save the log (to slave.log) check the last line of that, as I was getting similar issues with packet not found and error connecting there has also been a schema change. (I’ve just got mine up to date bow)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.