Massive connectivity issues

As you are probably aware, we’ve been having lots of network connectivity issues with all services hosted at Digital West in California (all of our projects, except ListenBrainz and AcousticBrainz).

Today we spent all morning trying to replace what we thought to be a faulty switch. That process didn’t go very well at all – we hit every conceivable issue that we could’ve hit. And a few more.

But, in this process we connected our gateway machines directly to our uplink (not through our switch) and the network issues persisted! After testing this setup with both of our machines, we’ve now conclusively eliminated all of our equipment as the possible source of trouble.

At this point our troubles lie in the hands of Digital West to fix. Thankfully the day staff will return to work in a few hours and hopefully we will make some progress on this issue then.

Sorry for all of this hassle. 😦

20 thoughts on “Massive connectivity issues

  1. PATATE12

    For the time being, I am using http:/musicbrainz.org (not HTTPS), it works. 🙂

  2. Dave

    will this be causing timeouts for musicbrainz server replication? server slave.sh is failing: 550 cant connect to metabrainz.org:443

  3. TCosbyJr

    After trying every trick online I could find to get Picard to load my albums today (as it always worked before), I stumbled across this. Hopefully you get the site up and running again soon. Strange that I feel lost now as I am unable to keep working on my mp3s – as I’ve been at it for several days 12+ hours daily till today as I have nearly 20000 that I have ripped from vinyl/8-track/cassette/cds and was hoping to finally “finish” fixing tags and getting rid of duplicates. Hurry back.

  4. oceanian

    Looking like the Digital West day-shift wasn’t enough.
    http:/musicbrainz not working here.

  5. InvisibleMan78

    @Dave: Yes, the same here, no more replication for a virtual machine slave server.

  6. TCosbyJr

    The website is loading fine now and so is my albums in Picard – looks like you got it solved. Glad it’s back. 🙂

  7. Kelsey Bjarnason

    As a regular user, allow me to say thank you for the service, thank you for the update, and been there done that on the outage. We get it, no criticism.

  8. stragu

    It seems the front page, entities and editing all work at the moment. However, documentation (i.e. the wiki) are not accessible. I recommend anyone needing to access the documentation to use the Internet Archive’s Wayback Machine, like so: https://web.archive.org/web/20160609042135/https://beta.musicbrainz.org/doc/Style/Titles#Extra_title_information

    Thanks for the hard work, sorry to hear about the amount of issues in the last few weeks, but great work on how transparent you are about it.

  9. Michael

    I’m having issues with replication. I had set up a new VM server and replication had made it to about April I think. (I tried the fresh database load and it didn’t work, so I was doing replication cause it was working). Now I’m stuck at: Replication packet #95626 not available. Did something with replication go down? It was working after this connectivity problem, but did some or one of the replication packets get lost from all of this?

  10. salvationtower

    Does your replication save the log (to slave.log) check the last line of that, as I was getting similar issues with packet not found and error connecting tonmetabrainz.org. there has also been a schema change. (I’ve just got mine up to date bow)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s