Category Archives: Development

The end of the replication nightmare!

I’m pleased to report that our nightmare of finding/reconstructing the missing replication packets is finally over!

Through many heroic hours of work, Bitmap and Chirlu have reconstructed the missing replication packets. All clients should now be on their way to being up to date. We’ve learned a number of lessons (some good, some bad — that’s life, right?) in this ordeal and we hope to avoid these issues in the future.

An integral part of this recovery process were a number of people from our community who helped us: Users mbcz, rembo10 and xeam sent us their complete DB dumps! Bitmap used these to sanity check and diff several other database to finally extract the missing packets. Thank you for dropping what you were doing and sending us a few GB of data over blazingly fast connections. Without you this would not have been possible; and this is not an exaggeration. Thank you!

After some more rest we’re going to continue to put out smaller fires that remain from the move to NewHost, but for now, the big fires are put out. Just in time for the weekend!

In the 11 year history of the replication stream we’ve had to have users restart their stream about 3-4 times because of problems on our end. Zero would’ve been nicer, but I’m proud that we’ve been able to make this system work for so long. On a daily basis we seem to have about 400 replicated copies of MusicBrainz running all over the world. Clearly this part of our service is well used and I sleep a little better at night knowing that our most critical data is backed up across the globe.

Just for fun, here is a graph of the replication API usage over the last 6 months:

hourly-api-usage.png

Towards the end the graph shows the week plus long break, then a small blip as some of our replicas got unstuck yesterday and the much larger spike shows the rest of the replicas getting unstuck. Now, as to what caused the blip in mid-October — I have no idea.

Anyways, please accept my apologies for the replication stream outage and keep replicating!

Thanks!

State of the Onion: MetaBrainz

In the past few weeks we’ve been hit with several traffic increases to MusicBrainz which is putting considerably more strain on our aging infrastructure than we’re happy with. If it seems that we’re not doing anything about it, that is because we’ve been busy behind the scenes trying to keep things moving forward. This sometimes doesn’t leave us a lot of time to keep the public informed on our work. Hopefully this blog post will fix this in the short term:

In 2011 we started to make plans to move MusicBrainz hosting into the cloud, but then out of the blue we were donated a pile of machines. There were so many machines that I postponed the cloud plans and prepared the donated machines for service. That has carried us for 4+ years with almost no hardware cost, which was really great. The plan was to move to the cloud sometime around 2015, but then I spent most of 2014/2015 dealing with conflicts in the team, putting us seriously behind schedule while our hardware decayed.

On top of that, we’ve recently had some “bad luck”. We have had some disrespectful commercial customers hit us really hard and we had to find and block them. We have had unexpected traffic spikes and when trying to address these unexpected traffic spikes, we had two more machines fail on us. These were the donated machines that we kept in reserve for just this moment. The loss of two machines caught us short on capacity to handle the increased demands on our servers.

So, now we face the tough question: Do we buy expensive hardware that we might use for 6 months (~$5000) or do we try and save the money and tough it out? I’d rather not spend so much money on such short term use if we can avoid it. We’re going to try and move to a new hosting facility somewhere in the EU, since that is where most of our users are.

Moving to a new hosting facility has an incredible number of dependencies that Christina (our Biz Dev manager), Zas and I have been working through. It may not seem like we have a plan, but we do, and we’re incredibly busy trying to make the plan happen. To give you a taste of what we’re up against:

  1. We want to move our hosting to Europe and have a business presence in Europe in order to reduce the costs and inefficiencies of being a solely US based business. A lot of our traffic, customers and contractors are in the EU and it simply makes sense to have a presence here.
  2. To establish a presence in the EU I needed local help to help with the business matters as well as researching and establishing an EU organization. So I needed to find a Biz Dev manager and that person is Christina.
  3. Once Christina was on board she researched our options about what was suited for us. Getting that process moving involved getting certified documents from California, board approval for spending funds to establish the organization, EU labor law research, (and we needed to swap a board member, too!), hiring help to establish the org. and generally navigating the Spanish bureaucracy. (See this only slightly exaggerated short film for some clues of our ordeal.)
  4. Once the org. had been established we needed to convince the bank to open a bank account for us. The draconian US banking laws extend worldwide and the local bank had to ensure that they were not opening themselves up to thousands of $$$ in accounting hassles just to allow a tiny non-profit to open a bank account. We finally have a bank account and have started paying our contractors with it!
  5. At the same time we’re also working to set up an office for the growing team here in Barcelona. That required a byzantine process that barely started when you sign the lease. Getting power, internet and water set up has taken a frustratingly long time. Had I known how long, I would have stayed at my co-working space for a while longer while addressing hosting issues.
  6. While Christina has been focused on the hardcore paperwork, Zas is keeping the site running, which itself requires many heroics. Zas and I have started planning the move to the EU hosting provider. We’ve got a 5-page document that collects some of the open questions and requirements around this process: https://docs.google.com/document/d/16KNm4KksNwz29Opk1aILOMtCmPIeXFuxxUjMoPT3th0/edit#heading=h.dpfvoz1idcro. Right now Zas and Bitmap are here in Barcelona and we’re going to work on establishing a formal plan for moving to the new hosting company. We’re currently comparing hosting company offerings – see what we’ve collected so far if you care to follow along. The amount of work required to make this happen is making my head hurt. (A special shoutout to KodeStar, lead developer of FanArt.tv, for providing a lot of useful feedback about our various options.)
  7. While Christina, Zas and I have our hands full, Bitmap and Gentlecat continue to release new features and work on the schema change. Not to mention all the contributions from Freso and Reosarevok to keep the community happy and polite while we deal with less than optimal site conditions. That said, I am really happy and proud of my team, trying to keep things running in sub-optimal conditions.

This is just a snapshot of everything that is happening behind the scenes that will culminate with the goal of moving to a new hosting company and being set up in the EU. And mind you, we’re doing this with a minuscule budget trying to be careful of how we spent our money.

Announcing python-musicbrainzngs, release 0.6

From the better late than never department…

After more than 2 years we’ve finally released version 0.6 of python-musicbrainzngs, a library for accessing the Musicbrainz webservice from python.

After such a long time we have perhaps too many new changes to describe. Some major changes include:

  • Better handling of authentication private user collections
  • Support for loading all types of user collections (artist, event, place, recording, release, work)
  • Work attributes
  • Support for the Cover Art Archive
  • Support for Events, Instruments, Places, and Series

And numerous other bug fixes and small changes. See the CHANGES file  for more information.

This release contains contributions by Alastair Porter, Corey Farwell, Ian McEwen, Jérémie Detrey, Johannes Dewender, Pavan Chander, Rui Gonçalves, Ryan Helinski, Shadab Zafar, and Wieland Hoffmann. Thank you everyone!

 

The new version can be downloaded from github, pypi, or installed with pip

Sandbox upgrade and downtime: Tuesday, 24 November, 2015 @ 17:00 UTC

Tomorrow, Tuesday, 24 November, 2015 at 17:00 UTC, we will be bringing down rika, our sandbox server, to update the operating system. We expect this to take approximately two and a half hours, during which time rika will be updated to Ubuntu 14.04 LTS. If you are running any services (such as a MusicBrainz sandbox) on rika, remember to back up your data (like the MOTD says!), and be aware that you will need to restart these services once the server is available. Thank you for understanding the brief inconvenience as we put shiny new things in place.

UPDATE: Unfortunately, something has gone wrong during the update process. rika is responding to pings, but no services have come up. We will do our best to resolve this quickly, but there is currently no ETA.

Updated libcoverart-1.0.0.tar.gz

It was brought to my attention that the tarball for the libcoverart 1.0.0 release wasn’t built correctly, and contained files within a confusingly named subdirectory.

I’ve rebuilt the tarball, and made it available via GitHub as usual. Please note that as a result of this, the MD5 checksum of the tarball has changed. The file content of the archive is, however, unchanged.

The new tarball is available here: libcoverart-1.0.0.tar.gz (MD5 checksum: 856d83a4e57a2325c168eb979b9c00d8)

The MusicBrainz documentation page for the library was also updated to reflect this.

Roman Tsukanov joins the MetaBrainz team

I’m pleased to announce that last week we officially hired Roman Tsukanov, AKA Gentlecat to be a part time developer for MusicBrainz!

Gentlecat has already established himself firmly in our community: Last year he rocked the CritiqueBrainz project for Summer of Code and this summer he rocked AcousticBrainz. And he’s written our shiny new MetaBrainz web site! He is now in the process of learning perl and has started to help Bitmap review existing code reviews. And he has even fixed a couple of issues already. In other words he lives up to his name: To Gentlecat something means to rock it!

I’m quite happy to have such a capable developer participating in MusicBrainz. Welcome to the team Gentlecat!

Poisonous people

As I work to re-shape our community, I’ve been brushing up on open source philosophy. I’ve already mentioned “Producing Open Source Software” book, which is great, but can be somewhat dry at times. A more recent book that covers more of the team related issues is the excellent “Team Geek“. In particular the chapter on “Poisonous people” is really well worth reading.

One passage summarizes some underlying themes of my current series of blog posts:

“But there’s a danger here. In general, it’s not healthy to spend one’s time stewing in an ocean of negativity—it tends to eat you up and can create more conflicts in the long run. The term poisonous person is a nasty label and automatically creates a dividing line between “us” (the good guys) and “them” (those nasty jerks). There’s a better way to think about the problem. Instead of running your team as an elite fraternity with a mission to repel mean people, it’s healthier to create a culture that simply refuses to tolerate certain negative behaviors.”

Our community currently has a number of disrespectful people who could be classified as “poisonous”. However, I have good news for these people: I’m wiping the slate clean! Everyone gets a fresh start. As of today no one is poisonous! Hooray! \ø/

However, if you decide to go against the updated principles I’m stating in this blog post series, you should expect that people will confront your behaviour. One point of these blog posts is to provide bite-sized chunks of information that can be used to succinctly state updated project policies. If you encounter a person who is communicating aggressively, is being disrespectful or is hindering the community from moving the project forward, please remind them of these blog posts. When you encounter a problematic person, post the link to this blog entry and respectfully remind them to improve their behaviour.

I would also like to propose that we organize a group of volunteers to form what could be called: “Team Respect“. If someone in the community finds a person who is going on a “no-vote spree” or is entering useless and aggression filled tickets into jira, then they can call on Team Respect to jump in. Team Respect should engage this person, link to the relevant blog posts/project policies and remind them to be respectful. The team should also review the actions of the person — if the team feels that the actions by this person were uncalled for, the team should work to counter-act their actions. For instance, if someone goes an on unjustified no-vote spree, the team members should evaluate the edits in question and if they “improve the database” then they should vote yes on the edits, improve the edits to remove the objections or even approve the edits outright if they are auto-editors. Use your best judgement, of course!

The basic idea behind Team Respect is to make being a disrespectful person no fun and frustrating. At the same time Team Respect can raise positive feelings in the community by working together and protecting the work for polite, respectful people. And the polite, respectful people whose work was protected will be more inclined to contribute in the future.