The MetaBrainz ticket tracker (which, incidentally, received a long-needed upgrade recently – thanks, Gentlecat!) is an important tool for all of our projects. It collects all kinds of bug reports, feature requests and other tasks to be done and makes sure none are forgotten.
One of its auxiliary features is the possibility for users to vote for a ticket, to indicate which tickets they consider particularly important. (There are only upvotes; you can’t vote against a ticket.) This may factor in when MetaBrainz employees decide on which tickets to tackle next, although there are other factors as well such as the complexity and the impact of a particular issue.
In the past, who voted for which tickets has been private, mostly because that is the default setting in JIRA, the ticket tracker software used. Only administrators can see the list of voters for a ticket; regular users just see the number of votes.
Now, we have decided to change that: In the future, all logged-in users will be able to see who voted for a ticket. This should not be sensitive information; whoever expressed their support for a ticket by commenting on it instead of (or in addition to) voting already was in the public eye. Still, it is a policy change. We’ve therefore decided to wait two weeks before implementing the new privileges, in order to give everybody the chance to remove any votes that they don’t want to be known with. The ticket tracker provides a list of all tickets that you have voted for.
I’m pleased to report that our nightmare of finding/reconstructing the missing replication packets is finally over!
Through many heroic hours of work, Bitmap and Chirlu have reconstructed the missing replication packets. All clients should now be on their way to being up to date. We’ve learned a number of lessons (some good, some bad — that’s life, right?) in this ordeal and we hope to avoid these issues in the future.
An integral part of this recovery process were a number of people from our community who helped us: Users mbcz, rembo10 and xeam sent us their complete DB dumps! Bitmap used these to sanity check and diff several other database to finally extract the missing packets. Thank you for dropping what you were doing and sending us a few GB of data over blazingly fast connections. Without you this would not have been possible; and this is not an exaggeration. Thank you!
After some more rest we’re going to continue to put out smaller fires that remain from the move to NewHost, but for now, the big fires are put out. Just in time for the weekend!
In the 11 year history of the replication stream we’ve had to have users restart their stream about 3-4 times because of problems on our end. Zero would’ve been nicer, but I’m proud that we’ve been able to make this system work for so long. On a daily basis we seem to have about 400 replicated copies of MusicBrainz running all over the world. Clearly this part of our service is well used and I sleep a little better at night knowing that our most critical data is backed up across the globe.
Just for fun, here is a graph of the replication API usage over the last 6 months:
Towards the end the graph shows the week plus long break, then a small blip as some of our replicas got unstuck yesterday and the much larger spike shows the rest of the replicas getting unstuck. Now, as to what caused the blip in mid-October — I have no idea.
Anyways, please accept my apologies for the replication stream outage and keep replicating!
In the past few weeks we’ve been hit with several traffic increases to MusicBrainz which is putting considerably more strain on our aging infrastructure than we’re happy with. If it seems that we’re not doing anything about it, that is because we’ve been busy behind the scenes trying to keep things moving forward. This sometimes doesn’t leave us a lot of time to keep the public informed on our work. Hopefully this blog post will fix this in the short term:
In 2011 we started to make plans to move MusicBrainz hosting into the cloud, but then out of the blue we were donated a pile of machines. There were so many machines that I postponed the cloud plans and prepared the donated machines for service. That has carried us for 4+ years with almost no hardware cost, which was really great. The plan was to move to the cloud sometime around 2015, but then I spent most of 2014/2015 dealing with conflicts in the team, putting us seriously behind schedule while our hardware decayed.
On top of that, we’ve recently had some “bad luck”. We have had some disrespectful commercial customers hit us really hard and we had to find and block them. We have had unexpected traffic spikes and when trying to address these unexpected traffic spikes, we had two more machines fail on us. These were the donated machines that we kept in reserve for just this moment. The loss of two machines caught us short on capacity to handle the increased demands on our servers.
So, now we face the tough question: Do we buy expensive hardware that we might use for 6 months (~$5000) or do we try and save the money and tough it out? I’d rather not spend so much money on such short term use if we can avoid it. We’re going to try and move to a new hosting facility somewhere in the EU, since that is where most of our users are.
Moving to a new hosting facility has an incredible number of dependencies that Christina (our Biz Dev manager), Zas and I have been working through. It may not seem like we have a plan, but we do, and we’re incredibly busy trying to make the plan happen. To give you a taste of what we’re up against:
- We want to move our hosting to Europe and have a business presence in Europe in order to reduce the costs and inefficiencies of being a solely US based business. A lot of our traffic, customers and contractors are in the EU and it simply makes sense to have a presence here.
- To establish a presence in the EU I needed local help to help with the business matters as well as researching and establishing an EU organization. So I needed to find a Biz Dev manager and that person is Christina.
- Once Christina was on board she researched our options about what was suited for us. Getting that process moving involved getting certified documents from California, board approval for spending funds to establish the organization, EU labor law research, (and we needed to swap a board member, too!), hiring help to establish the org. and generally navigating the Spanish bureaucracy. (See this only slightly exaggerated short film for some clues of our ordeal.)
- Once the org. had been established we needed to convince the bank to open a bank account for us. The draconian US banking laws extend worldwide and the local bank had to ensure that they were not opening themselves up to thousands of $$$ in accounting hassles just to allow a tiny non-profit to open a bank account. We finally have a bank account and have started paying our contractors with it!
- At the same time we’re also working to set up an office for the growing team here in Barcelona. That required a byzantine process that barely started when you sign the lease. Getting power, internet and water set up has taken a frustratingly long time. Had I known how long, I would have stayed at my co-working space for a while longer while addressing hosting issues.
- While Christina has been focused on the hardcore paperwork, Zas is keeping the site running, which itself requires many heroics. Zas and I have started planning the move to the EU hosting provider. We’ve got a 5-page document that collects some of the open questions and requirements around this process: https://docs.google.com/document/d/16KNm4KksNwz29Opk1aILOMtCmPIeXFuxxUjMoPT3th0/edit#heading=h.dpfvoz1idcro. Right now Zas and Bitmap are here in Barcelona and we’re going to work on establishing a formal plan for moving to the new hosting company. We’re currently comparing hosting company offerings – see what we’ve collected so far if you care to follow along. The amount of work required to make this happen is making my head hurt. (A special shoutout to KodeStar, lead developer of FanArt.tv, for providing a lot of useful feedback about our various options.)
- While Christina, Zas and I have our hands full, Bitmap and Gentlecat continue to release new features and work on the schema change. Not to mention all the contributions from Freso and Reosarevok to keep the community happy and polite while we deal with less than optimal site conditions. That said, I am really happy and proud of my team, trying to keep things running in sub-optimal conditions.
This is just a snapshot of everything that is happening behind the scenes that will culminate with the goal of moving to a new hosting company and being set up in the EU. And mind you, we’re doing this with a minuscule budget trying to be careful of how we spent our money.
From the better late than never department…
After more than 2 years we’ve finally released version 0.6 of python-musicbrainzngs, a library for accessing the Musicbrainz webservice from python.
After such a long time we have perhaps too many new changes to describe. Some major changes include:
- Better handling of authentication private user collections
- Support for loading all types of user collections (artist, event, place, recording, release, work)
- Work attributes
- Support for the Cover Art Archive
- Support for Events, Instruments, Places, and Series
And numerous other bug fixes and small changes. See the CHANGES file for more information.
This release contains contributions by Alastair Porter, Corey Farwell, Ian McEwen, Jérémie Detrey, Johannes Dewender, Pavan Chander, Rui Gonçalves, Ryan Helinski, Shadab Zafar, and Wieland Hoffmann. Thank you everyone!
The new version can be downloaded from github, pypi, or installed with pip
Tomorrow, Tuesday, 24 November, 2015 at 17:00 UTC, we will be bringing down rika, our sandbox server, to update the operating system. We expect this to take approximately two and a half hours, during which time rika will be updated to Ubuntu 14.04 LTS. If you are running any services (such as a MusicBrainz sandbox) on rika, remember to back up your data (like the MOTD says!), and be aware that you will need to restart these services once the server is available. Thank you for understanding the brief inconvenience as we put shiny new things in place.
UPDATE: Unfortunately, something has gone wrong during the update process. rika is responding to pings, but no services have come up. We will do our best to resolve this quickly, but there is currently no ETA.
It was brought to my attention that the tarball for the libcoverart 1.0.0 release wasn’t built correctly, and contained files within a confusingly named subdirectory.
I’ve rebuilt the tarball, and made it available via GitHub as usual. Please note that as a result of this, the MD5 checksum of the tarball has changed. The file content of the archive is, however, unchanged.
The new tarball is available here: libcoverart-1.0.0.tar.gz (MD5 checksum: 856d83a4e57a2325c168eb979b9c00d8)
The MusicBrainz documentation page for the library was also updated to reflect this.
I’m pleased to announce that last week we officially hired Roman Tsukanov, AKA Gentlecat to be a part time developer for MusicBrainz!
Gentlecat has already established himself firmly in our community: Last year he rocked the CritiqueBrainz project for Summer of Code and this summer he rocked AcousticBrainz. And he’s written our shiny new MetaBrainz web site! He is now in the process of learning perl and has started to help Bitmap review existing code reviews. And he has even fixed a couple of issues already. In other words he lives up to his name: To Gentlecat something means to rock it!
I’m quite happy to have such a capable developer participating in MusicBrainz. Welcome to the team Gentlecat!