MusicBrainz Blog

We’re nearly done implementing the SQL portions of our tickets for the upcoming schema change on 15 May, 2013. We’ve settled on the following tickets that we plan to release:

• MBS-5861: Dynamic work attributes
• MBS-3978: Support more than one barcode on same release
• MBS-4756: Move the wiki transclusion index to the database
• MBS-799: Location, venue and event support
• MBS-3985: Support multiple artist countries
• MBS-4925: Add country of birth and country of death to Artist (person)
• MBS-4115: Cover art archive: Support .png SQL changes
• MBS-1839: Track MBID SQL changes
• MBS-5809: Add a “description” field to collections
• MBS-5314: Drop the work.artist_credit column
• MBS-5302: Store International Standard Name Identifier (ISNI, ISO 27729) for artists and labels
• MBS-5528: Change short_link_phrase to long_link_phrase
• MBS-2229: Allow multiple release events per release
• MBS-2417: Support multiple countries/regions on a single release
• MBS-5772: Generate relationship documentation (semi-)automatically
• MBS-5848: Instrument credits

Each of the tickets above will give you a complete idea of how we plan to change our schema on May 15th. Questions? Post a question in the comments and we will answer it.

Finally, we are going to require that Postgres 9.1 will be the minimum version of Postgres going forward. I’ve spoken to many people about this and it seems that a large percentage of people are already using Postgres 9.1, so this should not be a major change.

Thanks!

A common problem for users of MusicBrainz is that of synchronizing a local collection against the main MusicBrainz servers. Our current rate limit stipulates that you make at most 1 request per second, which we understand is extremely limiting – especially if you’re trying to fetch thousands of releases! During our first hack weekend, we created the beginnings of a service to allow you to get a list of MBIDs that have been updated. We have finished the preliminaries of this service, and now we need to hear from you how you’d want to utilize this.

Change Logs

The most basic data we currently gather is a JSON document containing a list of MBIDs that have changed per hour. For each of our data replication packets, we generate a JSON packet that summarizes all of the MBIDs that have changed, either directly on indirectly (such as the addition of more relationships).

A “What’s Changed?” Service

The first piece of feedback we received was that people were not really interested in consuming this data stream, but would rather have a service that allows them to query what data has changed in a given window of time. Having to manually fetch packets and perform set intersections is not particularly difficult, but the more hoops people have to jump through, the less likely they are to even use the service. We’ve been pondering how best to implement this service, and we would like feedback on the following options:

1. Filter a list of MBIDs

   The service would allow you to POST a set of MBIDs, and would in turn return the subset of these MBIDs that have been changed. You are able to specify any date and have all changes since that date. For example, you could find all changes to all releases in your library since you last checked 2 weeks ago.

   Because every MBID would take 36 bytes to submit, there will be a limit on the amount of MBIDs that can be submitted in order to preserve bandwidth.

2. Provide client libraries

   Rather than having people craft their own web service requests, MusicBrainz should provide a library to do this. This will allow us to use more advanced techniques (for example, Bloom filters) to both conserve bandwidth, and allow for larger queries. In this scheme the web service will be documented, but users are not expected to consume it directly.

3. Support Both!

   MusicBrainz could offer a simplified API, which is based on option 1, while also supporting larger queries through option 2. For example, we might limit option 1 to have a maximum of 4000 MBIDs per request/response, while the service that depends on our client libraries could handle many more.

4. Allow filtering based on collections

   MusicBrainz already has the concepts of collections, which have an associated unique identifier, so these will be used to filter the list of changes. This limits the service to only deal with releases, and will require people set up collections before they can do queries. Again, due to the possibility of large collections, there will likely be pagination on responses – though the per-page limit will probably be fairly high.

These are the ideas that we’ve been debating, and we’d love to know which of these would work for you. If you have other ideas, we’re also very interested in hearing what those are!

It it time for us to start the process towards the next schema change release. Starting today and for the next two weeks, we’re going to seek people to be the champion (sponsor) of a ticket. If you feel strongly about a schema change ticket getting taken care of, you should consider championing this ticket. Once you’ve decided to do adopt a ticket, you should assign the ticket to yourself.

Then, over the next two weeks it will be up to you to do the following:

1. Drive consensus around the core concept of the ticket. If you go through the process of working up a ticket, but no one agrees with what you’re proposing, you’ve wasted your time. Make sure that you get buy in from others in the community. For instance, if Nikki doesn’t like it, chances are its not going to fly.
2. Each schema change feature requires two tickets: 1) An SQL ticket that implements the actual changes to the database and defines the queries used to fetch the data. 2) A UI change ticket that implements the UI portions of the schema change ticket.
3. Ensure that the ticket clearly states what needs to be done to implement the ticket. The ticket should essentially become or link to a requirements document. This requirements document should explain what the new feature should do. It should not explain how it should be done — we should leave the how to our developers who are going to implement the feature.
4. Provide as much supporting documentation as you can. Mock-ups for UIs are deeply appreciated (even if they delve into the how realm of things) and very useful for meaningfully discussing these tickets.
5. Have the ticket reviewed by a developer for clarity and completeness, then address any issues said developer may raise.

On 15 February, we’re going to look at the list of tickets that people have taken on and choose the ones that are clear enough to move forward. If you’ve done all the work outlined above, the chances are good that your ticket will be chosen to move forward. If your ticket is chosen to move forward, there will be more questions that the developers will raise — hopefully those can be tackled in the space of a week. After that we will take all of the well defined tickets and schedule them for implementation. All the other tickets that are not clear to implement will be rejected and will have to make another pass though this process in the autumn.

If you’re still interested, here is the list of schema change tickets that should be considered for this.

We’re going to follow the this schedule:

• 1 Feb: Schema change ticket selection starts
• 15 Feb: Select schema change tickets for implementation, start making tickets fully actionable
• 1 March: Tickets must be fully actionable. Tickets that are not actionable will be dropped from the 15 May release.
• 15 March: SQL tickets must be fully implemented.
• 1 May: UI tickets must be fully implemented, start final ticket testing phase
• 15 May: Release day

All of these dates have been added to our new community calendar.

Our current web service at the /ws/2 endpoint returns too much data in a lot of cases and in many cases we suspect that the programs making the calls to the service don’t actually consume all of that data. We’d like to reduce the amount of unused data our web service returns, in order to reduce our bandwidth costs. We propose that:

• The web service will no longer includes aliases and tags in relation elements. Regardless of what entity you may request, if the results of your request includes a relation element, any alias or tag elements that are currently returned will no longer be returned.
• The web service no longer includes aliases and tags in for the Various Artists artist anywhere, unless you specifically request the Various Artist from the /ws/2/artist endpoint.

We’ve mocked up these changes in the following XML files:

• Original XML file
• Reduced XML file
• Diff between the XML files

We think that this will have a minimal impact on our web service users. If you use our web service, please tell us what you think about this. If you know someone who is using our web service, but may not read this blog, please forward a link to this post to them.

For more background on our research into this topic, please take a look at this document.

We’ve just finished releasing a small set of bug fixes and improvements to the MusicBrainz website. As I’m sure you all noticed, we had a fairly shaky last couple of weeks, so this release is a bit smaller than planned as we focused on trying to keep things stable.

Thanks to Ian McEwen, Pavan Chander and the MusicBrainz team for their work in this release. Here’s what’s new:

Bug

• [MBS-1333] – URLs in annotations remaining as plain text
• [MBS-1351] – Ratings not displayed correctly after removing personal rating
• [MBS-3492] – When search rate limiter is hit, /ws/1 returns 200 with no body
• [MBS-3590] – Release editor session timeout ISE super-ticket
• [MBS-4058] – Tags with "/" not loading
• [MBS-4439] – Search edit with "on date YYYY-MM-DD" doesn’t work as intended
• [MBS-4713] – "None" radio button is missing on edit pages
• [MBS-4714] – User pages for users with names containing / are inaccessible.
• [MBS-4781] – Entering "bogus" birth date (0-1-1) in user profile causes an exception
• [MBS-4853] – $ENV{LANGUAGE} shouldn’t end up in xml:lang and lang attributes.
• [MBS-4943] – Type/Extra Types show as being unset in the RE
• [MBS-5007] – Rating > error 400

Improvement

• [MBS-776] – Allow birth dates without a year
• [MBS-2138] – Point people to the docs when loading /ws/2/ itself
• [MBS-2199] – Context help for release type when editing release
• [MBS-3518] – Remove Chrome frame
• [MBS-4674] – Use 500px cover images
• [MBS-4843] – Autoeditor-election status names are not translatable
• [MBS-5002] – Show (bot), (limited user), etc, next to edit notes too
• [MBS-5021] – Country/Language/Script domains should have iso code in an automatic comment
• [MBS-5032] – A lot of language statistics that are likely to never get used are being collected
• [MBS-5038] – favicon.ico includes both the 16×16 NGS image (which we use) and a larger 32×32 image left over from Mason

New Feature

• [MBS-3188] – Ability to re-request email verification

Task

• [MBS-5028] – Fill in missing statistics between 2012-07-11 and 2012-07-17
• [MBS-5045] – Activate Quenya and Sindarin languages (qya, sjn)

Sub-task

• [MBS-4117] – Add language switcher

The Git commit SHA for this release is 86e645395c8d58381a6 and the git tag is v-2012-08-06.

Stefan Sperber has just announced the open source release of musicbrainz-data, a Java library that uses Hibernate to interface to the MusicBrainz PostgreSQL database:

We at Last.fm are happy to announce that we are open-sourcing musicbrainz-data, our Java data bindings for the MusicBrainz Database.

The source code for musicbrainz-data and information on how to use it in your projects can be found on our GitHub:

https://github.com/lastfm/musicbrainz-data

Please report any issues at

https://github.com/lastfm/musicbrainz-data/issues

If you have any questions, suggestions or feedback please post them in the musicbrainz-devel mailing list. I will also attempt to be available on the #musicbrainz-devel IRC channel on Freenode (nick: stefans).

Thanks Stefan!

Ollie has just published his second post on the new edit system. This post looks at the actual database structure of NES, and the motivations that lead to this final design.

We’re encouraging more discussion on NES. The most active places for this are #musicbrainz-devel on irc.freenode.org and the musicbrianz-devel mailing list.

Long after we finally delivered our much anticipated Next Generation Schema, we’re finally ready to dive in and get to work on our New Edit System. You may recall that during the NGS re-write we opted to not tackle the re-write of our edit system. Instead, we decided to do the edit system overhaul once NGS became stable.

We finally reached this point when Ollie, Warp and I met in London last month to discuss this new project. We had a productive meeting reviewing all of the research that Ollie had done over the past months. We improved the terminology here and then, but in general approved for Ollie to continue his work. Part of his work was to write about it so that our users can get familiar with the concepts that will come about in NES.

Ollie had started blogging about that on his own blog:

• NES Does It Better: Introducing the New Edit System
• New Database Abstraction Layer

If you’re interested in the New Edit System, please take a moment to read these posts and consider following Ollie’s blog.