Notifications and messaging in MetaBrainz projects

During the last MusicBrainz summit in Barcelona we decided to start working on finding possible ways to implement two features that have been requested for a long time:

  1. Messaging between users
  2. Notifications about various actions in MetaBrainz projects

Since MetaBrainz is more than just MusicBrainz these days, we also want to integrate these features into other projects. That, for example, means when a user is reading reviews on CritiqueBrainz they can see notifications about comments on their edits on MusicBrainz. Same applies to messaging. These features are intended to encourage our communities to communicate more easily with each other.

Messaging

http://tickets.musicbrainz.org/browse/MBS-8721

The only ways of communication we have right now are two IRC channels, forums that we plan to replace with Discourse, and comments on individual edits. Sometimes we end up sending private emails to editors for one reason or another. Perhaps it is better to have our own messaging system for this purpose? I imagine it being similar to messaging systems on forums, reddit, etc. We would like to know what you think potential uses are for this and how it might look like to be useful.

Notifications

http://tickets.musicbrainz.org/browse/MBS-1801

Site-based notifications are another thing that people have been asking for a long time. For example, these notifications can be related to edits on MusicBrainz, reviews on CritiqueBrainz, datasets in AcousticBrainz, etc. It can be an addition or replacement for email notifications that we currently have in MusicBrainz. Maybe something similar to the inbox feature that the Stack Exchange network has. People should be able to choose if they want to keep receiving email notifications or only use the new site-based notifications.

Progress so far

We looked at a couple of ways to implement this functionality.

First suggestion was to use the Layer toolkit. The problem with it is that we don’t want to be dependent on closed software and another company’s infrastructure, especially in case of such important features.

Second was to use the XMPP protocol to handle communication and notifications. We tried to implement a proof of concept using this protocol and encountered several issues at the start:

  • It’s unclear how to store messages and process them later;
  • It can be problematic to reuse the same connection in different browser;
  • There are plenty of things that we’ll need to implement on top of this protocol ourselves (like authentication, storage, notifications).

Repository with everything that was implemented so far is at https://github.com/metabrainz/xmpp-messaging-server. Given these problems we started considering implementing our own server(s) for this purpose.

You can take a look at the document where we collect most information about current progress.

Feedback

There’s plenty of feedback on the site-based notifications feature request, and we have a pretty good understanding of what’s needed. This is not the case with the messaging feature. We explored several options for implementing this kind of functionality and decided that it’s time to refresh the list of requirements to get an idea of what needs to be done.

The goal of this blog post is to encourage discussion and gather ideas. If you are interested in these features, please share your thoughts and suggestions.

Server update, 2016-01-25

Backwards-Incompatible JSON Web Service Changes

In an effort to get our JSON Web Service out of “beta” status, we’ve made some backwards-incompatible changes to it in this release:

  • The video flag on recordings is now outputted as true or false instead of 1 or 0.
  • Empty relations arrays are not outputted for linked entities anymore, since linked entities never include relationships.
  • The iso_3166_1_codes, iso_3166_2_codes, and iso_3166_3_codes properties have been renamed to iso-3166-1-codes, iso-3166-2-codes, and iso-3166-3-codes, respectively. This only applies to lookup and browse requests; search requests already outputted these with hyphens.
  • The iso-3166- properties mentioned in the previous point are not outputted if they’re empty.

Some other changes to the web service have been made, but are considered additions (not changes to existing output), so hopefully shouldn’t cause any problems. You can review them in the changelog below.

Other Changes

An issue where entities deleted from the database (but still present in the cache) remained visible has hopefully been fixed. There are several other miscellaneous bug fixes linked below. Thanks again to Ulrich Klauer for his contributions. The git tag is v-2016-01-25.

Bug

  • [MBS-5676] – JSON relationships output doesn’t include target-type
  • [MBS-6166] – Deleted accounts can still have details edited
  • [MBS-7241] – Non-transactional cache means the cache can sometimes fail to delete entities that are gone at the database level
  • [MBS-7735] – ws/2: recording’s “video” flag inconsistent between xml and json
  • [MBS-7921] – Internal server error when requesting /ws/2/isrc as JSON
  • [MBS-8367] – ws2 JSON incorrectly returns non-included field as null value
  • [MBS-8396] – JSON output has no ordering key attribute for release group series
  • [MBS-8563] – Release & Release Group browse requests without type/status filters return results which contradicts the documentation
  • [MBS-8688] – Random tagged entity type display inconsistency in personal tag page
  • [MBS-8722] – Edit stuck trying to change the gender of a group
  • [MBS-8726] – Replicated updates don’t invalidate cache entries on slave servers
  • [MBS-8730] – Reordering of sub work parts causes unwanted reordering of main work parts
  • [MBS-8746] – JSON web service doesn’t distinguish between relationships not existing vs. not being loaded

Server update, 2016-01-11

Our first release of 2016 consists mainly of data-display fixes by Ulrich Klauer and a couple small improvements by Google Code-In students Caroline Gschwend and Ohm Patel. Notably, internationalized domain names are now displayed in decoded form: https://musicbrainz.org/url/2de1616a-7ca0-4688-92cc-0a8373190ede

Thanks once again to the above contributors. :) The git tag for today’s release is v-2016-01-11 and the changelog is below.

Bug

  • [MBS-4575] – Old add release label edit does not display
  • [MBS-5205] – Text diff incorrectly highlights first word that didn’t change
  • [MBS-7844] – Name variation marker not used for artists in tracklists in “edit medium” edits
  • [MBS-8012] – Release dates/countries are displayed strangely in edit release label edits
  • [MBS-8161] – Medium titles have no diff highlighting when displaying edits
  • [MBS-8210] – Multiple “Remove ISRC/ISWC” edits on one page interfere
  • [MBS-8330] – Another name variation check after HTML entity conversion
  • [MBS-8413] – Removed URLs in edits are badly encoded
  • [MBS-8692] – Expired Catalyst sessions remain (partially) in Redis
  • [MBS-8698] – Content negotiation for JSON-LD representation does not work with multiple MIME types in Accept header

Improvement

  • [MBS-6407] – Add username to our verification mails
  • [MBS-8683] – Display internationalized domain names in decoded form
  • [MBS-8709] – Mark up removed entities as usual in add medium/edit medium edits
  • [MBS-8713] – Block SoundCloud search and tags URLs

One month of Google Code-in

So today it is a month ago since the Google Code-in competition started and 18 days until it is ending. I wanted to take this opportunity to talk a bit about some of the things that have happened so far and where we’re at.

Google Code‐inSince December 7th when Google Code-in started, we have been in touch with 107 students on the Google Code-in site, of which 70 have completed at least one task and thus earned a digital certificate from Google. 11 students have so far earned themselves a t-shirt from Google by completing 3 or more tasks. The student with the highest number of completed tasks right now sits at 17 tasks, followed by one at 16 and another at 15 completed tasks. The student with the 10th most tasks completed has 3 tasks to their name.

Stanisław Szcześniak presenting about MusicBrainz

Stanisław Szcześniak, GCI student from Poland, presenting about MusicBrainz.

We have had 7 students do presentations on MusicBrainz in at least India, Romania, England, and Poland; about 50 reviews written for CritiqueBrainz with a few more in progress; a couple of MusicBrainz how to’s written for the wiki; one video tutorial made (which hasn’t been uploaded yet); a bunch of tests written for BookBrainz; updated and have had made a bunch of icons/logos in various places; a bunch of code patches and tests written for almost all our projects, as well as for beets (a 3rd party music file tagger and organiser heavily using MB data).

We have also had to report 3 students for plagiarising leading to their disqualification. :( However, compared to the amount of work and number of students, I think it’s a decently small number.

Overall, I am (still!) really excited about MetaBrainz finally being a part of Google Code-in, and I definitely think the lack of sleep the first week and newbie questions on IRC and on the GCI tasks are worth it. We’re getting some great stuff done, that we may not have gotten around to in any reasonable time ourselves, and we get to help all these students learn about programming, open source, open data, licenses, and a bunch of other things. I’m happy and I’m not looking forward to picking only 5 finalists and only 2 winners. There are definitely more than that I would personally like to see in both categories. :)

Have you had any experiences with or thoughts on our Google Code-in participation so far? Please do share them with us in the comments!

Server update, 2015-12-28

A new feature in today’s release is a list of recent notes left on your edits (accessible from the menu in the upper-right corner of the page: “My Data” -> “Notes Left on My Edits”).

Users will be notified in a banner message whenever they receive a new edit note, and be linked to that page. Hopefully this will improve communication with people who aren’t seeing their emails.

We also have a number of bug fixes/improvements from Ulrich Klauer and several Google Code-In students: Caroline Gschwend, Ohm Patel, and Stanisław Szcześniak. Thanks for all your work!

The git tag is v-2015-12-28 and the changelog is below.

Bug

  • [MBS-7572] – Edit work alias is not showing the sort name loss
  • [MBS-7873] – Edit search for editor causes editor name to be listed twice in recent items
  • [MBS-7882] – Strange empty parens after alias name on edits
  • [MBS-7907] – Warning about pending edits on a relationship is missing
  • [MBS-8051] – Editor field in the edit search does not behave correctly
  • [MBS-8167] – Display error for instruments in member rels on edits
  • [MBS-8241] – Mediums with pending edits not highlighted in release display
  • [MBS-8381] – Some symbols are double-encoded in inline search
  • [MBS-8414] – Ended flag is not correctly handled in edits
  • [MBS-8635] – Untranslatable strings
  • [MBS-8690] – “Add artist” edit can’t be rejected if the artist has subscriptions
  • [MBS-8691] – “Edits for subscribed editors” page timing out
  • [MBS-8702] – Page title shows “"”
  • [MBS-8705] – HTML code shown in edit summary

Improvement

  • [MBS-4565] – Search for Edits by editor notes.
  • [MBS-7070] – Missing icon (small version) for area, place and work
  • [MBS-8482] – “Add Disc” button in editor should be renamed to “Add Medium” or “Add Tracklist”
  • [MBS-8504] – Edits mandated by editing interface should be auto-edits
  • [MBS-8565] – Update the Discogs logo used in the sidebar
  • [MBS-8658] – Include tag name on user/tag/tagname pages
  • [MBS-8663] – Link to user’s profile in report emails
  • [MBS-8665] – Automatically add a space after ; in join phrases

New Feature

  • [MBS-8676] – Add a page to see all notes left on your own edits

Task

  • [MBS-8574] – Update the Google+ logo used in the sidebar
  • [MBS-8657] – Update the flag of Libya
  • [MBS-8672] – Update the flag of Lesotho
  • [MBS-8673] – Update the flag of Myanmar

Server update, 2015-12-14

  • The edit listings for your subscribed entities (and subscribed editors) now shows all edits entered within the past 7 days, by default (so, auto-edits are now visible, plus edits that may have passed by vote before you saw them). There’s a new toggle-able option on the page to only display open edits, to restore the previous behavior while voting. We’ll hopefully introduce some way to mark closed edits as reviewed/dismissed soon, to make the listings easier to digest.
  • Edit histories for large collections should hopefully load a bit faster now. Please comment on MBS-8368 if you have a collection that still consistently times out when viewing its edit history.

We’ve also made a slight change to our release process. Previously, our master branch was our “stable” branch which pointed to the most recent release (i.e., the code we run on musicbrainz.org itself). If this is what you expect, you should now be using the production branch instead. The master branch has become our main development branch, which means it may be slightly less stable from now on (of course, we’ll do our best to avoid that).

Thanks to Ulrich Klauer (chirlu) for his contributions to today’s release.

The git tag is v-2015-12-14 and the complete changelog is below.

Bug

  • [MBS-7012] – Join phrase cleanup not running
  • [MBS-8145] – Internal server error loading collection edits
  • [MBS-8368] – Edit queries for large collections time out
  • [MBS-8647] – “Edits for subscribed entites” doesn’t show auto-edits
  • [MBS-8660] – Editors can’t delete their location
  • [MBS-8661] – Adding and editing non-ended areas is broken

New Feature

  • [MBS-8664] – Show “last login/active date” somewhere on /user/ for account administrators

New business relations guru: Christina Smith

I am pleased to announce that I’ve just hired our first person responsible for looking after the day-to-day business operations of MetaBrainz! Christina Smith, a Canadian who also lives in Barcelona, has signed on as our part-time Business Development Guru (/Manager/Troublemaker/What have you).

This marks a significant step forward for us in a number of ways. First, it acknowledges that we have grown enough to warrant such a position. Second, Christina will improve how we communicate with customers and how much revenue we bring into the foundation. In the last two years we’ve added several new projects, but haven’t been able to hire new engineers to work on these projects, so hopefully having Christina will allow us to grow to support our new projects as well.

Christina’s responsibilities can be summarized as:

  • manage relationships with our supporters
  • provide timely, friendly and helpful answers and solutions for our supporters
  • increase revenues by connecting and building relationships with current and potential supporters
  • general accounting tasks including invoicing and payment tracking
  • periodic research, recommendations and help to address evolving business needs. (e.g. research the implementation of a company presence in Europe, including help in setting up a new headquarter in Barcelona)
  • research and suggestions on law and policy issues related to the business

I am super excited that Christina will be working with us — this removes a pile of tasks off my plate, which should allow me to focus more on running the foundation and managing our teams.

For the remainder of December, Christina will work on research tasks on how we can establish a better base in the EU. Then in January she will properly join the team and be present in our IRC meetings.

Welcome on board, Christina!