Tag Archives: ListenBrainz

Import your listens to ListenBrainz from Spotify!

Hullo!

We’ve been working on a system to import listens automatically to ListenBrainz from Spotify and we’ve recently deployed it to the ListenBrainz beta site. We would really appreciate it if you could help us test it out!

Please note that this is still beta software, there is a (very small) chance that we might miss a listen or two. So if you’re using this, please make sure that ListenBrainz is not the only service where you’re archiving your listens.

Another thing to note is that importing the same listens from two different sources such as Last.FM and Spotify may cause the creation of duplicates in your listen history. If you opt into our automatic Spotify import, please do not use the Last.FM import or submit listens from other ListenBrainz clients. This is a temporary limitation while we find better ways to deduplicate listens.

That’s it for the caveats, please go ahead and use the new shiny Spotify Importer. And feel free to report bugs on tickets.metabrainz.org or on IRC in #metabrainz on Freenode.

Thanks!

FOSS Independent Studies with ListenBrainz at the Rochester Institute of Technology

How to set up a ListenBrainz development environment

One of the first rites of passage when working on a new project is creating your development environment. It always seems simple, but sometimes there are bumps along the way. The first activity I did to begin contributing to ListenBrainz was create my development environment. I wasn’t successful with the documentation in the README, so I had to play around and work with the project before I was even running it.

The first part of this post details how to set up your own development environment. Then, the second half talks about the solution I came up with and my first contribution back to the project.

Continue reading

GSoC 2018: A way to associate listens with MBIDs

Hi, I’m Kartikeya Sharma, a postgrad student at National Institute of Technology, Hamirpur. I’ve worked on the project MessyBrainz as a student developer for GSoC 2018. Robert Kaye mentored me during this GSoC programme. The goal of my project is associating MBIDs to MSIDs and clustering together the MSIDs which represents the same MBID. The MBIDs represent MusicBrainz Identifier. It is an Universally Unique Identifier that is permanently assigned to each entity in the MusicBrainz database, MSID represents MessyBrainz Identifier which is associated with each unique recording, artist_credit and release in MessyBrainz database. In simple words MSIDs represents unclean metadata whereas MBIDs represent clean metadata.

This blog post summarizes the work that I did in my project, which was divided into three parts.

Processing the data already in MessyBrainz database

The first part involves creating clusters using the MBIDs already present in the MessyBrainz database. This involves creating clusters for recordings, artists, and releases. To implement this part I created the following three PRs #37, #41, and #44.

After that, I began to work on the second part of this which involves creating clusters using the artist MBIDs and release MBIDs and names fetched from MusicBrainz database. I needed to access MusicBrainz database, for that, I first had to work on BrainzUtils to have methods to access MusicBrainz database to fetch artist MBIDs using recording MBIDs and release name and release MBIDs using recording MBIDs. The part to fetch artist MBIDs was done during the community bonding period in PR #14 at BrainzUtils and to fetch releases I created PR #18 at BrainzUtils during GSoC coding period. After that, I created a PR to create clusters using the fetched artist MBIDs #47 and another one to create clusters using releases fetched #49.

I did write around 60 tests which proved to be vital in making sure that the code does what it’s supposed to do.

Processing the data as it is inserted into the MessyBrainz database

Creating clusters for the data inside the database requires a lot of resources. So, it was better to create clusters as recordings are inserted into the database but, even this type of clustering is not efficient. So, to cluster these recordings first these recordings are sent to rabbitMQ server and from that, these are sent to a clustering script which runs in a different container and runs continuously and clusters the incoming recordings. That way it does not slow down the process of submitting recordings to the database. For this I created PR #50.

Create endpoints to access MSIDs and MBIDs

I created two API endpoints in PR #51.One endpoint is to fetch MBIDs and MSID using an MSID. Another endpoint is to fetch MSIDs using an MBID. This way end users can access MBIDs and MSIDs which may be used for calculating different stats.

Apart from that with the help of my mentor, I did setup a VM to test the above code on the MsB datadump. This task had some challenges: first I had to create indexes for various fields to speed up the process of clustering. Without indexing, it would have taken approximately 37 days but after creating indexes on various fields It just took 3 hours. I found out that PostgreSQL does allow to create indexes on functions too which came into use while creating artist_credit clusters for which I created a custom function. Indexes were created in PR #53. When I ran the clustering code on a VM on which the whole MessyBrainz datadump was present I found out that we have fields in recording_json table which are supposed to store MBIDs but were pointing to empty strings. This was not supposed to happen initially as ListenBrainz is the only source of data for MessyBrainz currently. Submissions to MessyBrainz are restricted from users directly and ListenBrainz does validate listens for that. So, those recordings must have been inserted before that validation was present. To solve the problem I created PR #52.

The summer was a great learning experience for me. I started slowly as things were messy at the start. As at the start everything wasn’t crystal clear to me, I wasn’t sure on how exactly to write scripts that manipulate database and did write the scripts in the most trivial way possible. Here I was doing a query for every single MBID to first check if it’s present in the recording_cluster tables and if not then cluster the recording. Which is conceptually correct but not efficient by any means. And this could be done by executing a single query on the recording_json table to fetch only those recording MBIDs that are not present in recording_redirect table as those are unclustered. That way we don’t have to process the recording MBIDs that have been already processed making the process of clustering efficient.

With time I got an understanding of how clusters are created and how to handle anomalies. Such as James Morrison. In the end, the definition of anomaly can be put as an MSID represents an anomaly if it points to different MBIDs in entity_redirect table (entity can be artist_credit, recording, and release).

Work to be done ahead

The project is still in its initial stages and requires a lot of work to be done before moving it into production. We still need to write integration tests for ClusterWriter and API endpoints. After that, we can work on the Additional Ideas that I proposed in my proposal. We need to figure out some way to associate MBIDs to MSIDs for the artists, recordings, and releases where no MBIDs are present. This does not seem like a trivial task with so many anomalies to take care of.

Last three months have been a great experience for me. I would like to thank Robert Kaye, Param Singh, and Alastair Porter who helped me to solve a lot of problems that I encountered during the entire period. Working on their suggestions and reviews I was able to write good quality code which was efficient as well. The work culture at MetaBrainz inspired me a lot. At MetaBrainz we have weekly IRC meetings where we get to know what others are doing at the organization and also get a place to tell what we did in our past week. I would like to thank MetaBrainz and Google for giving me this chance to get involved in open source on such a cool project. The association of MSIDs to MBIDs can be used by ListenBrainz as stats are calculated on MSIDs which can then be mapped onto MBIDs which represents clean metadata. I would like to work on the project further because of the learning opportunities that are present in the project.

ListenBrainz release 18 March 2018

We received so few bug reports on the beta release of the ListenBrainz web site, that we decided to push those changes live and start working on new features. This release is substantially unchanged from our beta release.

The user facing changes that were released include:

  • Statistic infrastructure: We’ve created an infrastructure for creating graphs of user’s listening behaviour. So far we’ve only got an all-time top-artists graph to illustrate our setup, but soon we will work to create more graphs. Currently graphs will be generated every Monday starting at 0:00 UTC, if you logged in into your LB account during the last 30 days. If you haven’t logged in recently, you can request the calculation of your stats from your profile page.
  • Automatic data dumps: Now the ListenBrainz data will be dumped and synced to our FTP site twice a month. Currently this is scheduled for the 1st and the 15th of every month. The dumps will start being generated at 04:00 UTC and then copied to our FTP site and it will take a number of hours for the data dumps to appear on the FTP sites. Our documentation details how this data dump can be consumed.
  • Documentation improvements: Quite a few documentation bits have been improved since our last release, including better documentation on the Last.fm compatible API that ListenBrainz exposes.
  • Static page improvements: We’ve done some rearranging of our static pages and navigation bar to reflect the latest changes, including updating the data page and our roadmap page.
  • Listen count on home page: The home page now shows the current listen count.

We also made some internal/hosting changes that you can read about in our beta release blog post. The release from Friday has been tagged with v-2018-03-18.

Thanks to all those people who helped us put the beta site through its paces.

ListenBrainz winter 2018 beta testing

After many more months of hacking on core infrastructure and improving our codebase, we’re finally ready to have more people come and help us test the latest beta version of ListenBrainz. Also, we’ve recently reached a milestone of the 100th million listen in our database!

We’ve made a some internal changes to the project (that took quite a bit of effort):

  • Improve hosting setup that allows us to run both the production and beta version of the site at the same time. This means that any data submitted to the beta site will be submitted to the master listens database and will be available in the BigQuery data set as well. We are mimicking the setup that MusicBrainz has — the beta site use a live database so that testing the service can work with live data.
  • Improve internal container setup to allow for both dumping the listen data and private data for complete backups.
  • Improve the speed with which we process incoming listens.

These internal changes will allows us to move to more frequent updates of ListenBrainz in the future! More important are the changes to the site that are user visible:

  • Statistic infrastructure: We’ve created an infrastructure for creating graphs of user’s listening behaviour. So far we’ve only got an all-time top-artists graph to illustrate our setup, but soon we will work to create more graphs. Currently graphs will be generated every Monday starting at 0:00 UTC, if you logged in into your LB account during the last 30 days. If you haven’t logged in recently, you can request the calculation of your stats from your profile page.
  • Automatic data dumps: Now the ListenBrainz data will be dumped and synced to our FTP site twice a month. Currently this is scheduled for the 1st and the 15th of every month. The dumps will start being generated at 04:00 UTC and then copied to our FTP site and it will take a number of hours for the data dumps to appear on the FTP sites. Our documentation details how this data dump can be consumed.
  • Documentation improvements: Quite a few documentation bits have been improved since our last release, including better documentation on the Last.fm compatible API that ListenBrainz exposes.
  • Static page improvements: We’ve done some rearranging of our static pages and navigation bar to reflect the latest changes, including updating the data page and our roadmap page.
  • Listen count on home page: The home page now shows the current listen count.

If you’re interested in helping us test, please use the beta site and test everything you can see. See if anything misbehaves and if you do spot any problems, please report them to our bug tracker! Hopefully we can push this live next week.

NB: The beta site is connected to the live database, so any listens you submit to it, will be part of your official ListenBrainz listen history!

ListenBrainz Alpha disappearing in 30 days

Since we released the beta of ListenBrainz six weeks ago, people have moved over and imported their listen histories onto the beta site, which is great. While we think that everyone who needs to migrate listens off the old server has already done so, we’re going to give people another 30 days in case anyone hasn’t gotten around to it yet.

If you’ve never submitted original listens to the alpha server, this does not concern you! In fact, if this blog post is confusing to you, it probably means that you’re not affected by us turning off the alpha server on 18 October, 2017.

Thanks!

P.S. We’ve collected 50M listens on the beta site!

 

Expanding our team

As the world comes back to life after the summer break, we’re making some changes and expanding our team. First, Roman Tsukanov has decided to not renew his contract with us. During his tenure with MetaBrainz, Roman adopted and released CritiqueBrainz and also wrote our new MetaBrainz web page, which is helping us bring in new supporters. His contributions have been far from trivial — thank you for your efforts, Roman!

Due in part to the new MetaBrainz web site, we’ve got more financial support than ever, and this allows us to replace Roman with two engineers! I’m please to announce that we’re hiring two of our Summer of Code students who just completed the program:

Sambhav Kothari AKA samj1912: Sambhav started hacking on Picard earlier this year and knocked Picard out of dormancy, working towards a new release and then making Picard his Summer of Code project. He completed his project with flying colors and is working towards a major upgrade of Picard. On the MetaBrainz team he is going to look after the new search infrastructure and the maintenance and bug fixing of our Web Service in addition to hacking on Picard. A full plate, for sure!

Param Singh AKA iliekcomputers: About the same time that samj1912 arrived, Param arrived. He expressed interest in working on ListenBrainz — he too dove right in and started making improvements. ListenBrainz had quite a ways to go before he could aim to make a Summer of Code project out of it. Param and I embarked on a journey to revamp and improve the stability of ListenBrainz, which culminated in us releasing the new ListenBrainz beta a few weeks ago. Since then he’s been focusing on his Summer of Code project, which is also now complete. On the MetaBrainz team Param will be looking after ListenBrainz and also the new MetaBrainz web site.

Both Param and Sambhav will officially start working on the MetaBrainz team starting October 1, but I strongly suspect we’ll see them around and hacking on the projects as has become the norm this year.

Welcome aboard Sambhav and Param!