Category Archives: ListenBrainz

Delhi Mini-Summit 2018

Rob, Suyash, Param and I met in the bustling city of Delhi where “horns are applied very liberally” (it is a very noisy city!) for a mini summit. Some may even call it elaborate break-out sessions on ListenBrainz and CrtiqueBrainz. We had discussions over a span of two days over laptops and notebooks, riding on bumpy roads in tuk-tuks and over spicy chicken biryanis. Here is a summary of all that we discussed:

ListenBrainz
Data Visualizations
We started Day 1 with graphs for ListenBrainz. After a long marathon of heavy development weightlifting tasks by Param and Rob (how do we work with BigQuery correctlty?), we are finally at a stage, where we can have some really cool amazing visualizations out of our dataset. What will they be? Where will they be? How will we implement them? Can our community pitch in with requests and maybe even play around with code?

After scrounging through a lot of other websites which do music-y data visualizations, and the few responses on our user survey, we started listing various ideas, and went through ideas on our community forum. We ended up dividing the data visualizations (from now on, called graphs) into two categories:

User specific graphs: showcasing a user’s listening history and taste
Site-wide graphs: showcasing the overall listening patterns on ListenBrainz

We had to make some tricky calls based on technical constraints, but overall, for starters, we decided some cool user graphs. We have detailed 6 of them over the summit:

  1. Listening history of a user: how much have you listen-ed, what you have you listened too, listen counts, etc
  2. Your top artitsts
  3. Your tracklist (listen history)
  4. How much music did you explore
  5. Which artists are trending in what parts of the worlds
  6. Listener count across the world

All these graphs will be available over different time durations (last week, month, year) and will also have handles to manipulate them. They will also have tools to easily share them on social media networks. We think, our community will really enjoy tracking their listening history with these. We also discussed a few ideas of how we can create a sandbox so our community can pitch in with ideas, vote on ideas and send pull requests for new graphs. More on that later, as we get there!

Rating System
If you are listening to a tracklist while working over something, how possible it is that you will rate a track saying “This is 3.5? This is 4.2? That is 5 stars!” So you see, ratings on ListenBrainz are tricky. It is very dynamic and interactive in real time, unlike other dear *Brainz projects, so we think that a Last.fm-like rating i.e like and dislike makes sense for ListenBrainz. There was also some discussion about where the ratings should reside — is CritiqueBrainz the correct place?

Home Page
We worked on redesigning the “My Listens” page as well the home page. We now plan to include, apart from the graphs, an infographic explaining how ListenBrainz works and things you can do with it! I will further detail out the mockup later this week.

Potential Roadmap
After almost two days of discussions, we could chalk up a rough roadmap for ListenBrainz, which include data visualizations, ability to rate/like tracks, create collections, follow users, and more. This also includes encouraging cross brainz pollination!

CritiqueBrainz
With Suyash around (he worked on Critique Brainz as part of GSoC last year, and has been actively involved since), there were obviously a lot of discussions on reinvigorating the project. We discussed quite a few ideas, which included innovating ways of writing and sharing reviews, sharing it on social media, cross *brainz interactions, a few UI changes, etc. We’re considering allowing Quick Reviews that, like Twitter, are limited to 280 characters. What do you think? Suyash has written down his ideas for the same and would love some feedback from the community!

MessyBrainz
With all these talks, a critical need to build some matching and clustering infrastructure was highlighted. Rob has written a possible roadmap for the project trying to compose his thoughts!

And of course! We couldn’t let Rob’s first visit to India be all about work. After the sunset, we went exploring the city of Delhi. That included rides in tuk-tuks, spicy chicken biryanis, shopping for some colorful clothes and definetly, the Indian chaat 🙂

All in all, it was a very productive mini summit and definitely made us all, more excited to start working on the ideas we discussed. We will keep you updated and post more soon!

food-01.jpg

Some A lot of Indian food!

IMG_20180322_211308.jpg

The troope at India Gate

IMG_20180323_195125.jpg

Param is really into (a lot of) selfies.

ListenBrainz release 18 March 2018

We received so few bug reports on the beta release of the ListenBrainz web site, that we decided to push those changes live and start working on new features. This release is substantially unchanged from our beta release.

The user facing changes that were released include:

  • Statistic infrastructure: We’ve created an infrastructure for creating graphs of user’s listening behaviour. So far we’ve only got an all-time top-artists graph to illustrate our setup, but soon we will work to create more graphs. Currently graphs will be generated every Monday starting at 0:00 UTC, if you logged in into your LB account during the last 30 days. If you haven’t logged in recently, you can request the calculation of your stats from your profile page.
  • Automatic data dumps: Now the ListenBrainz data will be dumped and synced to our FTP site twice a month. Currently this is scheduled for the 1st and the 15th of every month. The dumps will start being generated at 04:00 UTC and then copied to our FTP site and it will take a number of hours for the data dumps to appear on the FTP sites. Our documentation details how this data dump can be consumed.
  • Documentation improvements: Quite a few documentation bits have been improved since our last release, including better documentation on the Last.fm compatible API that ListenBrainz exposes.
  • Static page improvements: We’ve done some rearranging of our static pages and navigation bar to reflect the latest changes, including updating the data page and our roadmap page.
  • Listen count on home page: The home page now shows the current listen count.

We also made some internal/hosting changes that you can read about in our beta release blog post. The release from Friday has been tagged with v-2018-03-18.

Thanks to all those people who helped us put the beta site through its paces.

ListenBrainz winter 2018 beta testing

After many more months of hacking on core infrastructure and improving our codebase, we’re finally ready to have more people come and help us test the latest beta version of ListenBrainz. Also, we’ve recently reached a milestone of the 100th million listen in our database!

We’ve made a some internal changes to the project (that took quite a bit of effort):

  • Improve hosting setup that allows us to run both the production and beta version of the site at the same time. This means that any data submitted to the beta site will be submitted to the master listens database and will be available in the BigQuery data set as well. We are mimicking the setup that MusicBrainz has — the beta site use a live database so that testing the service can work with live data.
  • Improve internal container setup to allow for both dumping the listen data and private data for complete backups.
  • Improve the speed with which we process incoming listens.

These internal changes will allows us to move to more frequent updates of ListenBrainz in the future! More important are the changes to the site that are user visible:

  • Statistic infrastructure: We’ve created an infrastructure for creating graphs of user’s listening behaviour. So far we’ve only got an all-time top-artists graph to illustrate our setup, but soon we will work to create more graphs. Currently graphs will be generated every Monday starting at 0:00 UTC, if you logged in into your LB account during the last 30 days. If you haven’t logged in recently, you can request the calculation of your stats from your profile page.
  • Automatic data dumps: Now the ListenBrainz data will be dumped and synced to our FTP site twice a month. Currently this is scheduled for the 1st and the 15th of every month. The dumps will start being generated at 04:00 UTC and then copied to our FTP site and it will take a number of hours for the data dumps to appear on the FTP sites. Our documentation details how this data dump can be consumed.
  • Documentation improvements: Quite a few documentation bits have been improved since our last release, including better documentation on the Last.fm compatible API that ListenBrainz exposes.
  • Static page improvements: We’ve done some rearranging of our static pages and navigation bar to reflect the latest changes, including updating the data page and our roadmap page.
  • Listen count on home page: The home page now shows the current listen count.

If you’re interested in helping us test, please use the beta site and test everything you can see. See if anything misbehaves and if you do spot any problems, please report them to our bug tracker! Hopefully we can push this live next week.

NB: The beta site is connected to the live database, so any listens you submit to it, will be part of your official ListenBrainz listen history!

ListenBrainz Alpha disappearing in 30 days

Since we released the beta of ListenBrainz six weeks ago, people have moved over and imported their listen histories onto the beta site, which is great. While we think that everyone who needs to migrate listens off the old server has already done so, we’re going to give people another 30 days in case anyone hasn’t gotten around to it yet.

If you’ve never submitted original listens to the alpha server, this does not concern you! In fact, if this blog post is confusing to you, it probably means that you’re not affected by us turning off the alpha server on 18 October, 2017.

Thanks!

P.S. We’ve collected 50M listens on the beta site!

 

Expanding our team

As the world comes back to life after the summer break, we’re making some changes and expanding our team. First, Roman Tsukanov has decided to not renew his contract with us. During his tenure with MetaBrainz, Roman adopted and released CritiqueBrainz and also wrote our new MetaBrainz web page, which is helping us bring in new supporters. His contributions have been far from trivial — thank you for your efforts, Roman!

Due in part to the new MetaBrainz web site, we’ve got more financial support than ever, and this allows us to replace Roman with two engineers! I’m please to announce that we’re hiring two of our Summer of Code students who just completed the program:

Sambhav Kothari AKA samj1912: Sambhav started hacking on Picard earlier this year and knocked Picard out of dormancy, working towards a new release and then making Picard his Summer of Code project. He completed his project with flying colors and is working towards a major upgrade of Picard. On the MetaBrainz team he is going to look after the new search infrastructure and the maintenance and bug fixing of our Web Service in addition to hacking on Picard. A full plate, for sure!

Param Singh AKA iliekcomputers: About the same time that samj1912 arrived, Param arrived. He expressed interest in working on ListenBrainz — he too dove right in and started making improvements. ListenBrainz had quite a ways to go before he could aim to make a Summer of Code project out of it. Param and I embarked on a journey to revamp and improve the stability of ListenBrainz, which culminated in us releasing the new ListenBrainz beta a few weeks ago. Since then he’s been focusing on his Summer of Code project, which is also now complete. On the MetaBrainz team Param will be looking after ListenBrainz and also the new MetaBrainz web site.

Both Param and Sambhav will officially start working on the MetaBrainz team starting October 1, but I strongly suspect we’ll see them around and hacking on the projects as has become the norm this year.

Welcome aboard Sambhav and Param!

 

GSoC 2017: Hacking on ListenBrainz

Namaste!

I am Param Singh, an undergraduate at the National Institute of Technology, Hamirpur, India, and I worked on ListenBrainz over the summer as part of the Google Summer of Code program. I started contributing code to ListenBrainz in January 2017 and have been working on new features and bug fixes since. I’ll be writing about the work I did and my experience working on LB in this blog post.

After a few of my patches had made it in and I was comfortable with the ListenBrainz codebase (which was a really nice example of software architecture for me), I talked with the LB team about what possible contributions I could make over the summer, and we decided that a Google BigQuery based statistics system is something that would be useful to have in ListenBrainz after we release a beta and have listen data that is permanently archived. I made a proposal for adding statistics to ListenBrainz which got accepted! During the community bonding period, we decided to try to get a solid and stable beta of ListenBrainz released before starting with the relatively large code additions that would be required by my project proposal. We tracked issues that we wanted fixed before a release in the MetaBrainz ticket tracker here. This work of fixing release blocking issues went into the coding period and we decided to continue working on a solid beta instead of adding new features for the time being.

I started with fixing bugs and adding new features to get a beta released as soon as possible. Some cool stuff I worked on during this time was dockerizing MessyBrainz (see PR here), migrating the codebases of MessyBrainz and ListenBrainz to Python 3 (PRs here and here) and improving the startup resilience of various parts of ListenBrainz to make sure that the server is able to self-heal (partially) if some part of it like RabbitMQ goes down (ticket here).

Later on, I did a big refactor of the LB code so that adding new modules would be easier in the future (PR here). I also spent a lot of time fixing bugs in our listen deduplication. Relevant pull requests for this are here and here.

Another feature I added to ListenBrainz while working on the beta was incremental imports. Earlier, LB didn’t keep track of previous imports of a user and did a full Last.FM import every time. However, now we keep track of the last time each user imported listens and only import new data since then. The PR adding incremental imports is here.

My mentor, Robert Kaye (ruaok) set up a test instance of the ListenBrainz server that was used by the community and as the community kept throwing their data at us, bugs kept popping up. A particularly weird bug caused LB to lose data for users with special characters in their usernames. The PR to fix this took a lot of time to create.

We kept on fixing bugs for a long time and the biggest thing I took away from this period of GSoC was the Ninety-ninety rule: «The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time.» This summer has drilled this into my mind.

As soon as the beta was released, I started with writing code for statistics, making schema changes (PR here) and adding some user stats (PRs here and here). I’ll be continuing on the stats work after Summer of Code. The basic foundation of stats is mostly done and soon I’ll start with showing statistics to the users.

By the end of the official GSoC coding period, I have made 266 commits in the ListenBrainz codebase and have opened a total of 111 pull requests. The current production ListenBrainz running on https://listenbrainz.org has 253 commits by me, most of which were made during the GSoC period.

Over the summer, I have fallen in love with the MetaBrainz community and have learned a lot of stuff. I’m really looking forward to adding more features to ListenBrainz soon, so that the data that the community is contributing becomes useful to everyone. I loved working on a really cool open-source project like ListenBrainz this summer and am very thankful to Google for providing me this opportunity. I would encourage everyone reading this to give the ListenBrainz beta a try and contribute to ListenBrainz if possible.

ListenBrainz data is live on BigQuery!

We’re pleased to announce that in cooperation with Google, we are live streaming our ListenBrainz data to Google’s BigQuery service!

ListenBrainz is a project is that has the potential to gather a lot of data quickly, which would require us to have a Big Data infrastructure, which can be expensive. In an effort to use our available cash wisely, we began to look around for ways to take advantage of other infrastructures with lower costs.

Two years ago at the Google Summer of Code mentor summit I met with a representative from the BigQuery team who said that Google was happy to host any public data set for free! I immediately took them up on this offer and started a conversation.  With much time passed, we finally managed to get the data set live!

If you wish to play with the data, please do!

https://bigquery.cloud.google.com/table/listenbrainz:listenbrainz.listen

You’ll need a Google account to log in with — once you’re logged in, every user gets 5TB of query traffic free per month. That is quite a lot for how large this dataset is currently. The schema for this table is defined here and what the data elements mean are defined in our API docs. To get you started, I’ve written a few sample queries:

BigQuery uses an SQL like syntax, so if you know some SQL then diving right in should be easy. The queries above should give you an idea of what you can do with this data. Now, please know that currently we have approaching 30M listens, so the dataset is still quite small. We’re very much interested to see what sort of things people can come up with in the near future.

Finally, some notes about openness and proprietary software: Given that we have limited resources, we aim to make the most things happen with the services that are at our disposal. Google has been extremely generous to us over the years and we’re very pleased to have access to BigQuery now.

That is not to say that we’re putting all of our eggs in one basket or forcing people to use BigQuery. Our InfluxDB database hosted on our own servers keeps the master archival copy of our listen data. Soon we hope to make dumps of this data available for anyone to download and play with using whatever tools they would like. With this setup we are not fully reliant on Google for keeping this project alive. We’re glad to have their support, but should circumstance change, we can find another BigData solution and load our master archival copy there.

Now, go play with this very promising data and post some of your favorite queries in the comments!