Tag: ListenBrainz

GSoC 2020: Manage your listens better with ListenBrainz

Hey! My name is Shivam Kapila (shivam-kapila on IRC) and I am a final year undergrad at National Institute of Technology Hamirpur. I have been working on the ListenBrainz project this Summer as a participant of the Google Summer of Code program. The past four months were full of fun, hacking and loads of music!!

Landing into the MetaBrainz Community!

My journey with MetaBrainz began in late January this year, when I introduced myself to the community. My first PR improving the developer documentation was by adding parts connected with setting up the Spark infrastructure on a local setup along with consolidating and improving bits of documentation. I delved into real code while implementing front end components for Deleting Listens. Over the next few months, I fixed various bugs like making the Importer Modal responsive, fixing the DB setup scripts, fixing pagination issues while browsing listens, handling stat calculation errors in the Spark Reader and flushing user stats when they delete their listens.

As a GSoC applicant, I proposed to add various Listen Management features like love/hate (aka feedback) and deleting individual listens in ListenBrainz. I also proposed a new design for the Listens page. This involved a lot of designing and research, going through UI/UX design guidelines and tuning colors, shades and shadows till we arrived at a presentable and subtle design.

And finally I onboarded the GSoC train šŸ™‚ .

Bonding with the community

I had been a part of the community since January so I was familiar with how things work in ListenBrainz. So I decided to contribute to the TimescaleDB migration where we moved our primary listen store from InfluxDB to TimescaleDB, opening up a ton of features for us to work on. Here is the final migration PR containing the commits of my contribution.

I also contributed to easing the testing infrastructure for devs to test the patches on their local setups. Following this I upgraded the postgres-client to PG12 version when we migrated to Postgres 12. I also fixed a minor font bug on the profile page.

The GSoC journey begins

Laying the base

As the official coding period began, I started working on my proposed tasks. The first question was: how to store the feedback? So I began implementing the database changes to store the recording feedback and applying the necessary changes in production. Following this I added a Python module to interact with the database and implemented a Pydantic model to validate the feedback records before they are stored in the database or served over the API. Then I added the necessary APIs to store and fetch the feedback for a given user or recording. This was followed by improving the efficiency of the DB module.

I also worked on dumping the recording feedback in the ListenBrainz public dumps. Since ListenBrainz had migrated the stats calculation infrastructure from Google BigQuery to Apache Spark I also removed the BigQuery references from the ListenBrainz website. Now that the timescale migration work became stable, I began working on Delete a Listen feature.

Pulling out the front end brushes

Now that the base was ready for us to work on, I started working on the React components so that the feedback and deletion feature could actually be presented on the website. Around the same time, the Timescale release day was also getting near, so I helped with a few tests and finished up the work for deleting listens. The front end components also started looking good and we were ready to associate the back end with them.

Rectifying & Reactifying

Itā€™s high time and the final phase started. Now that we were ready with a few components we needed some tweaks in some production components to make them subtle. Hence I shot an improvement PR to tweak some shadows, adjust some fonts, adjust heights of the components, sticking the footer to the bottom, and reactify the loading spinner. Then came the Listen Count Card denoting the number of listens for a user. Following this we moved to Card based design for displaying listens.

This was followed by the much awaited feedback controls and now we can love/hate the songs from our listen collection. Isnā€™t this amazing! There were some needed minor tweaks needed to handle the ā€˜playing nowā€™ listens correctly. At the same time, following the MetaBrainz guidelines to write quality code, I worked on making the SQL queries more readable. Then came the much awaited Delete a Listen feature and now we can finally get rid of the embarrassing listens!!

I also addressed some high priority tasks like giving the users an option to download their submitted feedback as JSON. We noticed some UI glitches and then came three back to back PRs to update feedback control shades, improving the listen time text and smoothing up the deletion animation. This is how the listen list looks like:

List of listens

Whatā€™s next??

Oh, now comes the time when we talk about the current scenario. The tasks currently on my radar are adding cover art support so that the page looks more alive and improving the Spotify imports to only import listens that were listened by the user after the latest Spotify listen we have for them.

After this I aim to work on the recommendation stuff thatā€™s being actively pursued by the team. Also Mr_Monkey and me had been working on some design concepts for the All New ListenBrainz. I am pretty excited to work on it. Wanna take a sneak peek?

A new fam

The journey with MetaBrainz has been so amazing, that I am so tempted to stick here. I feel ecstatic to be a part of GSoC with the best org šŸ™‚ . The best part is ā€“ it’s never all about code. Thereā€™s a lot to gain. Each day marked gaining maturity and thinking more and more like a real developer. I started feeling at ease with the communicate ā†’ code ā†’ integrate chain. It really feels fortunate to be a part of the MetaBrainz family where everyone is a ping away ā¤ .

GSoC marks the kickstart of my journey with MetaBrainz and I will be here lurking on IRC, shooting PRs to make the projects more and more awesome.

Heartiest Gratitude

GSoC 2020: Adding Statistics and Graphs for ListenBrainz Users and Community

Hey everyone! I am Ishaan Shah (ishaanshah), a sophomore at International Institute of Information Technology – Hyderabad, India. This summer, I worked on ListenBrainz as a participant in Google Summer of Code ā€™20. My project involved generating statistics and visualisations for users using Apache Spark. This blog is an overview about the work I did and my experience working with ListenBrainz.

I started contributing to ListenBrainz in January 2020. My first PR was for LB-179, a small Quality of Life improvement to the LastFM importer. My first major contribution was porting the LastFM importer to ReactJS. Over the next two months, I continued working on the frontend, where I mainly worked on improving the frontend infrastructure by adding support for automated testing, porting the codebase to TypeScript and standardising the frontend code using ESLint and Prettier.

After making a few patches, I understood how ListenBrainz worked and got comfortable with the codebase. I decided to make a proposal for adding statistics to ListenBrainz using Apache Spark. While writing the proposal, I referred to many other websites, blogs, as well as community discussions for different ideas about statistics which could be added. After some research, I narrowed down on the specific graphs and statistics that I wanted to calculate during GSoC.

Community Bonding Period

Since I had been working with the MetaBrainz community since January, I was familiar with how things worked in the community. So we decided to use the Community Bonding Period for fixing and updating the Top Artists charts for a user. The first task that I took up was to add an API endpoint for fetching the Top Artists data for a user programmatically. Until then, I had mostly spent my time working on the frontend, this task helped me in getting familiar with the backend architecture. Next, I worked on porting the Top Artist graph from d3 to nivo – a charting library built with ReactJS and d3. The Top Artists graph only supported All Time statistics before. I worked on adding support for more time ranges. This was the first time I worked with Apache Spark and the PR for this took quite some time, but it was essential that we got it right as most of the statistics we built further would use a similar workflow. After we were satisfied with the overall flow of the data from our Spark cluster to the web server, I started working on showing the stats for different time ranges on the website. Although this task seemed easy at first, it took much longer than expected. We encountered some bugs and received some user feedback when we deployed the graph to production. The rest of this period was spent on incorporating the user feedback and fixing the bugs.

Top Artist shown on the Charts page
Top Artists

First Coding Period

We now had a somewhat stable pipeline for calculating the stats and sending them to the server. I started working on the backend for Top Releases stats for a user. We ran into memory issues when calculating these stats on the cluster, so I spent some time finding the cause of the issue and realised that we were collecting the results all at once which was causing the driver to run out of memory. I fixed this by collecting the results for each user separately and tweaking some RabbitMQ parameters to make sure that messages arenā€™t dropped while sending them to the server (PR #897). After this, I added Top Recordings for a user. Now we had a brand new Charts page that displayed the userā€™s Top Artists/Releases/Recordings for different time ranges. Next I started working on temporal statistics for a user i.e, number of listens in a past time range. The query that I wrote for calculating this data turned out to be pretty inefficient for larger datasets. So I ended up writing two versions of the same query: one for large datasets and one for smaller ones. While working on displaying these stats on the frontend, I tried various representations of the data. I finally settled on displaying the data as bar graphs, as shown on this report view.

Listening Activity shown on Reports page
Listening Activity

Second Coding Period

I added two more graphs in this period: Daily Activity and Artist Origins. The Daily Activity graph shows the number of listens a user has at a particular time of the day. I implemented the query for calculating this data in a slightly different way compared to the Listening Activity query. This change improved the query speed significantly. I had some trouble finding a correct way to represent this data. My mentor helped me in this by suggesting the usage of a Heatmap, and the results turned out to be pretty good.

Daily Activity shown on Reports page
Daily Activity

Next, we worked on the Artist Origins graph, which provides an insight into the geographical diversity of a userā€™s musical taste. I had a lot of help from the ListenBrainz team for this graph and I couldnā€™t have done this graph without their help. This was by far the most interesting stat that I worked on during the project. Furthermore it laid a general framework to calculate statistics using the data from MusicBrainz. After deploying this map on production, we received feedback from the users that the map looked plain for most of them and there wasn’t much colour difference between different regions. This happened because people generally tend to listen more songs from their home country, so there is a huge difference between the country with maximum artists and average number artists from other countries. We fixed this issue by changing the colour scale from linear to logarithmic.

Comparison between linear and log scale in Artist Origins Map

Final Coding Period

We now turned our attention towards calculating some stats for the whole website. We decided to make a graph for the Top Artists over different time ranges. We thought that this would be relatively easy given that we had already done something similar for individual users before. However we hit an unexpected bump; the data we were calculating was not accurate, mainly because of various different sources of the artists and some minor changes in the artistsā€™ name or metadata resulted in a different entry with a different listen count for the same artist. Moreover, we found a couple of users spamming our website for self promotion and we did not have a solid way to deal with this. Around this time, my college resumed and the amount of time I could dedicate to LB reduced severely. So we decided to use the remaining time to work on improving the frequency at which stats are updated. I have an open PR (#1052) for doing this at the time of me writing this blog and we should be able to implement this functionality in the near future.

Artist Origins shown on reports page
Artist Origins

Experience

The past 4 months have taught me a lot of things. I learnt new technical concepts everyday. I started writing code as a developer rather than a programmer. I understood the importance of proper unit and integration testing (even though it was my least favourite part while adding a new functionality). I also found it much easier to talk and interact with people both online and in real life. Frequent deployments of new features to production helped us a lot. We were able to catch bugs when we still had some context over the code written and also received feedback from the users about how we could improve the new features added. It also kept me motivated to keep working on new graphs and statistics and gave me a sense of satisfaction when I saw them on the production server. I also learnt that things donā€™t always go the way we expect them to. More often than not, you will run into some bumps while adding new features so it is better to keep some extra time to deal with these issues.

GSoC gave me a wonderful opportunity to work with some amazing people from all over the globe. I was not able to complete all the graphs that I had planned for this summer, but I do plan to continue working on ListenBrainz to add more statistics and new features.

Special Thanks

  • Param Singh (iliekcomputers) for being an amazing mentor and helping me whenever I was stuck on an issue.
  • Robert Kaye (ruaok) for providing some really insightful feedback and the MusicBrainz data that was required for calculating the Artist Origin map.
  • Nicolas Pelletier (Mr_Monkey) for helping me with the frontend for the user Charts page and providing some amazing tips for ReactJS.

Migration to TimescaleDB complete!

Yesterday I posted about why we decided to make the switch to TimescaleDB and then later in the day we actually made the switch!

We are now running a copy of InfluxDB and a copy of TimescaleDB at the same time — in case we find problems with the new TimescaleDB database, we can revert to the InfluxDB database.

In the process of migrating we got rid of a pile of nasty duplicates that used to be created by importing from last.fm. We also got rid of some bad data (timestamp 0 listens) that were pretty much useless and were cluttering the data. If you find that you are missing some data besides some duplicates, please open a ticket.

The move to TimescaleDB allows us to create new features such a deleting a listen (which should be released later this summer) and various other features that because the underlying DB is much more flexible than InfluxDB. However, right this second there are no real new features for end users — more new features are coming soon, we promise!

Thank you to shivam-kapila, iliekcomputers and ishaanshah — thanks for helping with this rather large, long running project!

ListenBrainz moves to TimescaleDB

The ListenBrainz team has been working hard on moving our primary listen store from InfluxDB to TimescaleDB, and today at UTC 16:00 we’re going to make the switch.

We were asked on Twitter as to why we’re making the switch — and in the interest of giving a real world use case for switching, I’m writing this post. The reasons are numerous:

Openness: InfluxDB seems on a path that will make it less open over time. TimescaleDB and its dependence on Postgres makes us feel much safer in this regard.

Existing use: We’ve been using Postgres for about 18 years now and it has been a reliable workhorse for us. Our team thinks in terms of Postgres and InfluxDB always felt like a round peg in a square hole for us.

Data structure: InfluxDB was clearly designed to store server event info. We’re storing listen information, which has a slightly different usage pattern, but this slight difference is enough for us to hit a brick wall with far fewer users in our DB than we ever anticipated. InfluxDB is simply not flexible enough for our needs.

Query syntax and measurement names: The syntax to query InfluxDB is weird and obfuscated. We made the mistake of trying to have a measurement map to a user, but escaping measurement names correctly nearly drove one of our team members to the loonie bin.

Existing data: If you ever write bad data to a measurement in InfluxDB, there is no way to change it. I realize that this is a common Big Data usage pattern, but for us it represented significant challenges and serious restrictions to put simple features for our users into place. With TimescaleDB we can make the very occasional UPDATE or DELETE and move on.

Scalability: Even though we attempted to read as much as possible in order to design a scalable schema, we still failed and got it wrong. (I don’t even think that the docs to calculate scalability even existed when we first started using InfluxDB.) Unless you are using InfluxDB in exactly the way it was meant to be used, there are chances you’ll hit this problem as well. For us, one day insert speed dropped to a ridiculously low number per second, backing up our systems. Digging into the problem we realized that our schema design had a fatal flaw and that we would have drastically change the schema to something even less intuitive in order to fix it. This was the event that broke the camel’s back and I started searching for alternatives.

In moving to TimescaleDB we were able to delete a ton of complicated code and embrace a DB that we know and love. We know how Postgres scales, we know how to put it into production and we know its caveats. TimescaleDB allows us to be flexible with the data and the amazing queries that can be performed on the data is pure Postgres love. TimescaleDB still requires some careful thinking over using Postgres, it is far less than what is required when using InfluxDB. TimescaleDB also gives us a clear scaling path forward, even when TimescaleDB is still working on their own scaling roadmap. If TimescaleDB evolves anything like Postgres has, I can’t wait to see this evolution.

Big big thanks to the Postgres and TimescaleDB teams!

State of the Brainz: 2019 MetaBrainz Summit highlights

The 2019 MetaBrainz Summit took place on 27thā€“29th of September 2019 in Barcelona, Spain at the MetaBrainz HQ. The Summit is a chance for MetaBrainz staff and the community to gather and plan ahead for the next year. This report is a recap of what was discussed and what lies ahead for the community.

Continue reading “State of the Brainz: 2019 MetaBrainz Summit highlights”

GSoC 2019: An open-source music recommendation engine

Give me music that I like.

When you start discovering yourself, just know that you are at the right place and with the right people.
MetaBrainz is the one for me!

I am Vansika Pareek (pristine__ on IRC), an undergraduate student at National Institute of Technology, Hamirpur, India. I have been working on the ListenBrainz-Labs project for MetaBrainz as a participant in Google Summer of Code ’19. The end of GSOCā€™19 is a beginning for me. Cheers!

How it all started?

Continue reading “GSoC 2019: An open-source music recommendation engine”

Mini ListenBrainz update released today

Following up on our release from last week, we found a number of minor problems in production that were really hard to spot on our test setup. Sometimes you need to have real data flowing through your system before you can find the real problems.

The following pull requests were merged and released just now:

This should hopefully make the follow page work a little better for everyone. šŸ™‚

ListenBrainz: Our new follow page

As promised, here is another blog post about the exciting new Follow page. The goal of this page is to finally make use of the data we collect in ListenBrainz and expose a new feature designed to let our users discover more music.

To use this new feature, you’ll need to link your Spotify account to ListenBrainz. Ideally you should give permission to record your listens and to play Spotify content. But if you’re not ready to dive into recording your listens, start with playback first. N.B. In order to really take advantage of this new feature, you’ll need a premium Spotify account.

Then head over to the recent listens page and hover over the tracks that are listed there. If the user listened on Spotify, then a play button will appear and you can listen to the track. Please note that playing from this page will interrupt whatever you’re already playing on Spotify. If you find that a user is listening to interesting music and you’d like to follow the user, head to the follow page and use the Follow Users section to add this user to your follow list.

When a user in your follow list finishes listening to a track, that track will appear as a line in the Playlist. In theory, you’ll be able to keep listening to what your followed users are playing: the player will attempt to play as many tracks as it can play and to keep the music going. The player also has a previous and next track button that allows you to easily skip tracks that you don’t like. Our team has found this feature exciting and to some extent even has started DJing for each other!

We’re pushing into new territory trying to offer music discovery features and trying out new features that we’ve not seen before. Expect bugs, missing features, and reactions of “why didn’t they do X?”. To be honest, we’re not entirely happy with it and we know that there are features missing. But we felt it important to push this out in order to start getting feedback from you — and we are also excited about the Spotify integration! That said, please continue reading and if you feel that we screwed something up, please open a ticket!

Also, keep in mind that we’re pushing against the tide of the music industry. Established players want to keep everything closed, controlled and in their silo (Apple Music, Tidal, etc). Spotify is slightly more open and allows us to record user’s histories and music playback from web pages, so we focused on working on Spotify first.

This has the unfortunate side-effect of making these new features useful only if you have a premium Spotify account, and following users who are not on Spotify is useless: we don’t know how to play this content. This blows — we know it and we hate it ourselves. But we needed to start with something to show what we’re trying to do and to generate some interest. If people are interested, we can start working in supporting more services and making more of the music in our pages playable.

Finally, the recording user’s listens API endpoint at Spotify has an annoying tendency to fall behind sometimes, which means that the flow of listens from Spotify slows or stops altogether, which is… less than ideal. We’re prodding Spotify to keep the bits flowing if at all possible, but know that all of this is a work in progress.

In fact, the release has already generated a flurry of fixes that we’ll push live before too long. A lot of these sorts of fixes are for problems that you can only see when real-live data flows through the data pipelines: these are tricky features to debug!

Please play with the follow feature and tell us what you think! If you know other services that we can use to play music from the data we have available, please comment! If you find bugs or have suggestions for how we make these features better, please open a ticket!

Have fun and discover some new music,
The ListenBrainz Team

ListenBrainz release: Spotify account linking and our first music discovery features!

For the past few months we’ve been working on enabling ListenBrainz to record your Spotify listening history automatically and we’ve just now released this feature! If you would like ListenBrainz to record your Spotify listening history automatically (and make it public!), go here to link your Spotify account to ListenBrainz. We’ll take care of the rest!

We would like to encourage as many users as possible to record their listening histories in ListenBrainz. With the data we collect and safeguard for you, we will soon start building more music discovery features. Please help our mission and go connect your account now!

This release also adds two new pages: Recent listens and the “follow” page. The recent listens page shows the most recent listens that we’ve saved in ListenBrainz for any user. This is a convenient way for you to discover other users who are currently listening to music.

The follow page is the new feature that we’re really excited about — it allows you to listen to the music that other people are currently listening to — pick a number of users to follow and their recent listens will appear on the page. The new embedded Spotify player can start playing the music as the listens roll in. This allows you to follow your friends and learn about music that they love! We’re going to write another blog post that talks more about the follow page and how we plan to improve that going forward — stay tuned for that.

This release also re-organizes the menu layout a little, moving the most useful features so that they’re easily accessible. Behind the scenes we’ve upgraded to using Python 3.7, starting using some portions of React for our user interface and also found ourselves amazed that this release included 646 commits! We hope to go to a more regular schedule of releases from here on out — this was a big push for us with a lot of infrastructure improvements that were needed.

This release would not have been possible if Monkey (from BookBrainz) didn’t come and help us write the UI for the follow feature. Monkey, iliekcomputers and myself worked relentlessly for weeks trying to push out some exciting features that show off the first steps for what we have planned for ListenBrainz. We’re quite excited for this release and we hope that you’ll enjoy the follow page and discover new music!

Import your listens to ListenBrainz from Spotify!

Hullo!

We’ve been working on a system to import listens automatically to ListenBrainz from Spotify and we’ve recently deployed it to the ListenBrainz beta site. We would really appreciate it if you could help us test it out!

Please note that this is still beta software, there is a (very small) chance that we might miss a listen or two. So if you’re using this, please make sure that ListenBrainz is not the only service where you’re archiving your listens.

Another thing to note is that importing the same listens from two different sources such as Last.FM and Spotify may cause the creation of duplicates in your listen history. If you opt into our automatic Spotify import, please do not use the Last.FM import or submit listens from other ListenBrainz clients. This is a temporary limitation while we find better ways to deduplicate listens.

That’s it for the caveats, please go ahead and use the new shiny Spotify Importer. And feel free to report bugs on tickets.metabrainz.org or on IRC in #metabrainz on Freenode.

Thanks!