Hi Everyone!
I am Suvid Singhal (suvid on matrix), an undergraduate Computer Science student at Birla Institute of Technology and Science (BITS), Pilani. I took part in the Google Summer of Code 2025 and have been contributing to Metabrainz Foundation since December 2024. My GSoC project was to develop a file-based listening history importer for ListenBrainz. The project was mentored by Lucifer and Monkey.
Project Overview
Listenbrainz is a platform to track your music habits, discover new music and share your music taste with the community. A feature I missed after creating my ListenBrainz account and connecting Spotify was the ability to see my complete Spotify listening history. My project addresses this gap by allowing users to export their extended streaming history from Spotify and import it into ListenBrainz. Additionally, users can import backups from their old ListenBrainz accounts. With the foundation ready, it will be simpler to add support for more file importers in future. This makes transitioning to Listenbrainz easier.
The importer can be accessed here.
The goals of my project were to create:
- a modular backend to import listens from files.
- a frontend for users to create imports and view their status.
- comprehensive backend and frontend tests.
My Work
Firstly, I worked on the backend. I created the API endpoints to create a new import, view existing imports and cancel pending imports. Before building these endpoints, I had to make some changes to the database schema to store and manage import information. This ensured that both individual import details and the complete list of imports could be retrieved efficiently.
POST /1/import-listens
This is the most important endpoint for the importer. The endpoint accepts the service the file to be imported is from, and start and end date to filter the listens for import. The endpoint uses token based auth to allow users to access it directly using the API.
Auth Implementation
user = validate_auth_header(fetch_email=True, scopes=["listenbrainz:submit-listens"])
if mb_engine and current_app.config["REJECT_LISTENS_WITHOUT_USER_EMAIL"] and not user["email"]:
raise APIUnauthorized(REJECT_LISTENS_WITHOUT_EMAIL_ERROR)
if user["is_paused"]:
raise APIUnauthorized(REJECT_LISTENS_FROM_PAUSED_USER_ERROR)This snippet validates the request’s auth header and required scope, then enforces extra restrictions. It rejects submissions from users without an email (if configured) and from users whose accounts are paused. In short, it ensures only authorized, active, and properly set up users can submit listens.
Upon successful validation and authentication, a background import task is created.
Code to create the background task
This query creates an import:
query = """
INSERT INTO user_data_import (user_id, service from_date, to_date, file_path, metadata)
VALUES (:user_id, :service, :from_date, :to_date, :file_path, :metadata)
RETURNING id, service, created, file_path, metadata
"""
result = db_conn.execute(text(query), {
"user_id": user["id"],
"service": service,
"from_date": from_date,
"to_date": to_date,
"file_path": save_path,
"metadata": json.dumps({"status": "waiting", "progress": "Your data import will start soon.", "filename": filename})
})This query creates a background task after creating an import successfully:
query = "INSERT INTO background_tasks (user_id, task, metadata) VALUES (:user_id, :task, :metadata) ON CONFLICT DO NOTHING RETURNING id"
result = db_conn.execute(text(query), {
"user_id": user["id"],
"task": "import_listens",
"metadata": json.dumps({"import_id": import_task.id})
})A background task processor that runs in a separate process will soon pick this task up for processing.
Background Task Processor
There are 2 specific imports for Spotify and Listenbrainz export. The common functions performed by each importer are:
- Unzipping the file and checking for zip-bomb attacks
- Finding the relevant files in the zip archive
- Processing the file contents to extract listens
- Parsing the listens
- Submitting listens to RabbitMQ queue in batches
The importers for different services just differ in the parsing part as the file formats may be different for every service.
ListenBrainz requires three fields at minimum for a valid listen submission: the timestamp of the listen, track artist name and track name. The Spotify data archives do not contain the track artist name but the album artist name. This can often be different. To obtain the correct track artist name, we use the spotify identifiers in the data archive. Using these identifiers, we lookup metadata in an internal cache and then fallback to retrieving the data from Spotify metadata API.
GET /1/import-listens/<import_id>
Fetches details about a single import. This is used when showing information about a specific import or when refreshing its progress on the importers page. It helps track the current state of an ongoing import.
GET /1/import-listens/list
Fetches details about all imports. This is used to display the full list of past imports on the import listens page, giving users an overview of their entire import history.
POST /1/import-listens/cancel/<import_id>/
This is used to cancel a specific import in progress. It also deletes the listening history file uploaded after successfully canceling the import.
Code to delete an import
def delete_import_task(import_id):
""" Cancel the specified import in progress """
user = validate_auth_header()
result = db_conn.execute(
text("DELETE FROM user_data_import WHERE user_id = :user_id AND id = :import_id AND metadata->>'status' IN ('waiting') RETURNING file_path"),
{"user_id": user["id"], "import_id": import_id}
)
row = result.first()
if row is not None:
db_conn.execute(
text("DELETE FROM background_tasks WHERE user_id = :user_id AND (metadata->>'import_id')::int = :import_id"),
{"user_id": user["id"], "import_id": import_id}
)
Path(row.file_path).unlink(missing_ok=True)
db_conn.commit()
return jsonify({"success": True})
else:
raise APINotFound("Import not found or is already being processed.")Frontend
This is the final UI that I implemented.
The form submit button is disabled if a no file is selected or an import is in progress.
Code to disable the import button
<div style={{ flex: 0, alignSelf: "end", minWidth: "15em" }}>
<button
type="submit"
className="btn btn-success"
disabled={hasAnImportInProgress || !fileSelected}
>
Import Listens
</button>
</div>The blue box shows the import in progress. It shows the progress text and a refresh button to refresh the status. Clicking on the “Details button reveals the additional details about the imports. The cancel import option is available if the import has not already started.
Testing
This was the most frustrating part for me personally. This also taught me how to think about testing though. I wrote some of the backend tests which were then improved by Lucifer.
Frontend tests were mostly written by me but I faced a lot of challenges trying to make them pass. I encountered an issue with React Testing Library for file uploads testing and had to resort to using a hack for a specific test.
The Current State and Future Scope
Currently, the importer supports only 2 services: Spotify and Listenbrainz. It can be further expanded due to the modular class-based structure. The new implementations only need to add the logic for parsing the desired file format from the specific service.
Final Thoughts
As someone who listens to music for at least 3-4 hours a day, a service like ListenBrainz is a godsend. Working on it gave me a sense of satisfaction that I have contributed something meaningful to the application I care for.
Looking forward, I am excited to see people using the feature and migrating to ListenBrainz easily. I plan to continue fixing bugs and adding new features to ListenBrainz. I would be happy to contribute further to Metabrainz Foundation.
Working on this project was a very nice learning experience and it is the largest codebase I have worked with till date. It is hard to find such experience even in many internships. I worked on a feature that will be used by thousands of people, hence thorough testing was required.. The mentors were very supportive and also encouraged good practices. This led to the real development. They encouraged me to think like a user and take ownership.
Working with my fellow GSoCers was also a great experience. We helped each other, built a strong bond, and I also made some wonderful new friends along the way. Overall, it was a very nice learning experience. It was a summer well spent 🙂
Thanks for taking out time to read and I hope that you learnt something from this!