TasteMender

Privacy-first music discovery through acoustic similarity

Overview

TasteMender is a music discovery app that recommends songs based on acoustic characteristics such as danceability, energy, mood, and genre. You start with a song and shape the recommendations through genre and decade filters, audio-feature settings, and a balance between similarity and popularity. It takes a privacy-first approach: recommendations don't depend on a persistent profile of your listening habits.

Motivation

I believe that finding new media to enjoy nowadays is an increasingly arduous task. Consumers' fragmented tastes and the ever-growing breadth of available content make serendipitous discovery a rare event. Online platforms pigeonhole users into highly personalized but restrictive content bubbles and don't offer much direct control over how content recommendations are made. Often, it feels impossible to find interesting media that's outside your usual preferences.

I want TasteMender to be an "eject button" from that behaviour-driven loop. It's deliberately built to give music recommendations based on the intrinsic characteristics of the songs themselves. It isn't concerned with user behaviour; at the same time, it gives users as much direct control as possible over what gets recommended. You actively tell the system what you want rather than letting it predict that for you.

Note

The app was originally developed as a final project for the BSc Computer Science degree at Goldsmiths, University of London (available here). This repository continues that work, aiming to eventually provide a fully-featured music discovery experience.

Etymology

The name TasteMender brings together two core ideas:

• Taste, as in shaping your musical tastes through discovery.
• Mender, a nod to "recommender" and the idea of mending listening habits that have grown stale.

Technology

TasteMender uses a Django REST API with a React and TypeScript frontend. Audio features and metadata are extracted from the AcousticBrainz dataset, with tracks, artists and albums identified through MusicBrainz IDs (MBID).

For more information, see the development guidelines and testing guidelines.

How It Works

The dataset

AcousticBrainz is a project that ran from 2015 to 2022, and gathered acoustic information about music recordings through crowdsourcing. Its goal was to provide an open dataset that developers and researchers could use to build or study music recommendation engines.

The dataset contains almost 30 million submissions which, when deduplicated, cover 7.5 million recordings. Each submission is made up of:

• metadata - MusicBrainz IDs, title, artist, album, etc.
• low-level features - raw mathematical metrics extracted directly from the audio waves
• high-level features - statistical predictions generated by feeding the low-level features into machine-learning models

The high-level features are suitable for building a recommender while keeping resource usage manageable for a small project. They take up ~40 GB for the whole dataset (versus 600 GB for the low-level features) and contain categorical predictions that can easily be used in comparison algorithms. Below is an example of a prediction of whether a sound has a "party" mood.

"mood_party": {
  "all": {
    "not_party": 0.993278443813,
    "party": 0.00672157853842
  },
  "probability": 0.993278443813,
  "value": "not_party"
}

A complete submission with all predictions is included in high-level-sample.json.

Building the database

The crowdsourced nature of the dataset makes it difficult to map its data directly to a database. Popular recordings can have thousands of duplicates, and many submissions have missing or incomplete data. The ingest pipeline (backend/ingest/pipeline.py) is responsible for cleaning and normalizing the data. It also splits the data based on how it will be used:

• Track metadata is stored in a database because it maps well to the relational model.
• Audio features are stored in a feature matrix file (features_and_index.npz), which is loaded into RAM, avoiding repeated disk access during recommendation comparisons.

The two data stores are linked together through track MBIDs. They only need to be synced once, during ingest, as the dataset is stable and most likely won't receive any updates in the foreseeable future.

Generating recommendations

The feature matrix sits at the core of how recommendations are made. It contains unique entries for each track of the dataset, consisting of:

• A unique identifier: the MBID.
• A 16-dimensional audio feature vector built from high-level prediction probabilities, including danceability, moods, voice/instrumentalness, tonality, brightness, and MIREX mood clusters.
• Dortmund and Rosamerica categories used to group tracks by genre.
• Release year used to group tracks by decade (70s, 80s, 90s).

Making a recommendation starts by providing the service (services/recommender.py) with a target track. The feature matrix is then optionally filtered to a subset of candidates that have the same genre and were released in the same decade. The remaining candidates are ranked by calculating the cosine similarity between their feature vectors. This measures how closely their audio profiles align. Because the feature data is stored in RAM, the process usually takes under 100 ms.

Note

Cosine similarity is a standard and widely used metric for making recommendations. Currently, it is calculated between the target track and every candidate track. Approximate nearest-neighbour search, supported by libraries such as FAISS, Annoy, and Voyager, could speed up the recommendation engine, with a configurable trade-off between speed and retrieval accuracy.

Putting users in control of recommendations

Both the API and the frontend expose filters that allow users to adjust which tracks are recommended. They consist of:

• Weights applied to audio features in the similarity calculation determine how much each mood should matter.
• Guardrails for decade and genre restrict the subset of candidate tracks.
• Popularity versus similarity re-ranks tracks based on how many times they were submitted to the dataset and how similar they are to the target.

Results are limited to one track per artist in order to encourage variety. The filters are always shown in the frontend player to encourage users to experiment and discover.

Playback experience

The frontend player is built around the idea of a flow of discovery. You select a track, it becomes the target for recommendations, and you're shown a subset of similar tracks. When the current track finishes, the top recommendation is played and becomes the next target.

Limitations

The AcousticBrainz dataset is frozen at the June 2022 dump, and coverage is uneven, so some artists, genres, and release periods are better represented than others. Because the data is crowdsourced and deduplicated, metadata may be incomplete or inaccurate, including occasional mismatches between artists and songs.

The API is stateless, so filter preferences, listening sessions, and playback history are not persisted across page refreshes or devices.

There is no direct mapping between MBIDs and YouTube videos. Playback therefore depends on searching for {track.title} {artist_name}, which can return incorrect results. Strict YouTube Data API quotas also make it difficult to scale the app beyond a small number of users.

Repo Structure

• backend/
  • music_recommendation/ - the main Django project
  • recommend_api/ - recommendation API
    • api/ - endpoint implementations
    • services/
      • recommender.py - recommendation logic
      • youtube_sources.py - gets playable sources for tracks
    • tests/ - API and service tests
    • models.py - database models
    • serializers.py - API response and validation serializers
  • ingest/ - AcousticBrainz dataset processing and database-building tools
    • management/commands/ - Django commands such as build_db and recommend
    • tests/ - ingest pipeline tests
  • features_and_index*.npz - generated recommendation feature data
• frontend/ - React and TypeScript app that consumes the API
  • src/pages/ - application pages
  • src/components/ - reusable UI components
  • src/hooks/ - shared React hooks
  • src/api.ts - API client
• docs/ - development, testing, architecture, roadmap, and decision records