It ships as a standalone app, a VST3, and an AU plugin.

Built to live inside a plugin

SA3 Variations is built on Stable Audio 3. To get it running inside a JUCE plugin, I ported the inference code to MLX in C++, with bit-exact outputs against the Python implementation, so it could be embedded directly in the plugin rather than shelling out to a separate Python process.

Variations is just one application of that stack. The same C++ inference pipeline, MLX bindings, and pre-converted models work for any similar plugin you might want to build.

For audio developers

If you’re an audio developer curious about using this kind of approach in your own JUCE project, everything is open source and ready to build on:

• the C++ inference pipeline
• the MLX bindings
• pre-converted models on Hugging Face

Fork it, extend it, and build your own apps on top of SA3 in JUCE.

Links

• Code
• Releases
• App / plugin install guide
• Converted models

What’s next

Breaking open the model for some more fine-grained, musically interesting edits.

You can try it live at remixab.maxgraf.space.

The new Stable Audio 3 release is genuinely interesting, and audio-to-audio is the mode I got most drawn into. Rather than generating from scratch, it takes an existing track plus a text direction and edits the original by a controllable amount. The app uses Stable Audio 3 medium by default.

Keeping the vocals

Stable Audio’s vocals tend to come out weird — the model seems to have been trained mostly on instrumental material, so getting reliably good vocals out of it is hard. To work around this, the app runs source separation in the browser (locally) before anything goes to the GPU, strips the vocals out, sends only the instrumental off to be remixed, and then layers the original vocals back over the result.

The source separation runs entirely on your machine through my own demucs.onnx port to WebGPU — the same engine behind Demucs WebGPU.

Locking tempo and key

The model has no inherent sense of tempo or key, so a remix can easily drift to a different BPM or land in a different key than the source. To keep it anchored, a small analysis step runs the moment you drop a file, detecting the BPM and key, and both get appended to the prompt. Without that, the remix often ends up at a different tempo or in a different key than the original.

Try it

Drop in a track, describe how you’d like it to sound, and use the A/B player to compare the original against the remix. It’s a fun thing to play around with — pick something you’d like to hear in a different style and see what comes back.

Open Stable Audio Remix

What’s next

A version that runs fully on your own laptop, with no GPU round-trip at all.

WavNav is an audio sample explorer built to make large sample libraries easier to navigate. Instead of digging through folders and filenames, it maps sounds into a 2D visual space where similar samples sit near each other, so browsing becomes much more like exploring a map than managing files.

If you want the full product overview, check out the WavNav product page first. It is much more in depth than this post and covers the app in its proper context.

The project sits at the intersection of audio analysis, machine learning, and music production tooling. The current version focuses on fast local analysis, visual search, and practical studio workflows for producers working with big libraries.

Demo · click to play

What it does

• Visual browser: samples appear as points on a map, with similar sounds clustering together.
• Semantic search: search for sounds using text prompts like "snare", "bright pluck", or "dark pad".
• Search by sound: drop a sample into the app to find similar material in your library.
• Key and BPM filters: narrow results by musical key, tempo, or both.
• Folder management: add multiple sample folders and toggle them on or off without rebuilding everything from scratch.
• Local-first workflow: machine learning models and audio processing run on your own machine, and the site copy describes the product as fully local with no cloud uploads or data collection after download.

Workflow

WavNav is built around quick sample discovery:

• pan and zoom around the map to explore dense libraries visually
• hover to preview sounds
• click or inspect samples while searching
• copy file paths directly for use in a DAW
• open files in Finder from the app
• save folders and settings between sessions
• use right-click similarity search to find nearby alternatives

Performance and scale

WavNav is made for large-library use:

• built for 50,000+ samples
• about ~1 minute for a first load of a 50k library
• about <3s startup from cache
• 60fps navigation target
• GPU-accelerated rendering via OpenGL/Metal
• clustering and map-based exploration using audio embeddings and UMAP-style visualisation

Platform status

• macOS: supported on Intel and Apple Silicon, with a stated minimum of macOS 13.0
• Windows: supported in beta, x86 only, with a minimum of Windows 10 version 1803

Screenshot

Product page • Features • Screenshots • Buy WavNav

As part of my work on CORPUS, I wanted to see how far this could be pushed directly in the browser, and specifically whether WebGPU could make a serious difference. The answer was yes: with the right changes, Demucs can run surprisingly efficiently client-side and feel genuinely fast.

What I changed

The goal was simply to adapt the original project and extend it for practical WebGPU execution on the web.

That included:

• pushing inference through ONNX Runtime Web with the WebGPU execution provider
• building the surrounding DSP path around it with WASM
• keeping the full separation flow in-browser, from loading audio to rendering separated stems
• adding a simple interactive demo interface with waveform views, transport controls, stem muting/soloing, and downloads

I also added some practical web-facing work around model loading and asset handling so the whole experience feels more like a usable demo than a raw port.

Why it was interesting

What I liked about this project was that it makes a fairly strong point very quickly: heavyweight music ML models do not always need to stay trapped in native desktop code or server-side inference setups.

With WebGPU, the browser becomes a viable place to run this kind of model at useful speed. That opens up interesting possibilities for lightweight music tools, interactive demos, and more accessible deployment of audio ML systems. For instance, this exact system powers real-time source separation in Reef DJ.

Demo

You can try the live demo here:

Demucs WebGPU demo

Rather than focusing on one narrow task, the pipeline performs a broad multi-stage reading of a track. It can process folders in batch or work as an API, and it produces a unified metadata view that describes a piece of music from several angles at once.

What it does

The pipeline is designed to analyse musical audio in a way that is useful for real systems. Depending on the track, it can extract and combine:

• basic metadata such as title, artist, duration, file type, and sample rate
• musical attributes such as key, mode, tempo, and tempo changes
• stylistic descriptors such as genres, moods, instruments, and keywords
• descriptive text that summarises the overall character of a track
• vocal information, including whether vocals are present, lyric transcription, detected language, and singer-profile fields
• content safety signals for lyrics and potentially sensitive material
• non-music filtering, helping separate genuine music from speech-heavy or irrelevant uploads
• cover-song detection signals for identifying likely matches against known material

The result is a single structured representation of a track that can be consumed by other tools and interfaces.

Applications

The interesting part of the project is the fact that all of these views are brought together into one coherent pipeline. That makes it possible to work with music collections in a much richer way than simple file metadata allows.

For example, the system can support workflows such as:

• organising and indexing large music catalogues
• powering upload-time analysis for music platforms
• generating richer search and filter options
• preparing tracks for recommendation, discovery, or visualisation layers
• supporting moderation and rights-related review workflows
• feeding downstream interfaces that need structured musical descriptions

In other words, it turns raw audio into something much more searchable, inspectable, and operationally useful.

Design goals

From the start, I approached this as a production-minded analysis system rather than just a research prototype. That meant designing for:

• breadth across many kinds of musical descriptors
• speed and batch throughput
• reliability across multiple connected analysis stages
• observability, so processing can be monitored and diagnosed
• integration, both as a folder-based pipeline and as an HTTP API

The pipeline can analyse whole directories of music files, return JSON outputs, and also expose the same capabilities through service endpoints for other tools to call programmatically.

Practical output

What comes out is a fairly complete metadata package for each track. That package can include musical structure, descriptive tags, text summaries, vocal and lyric analysis, moderation signals, and auxiliary matching results.

Context

I developed this project through my involvement with CORPUS, where the focus is on building practical AI and music technology that can plug into larger systems and workflows.

It was heavily inspired by Google’s MusicFX DJ. The point was to see whether a similar kind of live, prompt-driven music interaction could be built without relying on proprietary closed models. Instead of waiting for a full track to render, Reef DJ continuously generates short musical segments in real time and streams them straight into the browser for uninterrupted playback.

Core idea

Reef DJ was built to show that interactive AI music systems do not necessarily need very large models or heavy cloud infrastructure to feel responsive and musically playful. In this case, the underlying generation approach was based on Meta’s MusicGen, which also made the prototype interesting from a research perspective: it suggested a path toward training and adapting this kind of system on our own material rather than treating the model as a fixed black box.

The project focuses on immediacy, controllability, and a clear performance metaphor.

At the interface level, it behaves a bit like a speculative AI DJ deck:

• you build up the generation prompt from multiple weighted text ideas
• you shape the overall musical character with high-level controls
• the system keeps streaming fresh music in short successive chunks
• stems can be manipulated in real time while playback continues

That makes the interaction feel less like submitting a prompt and more like steering a live evolving mix.

What it does

The prototype supports several performance-oriented features:

• prompt-weighted mixing, where multiple text prompts can be blended with slider-based emphasis
• high-level creative macros for overall density, brightness, and unpredictability
• tempo guidance through a direct BPM control
• continuous real-time playback in the browser
• stem-based remixing, with separate control over drums, bass, and the remaining musical material
• rolling audio capture, allowing the most recent minute of output to be downloaded

Because the system works with a rolling musical context rather than isolated one-shot outputs, it can produce longer developments and gradual stylistic shifts. Sometimes those transitions are surprisingly convincing, and sometimes they expose the model’s limits in a very audible way.

Motivation

What interested me about this project was getting a hands-on feel for the generation model itself, and also building the interaction design around it. Reef DJ asks what happens when AI music generation is treated as a responsive musical instrument rather than a series of text-to-music prompts.

That opens up a set of questions:

• how direct can prompt-based musical control feel?
• what kinds of abstractions are useful in a performance setting?
• how can generated music remain fluid enough for mixing and intervention?
• what is the smallest practical setup that still feels musically responsive?

Within my broader work, Reef DJ sits somewhere between creative tooling, generative music systems, and performance interface design.

Demo

Context

This project emerged as I went further down the rabbit hole of audio-domain music generation and started building a series of practical prototypes around it. It also became a natural bridge between my earlier XR music work and my then recent role at CORPUS, being one of the first prototypes I built for the project.

Netz is an XR musical instrument that grew directly out of my PhD research in AI and Music. For more information check out the website at netzxr.com. The instrument itself emerged from a research process focused on musical control, ergonomics, hand tracking, and expressive interaction in mixed reality.

Netz was co-created through a longitudinal participatory design process with a professional keyboard player and music producer. Across ten design sessions, we developed an instrument that keeps the playing area compact, anchors interaction to a tabletop for better physical grounding, and uses hand- and finger-level gestures for nuanced musical control.

Research idea

The core design goal behind Netz was to move away from large, fatiguing mid-air XR interfaces and toward something that feels more playable, learnable, and expressive for musicians. The result is a compact mixed-reality instrument that sits on a physical surface while extending interaction into 3D space above it.

Its note layout is based on a custom Tonnetz-inspired structure designed for harmonic exploration. That makes it possible to play notes, scales, chords, and chord inversions through spatial patterns rather than a traditional piano keyboard layout. Additional “bridge” notes were introduced during the design process to make major and minor scale playing more practical with one hand.

Screenshot

How it works

• Surface-anchored mixed reality: the instrument is placed on a real table, giving the player a stable physical reference while still allowing 3D gestures above the surface.
• Expressive note layout: the interface uses an isomorphic harmonic layout tailored for learning chord shapes, scales, and transposition.
• Hand-pose control: different hand poses are used to distinguish between single notes, chords, and inversions.
• Continuous expression: wrist and finger movements can shape pitch, timbre, dynamics, and portamento-like transitions between notes.
• On-device sound engine: the prototype was built as a self-contained instrument with synthesis running directly on the XR headset.
• Interactive machine learning: a personalised hand-pose model was integrated to reduce sensing errors and improve musical control.

On the product side, the current Netz website describes full XR integration, hand tracking and gesture recognition, DAW integration, customisable gestures, MPE-style note expression, and support aimed at Meta Quest headsets, with Apple Vision Pro support planned.

Under the hood

The prototype described in my thesis was implemented in Unity for Meta Quest passthrough mixed reality. It combines:

• a custom tabletop XR interface
• collision-driven note interaction
• a 3,7,4-Tonnetz-inspired note structure
• hand-tracking-based gesture control
• an on-device wavetable synthesizer
• MIDI MPE messaging for expressive per-note control

One major theme of the project was dealing with system errors in XR hand tracking. Netz became the basis for my later work on using interactive machine learning and, beyond that, multimodal sensing to make XR musical instruments more reliable and expressive.

Outcomes

Netz has already led to both academic and commercial outcomes.

• It won the 2023 MIDI Innovation Awards in the category Software Prototypes and Non-Commercial Products.
• It was showcased at NAMM 2024 as part of the MIDI Innovation Awards winners.
• It formed the basis of multiple research publications on XR musical instruments, hand tracking, and interactive machine learning.
• It supported follow-on commercial exploration through Queen Mary Innovation Impact Fund support and the Innovate UK ICURe Discover and ICURe Explore programmes.

The project was also featured in a Reuters article on AI and music.

Publications and links

Website • Demo video • When XR Meets AI (AES 2024) • Combining Vision and EMG-Based Hand Tracking (CMMR 2023) • Reducing Sensing Errors in a Mixed Reality Musical Instrument (VRST 2023)

The key idea is simple: the headset provides the global hand position and orientation, while a deep learning model predicts finger joint rotations from forearm muscle activity. That means the system can keep estimating finger articulation even when the camera loses a clear view of the hand.

Data preprocessing, feature extraction, and model pipeline

What the paper does

• combines Meta Quest 2 hand tracking with Myo armband sEMG sensing
• estimates the MCP and PIP joint rotations of the index, middle, ring, and little fingers
• uses a deep learning model to map windowed sEMG features to finger joint angles
• evaluates the multimodal system against the headset’s native vision tracking
• uses Leap Motion as a reference tracker for finger joint angle comparisons

This was a proof-of-concept study, but it established an important direction for the rest of my PhD: if vision-based XR tracking breaks under self-occlusion, adding a second sensing modality can make musical interaction more reliable.

Publication note

The version linked below under the original CMMR title is currently only publicly available as a preprint. After the conference, the organisers allowed authors to further revise and extend their papers, and a longer version of this work was later published as a Springer LNCS book chapter:

Multimodal Hand Tracking for XR Musical Instruments Using Electromyography
Music and Sound Generation in the AI Era (LNCS 15236), first published online on 10 October 2025.

Main result

The multimodal approach often produced lower finger-tracking error under self-occlusion than vision-only tracking. In full-view conditions, vision tracking remained competitive or better in several tasks, which is expected. The value of the approach is therefore not that sEMG replaces vision, but that it complements it when camera tracking becomes unreliable.

That distinction is important: the overall tracking pipeline is multimodal, but the finger-pose model itself is unimodal and uses only sEMG-derived input features.

Relevance for musical XR

For XR musical instruments, small tracking failures quickly become musical failures: wrong notes, unstable control, broken gestures, and loss of performance flow. This paper was one of my first serious attempts to address that problem at the sensing level rather than only at the interaction-design level.

It also directly fed into later work on:

• more reliable control for Netz
• interactive machine learning for gesture interpretation
• larger-scale investigations into multimodal hand tracking for XR musical performance

Video

The video below shows a side-by-side comparison between the vision-only tracker and the multimodal system.

Recognition

The paper was published at CMMR 2023 and was nominated for the best paper award.

Links

CMMR paper (preprint) • Extended LNCS book chapter • Repository record • Video • sEMG Unity code • Python pipeline • Unity bridge

The paper, published at NIME 2022, presents an early mixed reality musical interface (MRMI) built for the Microsoft HoloLens 2. The core idea was to investigate how musicians would actually respond to a virtual instrument arranged in physical space: how large it should be, how gestures should work, and whether hand-pose recognition could become musically useful rather than just technically novel.

Musical inteface and mapping controls captured from the XR device in mixed reality

What I built

The prototype was implemented in Unreal Engine and used a piano-inspired octave of 12 virtual musical objects floating in mixed reality. Each object could be touched, grabbed, and dragged through space.

The playing model split musical interaction across both hands:

• the right hand triggered and manipulated notes and arpeggios
• the left hand controlled chord quality through recognised hand poses
• the whole instrument could be repositioned, rotated, and scaled with two-handed gestures

Sound generation happened externally through OSC-to-MIDI control into a DAW instrument, while the XR device handled interaction, visuals, and hand tracking.

Motivation

This was not meant to be a finished instrument. It was an exploratory prototype study designed to surface design questions for future XR musical instruments:

• What instrument sizes and spacings feel playable in mixed reality?
• Can hand poses become a meaningful control layer for harmony and expression?
• Is lightweight real-time pose recognition robust enough for performance?
• Can interactive machine learning help performers personalise the system to their own hands?

To explore that last point, I integrated an IML workflow that let performers record their own hand poses, train a classifier on-device, and map those poses to different chord types. That made the instrument adaptive rather than fixed: players could reshape the control vocabulary to match their own hands and mnemonic strategies.

Study findings

I evaluated the prototype with 10 musically experienced participants. The study showed that the concept was promising, but also made several limitations very clear.

• Compactness matters: participants consistently wanted a smaller, more body-centred instrument.
• Familiar musical logic helps, but a piano-like layout in mid-air does not automatically feel like a piano to play.
• Pose-based chord control was compelling when performers could personalise it through IML.
• Tracking reliability, lack of haptics, and limited field of view made sustained performance harder.
• Several participants wanted the interface anchored to a physical surface for better timing, confidence, and reduced fatigue.

Those findings directly shaped the design direction of later projects, especially the move toward more compact, surface-anchored XR instruments and more robust sensing pipelines.

Legacy in the PhD

This project was the starting point for several later themes in my research:

• ergonomic design of XR musical instruments
• personalised hand-pose recognition
• mixed reality as more than a novelty interface
• the need for tactile or spatial anchoring in musical XR
• the gap between promising interaction ideas and reliable musical control

In that sense, this NIME paper is the point where the broader research trajectory really began.

Links

Paper • Code • IML demo video

Audio examples

Music generated from scratch:

Music generated from prompts:

Screenshot

Report (PDF) • Code