Turning a single piano note into a full, playable 88-key instrument that holds its timbre across the entire range.
Overview
The trained model works like a pitch-shifter tuned to a specific musical instrument and how it changes over its full range (in this case the piano). Since real instruments have inharmonic elements and are register-dependent - a naïvely shifted low C sounds nothing like the real high C of the same piano. I solved this by freezing a pre-trained neural audio codec (Descript's DAC) and training a small (~27M-parameter) adapter that translates a source latent to a target pitch under direct supervision. It trains end-to-end in under an hour on a single GPU and reaches a 0.323 validation loss.
Challenges & lessons learnt
I first tried two harder architectures and abandoned both: A VQ-VAE plus latent transformer, which failed because the encoder had no incentive to separate pitch from timbre - it baked pitch into the very latent the downstream model was meant to transform, and plateaued without ever producing clean transfer. An adversarial disentanglement scheme using a gradient-reversal layer to force a pitch-free latent. Elegant on paper, but the pitch classifier kept beating the encoder across every weighting I tried - disentangling a high-dimensional continuous latent with a small network turned out to be genuinely hard, not a tuning detail. The breakthrough was a reframe: I didn't need a pitch-invariant representation, I needed a pitch-translated output - a supervised regression instead of an adversarial game. From there, the two remaining bugs both turned out to live upstream of the model: a stubborn validation-loss floor caused by training pairs that varied in more than pitch (fixed by switching to deterministic VST-rendered data where only pitch changes), and a tonal "buzz" in note decay tails caused by a flat L1 loss under-weighting low-energy frames (fixed with a per-frame cosine-direction plus log-magnitude loss). The recurring lesson is that the biggest wins came from reframing the problem and fixing the data and objective, not from a bigger model.
Next Steps
This is an on-going project and there is still room for improvement in terms of the audio quality - in particular the lower registers of the piano require further tuning. The next big step will be to generalise this approach to other instrument types, allowing any recorded one-shot sample to be used to generate a full multi-sample one-shot instrument.
Project on github
https://github.com/offalWaffle/Multi-Salmon-Training