A compact latent-diffusion toolkit for text-to-audio and sound-effect (foley) generation, in plain PyTorch.
foleydiff implements the whole text → audio latent-diffusion stack from scratch and small enough to read in an afternoon: a mel-spectrogram VAE, a conditional 1-D U-Net denoiser, DDPM/DDIM samplers with classifier-free guidance, a dependency-free text encoder, and a Griffin-Lim vocoder. Because the vocoder and the text encoder need no pretrained weights, the entire pipeline installs with just PyTorch + NumPy and runs end-to-end on CPU.
⚠️ It's a framework, not a model zoo. The components ship with random weights. foleydiff gives you the machinery to train and sample latent audio diffusion; it does not bundle a trained checkpoint. Out of the box,generate()returns correctly-shaped, noise-textured audio.
Most text-to-audio repos are research dumps wired to multi-gigabyte dependency
stacks (a frozen T5, a HiFi-GAN, a pretrained VAE, diffusers, accelerate, …).
That's great for results and terrible for understanding. foleydiff goes the
other way:
- Two runtime dependencies. PyTorch and NumPy. WAV I/O uses the standard library; everything else is optional.
- Readable, from-scratch components. The diffusion algebra, the U-Net, the VAE and the samplers are each a single small, typed, tested module.
- Runs anywhere. No checkpoints to download, no CUDA required. CI generates
audio on a CPU runner with the
tinypreset. - Honest seams. Want a real text encoder or neural vocoder? Each is a clean, documented interface you can swap.
# CPU-only PyTorch keeps the install small; use a CUDA build for training.
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install foleydiffFrom source:
git clone https://github.com/hazelp343/foleydiff.git
cd foleydiff
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -e ".[dev]"from foleydiff import TextToAudioPipeline, save_wav
from foleydiff.presets import get_preset
pipe = TextToAudioPipeline(get_preset("tiny"))
audio = pipe.generate("rain on a tin roof", seconds=2.0, steps=40, seed=0)
save_wav("rain.wav", audio, pipe.config.mel.sample_rate)Or from the shell:
foleydiff generate "glass shattering" -o sfx.wav --preset tiny --seconds 2 --steps 40
foleydiff info --preset smallprompt ─► HashingTextEncoder ─► context
│ (cross-attention)
noise ─► UNet1D ◄─ timestep │ DDPM / DDIM reverse loop
└─────────────► latent ─► AudioAutoencoder.decode ─► log-mel
└─► exp ─► Griffin-Lim ─► waveform
A VAE compresses mel-spectrograms into a short latent sequence; the diffusion
model learns to generate that latent, conditioned on text via cross-attention;
the decoder and vocoder turn the sampled latent back into audio. See
docs/architecture.md and
docs/design-notes.md for the full story.
- 🎛️ Pluggable samplers — ancestral
DDPMSamplerand accelerated, deterministicDDIMSampler; register your own with one decorator. - 🧭 Classifier-free guidance built into both samplers.
- 🔀 Three parametrisations —
epsilon,v_prediction,sample— with the conversion algebra centralised so samplers stay agnostic. - 📐 Four beta schedules — linear, scaled-linear, cosine, sigmoid.
- 🔊 Self-contained vocoding via Griffin-Lim (no neural-vocoder download).
- 🧪 Reproducible — one seed drives sampler and vocoder.
- 🧰 Typed, linted, tested —
ruff,mypy,pytestacross Python 3.9–3.12.
scripts/check.sh # ruff + mypy + pytestSee CONTRIBUTING.md. Contributions welcome — small, tested PRs especially.
MIT © 2026 Mingyan Yang