EzAudio

What EzAudio is

EzAudio is a free, open-source diffusion model that generates audio from text prompts. It also supports audio editing and inpainting, while its ControlNet example combines a text prompt with an audio reference. The model works in the latent space of a one-dimensional waveform variational autoencoder. Its paper describes an optimized diffusion transformer for convergence speed, training stability, and memory use, plus classifier-free guidance rescaling for stronger prompt alignment at higher guidance scores. Training uses unlabeled audio, machine-generated captions, and human-labeled fine-tuning data. Browser demos for EzAudio and EzAudio-ControlNet are linked, and code and checkpoints are available for local use. Example code uses PyTorch and SoundFile, with checkpoints hosted on Hugging Face. Local setup requires cloning the repository and installing Python dependencies; CUDA is selected when available, otherwise CPU. EzAudio is not deployed by an inference provider.