MMAudio

What MMAudio is

MMAudio is a free, open-source project that generates audio from video or text and aligns generated audio with video frames. Its command-line demo and Gradio interface support video-to-audio and text-to-audio generation. The interface also offers experimental image-to-audio processing: the model was not trained for this mode, and the demo duplicates a still image into a video. The demo saves audio as FLAC and video as MP4; its default output duration is 8 seconds, and durations far from that may reduce quality. MMAudio runs on Linux or in a self-hosted setup. Installation requires Python 3.9 or later and PyTorch 2.5.1 or later with corresponding torchvision and torchaudio; the project says it has only been tested on Ubuntu. In 16-bit inference, the default large_44k_v2 model uses around 6 GB of GPU memory, according to project experiments. Generated results may include unintelligible speech-like sounds or background music, and the model may struggle with unfamiliar concepts. The code is MIT licensed, while pretrained checkpoints use CC-BY-NC 4.0; the project does not guarantee pretrained models are suitable for commercial use.