LatentSync
LatentSync is a free, open-source lip-sync method for Linux and self-hosted deployment.
What LatentSync is
LatentSync is a free, open-source lip-sync method for Linux and self-hosted deployment. It uses audio-conditioned latent diffusion models to align video mouth movement with audio. Whisper converts mel spectrograms into audio embeddings, which are supplied to the U-Net through cross-attention layers. The project offers inference through a Gradio application or a command-line interface. LatentSync 1.6 was trained on 512 × 512 resolution videos to address blurry teeth and lips. Minimum stated inference memory is 8 GB of VRAM for version 1.5 and 18 GB for version 1.6. Increasing inference steps can improve visual quality but slows generation; increasing guidance scale can improve lip-sync accuracy while risking distortion or jitter. The release includes inference code, checkpoints, a data-processing pipeline, and training code. U-Net training configurations require 20–55 GB of VRAM. The documented pipeline uses PySceneDetect and InsightFace landmarks.
Good
- Free and open-source release
- Gradio and command-line inference options
- Includes checkpoints and training code
- Supports self-hosted Linux deployment
Watch out
- Version 1.6 requires 18 GB inference VRAM
- Training configurations require 20–55 GB VRAM
- Higher guidance can cause distortion or jitter
- LatentSync 1.6 lacks Inference Provider deployment
Who it suits
It suits Linux users who can self-host an open-source lip-sync system and have the required GPU memory. It also fits developers who want inference, data-processing, and training resources.
Verdict
LatentSync combines open-source lip synchronization with both graphical and command-line inference. GPU requirements vary substantially by version, and generation settings involve quality, speed, and stability trade-offs.
Plans and prices
From LatentSync's own pricing page, checked 8 Oct 2026.
| Plan | Price | What it gives you |
|---|---|---|
| Open-source release | Free | Inference code and checkpoints · data processing pipeline · training code |
The facts
Each one read from LatentSync's own pages, with the date it was checked.
What it does
- Audio conditioning
- Whisper converts mel-spectrograms into audio embeddings that enter the U-Net through cross-attention.checked 21 Oct 2026
- Community complaint: GPU compatibility
- Open issues include requests for RTX 5090 support and reports that older PyTorch versions do not work with 50-series Nvidia cards.checked 21 Oct 2026
- Community complaint: output quality
- Open issues include reports of red faces, blurry mouths, face misalignment and visible teeth artifacts.checked 21 Oct 2026
- Community issue volume
- The GitHub issue tracker showed 211 open issues and 16 pull requests when opened.checked 21 Oct 2026
- Data pipeline
- Its processing pipeline resamples video to 25 FPS and audio to 16 kHz, detects scenes, splits clips into 5–10 second segments, aligns faces and filters low-quality samples.checked 21 Oct 2026
- Data processing tools
- The documented data pipeline uses PySceneDetect for scene detection and InsightFace landmarks to transform faces.checked 8 Oct 2026
- Dependencies
- Pinned dependencies include PyTorch 2.5.1, Diffusers 0.32.2, Transformers 4.48.0, Gradio 5.24.0 and GPU ONNX Runtime 1.21.0.checked 21 Oct 2026
- Developer extensibility
- The repository exposes scripts for preprocessing, inference, U-Net training and file-list generation, enabling custom training workflows.checked 21 Oct 2026
- Efficient training tradeoff
- The 20 GB efficient Stage 2 configuration may slightly reduce visual quality and temporal consistency compared with the 30 GB configuration.checked 21 Oct 2026
- Environment
- The setup script creates a Conda environment with Python 3.10.13, installs FFmpeg and requires the Linux package libgl1.checked 21 Oct 2026
- Generation trade-off
- The README says more inference steps can improve visual quality but slow generation, while higher guidance scale can improve lip-sync accuracy but cause distortion or jitter.checked 8 Oct 2026
- Hardware requirement
- The stated minimum inference VRAM is 8 GB for LatentSync 1.5 and 18 GB for LatentSync 1.6.checked 8 Oct 2026
- Hosted API inputs
- Layer's schema requires at least one source-video file and one reference-audio file and supports seed and guidance-scale fields.checked 21 Oct 2026
- Hosted API option
- Layer documents a POST inference endpoint that accepts a source video and reference audio through bearer-authenticated JSON requests.checked 21 Oct 2026
- Hugging Face availability
- The model page provides Diffusers usage instructions and links to Google Colab and Kaggle notebooks.checked 21 Oct 2026
- Independent description
- ExplainGit describes LatentSync as a ByteDance research tool that replaces a video's lip movements to match a new audio track using Stable Diffusion and Whisper.checked 21 Oct 2026
- Independent setup assessment
- ExplainGit rates setup difficulty as hard and estimates more than one hour to reach a first run.checked 21 Oct 2026
- Inference interfaces
- The repository provides both a local Gradio application and a command-line inference script.checked 21 Oct 2026
- Inference memory 1.5
- LatentSync 1.5 requires at least 8 GB of VRAM for inference.checked 21 Oct 2026
- Inference memory 1.6
- LatentSync 1.6 requires at least 18 GB of VRAM for inference.checked 21 Oct 2026
- Inference options
- The project provides a Gradio app and a command-line interface for inference.checked 8 Oct 2026
- Intended users
- The README describes an efficient stage-two training configuration as suitable for consumer-grade GPUs such as the RTX 3090.checked 8 Oct 2026
- License
- The source repository is licensed under Apache License 2.0.checked 21 Oct 2026
- Limit: hardware
- Zenn identifies relatively high hardware requirements and recommends about 20 GB of VRAM.checked 21 Oct 2026
Alternatives to LatentSync
- 1
Sync Labs
Free planAndroidiOSWebAPI8.0From $5 a month - 2
Morph Studio
Free planiOSWeb7.8From $12 a month - 3
VEED
Free planAndroidiOSWebExtension7.5Free plan - 4
Muse
Free planOpen sourceLinuxmacOSSelf-hostedWindows7.4Free plan - 5
CHAMELAION LipSync API
Free planWebAPI7.3From €5 a month - 6
lipsyn.cc
Free planWebAPI7.3€11.99 one-time