Diff2Lip

What Diff2Lip is

Diff2Lip is an open-source project that uses an audio-conditioned diffusion model to synchronize speech with face videos. It takes speech and face video as inputs and aims to generate lip-synced video while preserving the identity shown in the source frames. The repository includes inference scripts for audio-video pairs and for a single video. In cross mode, an audio source drives a video; in reconstruction mode, audio drives the video's first frame. The project lists movies, education, and virtual avatars as applications, with video conferencing described as a possible future use. Diff2Lip is not real time yet. The project reports reconstruction and cross-audio-video results on the VoxCeleb2 and LRW datasets and identifies the work as a WACV 2024 paper. A Google Colab notebook is linked for trying it. Setup instructions use Python 3.9, FFmpeg 5.0.1, and the repository's requirements file. Inference can use one or more GPUs, with settings above one running distributed generation. The code and text use a CC BY-NC 4.0 license, permitting sharing and adaptation with attribution for noncommercial purposes.