SyncNet

SyncNet learns joint audio-visual representations that can determine whether a video's mouth movements are synchronized with its audio track. It was trained on thousands of hours of talking face videos to develop a robust understanding of audio-visual correspondence.

In the lip sync field, SyncNet serves two critical purposes: as an evaluation metric to objectively measure how well a lip sync model's output matches the target audio, and as a discriminator or loss function during training to guide lip sync models toward producing more accurately synchronized results. Wav2Lip notably uses a SyncNet-based discriminator as a core component of its training pipeline.

SyncNet relates to several other concepts in the AI lip sync pipeline: Wav2Lip , and Lip Sync .