Images, video, voice and music
Lip sync
In short
Lip sync is the matching of visible mouth movements to spoken audio. In AI video, a model can change or generate a talking face so its lips follow a new speech track, even for a person not seen during training1.
- Also called
- Lip synchronisation, Speech-to-lip generation, Lip dubbing
- Topic
- Images, video, voice and music
- Level
- Beginner
A lip-sync system takes a video or still face and an audio track, then estimates how the mouth should move at each moment. It generates or modifies facial pixels so the visible speech matches the sound1.
The challenge is more than opening and closing the mouth. The system must handle different speakers, head movement, lighting and speech sounds. The Wav2Lip paper describes earlier systems as struggling with arbitrary identities in dynamic, unconstrained talking-face videos1.
Lip sync is not the same as text-to-speech. Text-to-speech makes audio from words; lip sync aligns a face to audio. It is also not proof that the person shown said the words, because the audio or face may have been generated or replaced1.
It appears in dubbed films, translated lessons, virtual presenters, music videos and accessibility tools. Wav2Lip is an example of a research model that accepts a talking-face video and target speech and was evaluated with lip-synchronisation benchmarks1.
Example
A language-learning app keeps the same presenter on screen but replaces the English voice with Spanish speech. Lip-sync software adjusts the presenter’s mouth movements to match the Spanish audio1.
Why it matters
Good lip sync makes dubbed or generated video easier to watch. Poor lip sync is an obvious sign that the voice and face do not belong together, while convincing lip sync can make impersonation harder to spot12.
Questions people ask
How does AI lip sync work?
It analyses speech and the face video, predicts mouth movements for the sounds, and generates or edits the face frames to match the audio. Wav2Lip uses a learned lip-sync discriminator to improve alignment1.
Is lip sync the same as dubbing?
No. Dubbing replaces the spoken audio, often with another language. Lip sync is the visual matching step that makes the mouth movements fit the replacement or generated audio1.
Can AI lip sync any face to any voice?
Some systems are designed for arbitrary identities and speech, but results are not guaranteed. The Wav2Lip research specifically studies unconstrained videos and reports performance on its own evaluation benchmarks1.
Related terms
Tools that use it
- Kling AI Avatar 2.0Kling feature that turns one picture and a voice track into a talking avatar video up to five minutes
- LiveAvatar by HeyGenHeyGen's API for real-time talking avatars that listen and reply on video inside other apps
- ElevenLabs Image & VideoElevenLabs feature for generating images and videos with outside models, then adding voice and music