Optical Flow

Optical flow computes a dense motion field describing how each pixel moves from one frame to the next, effectively creating a map of all motion in the scene. In lip sync, optical flow serves multiple purposes: it helps maintain temporal consistency by constraining how much the generated mouth region can change between frames, assists in blending the generated region with the original by matching motion patterns, and provides motion cues that complement audio information for predicting mouth movements.

Some lip sync architectures use optical flow as an explicit input alongside audio features, while others learn implicit motion representations.

Optical Flow relates to several other concepts in the AI lip sync pipeline: Temporal Consistency , and Motion Field .