Audio to video synchronization (also known as Audio video sync, Audio/video sync, AV-sync, lip sync – or lack of it: lip sync error, lip-flap) refers to the relative timing of audio (sound) and video (image) portions during creation, post-production (mixing), transmission, reception and play-back processing. Whenever sound and video has timing related cause and effect, AV-sync can be an issue in e.g.: television program, videoconferencing, video transcoding, dubbing, computer facial animation, look alike contest and playback singing.

There are various ways where the AV-sync can get mis-synchronized:

During creation AV-sync errors happens because of

Internal AV-sync error: Different processing delays between image and sound in video camera and microphone. The AV-sync interdelay is normally fixed.

External AV-sync error: If a microphone is placed remotely of the sound source, the image is out of sync, because the speed of sound is much lower than the speed of light. If the sound source is 340 meters from the microphone, then the sound arrives approximately 1 second later than the light. The AV-sync interdelay varies with distance.

During mixing of video-clips and sound-clips you normally need to delay either sound or video so they are synchronized. The AV-sync interdelay is static, but can vary with the individual clip.

Video editing effects.

A digital or analog audio video stream or video file ought to have some sort of explicit AV-sync timing. Either in the form of interlaced video and audio data - or by explicit relative time stamping of data. The processing of data shall respect the relative data timing by e.g. stretching between or interpolation of received data. If the processing do not respect the AV-sync error, it will increase whenever data get lost, because of transmission errors or because of missing or mis-timed processing.

Instances of transmission (broadcasting), reception and play-back, that can get the AV-sync mis-synchronized:

A video camera with built-in microphones or line-in ought to delay sound and video paths by the same amount of milliseconds. A video camera should have some sort of explicit AV-sync timing put into the video and audio streams. Solid state video cameras (e.g. CCD and CMOS image sensors) can delay the video signal by one or more frames.

If an audio video stream during transmission gets bit-errors because of electrical glitches (wired) or wireless interruptions - and the AV-stream either miss explicit AV-sync timing or the processing is not respecting the explicit AV-sync timing. The AV-sync interdelay normally increases with time.

There is extensive use of audio and video signal processing circuitry with significant delays in television systems. Particular video signal processing circuitry which is widely used and contributes significant video delays include frame synchronizers, digital video effects processors, video noise reduction, format converters and MPEG pre-preprocessing.

The video monitor processing circuit delay the video stream. Pixellated displays require video format conversion and deinterlace processing that can add one or more frames of video delay.

A video monitor with built-in speakers or line-out ought to delay sound and video paths by the same amount of milliseconds. Some video monitors contain internal user adjustable audio delays to aid in correction of errors.

MPEG-2: Presentation Time Stamp (PTS), Decode Time Stamp (DTS)

Presentation Time Stamps (PTS) can be embedded in MPEG-2 to avoid AV-sync drift. Unfortunately these time stamps are often added after the video undergoes frame synchronization, format conversion and pre-processing,thereforethose delays remain uncompensated.

Viewer experience of mis-synchronized AV-sync

The result typically leaves a filmed or televised character moving his mouth when there is no spoken dialogue to accompany it as it has been removed or changed somehow in post-production, hence the term "lip flap" or "lip-sync error". The resulting audio video sync error can be annoying to the viewer and even lead to detrimental effects on the viewer's enjoyment of the program and the program's effectiveness and can even lead to negative perceptions of the speaker.

The lack of effectiveness problems are of particularly concern when product commercials and political candidates are viewed. See the report "Effects of Audio-Video Asynchrony on Viewer's Memory, Evaluation of Content and Detection Ability" by Reeves and Voelker for more information (a non-copyrighted PDF is available ). Television industry standards organizations have become involved in setting standards for audio video sync errors. See for example ATSC Document IS-191.

Because of the annoyances, AV-sync error is of concern to the television programming industry, including television stations, networks, advertizers and program production companies.

Effect of no explicit AV-sync timing

When a digital or analog audio video stream do not have some sort of explicit AV-sync timing these effects will break AV-sync:

In film movies these timing errors are most frequently caused by worn films skipping over the movie projector sprockets because the film has torn sprocket holes.

Errors can also be caused by the projectionist misthreading the film in the projector, although this is rare with competent projectionists.

Audio to Video Synchronization is frequently corrected and maintained with an audio synchronizer. Television industry standards organizations have established acceptable amounts of audio and video timing errors and suggested practices related to maintaining acceptable timing.

A/V sync errors are becoming a significant problem in the digital television industry because of the use of large amounts of video signal processing in television production, television broadcasting and pixelated television displays such as LCD, DLP and plasma displays.

In the television field, audio video sync problems are frequently caused when significant amounts of video processing is performed on the video part of the television program.

Typical sources of significant video delays in the television field include video synchronizers and video compression encoders and decoders. Particularly troublesome encoders and decoders are used in MPEG compression systems utilized for broadcasting digital television and storing television programs on consumer and professional recording and playback devices.

A source of significant video delay is found in pixelated television displays (LCD, Plasma display, DLP)which utilize complex video signal processing to convert the resolution of the incoming video signal to the native resolution of the pixelated display, for example converting standard definition video to be displayed on a high definition display. Lip flap may exceed 200 ms at times.

In broadcast television, it is not unusual for lip flap or lip-sync error to vary by over 100 ms (several video frames) from time to time.

Sources/references


http://www.chiariglione.org/mpeg/faq/mp2-sys/mp2-sys.htm#mp2-19 MPEG-2 Systems FAQ: 19. Where are the PTSs and DTSs inserted?]
MPlayer-G2-dev: mpeg container's timing (PTS values)
birds-eye.net: DTS - Decode Time Stamp
svcd2dvd.com: Perfect AV Sync: Preparation is key...
http://www.pixelinstruments.tv/pdf/Articles/Effects%20of%20Audio-Video%20Asynchrony.PDF
http://www.atsc.org/standards/is_191.pdf
http://www.atsc.org/news_information/newsletter/ATSC_Newsletter-11.pdf
http://www.atsc.org/standards/is_191.pdf