Synchronization in MPEG-4 Systems

Abstract

The MPEG-4 standard was defined in the early days of broadband Internet, after successful deployments of digital television networks, with the goal of unifying both broadcast and broadband media architectures and protocols in a single standard, tackling natural media (audio, video, images) as well as synthetic 2D or 3D graphics and audio. As such MPEG-4 can be seen as one of the first attempt at building the so-called convergence of Web and TV. Some parts of the standard have changed the media world forever (AAC audio and AVC|H264 video compression, MP4 file format), and while other parts have not always met their markets in successful way, they paved the way for more recent works, including HTML5 media. In this chapter, we explain how the MPEG-4 standard manages playback and synchronization of audio-visual streams and graphics animations and how multiple timelines can be used to provide rich interactive presentation over broadband and broadcast.

Definitions

Timestamp

time label assigned to a block of data in a media delivery system; timestamps are usually integer numbers, converted back to seconds using a timescale or timestamp resolution value expressing the number of integers in a second; for example, a timestamp of 180000 in a timescale of 90000 indicates a time of 2 s.

Decoding Time Stamp

time at which a media frame has to be available for the decoder.

Composition Time Stamp (or Presentation Time Stamp)

time at which a media frame must be presented to the user; composition and decoding times usually only differ for video stream using bi-directional motion estimation schemes, hence requiring a coding order different from the display order.

Intra-stream Synchronization

process and signaling required to reproduce a time-accurate, jitter-free playback of a given media stream; for example, signaling and maintaining the exact time lapse between successive video frames in a fixed or variable frame rate stream.

Inter-stream Synchronization

process and signaling required to reproduce a time-accurate playback of multiple media streams; for example, signaling and maintaining stream timestamps for achieve lip synchronization between an audio and a video stream.

Clock Reference

value against which decoding (resp. composition) timestamps are compared to trigger the decoding (resp. display) process; depending on the delivery systems, the clock reference may be delivered together with the content, as is the case in broadcast environments, derived from the wall-clock time another time source such as internet time, or derived from any other local system clock/counter (CPU cycle count, audio output sample rate, etc.).