Synchronization and flow adaptation schemes for reliable multiple-stream transmission in multimedia presentations

https://doi.org/10.1016/S0164-1212(00)00093-5Get rights and content

Abstract

In the practical networking environments, an orchestration of distributed multimedia presentation (DMP) with multiple media streams, e.g., WWW multimedia presentations over the Internet, is always disturbed due to the unpredictable end-to-end delay. Presentation adaptation associated with multimedia synchronization and flow control is a feasible solution to achieve a smooth multiple-stream DMP. In the paper, we propose the Pause-And-Run approach for k-stream (PARK) multimedia presentations over Internet to achieve reliable transmission of continuous media. Main characteristics of the PARK approach are as follows. (i) To achieve reliable transmission for continuous media, PARK adopts TCP instead of UDP. (ii) Since the slow-start scheme is embedded in TCP, a novel flow adaptation scheme is proposed to reduce the overhead of the network and end hosts. The server adapts its transmission rates to the buffer situation of the client and prevents the client's buffers from overflow and underflow as much as possible. (iii) With the provision of multiple-stream synchronization and the multi-level adaptation control, the client achieves smooth multimedia presentations and achieves graceful presentation degradation when the resources are insufficient. This paper shows the evaluation of applying the PARK approach over Internet. The evaluation results reveal a suitable buffering control policy for the audio and video media, respectively.

Introduction

The evolution of Internet and multimedia technology significantly contributes to the progress of distributed multimedia presentations (DMPs). A DMP integrates multiple media streams, e.g., audio, video, image, and text streams, and possesses timeliness requirement of media units with respect to the presentation. Temporal relations of a multimedia presentation can be defined and be scheduled in a multimedia presentation schedule (Huang et al., 1998). Fig. 1 depicts an example of a multimedia presentation schedule. For example, the DMP has to immediately present the audio medium A2 at time t3, which is also the end time of presenting media units A1 and V1. At the same time, the DMP has to present the image medium I3. The goal of a DMP is to present all composed media streams according to the temporal presentation schedule. However, in a distributed environment, the diverse network devices/links and the network traffic bursts always disturb the temporal relations, and then deteriorate the QoS satisfaction of the clients (Chandra et al., 1998, Metz, 1999a, Rajan et al., 1999). In order to compensate for temporal anomalies, a DMP has to accomplish multimedia synchronization.

Multimedia synchronization is the essential requirement to achieve smooth coordination and cooperation among various media (De Lima et al., 1999, Li and Liao, 1997, So et al., 1999). Two types of media synchronization are the intra-medium synchronization and the inter-media synchronization. Intra-medium synchronization tries to reserve the temporal relations between media units within a medium stream. Inter-media synchronization tries to reserve the temporal relations among related media streams. Intra-medium and inter-media synchronization can be achieved by allocating some buffers to pre-deposit media units (Biersack and Geyer, 1999, Johnson and Zhang, 1999). The pre-allocated buffers effectively compensate for temporal anomalies under the provision of bounded network delay.

However, the current Internet is an IP best-effort network, in which the network delay is usually unpredictable. Due to the unpredictable network delay, the buffer overflow and underflow always occur, and thus the presentation quality is severely deteriorated. For example, when a medium stream experiences longer delay, the associated buffer at the client becomes empty. On the other hand, the other media streams may experience normal or tolerable delays such that the associated buffers are still good for presentations. Under such a situation, a multiple-stream multimedia presentation becomes complicated.

In order to have smooth multimedia presentations over unpredictable delay networks, many DMPs adopt the resource reservation technique and/or the presentation adaptation technique. Based on the resource reservation technique, a DMP reserves required resources, such as transmission bandwidth, memory, or CPU cycles, in advance according to the quality-of-service (QoS) requirement (Foster et al., 1999, Metz, 1999b, Zhang, 1997). Based on the presentation adaptation technique, the server (client) adaptively transmits (presents) media units according to some measured parameters, such as the buffer status, the transmission delay, and the lost rate. The purpose of presentation adaptation is to meet the QoS requirement as much as possible (Campbell et al., 1998, Waddington and Hutchison, 1998). Resource reservation seems a very useful scheme to achieve a smooth DMP. However, many practical networks, e.g., the current Internet, which are composed of many heterogeneous routers and most of them only provide the best-effort transmission, still cannot achieve effective resource reservation (Gecsei, 1997, Xiao and Ni, 1999). Thus, many researchers intensively work on various multimedia synchronization approaches with the adaptation technique (Chatterjee and Brown, 1999, Hafid and Bochmann, 1998, Hollfelder et al., 1997, Rothermel and Helbig, 1997). With the adaptation technique, the server dynamically decreases or increases the transmission rate according to the feedback messages sent by the client. When the network resources are insufficient, the client adaptively moves the presentation time of media units to produce graceful presentation degradation. At the same time, the client sends feedback messages to the server in order to lessen the degraded situation according to some measured parameters. The well-known IVS teleconferencing system achieves A/V transmission and presentations with adaptation control (Bolot and Turletti, 1998, Turletti and Huitema, 1996). In IVS, the server's transmission rate is based on the lost rate of the video stream. When the lost rate increases (decreases), the client sends feedback messages to decrease (increase) the server's transmission rate. The client and server achieve the adaptation control using the video encoding parameters, which include the refresh rate, the quantizer, and the movement detection threshold.

Traditionally, a DMP adopts UDP to transmit continuous media, i.e., audio and video, because of their real-time requirement and their tolerance of data loss. UDP is a best-effort communication protocol and does not provide any flow control mechanism. UDP-based DMPs are easy to have unperceived qualities when the network congestion occurs. To deal with the network congestion, UDP-based DMPs adopt the so-called TCP-like scheme (Bolot and Turletti, 1998, Kato et al., 1998, Tan and Zakhor, 1999). According to the TCP-like scheme: (1) the server quickly decreases its transmission rate when the network congestion occurs in order to release the congestion situation; (2) the server slowly increases its transmission rate when the network congestion is released in order to utilize available bandwidth as much as possible.

On the other hand, due to critical reliability requirement, some multimedia applications adopt TCP to achieve lossless transmission of continuous media. For example, telemedicine applications transmit medical image/video/audio frames or some MPEG-based applications transmit I-frames. The slow-start scheme is embedded in TCP to achieve the flow control (Stevens, 1994). Based on the slow-start scheme, the transport layer of the server decreases the transmission rate when the network congestion occurs and slowly increases the transmission rate until the network congestion is released. That is, the flow control is automatically achieved at the operation system (OS) layer when TCP is adopted. Therefore, the user/application layer doesn't need to activate another slow-start-like mechanism. On the top of the TCP's slow-start mechanism, the flow control should be network-aware (Bolliger and Gross, 1998). That is, based on some available measured parameters such as buffer status, packet delay, and lost ratio, the server can effectively adapt its transmission rate to the network situation. In IVS, TCP is used to achieve reliable transmission and the flow control is based on the media lost rates. However, the media lost rates are not easily available at the user/application layer when TCP is adopted.

In this paper, we propose the Pause-And-Run approach for k-stream (PARK) multimedia presentations, where k is equal to 1,2,3,…, or n. The PARK approach adopts the concept of streaming and adaptation to handle medium transmission and presentations. The main characteristics of the PARK approach are as follows. (i) To have lossless transmission for continuous media, PARK adopts TCP instead of adopting UDP. (ii) The server's transmission rate depends on the buffer situation of the client. When the client's buffer becomes low, the server increases the transmission rate of the corresponding medium stream. (iii) When the client's buffer becomes empty due to the really long delay, the whole presentation just pauses for a while in order to avoid the serious presentation Jitter between/among media streams. During the paused period, the server increases its transmission rate and the client continues to receive media units. (iv) The client resumes the paused presentation when the number of buffered media decreases to a threshold. On the other hand, when the client's buffer tends to be full, the server decreases its transmission rate or pauses the transmission to avoid the buffer overflow at the client site.

The rest of the paper is organized as follows. Section 2 describes the proposed PARK approach. Section 3 describes the synchronization and presentation adaptation schemes that are adopted in the PARK. Section 4 describes the abstract system architecture to develop the PARK approach. Section 5 describes the system development techniques of the PARK approach. Section 6 describes the evaluation results of applying the PARK approach over the Internet environment. Section 7 has some discussion and conclusion remarks.

Section snippets

The PARK approach

In the deployment of presentation adaptation, the client sends feedback messages to the server in order to utilize the available network bandwidth as much as possible. For the feedback control, the thresholds of the buffer determine the time of sending feedback messages and the transmission rate of the server. In order to have more stable transmission rate and to reduce the number of feedback messages, the PARK drives a graceful policy to send feedback messages based on the concept of the

Synchronization and presentation adaptation schemes

The timeliness requirement of continuous media and the heterogeneity between the continuous and static media complicate the multimedia synchronization control of a multiple-stream presentation. In this section, we describe the multimedia synchronization scheme that is suitable for a multiple-stream presentation. Furthermore, in order to adapt a presentation to the network situation, we propose a multi-level adaptation scheme to achieve graceful presentation degradation under a worse network

The abstract system architecture

Fig. 6 depicts the abstract system architecture based on the PARK approach. The server and client are composed of the Manager, the Synchronizer and some Actors. The Actors of the server communicate with the corresponding Actors of the client site via the dedicated media channels. The Actors of the server retrieve media units from the corresponding media bases and transmit them to networks. The Actors of the client receive media units from the network and present them in the corresponding

System development

In this section, we describe the system development of the PARK approach based on the proposed system architecture. The state transition diagrams are used to explain each system component. The Retriever captures media units from the medium base using different capture rates according to the buffer situation.

The Retriever uses three thresholds to adjust the capture rate. HT is used to avoid buffer overflow. LT is used to avoid buffer underflow. Middle Threshold (MT) is used to indicate the

Evaluation of the PARK approach

We evaluate the PARK approach over the practical Internet environment. Fig. 13 depicts the evaluated multimedia presentation, which is composed of the video, audio, image and text streams. Two SUN SPARC 20 workstations are used. Video compression/decompression is based on Parallax XVideo motion JPEG card. Audio acquisition/playout is performed through the SUN's audio hardware, which provides 8 bits μ-law audio at the 8k sampling rate. Medium transmission rate for video and audio is 15 media

Discussion and conclusion

In the current best-effort Internet, the orchestration of a multiple-stream DMP is always disturbed by the non-deterministic end-to-end delay. In the past, people always thought that audio and video transmission can tolerate loss and thus UDP is adopted in order to reach the real-time requirement. However, some applications do need lossless transmission of audio and video. Two examples are as follows. (i) Transmitting I-frames of an MPEG stream. Since the P-frames and B-frames depend on a

Chung-Ming Huang is a professor at Department of Computer Science and Information Engineering, National Cheng Kung University, Taiwan. He has received his BS degree in electrical engineering from National Taiwan University in 1984, and MS and PhD degrees from The Ohio State University, both in computer and information science, in 1987 and 1991, respectively. He has serviced as the chair of ISO special interested group for Open/Office Document Architecture in Taiwan, and as the General Secretary

References (27)

  • E. Biersack et al.

    Synchronized delivery and playout of distributed stored multimedia streams

    ACM Multimedia Systems

    (1999)
  • J. Bolliger et al.

    A framework-based approach to the development of network-ware applications

    IEEE Transactions on Software Engineering

    (1998)
  • J.C. Bolot et al.

    Experience with control mechanisms for packet video in the Internet

    ACM Computer Communication Review

    (1998)
  • A.T. Campbell et al.

    Transporting QoS adaptive flows

    ACM Multimedia Systems

    (1998)
  • Chandra, P., Fisher, A.L., Kosak, C., Steenkiste, P., 1998. Networking support for application-oriented QoS. In:...
  • Chatterjee, S., Brown, M., 1999. Adaptive QoS resource management in dynamic environments. In: Proceedings of IEEE...
  • De Lima, R.M., Junqueira, F.P., Goncalves, P.A.daS., Duarte, O.C.M.B., 1999. SAMM: An integrated environment to support...
  • Foster, I., Kesselman, C., Lee, C., Lindell, B., Nahrstedt, K., Roy, A., 1999. A distributed resource management...
  • J. Gecsei

    Adaptation in distribution multimedia systems

    IEEE MultiMedia Magazine

    (1997)
  • A. Hafid et al.

    Quality-of-service adaptation in distributed multimedia applications

    ACM Multimedia Systems

    (1998)
  • S. Hollfelder et al.

    A client-controlled adaptation framework for multimedia database systems

    Lecture Notes in Computer Science

    (1997)
  • C.M. Huang et al.

    MING-I: A distributed interactive multimedia document development mechanism

    ACM Multimedia Systems

    (1998)
  • T.V. Johnson et al.

    Dynamic playout scheduling algorithms for continuous multimedia streams

    ACM Multimedia Systems

    (1999)
  • Cited by (8)

    • FMF: Query adaptive melody retrieval system

      2006, Journal of Systems and Software
      Citation Excerpt :

      McCann et al. (2000) developed an audio delivery system called Kendra that uses adaptability with distributed caching mechanisms to improve data availability and delivery performance over the Internet. Huang et al. (2001) presented PARK approach for multimedia presentations over a best-effort network in order to achieve reliable transmission of continuous media such as audio or video. This section describes a schema for representing music and its metadata in the database and explains how to construct and maintain a dynamic index called FAI out of frequently queried melody tunes for fast music retrieval.

    • Towards realizing the importance of placing fog computing facilities at the central office of a PON

      2017, International Conference on Advanced Communication Technology, ICACT
    View all citing articles on Scopus

    Chung-Ming Huang is a professor at Department of Computer Science and Information Engineering, National Cheng Kung University, Taiwan. He has received his BS degree in electrical engineering from National Taiwan University in 1984, and MS and PhD degrees from The Ohio State University, both in computer and information science, in 1987 and 1991, respectively. He has serviced as the chair of ISO special interested group for Open/Office Document Architecture in Taiwan, and as the General Secretary of the Chinese Image Processing and Pattern Recognition Society. His research interests include QoS networking, personal wireless/mobile/WWW communication software, interactive distributed multimedia systems, and multimedia protocol engineering.

    Hsu-Yang Kung received the BS degree from Tatung University, the M.S. degree form National Tsing Hwa University, the PhD degree from National Cheng Kung University, Taiwan, all in computer science and information engineering, in 1987, 1989, and 2000, respectively. He is currently a lecturer at the Department of Management Information Systems, National Pingtung University of Science and Technology, Taiwan. His research interests are multimedia networking protocols and distributed multimedia systems.

    Jiong-Liang Yang received the BS degree in applied mathematics from National Sun Yat-Sen University, and the MS degree in computer science and information engineering from National Cheng Kung University, Taiwan, in 1996 and 1998, respectively. His research interests include multimedia synchronization and multimedia networking protocols.

    The research is supported by the National Science Council of the Republic of China under the grant NSC 89-2219-E-006-008.

    View full text