Overview of the MVC + D 3D video coding standard

Highlights

• MVC + D supports depth-image-based rendering for advanced 3D video use cases.

• MVC + D decoder can reuse H.264/AVC or MVC hardware decoder implementation modules.

• MVC + D supports package level bandwidth and device decoding capability adaptation.

• MVC + D allows asymmetric depth to have different spatial resolution as texture.

• MVC + D requires typically about twice the bitrate of 2D video coded by H.264/AVC.

Abstract

3D video services are emerging in various application domains including cinema, TV broadcasting, Blu-ray discs, streaming and smartphones. A majority of the 3D video content in market is still based on stereo video, which is typically coded with the multiview video coding (MVC) extension of the Advanced Video Coding (H.264/AVC) standard or as frame-compatible stereoscopic video. However, the 3D video technologies face challenges as well as opportunities to support more demanding application scenarios, such as immersive 3D telepresence with numerous views and 3D perception adaptation for heterogeneous 3D devices and/or user preferences. The Multiview Video plus Depth (MVD) format enables depth-image-based rendering (DIBR) of additional viewpoints in the decoding side and hence helps in such advanced application scenarios. This paper reviews the MVC + D standard, which specifies an MVC-compatible MVD coding format.

Introduction

3D video is becoming popular for example in cinemas, high definition (HD) TVs and smart phones, with different delivery mediums, such as Blu-ray discs, content delivery networks (CDN) and TV broadcasting channels. Interoperability of 3D video content over different applications and devices requires standardized 3D video data and compression formats.

In this section, we will first review the existing 3D video data and compression formats, intending to explain the relation of the MVC-based 3D video coding standard (MVC + D) to earlier coding standards and formats for 3D video. Then, in Sections 1.1 the target applications that may require MVC + D are described. Finally, we outline the contributions of this paper and introduce the structure of the paper in Section 1.3.

Stereoscopic 3D perception is achieved by presenting different views to the left and right eyes, which can either be facilitated with stereoscopic display technology requiring specific viewing glasses or auto-stereoscopic display technology whereby no viewing glasses are required. Consequently, coding of two video views is sufficient to obtain 3D perception with stereoscopic displays. Thus, data and compression formats to address stereoscopic (2-view) and multiview video coding have been developed and can be coarsely categorized into two families: frame-compatible stereoscopic video and inter-view-predicted multiview video coding, which are described in the next paragraphs.

In frame-compatible stereoscopic video, a picture of the left view and the corresponding picture of the right view, jointly referred to as the constituent pictures, are spatially packed into a single frame or temporally interleaved prior to encoding. Then, a regular 2D video codec is used to encode the frame-packed video signal. In the receiving end, the video bitstream is first decoded conventionally and then the pictures of the left and right views are extracted from the output pictures of the decoder to be rendered correctly on a stereoscopic display. Different types of frame packing have been proposed and are supported by various standards. As a common feature among most types of frame packing, the spatial packing of the constituent pictures typically results in reducing the resolution (e.g., horizontally or vertically) of the constituent pictures to half of the respective original resolution and hence keeping the resolution of the packed frames the same as that of a single view prior to encoding. The most commonly used frame packing formats include side-by-side and top–bottom arrangements, where the constituent pictures appear next to each other either horizontally or vertically within the packed frame. The applied frame packing type can be signaled by various means, such as with the frame packing arrangement supplemental enhancement information (SEI) message of H.264/AVC [1].

In inter-view-predicted multiview video coding, pictures for each view are coded as separate entities, which are referred to as view components in MVC [1], [2]. One of the views is designated to be the base view, which can be coded and decoded independently of the other view or views, referred to as non-base views. A picture within a non-base view may use the respective picture in the base view as a reference for inter-view prediction, which improves the compression efficiency of multiview video coding compared to that achieved by frame-compatible stereoscopic video where inter-view prediction typically (e.g., between the views in the same time instance) cannot be used [3].

Multiview video assumes fixed view positions and does not support viewpoint adjustment or additional viewpoint generation in the receiver-side and thus is not sufficient for achieving more flexible 3D perception. Consequently, data formats including depth or disparity information associated with coded video views have been studied, because they enable view synthesis through the depth-image-based rendering (DIBR) process [4]. A sequence of pictures representing depth or disparity or other ranging information for a particular viewpoint is called a depth view, while a regular color image sequence of the view is called a texture view. The pictures of a depth view may be referred to as depth maps regardless of the type of the ranging information actually represented by the depth view. In a format often referred to as 2D + Z, one depth view is coded with one coded texture view. The depth view can be regarded as monochromatic video and coded with any regular 2D video codec. The characteristics of the coded depth view, such as the represented depth range, must be indicated in order to use DIBR. The signaling of the depth view characteristics can be done using MPEG-C Part 3 [5], for example. The 2D + Z format provides no means for handling disocculusions, which are typical in the synthesized views, and hence it facilitates view synthesis within a limited viewpoint angle from the single coded view.

In order to facilitate view synthesis more flexibly than what 2D + Z can support, the Multiview Video plus Depth (MVD) format has been proposed [6], [7]. In MVD, each texture view is accompanied by a respective depth view. The texture views can be encoded as one bitstream and the respective depth views as another bitstream, each e.g., with an MVC codec, but then methods for encapsulating texture bitstream and depth bitstream into the same container, synchronizing transmission and decoding of texture and depth views, and buffer management among other things would have to be dealt within the application layer.

While coding of two views can be considered sufficient for achieving 3D perception on stereoscopic displays, it has been discovered that an optimal selection of camera baseline selection depends on the display size and viewing distance among other things [8]. Furthermore, it has been found that there are relatively large differences regarding the stereoscopic disparity preferences [8]. Moreover, auto-stereoscopic display technology, where a relatively large number of views have to be displayed simultaneously, is emerging. 3D video specifications, including MVC + D, are designed to address new application demands and technology trends that cannot be supported by existing standards.

As creation of the 3D video content is typically expensive, it is preferred that the same 3D video content can be experienced by users with heterogeneous devices, thus adaptation of the depth perception to different screen sizes or user preferences is required. Various application scenarios where adaptation of the 3D content is required are illustrated in Fig. 1 and described in the following paragraphs with reference to Fig. 1. Shown in Scenario (a) is the depth perception adaptation for stereo display; Scenario (b) includes the illustration for the use case involving auto-stereoscopic displays; Scenarios (c) and (d) present the backward compatibility functionalities to MVC and H.264/AVC decoders respectively; Scenario (e) is related to free-viewpoint video. A description of the mentioned scenarios is given below.

The same acquisition and post-production process is often used for 3D content production regardless of the target consumption platform or environment, which may vary from handheld 3D devices to movie theaters. Furthermore, particularly for broadcasting or streaming applications, the same encoded 3D video content is transmitted to any potential end-user devices, which may include for example high-definition 3D television sets (3D HDTVs) and 3D smartphones. However, 3D video content may have been produced in such a manner that the acquisition and post-production parameters, such as camera separation, were optimized for a particular consumption environment, such as 3D HDTVs. Therefore, the depth perception might be sub-optimal for other types of 3D screens or viewing environments.

To provide the best 3D perception across all types of devices, the 3D video content shall provide the possibility of adjusting the 3D perception, e.g., by reducing or increasing the disparity between two stereoscopic views for a display with a width that is different from the most suitable display width of the stereoscopic content. This capability is efficiently supported by coding the 3D video content with MVD.

As shown in Fig. 1, Scenario (a), after receiving the coded 3D video content and decoding the texture and depth pictures with an MVC + D decoder, the smartphone can perform DIBR to generate a view at a desirable location. The generated view, together with one of the transmitted views, can provide more suitable viewing experience for the smartphone. This scenario can also be extended in a way that the user can interactively select the best 3D perception and the position of the virtual view changes real-time.

If a conventional multiview video codec, such as MVC, was used to transmit 3D video content for multiview auto-stereoscopic displays, a substantial bandwidth proportional to the number of views in the display would be required. As tens of views, such as 28 for the Dimenco glasses-free 3D 55” display [9], may be displayed simultaneously, the 3D video content coded with e.g., MVC, for auto-stereoscopic displays may easily require more than 10 times of bandwidth compared to 2D video based on e.g., H.264/AVC coding. Hence, bandwidth reduction for 3D video content covering a relatively large viewing range/angle is a key for the delivery of 3D video content real-time while not degrading the 3D perception.

The MVD representation facilitates transmission of a small number of texture and depth views and rendering more views on an auto-stereoscopic display in the decoding end. For example, up to three views may be transmitted and a number of views in between can be rendered based on the transmitted views, since the DIBR technology provides the possibility to interpolate any view with a location between two horizontally neighboring views [6]. As shown in Fig. 1, Scenario (b), a view synthesis module can take advantage of the MVD representation and generate as many virtual views as needed for the auto-stereoscopic display.

The simulation results in Section 5 indicate that MVC + D requires roughly only twice the bit rate of the 2D video coded with H.264/AVC (e.g., the bit rate of the base view) to transmit 2–3 views, with each of them containing depth, so that any number of views can be rendered on an auto-stereoscopic display.

In Scenarios (c)–(e) of Fig. 1, the coded 3D video bitstream is pruned to contain a subset of texture and depth views according to the capabilities of the receiving device or according to user controls. For these Scenarios, the 3D video scene is firstly captured and then encoded by an MVC + D encoder. Either 2 or 3 views may be coded depending on whether any receivers are expected to have multiview auto-stereoscopic displays. A server transmits the coded bitstream(s) to different clients with different capabilities, possibly through media gateways. The possible client devices include 2D HDTVs, 2D/3D switchable stereoscopic HDTVs, 3D capable smartphones and HD auto-stereoscopic TVs. The media gateway is an intelligent device, also referred to as a media-aware network element (MANE), which may selectively forward the incoming video packets. At the final stage, the coded video is decoded and rendered with different means according to the application scenario and capabilities of the receiver.

Prior to displaying of the 3D video content, the reconstructed MVD representation of the 3D video scene may be processed by a 3D renderer, as part of the client device, to generate one or more virtual views real-time. This process, namely view synthesis, is outside the scope of any existing video coding standard similarly to other post-processing operations. Different devices may have different proprietary view synthesis solutions.

Currently the most popular 3D TVs are stereoscopic TVs, which display just two views. As shown in Fig. 1, Scenario (c), the MANE may forward only the packets belonging to the texture of two views. After receiving the MVC sub-bitstream, a stereoscopic TV with an MVC decoder, decodes and displays just views 1 and 3.

Furthermore, as shown in Fig. 1, Scenario (d), the MVC + D bitstream can be extracted to form an H.264/AVC 2D bitstream which is decodable for majority of the 2D TVs, so a 2D version of the content can be enjoyed.

In Fig. 1, Scenario (e), a regular 2D HDTV is equipped with MVC + D decoding capability and a User Interface (UI) that enables viewpoint selection. The UI can be based for example on a remote control, through which a user can manually select a viewpoint, or head tracking, whereby the viewpoint is automatically selected based on the user’s position relative to the screen. It the determined by the UI does not correspond to one of the transmitted views, view synthesis module provides the virtual view based on the MVC + D content. The virtual view can have a viewing angle or horizontal position between those of any two transmitted views, thus free-viewpoint video is realized [10].

This paper reviews the MVC + D standard, which specifies the encapsulation of texture views and depth views into the same bitstream. The mechanisms that extend the MVC features to MVC + D for depth views as well as the operation points involving depth views are described in this paper. The rest of this paper is organized as follows. Section 2 introduces MVC + D and the other currently ongoing standardization efforts for depth-enhanced multiview video coding and explains their differences and finalization timelines. In Section 3, we describe the fundamental normative features of MVC + D, such as the bitstream structure, while the optional features of the standard are presented in Section 4. Simulation results on the compression performance of MVC + D are presented in Section 5. Finally, Section 6 concludes this paper.