Comparative study of frame-compatible stereo 3D services and a novel method for spatial interleaving using HEVC

Highlights

• We compare spatial and temporal interleaving based frame-compatible methods.

• We benchmark the methods with MVC and present the results for both H.264 and HEVC.

• We propose to pack the views into a frame and encode the same in different tiles.

• We also propose to disable in-loop filters across the tile boundary.

• We benchmark the proposed method with the “no-tiles” case.

Abstract

There are a number of ways to realize frame-compatible stereo 3D services, which includes the spatial and temporal interleaving based methods. We first present the detailed analysis of these methods with respect to compression efficiency, backward compatibility and the ability to re-use existing infrastructure. Simulations have been set up to benchmark the Rate–Distortion (R–D) performance of these options with the state-of-the-art Multiview Video Coding (MVC) and results show that temporal interleaving based method performs better when compared to spatial interleaving in both H.264/AVC as well as in HEVC compression schemes, with MVC still outperforms both the methods. In the case of the spatial interleaving method using HEVC, we propose to encapsulate the two views in different tiles with filters turned off across the tile boundary. Simulation results show that the proposed method, when compared to “no-tiles” case, improves decoder performance by approximately 50% with no reduction in R–D performance.

Introduction

Success of movies like Avatar 3D along with the increase in the availability of cost effective 3D-ready devices (Televisions, Gaming consoles, Mobile phones) have renewed the interest in 3D and stereoscopic 3D video services are getting introduced across various industries. In the mobile industry, HTC Evo is one of the 3D mobile phones, which allow the user to capture the 3D video and stream the contents wirelessly to the 3DTV [1]. In the consumer industry, the Blu-ray consortium announced the support for 3D Blu-ray and almost all major TV manufacturers are producing 3D Blu-ray players. In the broadcast industry, Digital Video Broadcast (DVB) specification, which is used in a significant number of countries, provides the support for 3D. In the UK, the BBC introduced 3D broadcasting using the ‘spare’ available spectrum on BBC HD [2]. Hence, considerable momentum has been built up for 3D, with still the need to maintain backward compatibility with the existing 2D receivers (including set-top boxes (STB) and integrated receivers/decoders (IRD)).

RF spectrum is precious and the possibility of second digital dividend (UHF 700 MHz band) will make the spectrum even more precious [3]. Hence, apart from backward compatibility, the increased video data (right view of stereoscopic 3D) also have to be efficiently compressed, in addition to the ability to reuse the existing production and distribution infrastructures as much as possible. H.262/MPEG-2 [4] was the earliest standard, which provided support for the compression of stereoscopic 3D video material. It was not popular at that time, as display/compression technology was not advanced enough to adopt 3D video in the existing 2D infrastructure and workflow. Two decades later, the technologies have improved to an extent that H.264/AVC requires approximately half of the bandwidth when compared to MPEG-2 to deliver the same video content with almost the same quality [5] and one of the goals of HEVC [6] is to reduce this bandwidth requirement by further half. The state-of-the-art multiview video coding standard is the H.264 Multiview Video Coding (MVC) specification, which provides a compact representation of multiple views of a video scene and stereo 3D video has been considered a special case of MVC [7]. Though this specification has been adopted by the 3D Blu-ray consortium [8], unfortunately it has not been adopted in other industries such as broadcast. For example, DVB organization has adopted only the H.264/AVC and H.264 Scalable Video Coding (SVC) codecs [9], [10] for broadcast services from the H.264 family.

There are a number of ways to efficiently compress the stereoscopic 3D video material apart from using H.264 MVC. One approach is to squeeze the left and right views into a single frame and treat the result as a 2D frame (spatial interleaving). Alternatively, the left and right view frames could be temporally interleaved to form a 2D video sequence and compress the resultant 2D video using two temporal layers. In this scenario, enhancement layer contains the right view pictures and base layer contains the left view pictures. In addition to this, there are simulcast methods available which broadcasts both the 2D and 3D videos, where squeezing the left and right view videos into single 2D frame will form the 3D video. Apart from these methods, there are others, which include the support for encoding auxiliary information such as depth maps and camera parameters, along with the standards based compression for the efficient reconstruction of a scene. Though these methods are being studied in the context of H.264 AVC, H.264 MVC and HEVC [11], these are not standardized yet and hence not considered in the paper. All these methods have their own advantages and disadvantages (in terms of compression efficiency, backward compatibility, reuse of existing infrastructures, etc.) and to the best of the authors’ knowledge, no prior work has considered these aspects in the comparison of spatial and temporal interleaving based methods. In this paper, an attempt has been made to compare these methods in terms of the above-mentioned parameters and also proposes a novel method of encapsulating the two views in different tiles (for the case of spatial interleaving in HEVC) with filters turned off across the tile boundary, which enables parallel encoding/decoding of stereo video content.

The rest of the paper is organized as follows: Section 2 explains the various frame-compatible stereo 3D services, from the different perspectives as outlined in the last paragraph, Section 3 illustrates the proposed method where the two views are encapsulated in different tiles with filters turned off across tile boundary, Section 4 details the simulation setup and presents the simulation results, Section 5 provides the conclusions and finally references are listed at the end of the paper.