A novel holes filling method based on layered depth map and patch sparsity for complex-scene images

Abstract

The depth image based rendering (DIBR) is a key technology in the emerging 3D video and 3D-TV, which uses a single texture image and the corresponding depth map to generate the virtual views. A critical problem occurs when the virtual view generation refers to regions covered by foreground objects in the original view, which might be disoccluded in the synthetized view. In this paper, a holes filling method based on layered depth map and patch sparsity for complex scenes is proposed. The proposed approach consists of three parts. The first part refers to use of the depth information to divide the complex scene into simple scenes, which represents the separation of foreground and background with a special depth value using an adaptable layered method based on the depth map. The second part represents the holes filling in the depth map using the patch sparsity. Finally, in the third part, the background information and the filled depth map are used to fill the holes in the color map. The experimental results have indicated that compared to other methods, the proposed method provides better quality of synthetized complex scene and 3D scene.

Introduction

Recent success in 3D-TV and 3D video [1,2] indicates the growing interests in 3D contents. 3D video is captured and reconstructed from the real world in real time while the 3D attribute allows the video objects to be rendered at arbitrary viewpoint or merged with virtual 3D background. One of 3D research topics relates to acquisition and transmission of 3D video content. Therefore, a technique that virtually moves data through the scenes composed of limited number of viewpoints has been spotlighted.

In the past, this technique was difficult to realize since the depth camera was not common because of its high cost. However, in recent years, with the widespread emergence of high-performance and low-cost depth cameras, researchers have begun to consider using depth image-based rendering (DIBR) technology to generate virtual views of texture images and their corresponding depth images. The DIBR technique uses depth and camera information to back-project the pixels from the original image to 3D world, and the resulting 3D coordinates are then re-projected to the image plane of virtual views, which is known as 3D warping. Nonetheless, DIBR reduces the storage space and preserves the bandwidth.

However, in DIBR technique, a critical problem in generation of virtual views refers to regions of background occluded by foreground objects in the original views that might become invisible in the virtual views and cause the holes in the virtual views, which is known as disocclusion. Besides the disocclusion problem, the distance of pixels might also cause holes in virtual views because of an inaccurate depth map.

Generally, there are two categories of methods to fill the holes. One is to fill the hole with the information of itself without using of depth map, such as interpolation, texture synthesis and image inpainting [3]; and the other is to use both the information of hole and its depth information. The aim is to improve the method for image holes filling by using depth information to find the best-matched patch [[4], [5], [6]], or by using Gaussian Mixture Model (GMM), Foreground Depth Correlation (FDC) to obtain the background model. Nonetheless, the above mentioned methods are suitable only for simple scenes that can be easily divided into foreground and background. Their main disadvantage is that they cannot achieve good performance in complex scenes.

In this paper, an image hole filling approach based on layered depth image and patch sparsity primarily for complex scenes is proposed. In the proposed approach, the complex scene is divided into multiple simple scenes by using an adaptable layered method based on the scene depth map. In addition, an improved hole filling approach based on patch sparsity, wherein the filling order is from foreground to background and the depth map is a limit of filling process, is introduced.

The paper is organized as follows. In Section 2, the related work is presented. The proposed filling algorithm is explained in Section 3. The image preprocessing is introduced in Section 4. In Section 5, the hole filling approach based on layered depth image and patch sparsity is presented, and the hole filling in texture images based on patch sparsity combined with the filled depth map is introduced in Section 6. The experimental results are presented in Section 7. Finally, the conclusions and future work guidelines are given in Section 8.