Unsupervised virtual view synthesis from monocular video with generative adversarial warping

Highlights

• An unsupervised virtual view synthesis approach from monocular video.

• Embedding traditional DIBR into generative network.

• Introducing perception constraints to optimize the visual quality.

• Presenting plausible synthesized results in both objective and subjective studies.

Abstract

Virtual view synthesis from monocular video is challenging, as it aims to infer photorealistic views from single reference view. Previous work have achieved acceptable visual quality, however, are heavily relied on supervision information, such as depth or pristine virtual view, which are inadequate in practical application. In this paper, an unsupervised virtual view synthesis method is proposed to get rid of the supervision information. Firstly, it embed a spatiotemporal generative adversarial network into traditional depth-image-based rendering framework with no explicit depth information provided. Secondly, it utilized novel perceptual constraints without relying on pristine images, including the blind synthesized image quality metric and no-reference structure similarity. The entire framework is fully convolutional, producing hallucinated results in an end-to-end way. Particularly, the whole framework is independent of supervision information. Experimental results demonstrate that the proposed method produces pleasant virtual views in comparison with supervised methods, thereby can be beneficial to practical applications.

Introduction

The rapid development of mobile devices and wireless network makes interactive 3D graphic applications popular in recent years. Good examples include 3DTV, free-viewpoint video, 3D navigation and virtual environment roaming, as well as some recent attempts from the industry, namely Google Stadia, Nvidia GRID and Microsoft Project xCloud. To support the above applications, depth-image-based rendering (DIBR) [1], which is based on virtual view synthesis, is one of a most cost-effective solution. Major bottlenecks of DIBR are acquisition and transmission of reference views. To alleviate the problem, synthesizing virtual views from limited reference views is desirable, yet being challenging, especially from monocular video. To infer a novel view from monocular view is an ill-posed problem, which can be formulated as follows:

$$I^{v_{vir}}=F(I^{v_{ref}};\Phi )$$

where $I^{v_{ref}}$ denotes a reference view and $\Phi$ indicates supervision information, such as depth, disparity, or appearance flow to $I^{v_{ref}}$.

Existing virtual view synthesis methods can be generally categorized into geometry-based and learning-based approaches. The former ones transform pixels of a reference view to a virtual view with explicit geometry constraints, while the latter ones learn a parametric model of the scene and use it to generate novel views.

Geometry-based methods, including photometric stereo, depth estimation and appearance flow, rely heavily on explicit geometry of a scene structure, which may not be available in practice. Unfortunately, estimating scene geometry is a hard problem itself. For example, depth estimation may not work for regions with non-Lambertian reflectance and transparency. Moreover, estimated depth can describe spatial and occlusion relationship, only if the camera pose can be obtained. Besides, pixel transformation cannot infer the contents in disoccluded regions of synthesized views, resulting in severe geometric distortions. Post-processing, such as texture synthesis or image inpainting, can alleviate geometric distortions to certain extent, yet possibly inducing new distortions and time cost.

Learning-based methods do not need explicit geometry information, yet estimating implicit geometry information by utilizing feature representation based on convolutional neural network (CNN). However, the training of a generative network desires pristine virtual views as supervision. Taking Deep3D [2] as an example, the $l_1$ pixel error based loss function relies on the ground truth of a synthesized image. In practice, such pristine virtual views are rarely available to satisfy the training, thereby limiting its generalization.

The paper alternatively transforms the virtual view synthesis task into a generative adversarial way. With depth-image-based rendering as the backbone, we embed a spatiotemporal generative adversarial network (GAN) into virtual view synthesis, together with implicit depth estimation, to produce a novel view. To make the result hallucinated, it designs novel perceptual constraints, especially a blind synthesized image quality metric, to optimize the model.

The training is independent of either geometry prior or ground truth of a virtual view. Besides, the whole framework is end-to-end with no extra post-processing. The proposed method have been evaluated on multiple datasets with both subjective and objective evaluations. The main contributions of this paper are addressed as follows:

• A novel virtual view synthesis framework in an unsupervised way is proposed, which deserves no extra geometry information or pristine virtual view.

• A spatiotemporal generative adversarial network is embedded into the traditional depth-image-based rendering. The proposed framework implicitly combines depth estimation, differential 3D warping, hole filling and spatiotemporal discriminator in an end-to-end way.

• To make the novel view hallucinated, new perceptual constraints are proposed, including a pre-trained blind synthesized image quality metric as well as a no-reference structure similarity metric.

The rest of this paper is organized as follows. Related works are reviewed in Section 2. Section 3 introduces the framework. Sections 4 Unsupervised virtual view synthesis network, 5 Perceptual constraints without pristine image present the details of proposed method. In Section 6, the proposed method is evaluated with self investigation and comparison. Finally, a conclusion is presented in Section 7.