Self-supervised multi-scale pyramid fusion networks for realistic bokeh effect rendering

Highlights

• A self-supervised multi-scale pyramid fusion network is proposed for realistic bokeh effect rendering.

• ’Circle of confusion’ is taken as task-specific information for network training.

• The proposed network achieves new state-of-the-art performance on bokeh benchmark.

Abstract

Images with visual pleasing bokeh effect are often unattainable for mobile cameras with compact optics and tiny sensors. To balance the aesthetic requirements on photo quality and expensive high-end SLR cameras, synthetic bokeh effect rendering has emerged as an attractive machine learning topic for engineering applications on imaging systems. However, most of bokeh rendering models either heavily relied on prior knowledge such as scene depth or were topic-irrelevant data-driven networks without task-specific knowledge, which restricted models’ training efficiency and testing accuracy. Since bokeh is closely related to a phenomenon called ”circle of confusion”, therefore, in this paper, following the principle of bokeh generation, a novel self-supervised multi-scale pyramid fusion network has been proposed for bokeh rendering. During the pyramid fusion process, structure consistencies are employed to emphasize the importance of respective bokeh components. Task-specific knowledge which mimics the ”circle of confusion” phenomenon through disk blur convolutions is utilized as self-supervised information for network training. The proposed network has been evaluated and compared with several state-of-the-art methods on a public large-scale bokeh dataset- the ”EBB!” Dataset. The experiment performance demonstrates that the proposed network has much better processing efficiency and can achieve better realistic bokeh effect with much less parameters size and running time. Related source codes and pre-trained models of the proposed model will be available soon on https://github.com/zfw-cv/MPFNet.

Introduction

“Bokeh” is Japanese in origin and refers to a blurry quality. In photography, it is a very recognizable technique, which can lead pleasing visual aesthetic photos, as shown in Fig. 1.

In practice, images with visual pleasing bokeh effect are often produced through professional DLSR camera with large aperture and long focal length. However, they are often unattainable for mobile cameras with compact optics and tiny sensors. To balance the aesthetic requirements on photo quality and expensive high-end SLR cameras, bokeh effect has to be simulated computationally. Therefore, synthetic bokeh effect rendering has emerged as an attractive machine learning technology [1], [2] for engineering applications on imaging systems.

During the past years, many methods on synthetic bokeh effect rendering relied heavily on prior knowledge such as scene depth. Among these depth-based methods, some methods adopted to estimate scene depth through utilizing hardwares like the dual-pixel autofocus system [3] on Google Pixel devices, the dual-lens on iPhone7+ and the Time-of-Flight (TOF) lens on Huawei P30+ smartphones. However, since these specialized hardwares are expensive, they are often not supported on low-end commercial systems. Moreover, for images already captured using monocular cameras, the accurate depth information are not available either. Therefore, many other methods proposed to employ pre-trained models such as MegaDepth [4] to estimate the depth.

To some degree, incorporating prior knowledge to simulate realistic bokeh blur has potentials to improve visual effect of the final generated image. However, every thing has the pros and the cons. The typical limitations of prior-based methods are: (1) the depth sensor related hardwares are not always available on mobile devices; (2) pre-processing prior information by software is generally time-consuming; (3) when the prior information estimated does not work, unexpected out-of-focus blurriness conversely deteriorates the quality of ultimate synthetic image.

Witnessed the impressive success on image-to-image translation tasks, in recent years, many researchers started to consider bokeh simulation as a subtask of image translation [5], [6], [7]. Therefore, routines based end-to-end multi-scale encoder–decoder architecture are commonly adopted as camera-independent solutions. The nonlinear mappings between the low- and high-aperture photos captured with high-end DSLR camera are directly modeled in data-driven way. However, though much progress has been achieved, the majority of these models are topic-irrelevant networks. In other words, it means in these cases, task-specific knowledges like the intrinsic mechanism of bokeh generation are often neglected without fully utilization for more effective solutions.

According to the principle of optical imaging [8], [9], bokeh generation is closely related to a phenomenon called “circle of confusion (CoC)”. The “CoC” approximately brings different sizes of disk blurs on out-of-focus areas. Therefore, following the task-specific knowledge, we propose a novel self-supervised multi-scale pyramid fusion network for bokeh rendering. In the proposed network, blurred images after disk convolution kernels of different radius provide self-supervised informations for bokeh component learning. The final bokeh image is a weighted combination of the learned factorized bokeh components. Therefore, different from existing heavy prior-dependent algorithms, the proposed network does not have to preprocess time-wasting priors during training and testing in practice. At the same time, in contrary to some topic-irrelevant end-to-end networks, the proposed network employs task-specific knowledge as training guidance. With more clear training purpose, it can effectively boost network training’s efficiency and accuracy.

The proposed network has been evaluated and compared with several state-of-the-art methods on a public large-scale bokeh dataset- the “EBB!” Dataset [5]. The experiment performance demonstrates that the proposed network has much better processing efficiency and can achieve better realistic bokeh effect with much less parameters size and running time.

The contributions of this paper are summarized as followings:

• An effective multi-scale pyramid fusion network is proposed for realistic bokeh effect rendering. The proposed network can achieve new state-of-the-art performance on a large-scale bokeh benchmark dataset with relatively small parameter size and realtime processing speed.

• Task-specific knowledge which mimics the “circle of confusion” phenomenon through disk blur convolutions is utilized as self-supervised information for network training. Structure consistencies are employed to emphasize the importance of respective bokeh components. With more clear training purpose, it can effectively boost network training’s efficiency and accuracy.

In the following sections, related work will be summarized in Section 2. Details on the proposed network will be described in Section 3. Then experiment results and analysis are demonstrated in Section 4. Finally, conclusions will be given in Section 5.