Semi-supervised Learning to Remove Fences from a Single Image

Abstract

When taking photos in the outdoors, e.g., playgrounds, gardens and zoos, fences are often inevitable to interfere with the perception of the background scenes. Thus it is an interesting topic on removing fences from a single image, which is dubbed image de-fencing. Generally, image de-fencing can be tackled within the two-stage framework, i.e., first detecting fences and then restoring background image. In existing methods, detecting fences mask cannot guarantee accuracy due to the diverse patterns of fences. Meanwhile, state-of-the-art supervised learning-based restoration methods usually fail in well handling inaccurate fences mask, yielding artifacts in restored background image. In this paper, we propose a semi-supervised framework for removing fences from a single image, where a simple yet effective recurrent network is proposed for supervised learning to detect fences mask and unsupervised learning is employed for robust restoration of background image. Specifically, we propose a recurrent network for fences mask detection, where convolutional LSTM is adopted for progressive detection. Instead of strict fences mask detection using standard $\ell _1$ loss, we adopt an asymmetric loss to make detected fences mask be more inclusive of true fences. Then motivated by the success of deep image prior (DIP) for several image restoration tasks, we propose to employ unsupervised learning of three DIP networks for modeling background image, fences layer and fences mask, respectively. Moreover, we propose to adopt Laplacian smoothness loss function for refining the detected fences mask, making the restoration of background image be more robust to detection errors of fences mask. Experimental results validate the effectiveness of our semi-supervised image de-fencing approach.

The first author is a second year master student.