Inpainting of Depth Images Using Deep Neural Networks for Real-Time Applications

Abstract

Depth sensors enjoy increased popularity throughout many application domains, such as robotics (SLAM) and telepresence. However, independent of technology, the depth images inevitably suffer from defects such as holes (invalid areas) and noise. In recent years, deep learning-based color image inpainting algorithms have become very powerful. Therefore, with this work, we propose to adopt existing deep learning models to reconstruct missing areas in depth images, with the possibility of real-time applications in mind. After empirical tests with various models, we chose two promising ones to build upon: a U-Net architecture with partial convolution layers that conditions the output solely on valid pixels, and a GAN architecture that takes advantage of a patch-based discriminator. For comparison, we took a standard U-Net and LaMa. All models were trained on the publically available NYUV2 dataset, which we augmented with synthetically generated noise/holes.

Our quantitative and qualitative evaluations with two public and an own dataset show that LaMa most often produced the best results, however, is also significantly slower than the others and the only one not being real-time capable. The GAN and partial convolution-based models also produced reasonably good results. Which one was superior varied from case to case but, generally, the former performed better with small-sized holes and the latter with bigger ones. The standard U-Net model that we used as a baseline was the worst and most blurry.