Development of Real Environment Datasets Creation Method for Deep Learning to Improve Quality of Depth Image

Abstract

High quality depth images are required for stable and accurate camera tracking and 3D modeling. The method to improve the depth image quality using deep learning requires large and accurate datasets in advance. Datasets created by Middlebury Datasets which are typical depth image datasets do not always represent the feature of noise caused by depth cameras, and the number of datasets is not enough. In this study, the method for creating datasets for deep learning was developed and evaluated. The proposed method can improve the accuracy of the distance for each pixel by aligning the positions of pixels that capture the same part of the real world with multiple frames. In addition to super-resolution and denoising, the images are preprocessed such as patch division and data augmentation to eliminate the holes in correct depth images. By using this method, large number of real environment datasets can be automatically created. Two neural networks using Middlebury datasets and the datasets generated by the proposed method were trained respectively, and produced high quality depth images with them. In order to compare the Middlebury Datasets and the proposed method, we visually evaluated the hole filling and the smoothness of edges and surfaces of objects from the results. The result showed the network using the datasets created by the proposed method can remove noise rather than that using Middlebury Datasets since they include noise features caused by the performance limits of depth cameras.