RGB-Fusion: Monocular 3D reconstruction with learned depth prediction

Highlights

• Proposed a complete monocular 3D reconstruction pipeline with depth prediction.

• Integrated the PnP algorithm and ICP algorithm into the pose estimation module.

• Proposed a depth map refinement strategy based on uncertainty.

• Achieved a good reconstruction effect.

Abstract

Generating large-scale and high-quality 3D scene reconstruction from monocular images is an essential technical foundation in augmented reality and robotics. However, the apparent shortcomings (e.g., scale ambiguity, dense depth estimation in texture-less areas) make applying monocular 3D reconstruction to real-world practice challenging. In this work, we combine the advantage of deep learning and multi-view geometry to propose RGB-Fusion, which effectively solves the inherent limitations of traditional monocular reconstruction. To eliminate the confinements of tracking accuracy imposed by the prediction deficiency of neural networks, we propose integrating the PnP (Perspective-n-Point) algorithm into the tracking module. We employ 3D ICP (Iterative Closest Point) matching and 2D feature matching to construct separate error terms and jointly optimize them, reducing the dependence on the accuracy of depth prediction and improving pose estimation accuracy. The approximate pose predicted by the neural network is employed as the initial optimization value to avoid the trapping of local minimums. We formulate a depth map refinement strategy based on the uncertainty of the depth value, which can naturally lead to a refined depth map. Through our method, low-uncertainty elements can significantly update the current depth value while avoiding high-uncertainty elements from adversely affecting depth estimation accuracy. Numerical qualitative and quantitative evaluation results of tracking, depth prediction, and 3D reconstruction show that RGB-Fusion exceeds most monocular 3D reconstruction systems.

Introduction

Many researchers have paid great attention to the detailed 3D dense reconstruction of indoor scenes in recent years. Simultaneous Localization and Mapping (SLAM) aims to solve the navigation and map construction problems of unknown locations. It has been proven to be a practical dense 3D reconstruction approach and is widely employed in augmented reality [1]. Since the high popularity of RGB cameras on consumer devices, some researchers have focused on the monocular SLAM methods [2], [3]. These approaches performed feature matching on consecutive adjacent frames, employed stereo matching [4] to recover image depth information, and finally reconstructed the scene. However, the uncertainty of the absolute scale limits the application prospects of these methods. Even if all steps are carried out successfully, the ambiguous absolute scale still results in an unacceptable reconstruction. A solution [5] to scale vagueness was to integrate a monocular camera with an inertial measurement unit (IMU) to estimate the absolute scale of the scene through the measurement data of the IMU. However, when the acceleration is negligible, the low-performance IMU configured for consumer devices cannot provide data support for calculating the scene scale. Some scholars have proposed employ depth cameras [6], [7] for 3D reconstruction. However, most depth cameras have an unsatisfactory detection range and are sensitive to lighting conditions, making reconstruction less precise in uneven lighting environments.

Meanwhile, deep learning caused quite a stir in the area of 3D reconstruction. After training, the neural network can realize 3D object reconstruction from a single [8], [9], stereo [10], [11], or collection of images [12], [13]. Furthermore, deep learning obtains the absolute scale of the scene from images without relying on other auxiliary information. Due to the relatively complex spatial relationship, the accuracy of the end-to-end network is difficult to surpass the geometry-based 3D reconstruction methods.

After analyzing the characteristics of deep learning and geometry-based approaches, some scholars have suggested integrating them to yield satisfactory results. As shown in [14], the end-to-end 3D reconstruction system based on deep learning can accurately reconstruct the internal points of the scene, but it is easy to face the situation of blurred edges. In contrast, pipelines based on multi-view geometry have high accuracy at the edges of scenes with rich corners but fail to rebuild inside scenes with insufficient texture. Integrating deep learning with multi-view geometry is an exciting idea, but how to combine it efficiently is still worth exploring. Some methods [12], [15] have suggested inputting the structural model obtained from the geometry-based 3D reconstruction pipeline into the neural network, and then deep learning was utilized to perform model optimization. One limitation of these methods is that as long as the system processes new geometric information, the geometric model must be re-input to the network to perform complex calculations, which consumes a lot of calculation time.

This paper proposes RGB-Fusion, a new monocular surface reconstruction system that can support large-scale, high-quality reconstruction. Fig. 1 shows an example of our reconstruction results in the fr3/long_office_household sequence of the TUM RGB-D dataset [16]. RGB-Fusion leveraged the state-of-the-art algorithm DeepV2D [17] to predict the depth and completed the 3D reconstruction through the dense SLAM methods. We validated our approach by comparing three public datasets against many excellent monocular 3D reconstruction studies, focusing on tracking, depth prediction, and reconstruction accuracy. The results demonstrate that RGB-Fusion could achieve an average of 0.105 m surface reconstruction accuracy, an average of 31.255% depth prediction accuracy, and a 0.199 m absolute trajectory (ATE) root-mean-square error (RMSE) [16]. In general, our proposed pipeline outperforms most existing monocular 3D reconstruction systems by more than 10%.

The key contributions are summarized as follows:

• We integrate dense SLAM and deep learning into the new monocular 3D reconstruction pipeline, which can perform high-quality 3D reconstruction of large-scale scenes using only RGB images.

• We jointly optimize the ICP algorithm [18] and the PnP algorithm [19] to calculate the camera pose estimation and leverage the preliminary pose predicted by the depth module to initialize the least-square method to avoid trapping local minimums.

• We propose a depth refinement strategy that used the uncertainty of the depth map to improve the accuracy of depth prediction.

This paper is organized as follows: We introduce the development of related works in Section 2. Section 3 elaborates the techniques and critical strategies of our proposed monocular 3D reconstruction framework. The experiments and results on public datasets are presented in Section 4. We finally introduce our conclusions and the focus of future research in Section 5.