Robust self-supervised monocular visual odometry based on prediction-update pose estimation network

Abstract

Visual odometry aims at estimating the camera pose from video sequence, which is an important part of visual Simultaneous Localization and Mapping (SLAM). In this paper, we propose a novel prediction-update pose estimation network, PU-PoseNet, for self-supervised monocular visual odometry. It allows the network to use the effective information of the previous frame in estimating the current pose. The long-time pose consistency constraint-based motion weighted photometric loss is designed to make the network to pay more attention to the pixels of stationary objects and enhance the time consistency of estimation results. The depth image-based occlusion detection, the depth smoothness loss and auto-Masking are used to construct the depth consistency constraint loss term to reduce the influences of interferences such as occlusion. To further improve the robustness and the accuracy of the proposed method, both the depth consistency constraint and the variational auto-encoder are used for network training. For frame missing cases, a novel frame missing training strategy is used to make our method adapt frame missing cases. Extensive experiments on the KITTI dataset have validated the effectiveness of our proposed method.

Introduction

The autonomous driving vehicle has been one of the research hotspots in the field of artificial intelligence. It needs to determine their position during navigation. The traditional localization methods use Global Positioning System (GPS) or Real-Time Kinematic (RTK) positioning system. Recently, with the rapid development of computer vision technology, more and more researchers began to pay attention to the localization methods based on Visual odometry (VO), which estimates the camera’s position and pose from the video frames. As an important part of visual Simultaneous Localization and Mapping (SLAM) system, VO has been widely used in robot navigation, autonomous driving, augmented reality, etc. Existing VO methods can be divided into two categories: the traditional geometry-based methods and the learning-based methods. The traditional geometry-based VO methods first extract features from adjacent frames, and then match them to estimate the relative pose based on geometric relationships (Campos et al., 2021, Mur-Artal et al., 2015, Mur-Artal and Tardos, 2017). This kind of method has achieved satisfactory results in some practical applications, however it still suffers from the poor balance between computational performance and robustness. In recent years, the deep learning technique has developed rapidly and many methods using frontier deep learning methods, like federated learning, transfer learning, are proposed (Zhang et al., 2021a, Zhang and Li, 2022). With the successful application of deep learning technique in computer vision fields such as object detection, keypoint detection and feature extraction (DeTone et al., 2018, He et al., 2017, Liu et al., 2019, Ren et al., 2017, Sarlin et al., 2020), the learning-based VO methods have attracted more and more researchers’ attention. Compared with the traditional VO methods, the deep learning-based VO methods can learn the prior knowledge from a large number of images, and their performances do not depend on the accuracy of feature detection and image matching. So, the deep learning-based monocular VO methods are worthy of further study.

The learning-based VO methods can be divided into the supervised learning-based methods and the self-supervised learning-based methods. The supervised learning-based methods use the real poses as supervised signal to obtain high precision pose estimation results, but the acquisition of real poses requires additional expensive equipment, such as lidar or GPS systems. The self-supervised learning-based VO methods do not require the acquisition of real poses. So, they are more flexible than the supervised learning-based methods. In recent years, many self-supervised learning-based VO methods have been proposed. These methods mainly use the geometric relationship between depth and camera pose to train the networks. However, the performance of these methods is still somewhat poorer than supervised learning-based methods.

Most existing learning-based VO methods have not considered the association between consecutive frames. So the effective information from adjacent frames has not been fully used. In addition, the moving objects make it difficult to describe all motions in the scene with a single Euclidean transformation. Usually, the motions of moving objects are not consistent with the camera trajectories. So, it is unreasonable to treat all pixels equally in network training. Furthermore, most existing methods will fall when facing frame missing cases. To solve the above problems, a robust self-supervised monocular VO method based on prediction-update pose estimation network is proposed.

To better use the related information of adjacent frames and make the method adapt to frame missing cases, we introduce Kalman filter’s prediction-update idea into the design of the pose estimation network. The contributions of this paper can be summarized as follows.

(1) An end-to-end self-supervised monocular VO framework is proposed. We design a novel pose estimation network based on prediction-update mechanism, which can use the extracted information from previous moments to guide the estimation of the pose of current moment and adapt frame missing cases by applying a novel training strategy.

(2) We propose a novel loss function for network training to improve the robustness and accuracy. The long-time pose consistency constraint-based motion weighted photometric loss term is designed to reduce the impact on moving objects and enhance the time consistency of estimation results. The occlusion detection, depth smoothness and auto-Masking are used to construct the depth consistency constraint loss term to reduce the influences of interferences such as occlusion.

(3) Extensive experimental results on the KITTI dataset show that our proposed method achieves state-of-the-art accuracy, and the ablation study validates that the restructured image-based motion weighting, the depth consistency and the depth image-based occlusion detection can all bring improvement for our proposed VO method.