KVNet: An iterative 3D keypoints voting network for real-time 6-DoF object pose estimation

Abstract

Accurate and efficient object pose estimation holds the indispensable part of virtual/augmented reality (VR/AR) and many other applications. While previous works focus on directly regressing 6D pose from RGB and depth image and thus suffer from the non-linearity of rotation space, we propose an iterative 3D keypoints voting network, named as KVNet. Specifically, our method decouples the pose into separate translation and rotation branch, both estimated by Hough voting scheme. By treating the uncertainty of keypoints’ vote as the Lipschitz continuous function of seed points’ fused embedding feature, our method is able to adaptively select the optimal keypoints vote. In this way, we argue that KVNet bridges the gap between the non-linear rotation space and linear Euclidean space, which introduces inductive bias for our network to learn the intrinsic pattern and infer 6D pose from RGB and depth images. Furthermore, our model will refine the initial keypoints localization with iterative fashion. Experiments show that across three challenging benchmark datasets (LineMOD, YCB-Video and Occlusion LineMOD), our method exhibits excellent performance.

Introduction

This paper focuses on the research of 6-DoF (degrees of freedom) pose estimation, i.e. the 3D position and orientation of objects in a canonical frame. Many real-world applications inevitably require accurate, efficient and robust pose estimation of specific objects in 3D space, such as virtual and augmented reality (VR/AR) [1], [2], [3], autonomous driving [4], robotic manipulation [5], [6] and so on. However, this problem has been proven quite challenging due to sensor noise, occlusion, variations of shape and texture and demand for low latency.

Traditionally, for most existing methods [7], [8], [9], [10], template matching is an attractive solution achieved by establishing the correspondences between the object point clouds (or images) and the object 3D mesh model. Statistical properties encoded in these handcrafted features cannot show enough robustness, especially when the point clouds or pixels data exhibit significant changes in occlusion, illumination, noise corruption and so on. Recent work has therefore focused on the data-driven method feeding RGB or RGB-D images to a deep net architecture for 6-DoF pose estimation. Given the context of a single RGB image, PVNet [11] and other RGB-based methods [12], [13], [14], [15] recover the pose of each object instance with a two-stage pipeline covering the 2D keypoints localization in the image coordinate and one standard or improved perspective-n-point (PnP) algorithm. However, it’s not trivial to accurately predict the 2D keypoint position in the low resolution, poor illumination or overlapped inputs, and thus incurs a noisy set of 2D-to-3D correspondences. And the projection of 3D space to 2D image plane may partly lose the object’s intrinsic geometric pattern essential for 6D pose estimation. To overcome above limitations, the advent of cheap RGB-D cameras enables the research in 3D space to infer the 6D pose of objects. Assisted with an additional depth information, a popular approach is to directly regress the 3D rotation and translation using the deep learning network. But in this case, these holistic methods [16], [17], [18] lag behind in estimating rotation due to the non-linearity of rotation space. Alternatively, as robust and discriminative point sets, 3D keypoints can be treated as a compact and abstract representation and therefore can be used to establish 3D-3D correspondences for 6D pose estimation.

In this work, we propose a simple but significantly efficient 3D keypoint voting approach for real-time 6D pose estimation of known and rigid objects, named as KVNet, with an iterative fashion. We decouple the 6D pose $\mathit{SE}(3)$ into translation $t\in R^3$ and rotation $R\in \mathit{SO}(3)$ predicted by regressing 3D keypoints in separate network branch. Our method chooses the DenseFusion [18] to be the basic feature embedding learner encoding the texture appearance and geometric information at per-pixel level. The deep network optimizes a set of criteria to learn the pointwise 3D offset and vote for 3D keypoints which bridge the gap between linear Euclidean and non-linear rotation space.

One issue not yet considered in other works is the selection of the optimal keypoint vote. We argue that the confidence of the keypoints voting from point sets in the object surface, defined as the uncertainty of the voting in our method, varies with the spatial distribution of point sets. And the uncertainty of the voting can be treated as the Lipschitz continuous function of the pointwise fused embedding since the uncertainty distributes across [0, 1]. The continuous function can be approximated by a multi-layer perceptron (MLP) benefited from its powerful mapping ability. By integrating the uncertainty into the criteria function, the deep network therefore adaptively achieves the optimal positions of keypoints with the highest confidence of the uncertainty in 3D space. To make progress towards our goals, we also take into account the post-processing that attempts to refine keypoints positions iteratively. Note that the refinement module is continuously differentiable and thus can be trained jointly with our main architecture.

Experiments conducted on three widely-used benchmark datasets (LineMOD [19], YCB-Video [20] and Occlusion LineMOD [46] datasets) show that our method exhibits the excellent performance while achieving the real-time inference speed compared with other 6D pose estimation techniques based on RGB and RGB-D images.

In summary, the main contributions of this paper are threefold:

• We propose a novel uncertainty-driven keypoints voting scheme to recover 6D pose from RGB and depth images.

• We introduce a keypoints refinement procedure for establishing the most precise 3D-3D correspondences but only with mild cost of inference speed.

• We contribute in-depth experimental and mathematical analysis for demonstrating the excellent performance of our method.

Since keypoints detection is an open and representative problem in 3D data processing and computing, we expect that our idea could be transferred to many potential domains, such as 3D object detection, 3D reconstruction and so on.