Geometrical and statistical incremental semantic modeling on mobile devices

Highlights

• A technique that performs semantic modeling and tracking of primitives on sparse point clouds on desktop and mobile devices.

• The method allows the modeling and tracking of planes, spheres, and cylinders.

• A dataset with sparse point clouds of primitives.

• A guideline to evaluate computer vision techniques on mobile devices.

Abstract

Improvements on mobile devices allowed tracking applications to be executed on such platforms. However, there still remain several challenges in the field of mobile tracking, such as the extraction of high-level semantic information from point clouds. This task is more challenging when using monocular visual SLAM systems that output noisy sparse data. In this paper, we propose a primitive modeling method using both geometric and statistical analyses for sparse point clouds that can be executed on mobile devices. The main idea is to use the incremental mapping process of SLAM systems for analyzing the geometric relationship between the point cloud and the estimated shapes over time and selecting only reliably-modeled shapes. Besides that, a statistical evaluation that assesses if the modeling is random is incorporated to filter wrongly-detected primitives in unstable estimations. Our evaluation indicates that the proposed method was able to improve both precision and consistency of correct detection when compared with existing techniques. The mobile version execution is 8.5 to 9.9 times slower in comparison with the desktop implementation. However, it uses up to 30.5% of CPU load, which allows it to run on a separate thread, in parallel with the visual SLAM technique. Additional evaluations show that CPU load, energy consumption and RAM memory usage were not a concern when running our method on mobile devices.

Introduction

Several applications on computer vision and augmented reality require camera pose tracking. However, determining the device position in relation to the real environment can demand a lot of computational power and memory depending on the approach and the required information. Along with all the improvement on processing power and memory on mobile devices itself, several tracking techniques that are capable to run on such devices were released in the last couple of years. There are examples in the academy and in the industry. The most distinguished ones are ARCore¹ and ARKit,² by Google and Apple, respectively.

Although several techniques have been released, there are still many remaining research issues in the field of tracking. One is to extract and track high-level semantic information from the environment. Different types of semantics can be collected, from geometric primitives of objects to the model of a piece of furniture. This process is referred to as semantic modeling. Shape parameters of geometric primitives and the relationship among them are valuable knowledge to be estimated especially in man-made environments such as a house and a factory. This data can also be gathered in different ways: from the input image, the scene map or a combination of both.

There are several benefits of detecting primitives from the scene map. For instance, objects are usually over-represented when defined using a point cloud because it is not necessary to have so many points to describe them. Therefore, an implicit representation can replace redundant points, which is particularly helpful when targeting devices with memory restrictions, such as robots or unmanned aerial vehicles (UAV). Additionally, a tracking system can use these primitives to denoise the reconstructed map or constrain its optimization [1], which can reduce tracking errors. Furthermore, it can be used to provide haptic feedback on augmented reality applications [2].

Mobile devices are reducing the gap to desktop computers in terms of processing power and memory space [3]. This improvement allowed the development of more complex algorithms for mobile devices, including tracking techniques, which are one of the foundations of augmented reality. This can be linked to the improvement and popularization of augmented reality solutions for such devices.

Some of these recent advancements in tracking techniques involve removing the necessity of any external marker, improving the execution time to achieve real-time performance and having a more stable tracking that is not harmed by jitter or drift. For instance, the improvement of the device’s sensors allowed the development of visual-inertial trackers. However, there is not much progress regarding the extraction of different kinds of semantics from the 3D map of the scene.

Considering all the benefits mentioned above, the main goal of this study is to model and track primitives aiming to obtain semantics from sparse point clouds. In order to do that, we present Geometric and Statistical Incremental Semantic Tracking, or simply GS-IST, which is a method for incrementally modeling and tracking planes, spheres, and cylinders on sparse point clouds. In summary, this method uses the generating process of point clouds on SLAM effectively and relies on geometric and statistical analyses to filter unreliable shapes. Additionally, this method was ported and evaluated on mobile devices, showing that it was feasible to extract and track basic primitives on such platforms. This paper is an extended version of the work published in Roberto et al. 2018 [4] and its main contributions are summarized as follows:

• A technique that uses geometric and statistical evaluation to incrementally perform semantic modeling and tracking of primitives on sparse point clouds (Section 3);

• Evaluations of the proposed method in comparison with existing techniques, showing that it improves semantic modeling precision. This evaluation also includes a dataset with sparse point clouds of primitives and a metric precision evaluation that was not available in the preliminary work (Section 4);

• The port and evaluation of the proposed approach to mobile devices and a guideline to evaluate computer vision techniques on such platform, which is also an enhancement from our previous paper (Section 5).