Ground and aerial meta-data integration for localization and reconstruction: A review

Highlights

• Methods of integrating ground and aerial meta-data for localization and reconstruction are reviewed.

• Localization methods are reviewed in terms of image based methods and structure based methods.

• Reconstruction methods are reviewed in terms of image based methods and laser based methods.

Abstract

Localization and reconstruction are two highly related research areas. Both of them have developed rapidly in recent years. Apparently, with the help of ground and aerial meta-data integration, the performance of both localization and reconstruction can go a step further. For localization, aerial meta-data provides a global reference, by which the ground query can achieve a cumulative error free absolute localization. As for reconstruction, a complete and detailed model can be reconstructed by integrating ground and aerial meta-data. Though with many advantages, the integration itself is non-trivial. It is difficult to obtain ground-to-aerial correspondences neither in 2D manner nor in 3D manner. That is because: (1) The differences between the ground and aerial images in viewpoint, scale, illumination, etc. are notable; (2) The discrepancies between the ground and aerial point clouds in terms of point density, accuracy, noise level, etc. are very large. To deal with these problems, lots of methods have been proposed recently. In this paper, the methods of integrating ground and aerial meta-data for localization and reconstruction are reviewed respectively. Though many intermediate results with high quality have been achieved, we hope that inspired by the reviewed methods in this paper, more thorough methods and impressive results would emerge.

Introduction

Localization and reconstruction are two highly related research areas. Localization is to determine the location in the scene of the data acquisition equipment, e.g. camera or laser scanner; while reconstruction is to recover the 3D information of the scene using the acquired meta-data, e.g. images or light detection and ranging (LIDAR) data.

Currently, approaches that tackle the localization problem are roughly divided into two categories: image based methods and structure based methods. Image based methods cast the localization problem as an image retrieval task and represent a scene as a database of geo-tagged images [62], [83], [98]. The location of the query image is usually approximated to the geo-tag of the most relevant retrieved image. This process is also termed image geo-localization. In contrast, structure based methods cast the localization problem as a camera resection task [84], [94], [116]. By matching features, the pose (location and orientation) of the data acquisition equipment (camera or laser scanner) is recovered. The estimated pose is relatively accurate and is usually used for robot localization or augmented reality (AR).

According to the involved meta-data, scene reconstruction approaches broadly fall into two types: image based methods and laser based methods. The pipeline of image based methods contain: (1)  structure-from-motion (SfM) [11], [16], [17], [18], [86] to calibrate camera poses, (2) multi-view stereo (MVS) [87], [92] to get dense point cloud and (3) surface reconstruction [71], [101] to obtain surface mesh. In order to achieve more delicate scene reconstructions, sometimes LIDAR data is involved [66], [103], [117]. Compared with image based ones, though the laser based methods are more accurate and less dependent on the circumstances, they are costlier and less flexible.

Though both localization and reconstruction methods have achieved impressive improvements in recent years, most methods only consider single-source (ground or aerial) meta-data. However, there are several advantages of integrating ground and aerial meta-data for both localization and reconstruction tasks. Here, two examples are given. (1) For geo-localizing a ground image, aerial meta-data is more appropriate to be the localization reference. That is because compared with the non-uniformly sampled ground met-data, aerial meta-data provides more complete coverage of the scene. (2) For reconstructing an architectural scene, ground and aerial meta-data is complementary as they provide close-range observation and large-scale coverage of the scene respectively. By integrating them, both detail and completeness of the scene are guaranteed. Note that in this paper, the ground meta-data considered includes images (e.g. street-view images or images downloaded from the Internet) and LIDAR data. While the aerial meta-data also includes images (e.g. satellite images or oblique bird’s eye view images) and LIDAR data.

Though with many advantages, the integration of ground and aerial meta-data is non-trivial. There are several difficulties need to be considered. For finding 2D point matches (image matching), the differences between the ground and aerial images in viewpoint, scale, illumination, etc. are notable. As a result, the traditional 2D local features, e.g. scale invariant feature transform (SIFT) [54], speeded-up robust features (SURF) [4] and affine SIFT (ASIFT) [63], are inadequate to handle this image matching problem. In addition, the local geometrical constraint based robust feature matching modules [55], [56] would not achieve desirable results either, as they do not focus on handling the differences between the ground and aerial images, but on removing large proportions of outliers from the putative correspondence set. For obtaining 3D point correspondences (model registration), the discrepancies between the ground and aerial LIDAR data or the point clouds generated from ground and aerial images in terms of point density, accuracy, noise level, etc. are very large as well. Thus, neither the traditional 3D local features, e.g. spin images (SI) [33], fast point feature histograms (FPFH) [82] and rotational projection statistics (RoPS) [28], nor the commonly used iterative closest point (ICP) algorithm, e.g. point-to-point ICP [6] and point-to-plane ICP [12], can be used to deal with the model registration problem. Apart from these low-level feature based methods, points correspondence could also be treated as a graph matching problem [112], [113]. Though effective in many cases, these graph matching based methods are not appropriate for finding correspondences between ground and aerial meta-data with notable differences.

Recently, due to practical needs in areas such as autopilot, outdoor AR, and so on, the integration of ground and aerial meta-data for localization and reconstruction has been paid more and more attention. Lots of methods have been proposed to deal with the localization or reconstruction problem and achieved exciting results, which are reviewed respectively in this paper. The rest of this paper is organized as follows. Section 2 reviews the methods of integrating ground and aerial meta-data for localization, and Section 3 reviews the methods of integrating ground and aerial meta-data for reconstruction. Section 4 discusses the current datasets, evaluation metrics and future trends in integrating ground and aerial meta-data for both localization and reconstruction. Finally, Section 5 presents some concluding remarks.