Divide and conquer: A hierarchical approach to large-scale structure-from-motion

Highlights

• A divide and conquer based large-scale 3D reconstruction pipeline is proposed.

• The method partitions a large dataset into smaller well constrained clusters.

• Each cluster is independently reconstructed using global SfM techniques.

• Registration method using epipolar geometry is proposed to merge reconstructions.

Abstract

In this paper we present a novel pipeline for large-scale SfM. We first organise the images into a hierarchical tree built using agglomerative clustering. The SfM problem is then solved by reconstructing smaller image sets and merging them into a common frame of reference as we move up the tree in a bottom-up fashion. Such an approach drastically reduces the computational load for matching image pairs without sacrificing accuracy. It also makes the resulting sequence of bundle adjustment problems well-conditioned at all stages of reconstruction. We use motion averaging followed by global bundle adjustment for reconstruction of each individual cluster. Our 3D registration or alignment of partial reconstructions based on epipolar relationships is both robust and reliable and works well even when the available camera-point relationships are poorly conditioned. The overall result is a robust, accurate and efficient pipeline for large-scale SfM. We present extensive results that demonstrate these attributes of our pipeline on a number of large-scale, real-world datasets and compare with the state-of-the-art.

Introduction

In structure-from-motion (SfM) we recover both 3D geometry of points and camera motion parameters from correspondences across images. Despite the impressive performance of contemporary pipelines (Snavely, Seitz, Szeliski, 2006, Wu, 2013), there are some significant issues in solving large-scale SfM as both data size and problem complexity grow substantially. While the SfM literature is voluminous, we confine ourselves here to a discussion of the challenges in solving really large-scale problems (more than 1000 images). In this paper we present an efficient and robust pipeline for solving large-scale SfM problems in a hierarchical manner. Our method combines the benefits of both local and global approaches to SfM which form two prototypical extremes for SfM methods. We demonstrate that our method is more robust with respect to failures when compared to the state-of-the-art and offers at least one order of magnitude speed-up without sacrificing accuracy.

Many of the successful large-scale approaches treat SfM as one of minimising the reprojection error using bundle adjustment (Triggs et al., 2000) and its efficient modern variants (Agarwal, Snavely, Seitz, Szeliski, 2010, Havlena, Torii, Pajdla, 2010, Snavely, Snavely, Seitz, Szeliski, 2006, Snavely, Seitz, Szeliski, 2008, Wu, Agarwal, Curless, Seitz, 2011). As large-scale optimisations do not easily converge, these methods use a variety of heuristics and approximations to achieve a valid solution. In Agarwal et al. (2009), a high level of parallelism is utilised in the pre-processing stages, following which the SfM optimisation is based on a reduced skeletal set. This approach is further refined in Agarwal et al. (2010) by more efficient optimisation. Parallelism is also exploited in Wu et al. (2011). A similar approach of skeletal graph construction out of ‘iconic’ images is presented in Frahm et al. (2010). In sequential approaches such as Snavely et al. (2006) and Wu (2013), the difficulty of large-scale SfM is handled by incrementally adding one image at a time to a solution based on bundle adjustment seeded with a few images. To reduce the matching time, Wu (2013) (henceforth VSFM) proposes preemptive matching to reduce the number of pairs to be matched. Wu (2013) also introduces the concept of delay in optimisation where all cameras and 3D points are optimised only after a certain number of new cameras are incorporated in iterative bundle adjustment. This heuristic introduces some global view in an otherwise sequential method giving it additional robustness with respect to failures when compared to purely incremental methods (Snavely, 2008).

Despite their success, current bundle adjustment approaches suffer from some significant drawbacks. The computational cost for such approaches is very large and these methods have to carefully avoid getting trapped in poor local minima. While sequential methods reduce computational time, being unable to recover from earlier decisions, methods such as Wu (2013) fail to reconstruct on occasion resulting in the final reconstruction being broken into multiple parts. They are also susceptible to the problem of drift and accumulation of errors (Moulon, Monasse, Marlet, 2013, Wu, 2013) and are difficult to parallelise because of their inherent sequential nature.

An alternate solution is provided by motion averaging wherein the geometry of all camera motions is estimated by using pairwise camera motion estimates (Govindu, 2004). Recent work has shown that the motion averaging approaches have met with much success and can efficiently and robustly solve 3D rotations (Chatterjee and Govindu, 2013) and translations (Wilson and Snavely, 2014). But for large SfM datasets, global methods (Jiang, Cui, Tan, 2013, Moulon, Monasse, Marlet, 2013) need to be exceedingly robust to achieve acceptable performance. However, such solutions can effectively provide a good initial estimate for a batch approach to bundle adjustment (henceforth BBA), thereby reducing computational time (Crandall et al., 2013). We also remark that in all these approaches, the largest computational load is actually incurred in the pre-processing step of matching camera pairs. This is true even for methods that employ preemptive matching (Wu, 2013).

Farenzena et al. (2009), Toldo et al. (2015) and Gherardi et al. (2010) propose to merge smaller reconstructions in a bottom up dendrogram. Motivated by their work we adopt a hierarchical divide-and-conquer strategy that is designed to mitigate some of the above problems. In essence, our approach partitions the full image dataset into smaller sets that can each be independently reconstructed using global motion averaging followed by batch mode bundle adjustment. Subsequently, by utilising available geometric relationships between cameras across the individual partitions, we solve a series of global registration problems as we move up a hierarchical tree that correctly and accurately place individual 3D reconstructed components into a single global frame of reference. In what follows we show that this new approach is not only more robust with respect to failures in reconstruction but also gives significant improvements over state-of-the-art techniques in terms of computational speed. The main contributions of our paper are:

• We use hierarchical agglomerative clustering to partition a large dataset into smaller well constrained base clusters. The bottom-up agglomerative clustering starts with individual images as clusters, and pairs of clusters are merged as we move up the hierarchy until base clusters of pre-defined number of images are formed. The merging is based on the robustness of the epipolar geometry estimation and the number of epipolar inliers of the images across the clusters. Thus, by definition of agglomerative clustering, each cluster (at any level) represents as tightly constrained SfM problem as possible, given the problem size. Unlike in Farenzena et al. (2009), Gherardi et al. (2010) and Toldo et al. (2015) wherein the process of 3D reconstruction is initiated at the leaf nodes of the agglomerative tree using pairwise bundle adjustment, we initiate the 3D reconstruction at a node higher up in the tree representing a base cluster of a suitable size. This provides a more global view and yet presents a small enough SfM problem that can be solved reliably and robustly. Our experiments clearly demonstrate that this global view is crucial for a hierarchical strategy to be successful for large-scale SfM problems.

• We propose a reconstruction method for base clusters based on global motion averaging followed by batch mode bundle adjustment. We first use the pairwise epipolar relationships of the images within a base cluster to obtain initial estimates of pairwise rotations and translation directions of the cameras and subsequently apply global rotation (Chatterjee and Govindu, 2013) and translation (Wilson and Snavely, 2014) averaging to obtain consistent estimates of the camera poses. We use the camera poses and the feature correspondence tracks to compute an initial estimate of the 3D structure using triangulation. The camera poses and the 3D structure thus obtained are finally refined using a few iterations of batch mode bundle adjustment. Our experiments clearly demonstrate that for the typical sizes chosen for base clusters, the global approach to SfM is more reliable and robust as compared to both sequential approaches like (Wu, 2013) which have a limited global view and other hierarchical approaches like Farenzena et al. (2009), Gherardi et al. (2010), Ni and Dellaert (2012), Toldo et al. (2015) and Zephyr (2016) which are purely local in nature.

• We propose a method for registering the point clouds corresponding to the independently reconstructed base clusters as well as intermediate clusters using pairwise epipolar geometry relationships (Bhowmick et al., 2014). The epipolar based registration technique proposed in this paper is more robust than the standard techniques for registering point clouds using 3D-3D or 3D-2D correspondences. Registration methods based on 3D point correspondences do not use all available information (image correspondences) and may fail when the point clouds do not have a sufficient number of 3D points in common. 3D-2D based methods, such as a sequential bundle adjustment (Snavely, Wu, 2013), often result in broken reconstructions when the number of points available are inadequate for re-sectioning or when common 3D points are removed at the outlier rejection stage (Wu, 2013). Our experiments clearly demonstrate that the proposed registration algorithm using pairwise epipolar geometry alleviates this problem.

• We propose an effective method for merging and refining the 3D reconstructions as we move up the hierarchical agglomerative tree. After epipolar registration of clusters we selectively apply a step of batch mode bundle adjustment on the combined clusters. We show that often some clusters that could not be merged effectively lower down in the hierarchy can be merged reliably as we move up the hierarchical tree as larger parts of the scene gets accumulated in the reconstruction. Similarly, points that are removed as outliers during reconstruction at the early stages can often be reclaimed by re-sectioning at the later stages as we move up the hierarchical tree.

The two other methods that are closest to our approach are the hierarchical SfM of Farenzena et al. (2009), Gherardi et al. (2010), Toldo et al. (2015) and Zephyr (2016) and the divide and conquer method that we had proposed in Bhowmick et al. (2014).

Farenzena et al. (2009), Gherardi et al. (2010) and Toldo et al. (2015) initiate the reconstruction process using pairwise bundle adjustment at the leaf nodes. Because of the purely local nature of the approach without any global look-ahead, the method often fails to grow the reconstructions by merging or re-sectioning. Their use of reprojection errors of common 3D points for merging appear to be unsuitable for very large datasets. Results in Farenzena et al. (2009), Gherardi et al. (2010) and Toldo et al. (2015) are limited to reconstructions with a maximum of 477 images and our experiments with the codes available for both 3DF Samantha (Toldo et al., 2015) and 3DF Zephyr (trial version) Zephyr (2016) failed to produce results for some of the larger datasets that we consider (see the experimental results in Figs. 4, 5 and 6).

In Bhowmick et al. (2014) we had proposed a divide and conquer based pipeline wherein we used the normalised cut (Shi and Malik, 2000) for obtaining independent base clusters and used (Wu, 2013) for solving the within base cluster SfM problems. The individual reconstructions were merged using a preliminary version of the epipolar registration technique proposed in this paper. As a result, the method of Bhowmick et al. (2014) was not hierarchical. We find hierarchical agglomerative clustering to be more suitable than normalised cut because normalised cut gives poorer control on the size of each component, whereas hierarchical clustering produces balanced and well conditioned base clusters resulting in more robust global reconstruction. Also, the cut edges of the multi-way normalised cut partitions are sometimes too weak for epipolar registration, especially when the base clusters’ sizes are poorly balanced. In addition, our experiments demonstrate that for the typical base cluster sizes that we choose for reconstruction, our global reconstruction framework is more robust than VSFM.

The rest of the paper is organised as follows. In Section 2 we give an outline of our hierarchical approach. In Section 3 we provide the details of construction of the agglomerative tree and clustering. In Section 4 we describe our method of global 3D reconstruction based on motion averaging and batch mode bundle adjustment. In Section 5 we describe our technique of epipolar registration and in Section 6 we present our method of hierarchical merging and refinement along with re-sectioning. In Section 7 we present experimental results on several large datasets and finally, in Section 8, we conclude the paper.