Efficient guided hypothesis generation for multi-structure epipolar geometry estimation

Highlights

• A sampling method EGHG is proposed for multi-structure geometry estimation.

• EGHG combines the benefits of a global and a local sampling strategy.

• The global sampling strategy is designed to rapidly obtain promising solutions.

• The local sampling strategy is designed to efficiently achieve accurate solutions.

• Experimental results show the effectiveness of EGHG on public real image pairs.

Abstract

We propose an Efficient Guided Hypothesis Generation (EGHG) method for multi-structure epipolar geometry estimation. Based on the Markov Chain Monte Carlo process, EGHG combines two guided sampling strategies: a global sampling strategy and a local sampling strategy. The global sampling strategy, guided by using both spatial sampling probabilities and keypoint matching scores, rapidly obtains promising solutions. The spatial sampling probabilities are computed by using a normalized exponential loss function. The local sampling strategy, guided by using both Joint Feature Distributions (JFDs) and keypoint matching scores, efficiently achieves accurate solutions. In the local sampling strategy, EGHG updates a set of current best hypothesis candidates on the fly, and then computes JFDs between the input data and these candidates. Experimental results on public real image pairs show that EGHG significantly outperforms several state-of-the-art sampling methods on multi-structure data.

Introduction

Epipolar geometry estimation is an important task in computer vision and has been successfully employed in image registration (Kushnir and Shimshoni, 2014), image segmentation (Wang et al., 2012), motion segmentation (Jung et al., 2014), etc. RANSAC (Fischler and Bolles, 1981) has been extensively used in computer vision, especially for recovering the epipolar geometry from a pair of images. The success of RANSAC and its variants (such as Chum, Matas, Kittler, 2003, Lebeda, Matas, Chum, 2012, Rabin, Delon, Gousseau, Moisan, 2010, Raguram, Chum, Pollefeys, Matas, Frahm, 2013, Raguram, Frahm, 2011, Sattler, Leibe, Kobbelt, 2009, Torr, Zisserman, 2000, Wang, Chin, Suter, 2012) depends on whether these methods can sample at least one all-inlier minimal subset (i.e., one clean minimal subset). The sampling strategy used in RANSAC is random sampling. Random sampling has been used in many recently proposed fitting methods (e.g., Isack, Boykov, 2012, Isack, Boykov, 2014, Mittal, Anand, Meer, 2012, Wang, Chin, Suter, 2012) due to its simplicity, so it is a rather ubiquitous element of robust fitting methods.

A main deficiency of random sampling lies in that it is not an effective (in terms of efficiency and accuracy) sampling strategy. A robust fitting method based on random sampling needs to exponentially increase the number of samples to ensure at least one clean minimal subset is sampled when the dimension of the model, or the outlier ratio of the input data, grows. Moreover, in practice, data in computer vision are usually composed of multiple structures (i.e., model instances) (Stewart, 1999). In such a case, the outliers of a structure contain both gross outliers and pseudo-outliers (i.e., inliers belonging to the other structures are outliers to the structure), which often leads to high outlier rates and thus severely affects the performance of random sampling.

To reduce the deficiency of random sampling, many recently proposed sampling methods (e.g., Chum, Matas, 2005, Fragoso, Sen, Rodriguez, Turk, 2013, Tordoff, Murray, 2005) accelerate promising hypothesis generation by exploiting prior information from keypoint matching scores. However, these sampling methods only consider the single structure case: They can efficiently find clean subsets for single-structure data (even with a large fraction of outliers) benefiting from accurate matching scores, but they usually fail to find a “clean” solution for multiple-structure data in a reasonable time (e.g., 10 s),¹ where a “clean” solution means that at least one clean minimal or larger-than-minimal subset is sampled for each model instance in data. This is because the data points in the subsets sampled by these methods usually have high matching scores, but they might be from different structures. This may lead to overfitting (the inliers of one fitted hypothesis contain the inliers of multiple structures).

To efficiently sample for multi-structure fitting methods² (e.g., Lazic, Givoni, Frey, Aarabi, 2009, Magri, Fusiello, 2014, Schindler, Suter, 2006, Thakoor, Gao, 2008, Toldo, Fusiello, 2008, Wang, Chin, Suter, 2012, Yu, Chin, Suter, 2011), which aim to simultaneously estimate all model instances of multi-structure data, some sampling methods (e.g., Chin, Yu, Suter, 2012, Wong, Chin, Yu, Suter, 2013) have been proposed to simultaneously generate promising hypotheses for all the structures of multi-structure data, by residual sorting. However, in a reasonable time budget, these methods may not obtain any accurate solution for data with a low inlier rate, because using all the generated hypotheses, via residual sorting, to guide the subsequent hypothesis generation is time-consuming. Although some other sampling methods (e.g., Kanazawa and Kawakami, 2004) can also obtain clean solutions for multi-structure data (even with a low inlier rate), the clean solutions obtained by these methods may be inaccurate. This is often because the spans of minimal subsets sampled by these methods are small due to the use of local constraints (Tran et al., 2014).

In this paper, we focus on simultaneously and efficiently sampling promising hypotheses for multi-structure data. We propose an Efficient Guided Hypothesis Generation (EGHG) method, which contains two guided sampling strategies (i.e., a global sampling strategy and a local sampling strategy), for epipolar geometry estimation, which aims to simultaneously estimate all model instances of multi-structure data.

Although the global sampling strategy in EGHG can rapidly obtain clean solutions, the obtained solutions may not be accurate enough. The local sampling strategy, on the other hand, can efficiently generate more accurate solutions by using Joint Feature Distributions (JFDs) (Triggs, 2001), which use the epipolar constraint to accurately characterise the dependencies among keypoint matches. However, the local sampling strategy needs the clean solutions (i.e., the clean subsets) to compute JFDs. Thus, we combine the global sampling strategy and the local sampling strategy, based on a Markov Chain Monte Carlo (MCMC) process, to rapidly generate promising hypotheses for multi-structure data.

In detail, the sampling strategies are given as follows:

The global sampling strategy: We propose to sample minimal subsets by using both keypoint matching scores and computationally cheap spatial sampling probabilities in the global sampling process to rapidly “hit” clean solutions.

The local sampling strategy: A set of current best hypothesis candidates (corresponding to different model instances in data) is updated from the generated hypotheses by using a nonparametric kernel density estimate technique, on the fly. Then a hypothesis is selected from these candidates in each local sampling process, where the hypothesis is generated from a sampled subset $\mathcal{S}$. Following this, the JFDs between the input data and $\mathcal{S}$ are computed. Finally, we propose to sample subsets guided by using both JFDs and keypoint matching scores in the local sampling process, to effectively generate accurate solutions. The process of updating the hypothesis candidates and computing the JFDs is efficient.

The rest of the paper is constructed as follows: In Section 2, we review related work. In Section 3, we present the global sampling strategy. In Section 4, we describe the local sampling strategy. In Section 5, we propose the complete EGHG method, and in Section 6 we show the experimental results on real image pairs. Finally, we draw conclusions in Section 7.