TSSN-Net: Two-step Sparse Switchable Normalization for learning correspondences with heavy outliers

Abstract

In this paper, we solve the problem of feature matching by designing an end-to-end network (called TSSN-Net). Given putative correspondences of feature points in two views, existing deep learning based methods formulate the feature matching problem as a binary classification problem. In these methods, a normalizer plays an important role in the networks. However, they adopt the same normalizer in all normalization layers of the entire networks, which will result in suboptimal performance. To address this problem, we propose a Two-step Sparse Switchable Normalization Block, which involves the advantage of adaptive normalization for different convolution layers from Sparse Switchable Normalization and robust global context information from Context Normalization. Moreover, to capture local information of correspondences, we propose a Multi-Scale Correspondence Grouping algorithm, by defining a multi-scale neighborhood representation, to search for consistent neighbors of each correspondence. Finally, with a series of convolution layers, the end-to-end TSSN-Net is proposed to learn correspondences with heavy outliers for feature matching. Our experimental results have shown that our network achieves the state-of-the-art performance on benchmark datasets.

Introduction

Establishing reliable feature correspondences is a fundamental problem in computer vision [1], [2], [3], e.g., Structure from Motion (SfM) [4], [5], Simultaneous Localization and Mapping (SLAM) [6], [7], Image Fusion [8] and geometric model fitting [9], [10], [11], [12]. This problem relies on two steps, i.e., correspondence generation and correspondence selection [13]. The first step can be dealt with matching local key-point features (e.g., SIFT [14], Hessian-Affine Detector [15]). However, the initial correspondences are often inevitable to be contaminated by outliers (see Fig. 1) due to various problems, e.g., local key-point localization errors and ambiguities of the local descriptors. Thus, many methods (e.g., [13], [16], [17], [18], [19], [20], [21], [22], [23]) have been proposed to remove outliers.

Among current existing feature matching methods, deep learning based methods are the most popular technique due to its effectiveness, e.g., LGC-Net [20] and NM-Net [13]. LGC-Net introduces Multi-Layer Perceptrons (MLPs) and Context Normalization (CN) for feature matching. NM-Net includes a compatibility-specific mining algorithm to successively extract and aggregate local features for a hierarchical deep learning network. Despite their great successes, they adopt the same normalizer in all normalization layers of the entire networks, which will result in suboptimal performance.

Recently, Sparse Switchable Normalization (SSN) [24], which formulates the optimization problem into the differentiable feed-forward computation problem by a sparse learning algorithm, is proposed to adaptively select normalizers for the tasks of image classification, semantic segmentation and action recognition. Specifically, SSN employs a sparse learning algorithm to adaptively select the most suitable normalization, from Batch Normalization (BN) [25], Instance Normalization (IN) [26], and Layer Normalization (LN) [27], for each normalization layer of a deep network. As we know, different normalizers have their own advantages and disadvantages. For example, BN acts as a feature regularizer and improves generalization of the network, but it is sensitive to the value of batch size (i.e., the number of samples per GPU) since the mean and standard deviation of BN are derived from the batch size. In contrast, IN and LN are not sensitive to the value of batch size but they have a limited generalization ability of the network for feature matching. Thus, through adaptively select normalizers, SSN can inherit all benefits of normalizers (i.e., BN, IN, LN) and also has good generalization ability.

In this paper, we propose a hierarchical deep learning network (called TSSN-Net) to learn correspondences with heavy outliers for feature matching. The architecture of the proposed TSSN-Net is shown in Fig. 2. Specifically, we propose to add SSN into the deep network, to improve the performance of current existing deep networks for feature matching. Note that the performance of SSN depends on the effectiveness of the sparse learning algorithm, which trains deep models with sparse constraints. However, the input data for feature matching problem has not obvious sparse space. To address this issue, we propose a novel Two-step Sparse Switchable Normalization block (TSSN Block), which introduces CN to construct the sparse space for SSN. Thus, TSSN Block is able to inherit all benefits from different normalizers for feature matching.

In addition, to extract local features, we search for consistent neighbors of each correspondence according to the compatibility-specific distance, which is able to find more reliable neighbors than spatial distance for feature matching. To further improve the generalization ability of the proposed network, we propose a Multi-Scale Correspondence Grouping algorithm, which searches for neighbors of each correspondence and integrates correspondence and neighbors to a graph, by a novel multi-scale neighborhood representation.

The contributions are summarized as follows:

• We develop an end-to-end network to handle the feature matching problem with heavy outliers. The proposed network is able to achieve the state-of-the-art performance on three benchmark datasets with various proportions of outliers.

• We propose a novel normalization block, which effectively involves the global context information and adaptively switches normalizers in different convolution layers, for feature matching. The proposed block is able to significantly improve the performance of our network since it inherits all benefits of from different normalizers.

• We propose a correspondence grouping algorithm to capture local information of correspondences. The proposed algorithm searches for neighbors based on compatibility-specific distance and integrate correspondences with neighbors to a graph, which is permutation-equivariant to initial correspondences. Therefore, our network is insensitive to the order of correspondences and has good generalization ability for feature matching.

The rest of the paper is organized as follows: We briefly review the related work and background material in Section 2. We present our network for feature matching in Section 3. In Section 4, we describe and analyze the experimental results on three benchmark datasets. At last, we draw conclusions in Section 6.