Spectral-invariant matching network

Highlights

• The method is based on domain conversion for matching between VIS and NIR images.

• The method uses metric learning to integrate information from the two domains.

• This work studies the effect of the domain conversion method and loss function.

• This work achieves noteworthy performance in three public datasets.

• The result can be adapted to various applications such as depth estimation.

Abstract

As the need for sensor fusion systems has grown, developing methods to find correspondences between images with different spectral ranges has become increasingly important. Since most images do not share low-level information, such as textures and edges, existing matching approaches fail even with convolutional neural networks (CNNs). In this paper, we propose an end-to-end metric learning method, called SPIMNet (SPectral-Invariant Matching Network) for robust cross- and multi-spectral image patch matching. While existing methods based on CNNs learn matching features directly from cross- and multi-spectral image patches, SPIMNet transforms across spectral bands and discriminates for similarity in three steps. First, (1) SPIMNet is adjusted for a feature domain by introducing a domain translation network; then (2) two Siamese networks learn to match the adjusted features with the same spectral domain; and (3) the matching features are fed to fully-connected layers to determine the identity of the patches as a classification task. By effectively incorporating each step, SPIMNet achieved competitive results on a variety of challenging datasets, including both VIS–NIR and VIS–Thermal image pairs. Our code is available at https://github.com/koyeongmin/SPIMNet.

Introduction

Many researchers and industries utilize sensors of various domains to get more information about targets. For managing information from each sensor, proper sensor fusion methods are required. In the field of computer vision, cross-spectral (i.e.visible–near infrared (NIR)) and multi-spectral (i.e.visible–thermal) image matching are being actively studied because the different spectral domains can provide complementary information [1]. As an example, visible and thermal images can mutually compensate for rich color information and high textural structures in low-light conditions, making these images suitable for all-day vision systems [2]. All-day vision or fusing multi-spectral images has become an essential and significant task for sensor fusion systems that conduct facial expression recognition [3], [4], material classification [5], [6], medical image analysis [7], pansharpening [8], and pedestrian detection [9], [10], [11].

Since cross- and multi-spectral images capture different wavelength spectral ranges, the images appear significantly different in both intensity and pixel levels. Even with well-known local feature descriptors [12], [13], the relationship between images across spectral domains cannot be accounted for, which results in severe performance drops in matching tasks. Recently, convolutional neural networks (CNNs) have demonstrated some ability to address this issue, by leveraging semantic information along with low-level features. Siamese structures overcome the somewhat challenging matching problems among various spectral domains [14], [15], [16], [17]. In most siamese structures, the same deep neural network is applied to both multi-spectral image patches and extracts each feature. Their loss functions make the distance between two positive patches short, otherwise far. Encoder–decoder structures are also utilized to extract common features between multi-spectral image patches [18], [19].

Although these previous methods show that their methods work for fusing cross- and multi-spectral images, these methods separately extract features from each image, and just aggregate the features to fuse information or to predict similarities. We have observed that the methods are not suited to dealing with both intensity- and pixel-level differences because these differences can be one reason to make a fusion between multi-spectral images hard and previous methods do not have any module to reduce the difference. In this paper, we present a SPectral-Invariant Matching Network (SPIMNet), an end-to-end CNN framework for robust image patch matching across different spectral domains. These are the primary contributions of this study:

• In contrast to previous methods that extract features directly from input patches, SPIMNet learns the spectral translations of input patches using a domain conversion network. We can use similar features to compare image patches across different spectral domains.

• SPIMNet utilizes a dual-Siamese network for feature extraction from each translated piece of information to predict the matching label through a fully connected network.

• The proposed end-to-end method can be trained from scratch without any pre-trained backbone network, and we obtain competitive results over several standard datasets, including both visible–NIR and visible–thermal images.

• Additionally, ablation studies indicate that each of these technical contributions leads to appreciable improvements in matching accuracy, and we show that the proposed method can be applied to various applications such as stereo matching and image enhancement.