Hierarchical multi-view metric learning with HSIC regularization

Abstract

As the information era develops rapidly, it’s common to utilize multiple features from different sources to represent one object. Measuring the similarity between multi-view objects is the fundamental task in multi-view learning. To effectively measure the similarity between multi-view samples, multi-view metric learning has gained extensive attention recently. Nevertheless, most existing methods merely focus on the closeness of similar pairs and the separability of dissimilar ones inside each view, so that rich consensus properties existing in multi-views data might be ignored to some extent. To mitigate this issue, we come up with a novel method entitled Hierarchical Multi-view Metric learning with HSIC regularization (HM²H). HM²H aims to simultaneously maintain the closeness of similar points and the separability of dissimilar ones in intra-view and inter-view. Since multiple views depict different perspectives of the same object, the shared metric is introduced to capture the consensus information among those views. Moreover, we take advantage of the Hilbert–Schmidt Independence Criterion to seek the maximum distribution agreement of the multi-view dataset. Correspondingly, an algorithm based on Alternating Direction Method is provided to solve the proposed HM²H. Finally, various experimental results on five visual recognition datasets confirm the effectiveness and feasibility of our proposed method.

Introduction

The increasing interest in feature extraction techniques has heightened the need for tools suitable to cope with multi-modality (multi-view) data [1]. Of particular fundamental is the Multi-view Metric Learning (MvML) techniques, which attempt to concurrently learn multiple Mahalanobis metrics from multi-view features to reveal the semantic similarity of multi-view data [2]. With proper distance metrics, the performance of various similarity-based multi-view applications can be significantly improved, such as image classification [3], face verification [4], and re-identification [5], [6].

Distance Metric Learning (DML) aims to learn a Mahalanobis metric matrix from the training data, such that samples from the same classes are aggregated close to each other while those from different classes are detached by a large margin [7]. Since DML takes weights and correlations of features of objects into consideration when evaluating the distance between samples, it can better measure the semantic similarity between data points than the Euclidean one. Advantages of DML have been discovered and validated from various perspectives [8]. Despite most DML methods having been widely utilized in practical applications, these studies can only be applied to single-view scenarios. With that in mind, researchers have proposed to extend the classical DML methods to multi-view scenarios, and simultaneously learn multiple Mahalanobis matrices from side information [9]. A simple yet effective manner is to concatenate all the multi-view features into a compact vector and then employ DML techniques directly in this case. Nevertheless, the concatenated vector lacks physical meaning and is prone to incur the curse of dimensionality problem. Hence, McFee and Lanckriet [10] adopted multiple kernel functions for different views to project the multi-view features into reproducing kernel Hilbert space (RKHS), and then multiple projections and the corresponding weights were concurrently learned under the supervision of human perceptual similarity information. Afterwards, many MKL-based MvML approaches have been developed [11], [12]. Although MKL based methods could capture the heterogeneous multi-view features structure, these methods might suffer from poor scalability problems.

Besides MKL techniques, there have also been some late fusion approaches. These methods combine outputs of the single-view DML methods constructed from different views and can be recognized as the naive extension of the classical single-view DML method under the multi-view setting [13], [14], [15]. For instance, EMGMML [13] was presented to extend GMML [16] into a multi-view scenario, in which the weight and the metric inside each view were collectively obtained. Whereas, late fusion methods only concentrate on the closeness of samples from the same classes and the separability of those from different classes inside each view, leaving the rich consensus information among multiple views unconsidered. Notably, most existing MvML methods optimize the model via the naive euclidean gradient descent algorithm [17], [18]. It has been validated by various studies that directly optimizing the loss functions constrained by the low-rank matrices in the Euclidean space are numerically unstable and are easily trapped into the bad local optimal solutions [19], [20].

In light of the aforementioned limitations and deficiencies of the existing MvML models, we put forward a novel MvML model entitled Hierarchical Multi-view Metric Learning with HSIC Regularization (H${\mathrm{M}}^2$H), which aims to simultaneously optimize the closeness and the separability in intra-view and inter-view to improve the overall discriminative ability. For comprehensibility, we plot the main flowchart of H${\mathrm{M}}^2$H in Fig. 1. H${\mathrm{M}}^2$H adopts a hierarchical mechanism to build multiple metric matrices, where each metric is composed of a view-specific projection matrix and a Symmetric Positive-definite (PSD) metric shared by all the views. The view-specific projection matrix is exploited to extract the consensus information inside that view, and the shared PSD matrix is in charge of characterizing the correlations of the features in the shared latent space. Importantly, a regularization term based on the Hilbert–Schmidt Independence Criterion (HSIC) [21] is further come up with to calibrate the enormous distribution difference among different views. Accordingly, an alternating direction method is employed to settle the minimization problem. Instead of using the Euclidean gradient descent method to solve the sub-problem, we propose to view the sub-problem as an unconstrained minimization one on the Riemannian manifold and handle it efficiently by means of the Riemannian optimization technique. Finally, extensive experiments on visual recognition datasets confirm the effectiveness of our proposed method. In summary, the major contributions of this paper can be listed as follows:

• We propose a novel MvML method to simultaneously maximize the discriminative ability in intra-view and inter-view, which aims to amply explore the consensus information among multiple views.

• An alternating direction strategy is developed to seek the feasible solution of H${\mathrm{M}}^2$H and the optimization of the sub-problem is further efficiently tackled by the Riemannian optimization technique.

• Extensive visual recognition experiments on five datasets validate the effectiveness of the proposed method over the compared MvML methods used in experiments.

The rest of the paper is organized as follows. In Section 2, we review the related work, and describe the main motivation of H${\mathrm{M}}^2$H in Section 3. In Section 4, based on Riemannian optimization techniques, we develop an effective Riemannian gradient descent algorithm to solve the proposed method, followed by the visual recognition experimental results in Section 5. In Section 6, we conclude this paper.