Multiple kernel dimensionality reduction via spectral regression and trace ratio maximization

Abstract

The performance of kernel-based dimensionality reduction heavily relies on the selection of kernel functions. Multiple kernel learning for dimensionality reduction (MKL-DR) has been recently proposed to learn a convex combination from a set of base kernels. But this method relaxes a nonconvex quadratically constrained quadratic programming (QCQP) problem into a semi-definite programming (SDP) problem to specify the kernel weights, which might lead to its performance degradation. Although a trace ratio maximization approach to multiple-kernel based dimensionality reduction (MKL-TR) has been presented to avoid convex relaxation, it has to compute a generalized eigenvalue problem in each iteration of its algorithm, which is expensive in both time and memory. To improve the performance of these methods further, this paper proposes a novel multiple kernel dimensionality reduction method by virtue of spectral regression and trace ratio maximization, termed as MKL-SRTR. The proposed approach aims at learning an appropriate kernel from the multiple base kernels and a transformation into a lower dimensionality space efficiently and effectively. The experimental results demonstrate the effectiveness of the proposed method in benchmark datasets for supervised, unsupervised as well as semi-supervised scenarios.

Introduction

In real applications, many machine learning algorithms, such as spectral clustering, usually behave badly when faced with high-dimensional data. Hence, finding a way of transforming them into a unified space of lower dimension can facilitate the underlying tasks such as pattern recognition or regression problems. According to different assumptions about the data distribution, these methods can be classified into two categories, i.e., linear and nonlinear methods [1], [2], [3], [4], [5], [6]. If the training data is embedded in a linear subspace, linear dimensionality reduction methods are commonly used to discover the dimensionality of the subspace. Correspondingly, if the data is sampled from a nonlinear low dimensional manifold which is embedded in the high-dimensional ambient space, nonlinear dimensionality reduction methods are usually utilized to preserve the manifold structure.

In recent years, nonlinear dimensionality reduction methods based on manifold assumption have been attracting many researchers. These methods mainly take advantage of various manifold learning techniques, such as isometric feature mapping (ISOMAP) [6], locally linear embedding (LLE) [7] and Laplacian Eigenmap [8] (LE), which reduce the dimensionality of a fixed training set in a way that maximally preserve certain inter-point relationships. But, one of the major limitations of these methods is that they do not generally address the out-of-sample problem. There are a lot of approaches that try to provide natural out-of-sample extensions of Lapalcian Eigenmaps, LLE and Isomap, such as locality preserving projections (LPP) [9] and regularized regression neural networks, they address this issue by explicitly finding an embedding function when minimizing the objective function [9], [10], [11]. The computation of these methods generally involves eigen-decomposition of dense matrices which is expensive in both time and memory. Some other approaches interpret these spectral embedding algorithms as learning the principal eigenfunctions of an operator defined from a kernel and the unknown data generating density. Thus, they address this issue through a kernel view of LLE, Isomap and Laplacian Eigenmaps [12], [13]. To obtain the embedding result of an unseen example, the kernel function values of this unseen example with all the training samples have to be calculated, which may not be possible in some situations. In addition, several fast extension methods based on landmarks have been developed to dramatically reduce the computational burden of these methods. But, these methods use landmarks to increase speed at the cost of some accuracy. Take Landmark-Isomap for example, it is a variant of Isomap which is faster than Isomap. However, the accuracy of the manifold is compromised by a marginal factor [14]. Fortunately, Spectral regression (SR) subtly avoids eigen-decomposition of dense matrices by means of regression and spectral graph analysis. Meanwhile, it can enhance the performance of dimensionality reduction by introducing the regularization term [15], [16]. Moreover, it can be performed either in supervised, unsupervised or semi-supervised settings. These dimensionality reduction techniques could be unified under a common framework called graph embedding [17].

Recently, multiple kernel learning was incorporated into graph embedding called multiple kernel learning for dimensionality reduction (MKL-DR) [18], which aims to learning a linear transformation in the nonlinear space induced by a set of base kernels. MKL-DR provides the ability of automatically selecting optimal kernels by identifying a linear combination of base kernels. The advantage of using multiple kernels instead of only one kernel in dimensionality reduction has been demonstrated [17]. But, this method needs to iteratively solve a generalized eigenvalue problem, which is time-consuming. In addition, it replaces a QCQP optimization problem by a SDP problem to obtaining kernel weights, which could have a negative effect on its performance. To speed up MKL-DR, a multiple kernel dimensionality reduction algorithm based on spectral regression, called MKL-SR, was proposed in Ref. [19]. It transforms eigen-decomposition of dense matrices in the first step of MKL-DR into a linear regression problem by means of spectral regression. But, MKL-SR still uses the convex relaxation technique to optimize the kernel weights. Instead of convex relaxation, a multiple kernel learning framework called MKL-TR was recently proposed to avoid relaxing the primal problem [20]. MKL-TR learns a transformation into a space of lower dimension by converting a trace ratio maximization problem into single trace maximization. But, the computational bottleneck of both MKL-DR and MKL-TR is in iteratively computing generalized eigen-decomposition of regularized dense matrices, which has high computational complexity of $O\left(n^3\right)$ (n denotes the number of training data).

Since SR, MKL-DR, MKL-SR and MKL-TR are all based on graph embedding, it is natural to make good use of SR and the trace ratio optimization technique to overcome the restrictions of MKL-DR, MKL-SR and MKL-TR. Consequently, we present a framework, termed as MKL-SRTR, which not only incorporates spectral regression into multiple kernel learning for dimensionality reduction, but uses the trace ratio optimization algorithm to obtain kernel weights. On one hand, spectral regression does not increase speed at the cost of some accuracy, on the other hand, the trace ratio optimization algorithm can avoid the convex relaxation of MKL-DR and MKL-SR. The formulation of MKL-SRTR with graph embedding will be illustrated, which not only extends any DR technique expressible by graph embedding to multiple kernel settings, but selects optimal kernels more efficiently than other multiple kernel-based dimensionality reduction methods. Specifically, the proposed approach decreases the computational complexity to $O\left(n^2\right)$. The experimental results demonstrate the proposed method achieves better or similar performance compared to other algorithms for supervised, unsupervised as well as semi-supervised scenarios. In addition, as other multiple kernel-based dimensionality reduction methods would do, it can also solve the out-of-sample extension problem.

The paper is structured as follows: Spectral regression and dimensionality reduction with multiple kernels are introduced in Section 2 Spectral regression algorithm, 3 Dimensionality reduction with multiple kernels, respectively. In Section 4, we provide the MKL-SRTR framework and the optimization process. The experimental results are listed in Section 5. Finally, we give the related conclusions and a discussion of future works in Section 6. In order to avoid confusion, we give a list of the main notations used in this paper in Table 1.