Semi-supervised adaptive kernel concept factorization

Abstract

Kernelized concept factorization (KCF) has shown its advantage on handling data with nonlinear structures; however, the kernels involved in the existing KCF-based methods are empirically predefined, which may compromise the performance. In this paper, we propose semi-supervised adaptive kernel concept factorization (SAKCF), which integrates the data representation and kernel learning into a unified model to make the two learning processes adapt to each other. SAKCF extends traditional KCF in a semi-supervised manner, which encourages the high-dimensional representation to be consistent with both the limited supervisory and local geometric information. Besides, an alternating iterative algorithm is proposed to solve the resulting constrained optimization problem. Experimental results on six real-world data sets verify the effectiveness and advantages of our SAKCF over state-of-the-art methods when applied on the clustering task.

Introduction

Nonnegative matrix factorization (NMF) [1], [2] has shown its effectiveness for the representation of linear data, and many extended NMF algorithms have been proposed to improve its performance on data representation [3], [4], [5], [6], [7] and clustering [8], [9], [10] in the past decade. Particularly, as a variant of NMF, concept factorization (CF) [11] models its basis as a linear combination of the input data samples in the original space, which inherits the strengths of NMF, such as the physically meaningful representations and effectiveness for representing linear data. To better capture the underlying structure, many CF-based models incorporating local geometric information were proposed. For examples, Cai et al. [12] proposed locally consistent CF, in which the low-dimensional representation preserves the manifold structure by introducing a graph regularization. Liu et al. [13] imposed a locality constraint into CF and proposed local-coordinate CF, where the sample is represented by only a few nearby bases. Pei et al. [14] improved the locally consistent CF by jointly learning an adaptive weight matrix for graph regularization and factored matrices for CF. Zhang et al. [15] introduced an auto-weighted reconstruction graph learning strategy into local-coordinate CF, where L${}_{2,1}$ norm is imposed to enhance the robustness of CF to noise. Instead of using F-norm in locally consistent CF, Peng et al. [16] proposed a correntropy based CF model which adopts correntropy to measure the similarity between the input data matrix and reconstructed one. Ma et al. [17] improved the conventional CF by introducing both manifold and adaptive neighbor structures into a unified model.

In addition, considering that a small amount of supervisory information may be available in practical applications, CF has been extended in a semi-supervised manner. Liu et al. [18] proposed constrained CF, which introduces a label indicator matrix into the low-dimensional representation to enforce the data samples in the same class to share the identical low-dimensional representation. Based on [18], Shu et al. [19] proposed a local regularization constrained CF, which takes both geometric and discriminative information into account. Zhang et al. [20] relaxed the reconstruction error of the constrained CF, and proposed a joint classification and data representation framework. Zhou et al. [21] proposed a novel robust semi-supervised CF, which integrates the robust adaptive embedding and maximum correntropy criterion based CF into a unified model.

Technically, to ensure basis and original data sample are in the same space, CF aims to represent basis by the linear combination of original data samples. However, when applied on nonlinearly distributed data, the way that CF formulates the basis may lead to an uninterpretable basis. As an example, Fig. 1 intuitively illustrates the linear combination of the data samples in the original data space, where the weight of linear combination is set to 1. Since kernel methods can deal with nonlinear correlation in data, both CF [11] and locally consistent CF [12] have been extended into their kernel versions, respectively, which improve the clustering performance. Li et al. [22] proposed a manifold kernel concept factorization (MKCF), which introduces the manifold kernel learning into CF to encode the geometric information in the kernel space. Similar to MKCF, Li et al. [23] presented a manifold kernel based local-coordinate CF, which utilizes a manifold kernel to improve the capability of local-coordinate CF on handling nonlinearly distributed data. Instead of selecting an optimal kernel, Li et al. [24] proposed a multi-kernel based CF, which takes advantage of several kernels by linearly fusing them.

Although the existing kernel CF (KCF)-based methods have demonstrated their advantages and effectiveness on handling linearly inseparable data, they construct the kernel empirically, and it is not clear whether the predefined kernel is able to adapt the subsequent learning of the low-dimensional representation well. To this end, in this paper, we propose a model, namely semi-supervised adaptive kernel concept factorization (SAKCF), to jointly learn an adaptive kernel and low-dimensional data representation. In addition to incorporating the two learning processes into a unified model, we consider the situation where the limited supervisory information is given, and formulate the proposed method in a semi-supervised scenario. As illustrated in Fig. 2, the proposed SAKCF performs CF in a learned high-dimensional kernel space, in which the high-dimensional representation is consistent with the limited supervisory and local geometric information. Different from locally consistent CF (LCCF), constrained CF (CCF) or local regularization constrained CF (LRCCF), the proposed SAKCF encodes supervisory and geometric information into the kernel space to obtain an informative kernel, rather than directly uses these two types of information to regularize the low-dimensional representation. Additionally, the way that the proposed model utilizes supervisory and geometric information can be extended to LCCF, CCF or LRCCF as well.

The main contributions of SAKCF lie in the following three-fold:

• SAKCF performs CF in an adaptively learned kernel space, where the limited supervisory and local geometrical information are encoded. The proposed model leverages the power of kernel and limited supervisory information, and such a semi-supervised learning framework can be naturally extended to several variants of CF (e.g., LCCF and CCF).

• SAKCF integrates the data representation and kernel learning in a unified model to make the two learning processes adapt to each other.

• We also propose an iterative updating optimization scheme for solving the optimization problem. The theoretic analysis including convergence and complexity analyses of the proposed algorithm is presented.

The rest of this paper is organized as follows. In Section 2, we briefly review the rationales of NMF, CF, LCCF, and CCF. In Section 3, we present the proposed SAKCF. We evaluate the performance of SAKCF when used for the clustering task in Section 4. Finally, Section 5 draws the conclusion of this paper.

Notation: Throughout this paper, we use lowercase letters, boldface lowercase letters and uppercase letters to denote scalars, vectors, and matrices, respectively. The elements of a matrix are denoted by lowercase letters with two subscripts, which indicate the row and column indices, respectively. For a matrix $A$, we use ${\parallel A\parallel }_F$, Tr($A$), and $A^{\mathsf{T}}$ to denote its Frobenius norm, trace, and transpose, respectively. ${\mathcal{N}}_k(·)$ indicates the set containing the $k$ nearest neighbors of a typical data sample. For some space $\mathcal{F}$, ${\parallel ·\parallel }_{\mathcal{F}}$ and $<·,·{>}_{\mathcal{F}}$ denote the distance metric and inner product in this space, respectively. $I_l$ denotes an identity matrix of size $l\times l$, and $\mathit{0}$ is a matrix with all entries being 0.