Confidence level auto-weighting robust multi-view subspace clustering

Abstract

For the subspace clustering task of multi-view data, it is an important research content to mine the complementary feature across all views. However, when learning the consensus representation of all views, each view may have different confidence levels. In addition, owing to the non-linearity and noise pollution of the data, different samples in the same view may have different confidence levels. Unfortunately, most of the existing methods only assign a uniform weight to each view, and may only obtain a suboptimal solution. In this work, we put forward a confidence level auto-weighting robust multi-view subspace clustering (CLWRMSC) model. Specifically, we designed an adaptive sample weighting strategy that enables our model to focus on the confidence of both views and samples while learning the consensus representation of all views. At the same time, using the self-expression property of subspace, an adaptive low-rank multi-kernel learning (MKL) strategy is designed. Further, the Weighted Truncated Schatten p-Norm (WTSN) is proposed to better solve the optimization problem of low-rank constraints. A lot of experiments prove that the proposed model is a superior clustering algorithm.

Introduction

In reality, the rapid growth of data scale makes it very difficult to directly process and understand raw data. Fortunately, by using the data structure characteristics to mine the compact representation of data, we can understand the raw data at minimal cost. As we all know, high-dimensional data can usually be obtained by modeling multiple samples of linear subspaces. So, subspace clustering has been widely used [1], [2], [3]. Recently, scholars have proposed various subspace clustering algorithms, such as statistical-based, iterative-based, algebraic-based, and spectral-based methods. Among them, the spectral-based methods have received extensive attention because of their effectiveness for data clustering. The typical ones are Sparse Subspace Clustering (SSC) [4], Low-Rank Representation (LRR) [5] and Block Diagonal Representation (BDR) [6]. They have been applied to image classification, motion segmentation, document clustering and so on. In general, the key point to improve clustering performance is how to use standard spectral algorithm to construct a robust affinity matrix (or graph). This is also the goal of the proposed algorithm. Before proceeding, let us introduce our notation conventions in Table 1.

For linear subspaces, data in the same subspace can be represented linearly with each other, i.e., subspace self-expression property, which can be mathematically expressed as:

$${\displaystyle \underset{\mathit{Z}}{\min }}\frac{1}{2}\Vert \mathit{X}-\mathit{XZ}{\Vert }_F^2+\lambda \mathfrak{R}(\mathit{Z})s.t.{\mathit{Z}}_{\mathit{ii}}=0$$

where $\lambda >0$ is called trade-off parameter, $\mathfrak{R}(\mathit{Z})$ is called penalty (or regularization) term. There are currently many options for $\mathfrak{R}(\mathit{Z})$, such as $\Vert \mathit{Z}{\Vert }_1$ [7], [8], $\Vert \mathit{Z}{\Vert }_{\ast }$ [9], [10], $\Vert \mathit{Z}{\Vert }_F^2$ [11], $\Vert \mathit{Z}{\Vert }_k$ [6] and so on.

To better treat the non-linearity and improve the accuracy of data reconstruction, some kernel-based SSC (KSSC) methods are designed [12], which perform clustering tasks in the linear feature space obtained by mapping the original data, i.e.,

$${\displaystyle \underset{\mathit{Z}}{\min }}\frac{1}{2}\Vert \varphi (\mathit{X})-\varphi (\mathit{X})\mathit{Z}{\Vert }_F^2+\lambda \mathfrak{R}(\mathit{Z})s.t.{\mathit{Z}}_{\mathit{ii}}=0$$

This problem is equivalent to

$${\displaystyle \underset{\mathit{Z}}{\min }}\frac{1}{2}\mathit{tr}[(\mathit{I}-2\mathit{Z}-{\mathit{ZZ}}^{\top })\mathit{K}]+\lambda \mathfrak{R}(\mathit{Z})s.t.{\mathit{Z}}_{\mathit{ii}}=0$$

where $\mathit{K}={\varphi (\mathit{X})}^{\top }\varphi (\mathit{X})$ is the unknown kernel Gram matrix.

In order to ensure that the space obtained by using the kernel trick is composed of multiple low-dimensional subspaces, LrKSC [13] adds a low-rank kernel constraint $\Vert \varphi (\mathit{X}){\Vert }_{\ast }$ within the SSC framework. Zhang et al. [14], [15] use more advanced low-rank constraint methods to optimize LrKSC (i.e., $\Vert ·{\Vert }_{S_p}^p$ and $\Vert ·{\Vert }_{\mathit{w},{\mathit{S}}_{\mathit{p}}}^p$), and then design two superior subspace clustering algorithms. Unfortunately, these methods are designed based on single kernel learning, so their dependence on kernel selection limits their clustering performance.

As a result, some scholars optimize their algorithms and propose multiple kernel learning (MKL) methods [16], [17], [18]. In recent years, as the superiority of MKL technology has been continuously highlighted, its application in subspace clustering tasks has become more and more extensive [19], [20], [21]. The advantages of MKL are usually achieved by assigning appropriate weights to each basic kernel. In [22], a spectral clustering model (AASC) is proposed by adopting the multiple affinity matrices, which is a successful attempt of multi-kernel clustering. Combining the MKL and K-means algorithms, [23] designs a robust multi-kernel k-means (RMKKM) method, which can independently select the appropriate kernel from the basic kernel pool and improve the efficiency of kernel design or selection. Inspired by this, an advanced method is designed and used for clustering tasks (SMKL) [24]. Similar to RMKKM, Kang et al. design a MKL clustering method (SCMK) by learning an optimal kernel [25]. Aiming the robustness of the model, Yang et al. [26] design a correntropy metric weighting method for MKL and achieve ideal clustering results. Paying full attention to the inner neighborhood structure between the base kernels, a neighbor-kernel-based MKC algorithm is proposed [27]. In short, a large number of research results prove that the learning effect and agility of MKL are better than single-kernel learning. Unfortunately, these multi-kernel clustering methods do not pay special attention to whether the data space obtained by using the kernel trick contains multiple low-dimensional subspaces.

Because multi-view data describes characteristics of the same target, there is naturally a strong correlation between them. So, fully mining the inherent features of data can effectively enhance the capability of multi-view subspace clustering (MSC) algorithms. Many scholars have done a lot of researches for this purpose [28]. Using collaborative learning, a consensus representation is obtained successfully between views (Co-Reg) [29]. Of course, a large amount of redundant features in the data will reduce the stability of clustering. In this regard, a robust clustering method (RMSC) is designed via applying Markov chain [30]. By using a common clustering structure to ensure consistency between different views, Gao et al. [31] propose to simultaneously cluster the subspace representations of each view. Solving models efficiently is also a concern of researchers, for this, Lu et al. [32] present a tight relaxation for SSC and propose a convex pairwise SSC model (CSMSC). Inspired by Co-Reg, [33] simultaneously performs $\Vert ·{\Vert }_{\ast }$ and $\Vert ·{\Vert }_1$ on the obtained consensus representation matrix (MLRSSC). To suppress noise in data and deal with nonlinear structures, a low-rank kernel-based MSC model (RLKMSC) is proposed, which has good clustering performance and robustness [34]. In order to overcome the influence of noise and improve the ability of mining the underlying subspace structure of the model, a robust energy preserving embedding (REPE) method is proposed [35]. However, existing methods only assign a uniform weight to each view, but the damage degree of each sample in the same view may be different, which means that the confidence of each sample may be different (for example, the confidence levels of outliers or noisy samples should be lower than that without being destroyed), therefore, considering the uniform weight of a view may lead to a suboptimal solution.

Recently, with the rise of deep learning (DL) again, many researchers have applied it to single-view subspace clustering tasks. In [36], based on the deep autoencoder, a deep subspace clustering networks (DSC-Nets) is proposed, which creatively introduces a novel “self-representation layer” between the encoder and the decoder. Peng et al. [37] propose a deep extended structure of sparse subspace clustering, called ${\ell }_1$ norm deep subspace clustering. Of course, some scholars have designed some deep subspace clustering networks for multi-view data [38], [39]. These methods have achieved amazing results. Generally speaking, the clustering performance of DL-based methods is better than traditional methods. However, the DL-based methods have higher requirements for the number of samples and computer hardware, which limits their applicability. The focus of this work is different from the DL-based methods, so we will not discuss too much about DL-based methods here.

In conclusion, there are many studies on dealing with non-linear structures and complex noises in data, but there is not yet a method to solve the problem that can simultaneously deal with the problem that samples may have different confidence levels under the same view. Therefore, we aim to design a novel robust MSC model that can learn the weights of the samples and consensus representation jointly. Considering the nonlinearity of high-dimensional data and the possibility of noise pollution, we propose a weighted truncated Schatten p-norm (WTSN) regularization, and design an adaptive low-rank multi-kernel learning strategy via WTSN and mixture correntropy [40].

Simply put, the main contributions are the following four aspects:

• To highlight the confidence level of pure samples, we design a unique weighting strategy to consider the confidence of both views and samples.

• To better cope with the nonlinearity and noise pollution of data, we design an advanced MKL strategy. The weights of all basic kernels are automatically allocated based on their similarity to the consensus kernel.

• To obtain a better low-rank approximation, we propose a weighted truncated Schatten p-norm (WTSN), and give the corresponding optimization procedure.

The remaining parts of this paper include three sections. Section 3 provides the relevant preliminary knowledge used in this work. Section 3 proposes the model CLWRMSC, and introduces its’ optimization and solution. Meanwhile, the complexity and convergence of the model are analyzed. Section 4 introduces the relevant experiments in detail, and analyzes the results carefully. Section 5 summarizes this work.