Robust subspace clustering network with dual-domain regularization

Abstract

The field of deep subspace clustering has advanced rapidly in recent years. Ideas such as self-expression and self-supervision have led to innovative network design and improved clustering performance. However, it is observed that the nonlinear low-dimensional manifold constraint is valid in not only the ambient but also the latent feature spaces. Meanwhile, the issue of robustness has been largely overlooked in the literature of deep subspace clustering despite previous studies on robust model-based clustering. Based on these two observations, we present a robust subspace clustering network (RSCN) based on a novel hybrid loss function with dual-domain regularization. On the one hand, we propose to replace the existing $L_2$ loss by a robust hybrid function inspired by half-quadratic minimization; on the other hand, we come up with a novel strategy of sparsity regularization in the dual domain (both ambient and feature space). To the best of our knowledge, this is the first attempt to incorporate dual manifold constraints into deep subspace clustering. Experimental results show that our new network outperforms the existing state-of-the-art on several widely-studied datasets such as Extended Yale B, COIL20, and COIL100. The performance gain of our RSCN over several other competing approaches improves dramatically in the presence of noise contamination.

Introduction

Subspace clustering - a problem of clustering high-dimensional data into multiple groups satisfying a subspace constraint – has been extensively studied in the past decades (please refer to the tutorial and reviewing articles [1], [7], [27], [36]). Early works [1], [27] focused on the curse of dimensionality and tackled the challenge by feature transformation techniques [27]. Subspace clustering can be viewed as an extension of feature selection targeting at finding clusters in well-structured subspaces [1]. Inspired by the ideas of sparse coding or sparse representation [5], [43], sparse subspace clustering (SSC) has been proposed to cluster data in [6], [7], [40] based on the observation that each point in a union of subspaces admits a sparse representation with respect to the dictionary formed by all other data points. This line of research was further developed by connecting with low-rank matrix decomposition [28], information theoretic formulation [12], orthogonal matching pursuit [47] and elastic net theory [46]. The framework of SSC has been extended into kernel SSC in [29] and structured SSC in [19].

Most recently, breakthroughs in deep learning [8], [17] have made a splash in the field of subspace clustering, leading to a flurry of deep subspace clustering networks [3], [4], [15], [30], [31], [32], [48], [50]. In [32], a deep learning model named PARTY was developed to progressively transform the input data into nonlinear latent space; in addition to being adaptive to the local and global subspace structure simultaneously, PARTY incorporates a sparsity prior in order to preserve the sparse reconstruction over the whole data set. In [15], the self-expressiveness property - originated from SSC [6] – was introduced into the architecture of deep convolutional auto-encoder, which produced a novel deep subspace clustering network (DSCN). In [50], the concept of adversarial learning [9] was leveraged into subspace clustering, leading to a deep adversarial subspace clustering (DASC) network. DASC consists of a subspace clustering generator and a quality-verifying discriminator, which learn against each other in a generative adversarial network (GAN)-like manner [9]. In [48], an end-to-end trainable framework consisting of three modules (feature learning, subspace clustering and self-supervision as shown in Fig. 1) was developed and demonstrate state-of-the-art performance on several well-known data sets. Most recently, multiscale fusion and multilevel representation have been studied for subspace clustering in [4], [16] respectively.

The motivation behind this paper is two-fold. On the one hand, it has been observed that not only the original high-dimensional image data reside on a nonlinear low-dimensional manifold in the ambient space, but also their feature representations lie on a manifold in the latent space [45]. Such observation has inspired the regularization strategy constructed on a dual graph aiming at simultaneously exploiting the geometric structures of both data manifold and feature manifold. Unfortunately, similar ideas have not been considered for the class of deep subspace clustering networks. On the other hand, the issue of robustness – even though it has been well addressed in conventional frameworks such as robust PCA (RPCA) [2], [42] and robust subspace clustering [34], has not been formally addressed under the framework of deep subspace clustering (with the only exception of [31]). The presence of outliers - either due to errors in observation data or the deviation from a linear subspace model - could have severe impact on the performance of subspace clustering techniques on real-world data [48].

Inspired by previous works of nonconvex penalty function [44] and half-quadratic minimization [49], we propose to replace the $L_1$/$L_2$ norm in the original DSCN by a robust loss function consisting of both $L_1$ and $L_2$ terms. In particular, the $L_1$-regularization term reflects the sparsity constraint on modeling errors, which is well aligned with the ideas in previous model-based approaches such as RPCA [2], [42]. Meanwhile, we suggest that such a hybrid loss function can be combined with dual-domain regularization for greater flexibility and robustness. As far as we know, this work represents the first attempt to leverage a hybrid robust loss function involving dual domain (both ambient domain and feature space) regularization into a deep learning-based subspace clustering framework. By contrast, the recent work on deep subspace clustering [31] considers $L_1$-norm in the ambient domain only.

We have validated the proposed robust subspace clustering network (RSCN) with dual domain regularization on several well-known datasets including Extended Yale B, COIL20, and COIL100. Experimental results have demonstrated that the proposed method outperforms existing state-of-the-art methods including DSC-Net [15] and Self-Supervised Convolutional Subspace Clustering Network [48] on most datasets. The gain (in terms of error rate reduction) improves as the number of different subjects increases, which verifies the effectiveness of the proposed network architecture. To further verify the robustness of RSCN to noise interference, we have artificially contaminated the dataset by additive white Gaussian noise. The performance gain of our RSCN over other competing methods is dramatically larger in the presence of noise contamination, which justifies the robustness of our approach.