Weighted Schatten p-norm minimization with logarithmic constraint for subspace clustering

doi:10.1016/j.sigpro.2022.108568

Signal Processing

Volume 198, September 2022, 108568

https://doi.org/10.1016/j.sigpro.2022.108568 Get rights and content

Abstract

Rank minimization-based subspace clustering methods have been widely developed in the past decades. Although some smooth surrogates, such as the nuclear norm and Schatten- $p$ norm mitigate the NP-hard issue to some extend, these existing methods may yield unsatisfactory results, due to the singular values of the coefficient matrix not being further suppressed. To tackle this, in this paper, we propose a novel non-convex low-rank approximation based on weighted Schatten- $p$ norm jointed logarithmic constraint, which can suppress the small and large singular values flexibly with a tighter relaxation. Specifically, we firstly proposed a low-rank approximation termed SLog by utilizing the logarithmic to tighten the Schatten- $p$ norm, which can shrink the large singular values in a similar trend of the real rank minimization. Furthermore, to suppress the small singular values simultaneously, usually considered noise, we propose a weighted Schatten $p$ -norm minimization named WSLog based on SLog by introducing the weight $w$ , which behaves more robust to the sparse noise, especially $w < 1$ . Compared with recent-proposed methods, extensive experiments in subspace clustering on real datasets demonstrate the effective performance of our methods.

Introduction

Subspace clustering has significantly facilitated image processing tasks such as face clustering [1] and motion segmentation [2], [32], whose purpose is to explore the multiple underlying low-dimensional subspaces and segment the high-dimensional data into these relative low-dimensional subspaces. Over the past few decades, numerous subspace clustering algorithms have been extensively studied. Most of them can roughly fall into four categories: factorization-based methods [5]; algebraic methods [3], [4]; statistical methods [6]; self-expressive methods [8], [9], [10], [11]. In this paper, we focus on the self-expressive methods owing to its promising property.

In self-expressive methods, each data sample drawn from a union of low-dimensional subspaces can be linearly represented by samples in the same subspace. To learn a more discriminative affinity matrix, the self-expressive methods commonly employ the sparse or low-rank prior as the regularization of the coefficient matrix. For example, the SSC [7] get a sparse coefficient matrix, which utilize the $l_{1}$ norm. However, the coefficient matrix in SSC may be too sparse to poor result, because it uses only the local structure. To overcome this, many self-expressive methods based on low-rank representation have been widely developed, which integrate the global information, such as LRR [10], FLRR [22], WNNM [34]. Instead of directly solving the rank function with NP-hard complexity, these low-rank methods replace it with the nuclear norm, which is the convex envelope of the rank function over the unit ball of matrices [12]. However, the results of the nuclear-norm minimization may deviate from that of the real rank minimization. That is because the real rank minimization considers that the contributions of all nonzero singular values of the coefficient matrix are equal, while the nuclear-norm minimization simply adds all the nonzero singular values [13].

To conquer the above issue, some non-convex low-rank surrogates [14], [15], [16], [17] have been developed to further replace the nuclear norm. These low-rank methods treat the nonzero singular values in a soft way, where the contributions of the large singular values are declined compared with the nuclear-norm minimization. For example, the Schatten- $p$ norm [15], [16], [18] and the logarithm of the determinant [13], [19], [20] are two representative non-convex low-rank surrogates. In detail, Schatten- $p$ norm achieves to apply more suppression on the large singular values. By adjusting the parameter $p$ , a family of non-convex low-rank surrogates can be obtained, which are between the nuclear norm and real rank function. For example, the Schatten- $p$ norm can be treated as the rank function at $p = 0$ . Nevertheless, the Schatten- $p$ norm will promote the contribution of the small singular values that are generally regarded as noise. As for the logarithm of the determinant, it not only can reduce the contribution of large singular values, but also urge the small singular values to zero. However, the logarithm of the determinant, as one non-convex low-rank surrogate, can not infinitely approximate the rank function. Besides, these non-convex low-rank surrogates may yield unsatisfactory results, because they ignore that the singular values of the coefficient matrix can be suppressed to a greater degree.

Motivated by the advantages and weaknesses of the Schatten- $p$ norm and the logarithm of the determinant, we propose two novel non-convex low-rank surrogates named SLog and WSLog. That is, our methods can combine the advantages of the two previous methods and make up for their weaknesses at the same time. On the other hand, combining the Schatten-p norm and the logarithm of the determinant can suppress the small and large singular values flexibly with a greater degree. Specifically, we firstly proposed SLog by utilizing the logarithmic to tighten the Schatten- $p$ norm. As shown in Fig. 1(a), our method SLog can approximate the rank function more closely in comparison with other surrogates. Meanwhile, to suppress the small singular values simultaneously, we propose a weighted Schatten $p$ -norm minimization named WSLog based on SLog by introducing the weight $w$ . Our method WSLog can shrink the small singular values to a greater degree, especially when $w < 1$ . Extensive experimental results demonstrate that the two proposed methods achieve comparable and even better accuracy. Our contributions are summarized as follows:

1) We firstly propose SLog by utilizing the logarithmic to tighten the Schatten- $p$ norm, which can shrink the large singular values in a similar trend of the real rank minimization, and approximate the rank function more closely.

2) To shrink the small singular values to a greater degree simultaneously, we propose a weighted Schatten $p$ -norm minimization named WSLog based on SLog by introducing the weight $w$ to mitigate the negative contribution of noise, especially $w < 1$ .

3) We present an optimization algorithm to solve SLog and WSLog problems. Extensive experiments demonstrate that the proposed methods outperform other methods across most of datasets.

Section snippets

Related works

Given a data matrix $X = [x_{1}, \dots, x_{n}] \in R^{m \times n}$ , $n$ denotes the number of samples and each sample $x_{i}$ is represented by an $m$ -dimensional column vector (i.e., $x_{i} \in R^{m}$ ). The low-rank problem for subspace clustering can be written as $min_{Z, E} r a n k (Z) + λ ∥ E ∥_{2, 1}, s.t., X = X Z + E .$ where the low-rank representation $Z \in R^{n \times n}$ denotes the coefficient matrix. $E \in R^{m \times n}$ is introduced as sparse noise, which is often regularized by $ℓ_{2, 1}$ norm, such as $∥ E ∥_{2, 1} = \sum_{j = 1}^{n} ∥ e_{j} ∥_{2}$ , where $e_{j}$ is the $j$ th column of $E$ . $λ$ represents the balanced parameter

Construction of SLog

In fact, the existing methods of non-convex low-rank approximation including Schatten- $p$ norm and the logarithm of the determinant ignore that the large singular values of the coefficient matrix should be suppressed with a greater degree, which may not be capable of exploiting the subspace structures of data faithfully, and hence may produce the unsatisfactory results. Thus, there is a need to design a closer surrogate function of the real rank function, which is expected to suppress these large

Connection to related works

Recently, Peng et al. [13] proposed the Log-det, which is written as $\begin{matrix} min_{Z, S, B, E} \sum_{i = 1}^{n} \log (σ_{i}^{2} + 1) + α ∥ S ∥_{ℓ} + β ∥ E ∥_{F}, \\ s.t., B = B Z, A = B + S + E . \end{matrix}$ In problem (20), the data matrix $A = B + S + E$ , where $B$ is the unknown underlying clean data matrix, $S$ denotes the sparse error, and $E$ represents the Gaussian noise. $Z$ denotes the low-rank representation, and $σ_{i}$ is the $i$ th singular value of $Z$ . $∥ \cdot ∥_{ℓ}$ denotes a proper norm to handle sparse noise or outliers. $α$ and $β$ are two positive balancing parameters.

Connection to Log-det: Both

Experiments

In this section, we conduct extensive experiments to evaluate the effectiveness of the proposed methods, which are conducted on an Intel Core i5-8500 workstation with 3.00-GHz CPU and 16-GB memory. In these experiments, we compare our methods with the classical LRR methods based on nuclear norm (i.e., LRR [10] and FLRR [22], where FLRR is a fast version of LRR.), and the non-convex low-rank approximations (i.e., Schatten- $p$ norm [15], [16], FULRR [11], RMC-NC [20], and Log-det [13]). For the

Conclusions

In this paper, we proposed a novel non-convex low-rank approximation by applying the weighted Schatten- $p$ norm with logarithmic constraint, which can suppress the small and large singular values flexibly with a greater degree, and be more powerful to exploit the subspace structures. Specifically, we firstly proposed SLog by utilizing the logarithmic to tighten the Schatten- $p$ norm, which can shrink the large singular values in a similar trend of the real rank minimization. Furthermore, to shrink

CRediT authorship contribution statement

Qiangqiang Shen: Writing – original draft, Conceptualization, Methodology, Software. Yongyong Chen: Writing – review & editing, Validation, Visualization. Yongsheng Liang: Supervision. Shuangyan Yi: Data curation, Software. Wei Liu: Software, Investigation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (34)

M. Abdolali et al.
Scalable and robust sparse subspace clustering using randomized clustering and multilayer graphs
Signal Process.
(2019)
X. Zhang et al.
Multiple kernel low-rank representation-based robust multi-view subspace clustering
Inf. Sci.
(2021)
Y. Guo et al.
Efficient sparse subspace clustering by nearest neighbour filtering
Signal Process.
(2020)
Y. Wang et al.
Provable subspace clustering: when LRR meets SSC
IEEE Trans. Image Process.
(2019)
P. Ji et al.
Shape interaction matrix revisited and robustified: efficient subspace clustering with corrupted and incomplete data
Proc. ICCV
(2015)
Y. Ma et al.
Estimation of subspace arrangements with applications in modeling and segmenting mixed data
SIAM Rev.
(2008)
A. Gruber et al.
Multibody factorization with uncertainty and missing data using the em algorithm
Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(2004)
S. Rao et al.
Motion segmentation in the presence of outlying, incomplete, or corrupted trajectories
IEEE Trans. Pattern Anal. Mach. Intell.
(2010)
E. Elhamifar et al.
Sparse subspace clustering: algorithm, theory, and applications
IEEE Trans. Pattern Anal. Mach. Intell.
(2013)
P. Farhad et al.
Efficient solvers for sparse subspace clustering
Signal Process.
(2020)

G. Liu et al.

Robust subspace segmentation by low-rank representation

Proc. 27rd Int. Conf. Mach. Learn

(2010)

Q. Shen et al.

Fast universal low rank representation

IEEE Trans. Circuits Syst. Video Technol.

(2021)

M. Fazel

Matrix rank minimization with applications

(2002)

C. Peng et al.

Subspace clustering using log-determinant rank approximation

Proc. 19th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining.

(2015)

Q. Yao et al.

Large-scale low-rank matrix learning with nonconvex regularizers

IEEE Trans. Pattern Anal. Mach. Intell.

(2019)

D. Kong et al.

Minimal shrinkage for noisy data recovery using Schatten- $p$ norm objective

Proc. ECML PKDD

(2013)

F. Nie et al.

Low-rank matrix recovery via efficient Schatten p-norm minimization

Proc. AAAI Conf. Artif. Intell.

(2012)

Cited by (8)

Faster nonconvex low-rank matrix learning for image low-level and high-level vision: A unified framework
2024, Information Fusion
This study introduces a unified approach to tackle challenges in both low-level and high-level vision tasks for image processing. The framework integrates faster nonconvex low-rank matrix computations and continuity techniques to yield efficient and high-quality results. In addressing real-world image complexities like noise, variations, and missing data, the framework exploits the intrinsic low-rank structure of the data and incorporates specific residual measurements. The optimization problem for low-rank matrix learning is effectively solved using the nonconvex Proximal Block Coordinate Descent (PBCD) algorithm, resulting in nearly unbiased estimators. Rigorous theoretical analysis ensures both local and global convergence. The PBCD algorithm updates blocks of variables iteratively with closed-form solutions, adeptly handling nonconvexity and promoting faster convergence. Notably, the framework incorporates the randomized singular value decomposition (RSVD) technique and introduces a continuous strategy for adaptive model parameter updates. These strategic choices reduce computational complexity while maintaining result quality. They offer fine-tuned control over the desired rank of the learned matrix and enhance robustness in a straightforward manner. Furthermore, the versatility of the proposed nonconvex PBCD algorithm extends to handling problems with multiple variables, as supported by theoretical analysis. Experimental evaluations, spanning various image low-level and high-level vision tasks such as inpainting, classification, and clustering, validate the effectiveness and efficiency of our framework across diverse databases. The source code is available at https://github.com/ZhangHengMin/FNPBCD_LR.
In a nutshell, our framework provides a unified solution to tackle both low-level and high-level vision tasks in images. By combining fast nonconvex low-rank matrix learning with adaptive parameter updates, we achieve efficient computation, yielding high-quality results that demonstrate robustness against various types of noise. The evaluations further endorse the reliability and applicability of our proposed framework.
Quaternion tensor completion with sparseness for color video recovery
2024, Applied Soft Computing
A novel low-rank completion algorithm based on the quaternion tensor is proposed in this paper. This approach uses the TQt-rank of quaternion tensor to maintain the structure of RGB channels throughout the entire process. In more detail, the pixels in each frame are encoded on three imaginary parts of a quaternion as an element in a quaternion matrix. Each quaternion matrix is then stacked into a quaternion tensor. A logarithmic function and truncated nuclear norm are employed to characterize the rank of the quaternion tensor in order to promote the low rankness of the tensor. Moreover, by introducing a newly defined quaternion tensor discrete cosine transform-based (QTDCT) regularization to the low-rank approximation framework, the optimized recovery results can be obtained in the local details of color videos. In particular, the sparsity of the quaternion tensor is reasonably characterized by $l_{1}$ norm in the QDCT domain. This strategy is optimized via the two-step alternating direction method of multipliers (ADMM) framework with convergence analysis. Numerical experimental results for recovering color videos show the obvious advantage of the proposed method over other potential competing approaches.
LatLRR for subspace clustering via reweighted Frobenius norm minimization
2023, Expert Systems with Applications
Subspace clustering has attracted much attention on account of its excellent performance in various tasks, such as computer vision and pattern recognition. Since the high-dimensional data may comprise redundant information and noises, low-dimensional subspace structure is difficult to be recovered exactly. Existing methods adopt the non-convex Schatten- $p$ norm to approximate the rank function closer. However, these methods are still unable to handle each singular value flexibly, which may yield unsatisfactory solutions. To address this deficiency, in this paper, we propose a iterative reweighted Frobenius norm regularized latent low rank representation (IRFLLRR) model, which can restrain the small and large rank components simultaneously with rational weights. Specifically, the reweighting strategy can preserve the desirable structure information, while removing the sparse noise and redundant information. Hence the constructed coefficient matrix can capture the global structure of data more adequately. To solve the proposed model, an efficient algorithm is developed via the alternating direction method of multipliers (ADMM), which ensures each subproblem can be optimized in closed-form. In addition, we provide detailed mathematical proofs to guarantee that the sequence generated by our algorithm convergences to a Karush–Kuhn–Tucker (KKT) point. Extensive experimental results reveal the superiority of the proposed method over several state-of-the-art methods. MATLAB code is available at https://github.com/wangzhi-swu/IRFLLRR.
Unified Framework for Faster Clustering via Joint Schatten p-Norm Factorization With Optimal Mean
2024, IEEE Transactions on Neural Networks and Learning Systems
Image Segmentation Based on Fuzzy Low-Rank Structural Clustering
2023, IEEE Transactions on Fuzzy Systems
Multi-Mode Tensor Factorization for Image and Video Inpainting
2023, SSRN

View all citing articles on Scopus

^☆: This work was supported in part by the National Natural Science Foundation of China under Grant No. 62031013, No. 61871154, and No. 62106063, Shenzhen College Stability Support Plan under Grant No. GXWD20201230155427003-20200824113231001, Youth Program of National Natural Science Foundation of China under Grant No. 61906124, and Basic and Applied Basic Research Fund of Guangdong Province under Grant No. 2019A1515011307.

View full text

Weighted Schatten p-norm minimization with logarithmic constraint for subspace clustering☆

Abstract

Introduction

Section snippets

Related works

Construction of SLog

Connection to related works

Experiments

Conclusions

CRediT authorship contribution statement

Declaration of Competing Interest

Signal Process.

Inf. Sci.

Efficient sparse subspace clustering by nearest neighbour filtering

Signal Process.

Provable subspace clustering: when LRR meets SSC

IEEE Trans. Image Process.

Shape interaction matrix revisited and robustified: efficient subspace clustering with corrupted and incomplete data

Proc. ICCV

Estimation of subspace arrangements with applications in modeling and segmenting mixed data

SIAM Rev.

Multibody factorization with uncertainty and missing data using the em algorithm

Proc. IEEE Conf. Comput. Vis. Pattern Recognit.

Motion segmentation in the presence of outlying, incomplete, or corrupted trajectories

IEEE Trans. Pattern Anal. Mach. Intell.

Sparse subspace clustering: algorithm, theory, and applications

IEEE Trans. Pattern Anal. Mach. Intell.

Efficient solvers for sparse subspace clustering

Signal Process.

Robust subspace segmentation by low-rank representation

Proc. 27rd Int. Conf. Mach. Learn

Fast universal low rank representation

IEEE Trans. Circuits Syst. Video Technol.

Matrix rank minimization with applications

Subspace clustering using log-determinant rank approximation

Proc. 19th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining.

Large-scale low-rank matrix learning with nonconvex regularizers

IEEE Trans. Pattern Anal. Mach. Intell.

Minimal shrinkage for noisy data recovery using Schatten-p norm objective

Proc. ECML PKDD

Low-rank matrix recovery via efficient Schatten p-norm minimization

Proc. AAAI Conf. Artif. Intell.

Weighted Schatten $p$ -norm minimization with logarithmic constraint for subspace clustering☆

Minimal shrinkage for noisy data recovery using Schatten- $p$ norm objective