Cauchy greedy algorithm for robust sparse recovery and multiclass classification

Highlights

• We present a robust greedy algorithm called CauchyMP (Cauchy Matching Pursuit) for robust sparse recovery.

• We generalize CauchyMP for robust block and quaternion sparse recovery.

• We devise a CauchyMP based classifier for robust multiclass classification.

• Experimental results show that the proposed methods improve related competing algorithms with notable performance gains in presence of various gross corruption and outliers.

Abstract

Greedy algorithms have attracted considerable interest for sparse signal recovery (SSR) due to their appealing efficiency and performance recently. However, conventional greedy algorithms utilize the ℓ₂ norm based loss function and suffer from severe performance degradation in the presence of gross corruption and outliers. Furthermore, they cannot be directly applied to the recovery of quaternion sparse signals due to the noncommutativity of quaternion multiplication. To alleviate these problems, we propose a robust greedy algorithm referred as Cauchy matching pursuit (CauchyMP) for SSR and extend it for quaternion SSR. By leveraging the Cauchy estimator and generalizing it to the quaternion space to measure the residual error, our method can robustly recover the sparse signal in both real and quaternion space from noisy data corrupted by various severe noises and outliers. To tackle the resulting quaternion optimization problem, we develop an efficient half-quadratic optimization algorithm by introducing two quaternion operators. In addition, we have also devised a CauchyMP based classifier termed CauchyMPC for robust multiclass classification. The experiments on both synthetic and real-world datasets validate the efficacy and robustness of the proposed methods for SSR, block SSR, quaternion SSR and multiclass classification.

Introduction

Sparse representation (SR) has shown great potential in a variety of problems in signal processing and computer vision in the past decade [1]. For instance, SR methods have achieved great success in signal recovery [6], face recognition [1], and image restoration [8]. To be specific, given a dictionary matrix $\mathbf{D}\in {\mathbb{R}}^{m\times n}(m<n),$ SR aims to recover the target sparse signal ${\mathbf{x}}_0\in {\mathbb{R}}^n$ from its compressed measurement vector

$$\mathbf{y}=\mathbf{D}{\mathbf{x}}_0+\mathbf{n},$$

where $\mathbf{n}\in {\mathbb{R}}^m$ denotes the noise vector.

According to the mechanisms of inducing sparsity, existing SR approaches can be roughly divided into two categories: ℓ₁ minimization based approaches [5] and greedy algorithms [2]. A natural SR approach is to optimize the following problem related to the ℓ₀ constraint

$$\underset{\mathbf{x}}{\min }{\parallel \mathbf{y}-\mathbf{D}\mathbf{x}\parallel }_2^2\text{subject}\text{to}{\parallel \mathbf{x}\parallel }_0\le K.$$

where K denotes the sparsity parameter. However, the problem above is generally NP-hard due to the discontinuity and discrete nature of the ℓ₀. This makes it intractable to solve the problem (2) directly.To reduce such problem, ℓ₁ minimization based approaches consider the following surrogate optimization problem

$$\underset{\mathbf{x}}{\min }{\parallel \mathbf{y}-\mathbf{D}\mathbf{x}\parallel }_2^2+\lambda {\parallel \mathbf{x}\parallel }_1,$$

which is termed as Lasso [5]. Here the nonnegative scalar λ is the regularization parameter. While Lasso enjoys striking theoretical properties under appropriate conditions, most ℓ₁ minimization based approaches require heavy computation burden [6].

Unlike Lasso, greedy algorithms (GA) iteratively identify the indexes of nonzero entries of x₀ and estimate the sparse vector. Due to the low complexity and competitive performance, greedy algorithms have attracted increasing interest in recent years. Orthogonal matching pursuit (OMP) [2] is perhaps the most popular GA method because of its simplicity. Specifically, OMP identifies a column of the dictionary D in each iteration and estimates the sparse vector using the selected atoms in previous iterations. To improve the efficiency of OMP, the generalized OMP (GOMP) [2] selects multiple informative atoms in each iteration. The regularized OMP (ROMP) [4] algorithm tries to keep the advantages of both Lasso and OMP, i.e., the strong theoretical guarantees of Lasso and the high efficiency of OMP using a regularization rule. Analogously, the compressive sampling matching pursuit (CoSaMP) [3] also enhances OMP with an additional pruning step to provide strong theoretical guarantees that OMP cannot.

Despite their empirical success, most existing greedy algorithms explore the squared ℓ₂ norm as the loss function, which depends on the Gaussianity assumption of the noise distribution and sensitive to outliers. A violation of this assumption, e.g., missing entries, impulsive noise or random occlusions in face image data, may lead to severe performance degradation. To reduce the limitation, various robust SR approaches have been developed recently. The first category of robust SR methods aim to improve the robustness of lasso against outliers and gross corruptions. For instance, Carrillo and Barner [7] exploit the Lorentzian-norm to measure the residual error and introduce a geometric optimization problem for robust SR. Recent studies [8] have shown that leveraging the ℓ₁ norm based loss function in Lasso can lead to much better robustness compared with the conventional ℓ₂ norm and Lorentzian-norm. The resulting objective function is

$$\underset{\mathbf{x}}{\min }{\parallel \mathbf{y}-\mathbf{D}\mathbf{x}\parallel }_1+\lambda {\parallel \mathbf{x}\parallel }_1.$$

To solve the ${\ell }_1-{\ell }_1$ norm optimization problem above, many effective algorithms have been devised such as YALL1 [9]. In [10], the generalized ℓ_p-norm, 0 ≤ p < 2 is also adopted as the loss function for the residual error and the authors developed an alternating direction method based algorithm termed Lp-ADM for the corresponding optimization problem. Since these robust SR methods are still based on ℓ₁ norm regularization, these methods have heavy computational burden.

The second category of robust SR methods attempt to improve the robustness of greedy algorithms while keeping high efficiency. In [11], Razavi et al. attempt to robustify conventional greedy algorithms such as OMP by drawing on robust statistics and replacing least squares regression in OMP with robust regression. The robust OMP (RobOMP) [11] first calculates the so-called residual pseudo-values and selects a new atom which has the largest correlation with the residual pseudo-values in each iteration. In [12], Zeng et al. generalize conventional GA algorithms such as MP and OMP from inner product space to ℓ_p space (p > 0) for robust SR. The robust version of MP and OMP are called ℓ_p-MP and ℓ_p-OMP, respectively.

Another weakness of most existing SR methods is that they are designed for real or complex sparse recovery and cannot be directly applied to quaternion sparse signal recovery (QSSR). Because the product of two quaternions are noncommunicative in general, i.e., ${\dot{q}}_1{\dot{q}}_2\ne {\dot{q}}_2{\dot{q}}_1$. In addition, the definition and computation of the derivative (or gradient) of quaternion matrix function are much more complicated than those in ${\mathbb{R}}^n$ or ${\mathbb{C}}^n$ [13], [14]. This greatly increases the difficulty to tackle the quaternion optimization problems for QSSR.

In fact, quaternion has been widely used in various applications, including but not confined to, vector-sensor array signal processing [15], color face recognition [13], color image denoising, superresolution and inpainting [14]. Recent advances on quaternion image analysis [13], [14] show that quaternions are well adapted to color images by encoding the color channels into the three imaginary parts. In [14], Xu et al. extend OMP to the quaternion space and devise the quaternion OMP (QOMP) algorithm with application to color image restoration. In our previous work [13], we propose the quaternion Lasso (QLasso) model with quaternion ℓ₁ minimization for QSSR and color face recognition. However, both QOMP and QLasso rely on the quaternion ℓ₂ norm based loss function and may be sensitive to gross corruption and outliers.

In this paper, we develop a robust greedy algorithm referred as Cauchy Matching Pursuit (CauchyMP) by exploiting and generalizing the Cauchy estimator for robust sparse signal recovery (SSR) and quaternion SSR. By devising the half-quadratic theory [16] based optimization algorithm, CauchyMP can be viewed as an adaptive weighted OMP approach. The intuition behind CauchyMP is that it adaptively assign large weights on clean entries of y and small weights on noisy or outlying entries of y. Accordingly, the impact of corrupted entries and outliers can be well alleviated. The contributions of this work are summarized as below.

• We present a CauchyMP algorithm for robust sparse signal recovery. In the presence of gross corruption and outliers, CauchyMP can improve many prior greedy algorithms with notable performance gains.

• We generalize CauchyMP and devise the quaternion CauchyMP (QCauchyMP) algorithm for the recovery of quaternion sparse signals. Since the product of quaternions is noncommunicative in general, previous robust SR approaches cannot be directly applied to quaternion sparse signal recovery.

• We devise a CauchyMP based classifier termed CauchyMPC for robust multiclass classification and establish the theoretical analysis. Compared to the original sparse representation-based classification (SRC) [1], the proposed approach has better robust property and is more efficient.

The key differences between prior robust greedy algorithms (e.g., RobOMP and ℓ_p-OMP) and the proposed method lie in the following two aspects. First, the steps of identifying a new atom by these methods are different. Concretely, RobOMP identifies a new atom which has the largest correlation with the residual pseudo-values e_ψ and ℓ_p-OMP selects the atom that has the largest ℓ_p correlation with the residual. While CauchyMP selects a new atom most correlated with the residual in a reweighted version

$$j_k=\underset{j=1,\cdots ,n}{\text{arg}\max }\left|〈{\mathbf{d}}_j,{\mathbf{w}}_{k-1}\otimes \mathbf{r}〉\right|,$$

where ${\mathbf{w}}_{k-1}$ is the weight vector indicating the importance of each entry of r. For real vectors $\mathbf{u},\mathbf{v}\in {\mathbb{R}}^m,$ the inner product is defined as $〈\mathbf{u},\mathbf{v}〉={\mathbf{u}}^T\mathbf{v},$ while for quaternion vectors $\dot{\mathbf{u}},\dot{\mathbf{v}}\in {\mathbb{H}}^m,$ the inner product is defined by $〈\dot{\mathbf{u}},\dot{\mathbf{v}}〉={\dot{\mathbf{u}}}^H\dot{\mathbf{v}}$. Here ${\dot{\mathbf{u}}}^H={[{\overline{\dot{u}}}_1,\dots ,{\overline{\dot{u}}}_m]}^T$ denotes the conjugate transpose of $\dot{\mathbf{u}}$ and ${\overline{\dot{u}}}_i$ is the conjugate of the quaternion ${\dot{u}}_i$. Specifically, if the ith entry r_i of r is severely corrupted, it will receive small weight (the ith entry of ${\mathbf{w}}_{k-1},$ which is estimated adaptively) and the impact of noisy entries can be effectively suppressed. Thus the identification step of CauchyMP has clear and intuitive explanation. Second, the steps of estimating the sparse signal by these methods are also distinct. Specifically, both RobOMP and ℓ_p-OMP use the Iteratively Re-weighted Least Squares (IRLS) algorithm while CauchyMP estimates the sparse signal by the half-quadratic theory based optimization algorithm with guaranteed convergence.

The rest of this paper is organized as follows. In Section 2.2, we present the proposed methods for sparse signal recovery (SSR), block SSR, quaternion SSR and multiclass classification, respectively. Section 3 presents the experiments. Finally, Section 4 concludes the paper.

In this work, scalars, vectors and matrices are represented using italic letters (e.g., x), boldface lowercase letters (e.g., x), and boldface capital letters (e.g., X), respectively. For each vector $\mathbf{x}\in {\mathbb{R}}^n$ and an index set $\mathcal{J}\subset \{1,2,\dots ,n\},$ x_j denotes its jth entry and ${\mathbf{x}}_{\mathcal{J}}$ denotes a subvector of x containing entries indexed by the set $\mathcal{J}$. Analogously, for a matrix $\mathbf{X}\in {\mathbb{R}}^{m\times n},$ ${\mathbf{X}}_{\mathcal{J}}$ denotes a submatrix of X containing columns of X indexed by the set $\mathcal{J}$. Table 1 summarizes the key notations and acronyms used in this paper.