On a simple derivation of the complementary matching pursuit

Abstract

Sparse approximation in a redundant basis has attracted considerable attention in recent years because of many practical applications. The problem basically involves solving an under-determined system of linear equations under some sparsity constraint. In this paper, we present a simple interpretation of the recently proposed complementary matching pursuit (CMP) algorithm. The interpretation shows that the CMP, unlike the classical MP, selects an atom and determines its weight based on a certain sparsity measure of the resulting residual error. Based on this interpretation, we also derive another simple algorithm which is seen to outperform CMP at low sparsity levels for noisy measurement vectors.

Introduction

In this paper, we consider the problem of solving the following under-determined system of equations:

$$A\mathbf{x}=\mathbf{b}\text{,}$$

where the matrix A is of dimension $K\times N$, $K<N$, and $\mathbf{b}$ is some non-zero observation vector. The columns of A span the K-dimensional vector space ${\mathbb{R}}^K$. The system has infinite number of solutions, but here we are concerned with the solution vector $\mathbf{x}$ that has the minimum number of non-zero elements. In other words, we are interested in representing the given vector $\mathbf{b}$ as a linear sum of the fewest columns of A weighted by the non-zero elements of the solution vector $\mathbf{x}$. This problem is commonly known as sparse representation or approximation in the signal processing literature. The sparse representation problem thus can be posed as

$$\min \{\parallel \mathbf{x}{\parallel }_0:A\mathbf{x}=\mathbf{b}\}\text{,}$$

where the $L_0$ norm denotes the number of non-zero elements. The sparse approximation problem, which allows some approximation error, can be posed as

$$\min \{\parallel \mathbf{x}{\parallel }_0:\parallel A\mathbf{x}-\mathbf{b}{\parallel }_2\le \delta \}$$

for some $\delta >0$. The signals which represent the columns of the matrix A are commonly called atoms and the set of atoms is called a dictionary. Since the signal set is larger than necessary (i.e., $N>K$) and the signals span ${\mathbb{R}}^K$, the signals in the dictionary are said to form a redundant basis.

The exact solutions to above problems can be found through combinatorial optimization methods. Since such methods require high computational complexity, most of the research activity in this area is focused on finding approximate solutions with tractable complexity. In the literature, there are basically two approaches to arrive at a sub-optimal solution. One is a greedy approach which approximates the signal vector through a sequence of incremental approximations by selecting atoms suitably. Such an approach is known as matching pursuit (MP) [1], [2]. The other is known as the basis pursuit (BP), which relaxes the $L_0$ norm condition by replacing it by the $L_1$ norm and solves the problem through linear programming [3], [4], [5]. BP algorithms can produce more accurate solutions than the matching pursuit algorithms but require higher complexity. Recently, some other iterative algorithms such as the regularized orthogonal matching pursuit (ROMP) [6], the compressive sampling matching pursuit (CoSaMP) [7], and the subspace pursuit (SP) [8] have been proposed. These algorithms aim to provide the same guarantee as the BP but with computational complexity akin to orthogonal matching pursuit (OMP) [2], [9]. Another approach called gradient pursuit [10] is similar to the matching pursuit but updates the sparse solution vector at each iteration with a directional update computed based on the gradient or the conjugate gradient.

Recently Rath and Guillemot [11] have proposed the complementary matching pursuit (CMP) which is similar to the classical matching pursuit, but is performed in the coefficient space rather than the signal space. Through simulation results, they showed that by performing approximations in the coefficient space, the convergence speed and the sparsity of resulting vectors are improved. The improved performance of CMP over MP can be understood in a mathematically rigorous manner through the sensing dictionary formalism of Schnass and Vandergheynst [12]. In this paper, we provide a simple interpretation of the CMP based on the sparsity of the residual error. We believe that this provides us an intuitive understanding of the higher performance of the CMP over MP in a noiseless scenario. Following this derivation, we then present a simple pursuit algorithm which selects atoms based on the sparsity of the residual error resulting from MP algorithm. We compare it with the orthogonal CMP (OCMP) and the sensing dictionary method of Schnass and Vandergheynst [12] through simulation.