Sparse Coding

Synonyms

Sparse representation

Definition

Sparse coding is the act of expressing a given input signal (e.g., image or image patch) as a linear superposition of a small set of basis signals chosen from a prespecified dictionary.

Background

At a high level, the problem of sparse coding is one of representing a given input signal as efficiently as possible:

Given an input signal ${\boldsymbol y} \in {\mathbb R}^m$ (say an image or image patch) and a dictionary of basis signals $\boldsymbol a_1 \dots \boldsymbol a_n \in {{\mathbb R}}^m$, find a good approximation

$$ \boldsymbol y \approx x_1 \boldsymbol a_1 + x_2 \boldsymbol a_2 + \dots + x_n \boldsymbol a_n $$

in which most of the coefficients x _i are zero.

That is, we try to represent y as a linear combination of basis elements in which only a few of the coefficients are nonzero (i.e., the vector x = (x₁ … x _n ) is sparse). This deceptively simple problem arises repeatedly in signal processing, modern statistics, and machine...