Structured Overcomplete Sparsifying Transform Learning with Convergence Guarantees and Applications

Abstract

In recent years, sparse signal modeling, especially using the synthesis model has been popular. Sparse coding in the synthesis model is however, NP-hard. Recently, interest has turned to the sparsifying transform model, for which sparse coding is cheap. However, natural images typically contain diverse textures that cannot be sparsified well by a single transform. Hence, in this work, we propose a union of sparsifying transforms model. Sparse coding in this model reduces to a form of clustering. The proposed model is also equivalent to a structured overcomplete sparsifying transform model with block cosparsity, dubbed OCTOBOS. The alternating algorithm introduced for learning such transforms involves simple closed-form solutions. A theoretical analysis provides a convergence guarantee for this algorithm. It is shown to be globally convergent to the set of partial minimizers of the non-convex learning problem. We also show that under certain conditions, the algorithm converges to the set of stationary points of the overall objective. When applied to images, the algorithm learns a collection of well-conditioned square transforms, and a good clustering of patches or textures. The resulting sparse representations for the images are much better than those obtained with a single learned transform, or with analytical transforms. We show the promising performance of the proposed approach in image denoising, which compares quite favorably with approaches involving a single learned square transform or an overcomplete synthesis dictionary, or gaussian mixture models. The proposed denoising method is also faster than the synthesis dictionary based approach.

Appendix: Useful Lemmas

Here, we list three results (from (Ravishankar and Bresler 2014)) that are used in our convergence proof. The following result is from the Appendix of (Ravishankar and Bresler 2014).

Lemma 10

Consider a bounded vector sequence $\left\{ \alpha ^{k} \right\}$ with $\alpha ^{k} \in \mathbb {R}^{n}$, that converges to $\alpha ^{*}$. Then, every accumulation point of $\left\{ H_{s}(\alpha ^{k}) \right\}$ belongs to the set $\tilde{H_{s}}(\alpha ^{*})$.

The following result is based on the proof of Lemma 6 of (Ravishankar and Bresler 2014).

Lemma 11

Let $\left\{ W^{q_t}, X^{q_t} \right\}$ with $W^{q_t} \in \mathbb {R}^{n \times n}$, $X^{q_t} \in \mathbb {R}^{n \times N}$, be a subsequence of $\left\{ W^{t}, X^{t} \right\}$ converging to the accumulation point $(W^{*}, X^{*})$. Let $Z \in \mathbb {R}^{n \times N}$ and $L^{-1} = \left( ZZ^{T} + \lambda I \right) ^{-1/2}$, with $\lambda >0$. Further, let $Q^{q_t} \Sigma ^{q_t} \left( R^{q_t} \right) ^{T}$ denote the full singular value decomposition of $L^{-1} Z (X^{q_t})^{T}$. Let

$$\begin{aligned} W^{q_{t} + 1} = \frac{R^{q_{t}}}{2} \left( \Sigma ^{q_{t}} + \left( \left( \Sigma ^{q_{t}} \right) ^{2}+2\lambda I \right) ^{\frac{1}{2}}\right) \left( Q^{q_{t}} \right) ^{T}L^{-1} \end{aligned}$$

and suppose that $\left\{ W^{q_{t} + 1} \right\}$ converges to $W^{**}$. Then,

$$\begin{aligned} W^{**} \in \arg \min _{W} \left\| WZ-X^{*} \right\| _{F}^{2}+\lambda \left\| W \right\| _{F}^{2}- \lambda \log \,\left| \mathrm {det \,} W \right| \end{aligned}$$

(53)

The following result is based on the proof of Lemma 9 of (Ravishankar and Bresler 2014). Note that $\phi (X)$ is the barrier function defined in Section 4.

Lemma 12

Given $Z \in \mathbb {R}^{n \times N_{1}}$, $\lambda >0$, and $s \ge 0$, consider the function $g: \mathbb {R}^{n \times n} \times \mathbb {R}^{n \times N_{1}} \mapsto \mathbb {R}$ defined as $g(W, X) = \left\| WZ-X \right\| _{F}^{2} + \lambda \left\| W \right\| _{F}^{2}$ $- \lambda \log \,\left| \mathrm {det \,} W \right| + \phi (X)$ for $W \in \mathbb {R}^{n \times n}$ and $X \in \mathbb {R}^{n \times N_{1}}$. Further, let $(\hat{W}, \hat{X})$ be a pair in $\mathbb {R}^{n \times n} \times \mathbb {R}^{n \times N_{1}}$ satisfying

$$\begin{aligned}&2 \hat{W} Z Z^{T} - 2 \hat{X}Z^{T} + 2 \lambda \hat{W} - \lambda \hat{W}^{-T} = 0 \end{aligned}$$

(54)

$$\begin{aligned}&\hat{X}_{i} \in \tilde{H_{s}}(\hat{W}Z_{i}), \quad \forall \, 1 \le i \le N_{1} \end{aligned}$$

(55)

Then, the following condition holds at $(\hat{W}, \hat{X})$.

$$\begin{aligned} g(\hat{W}+dW, \hat{X}+\Delta X) \ge g(\hat{W}, \hat{X}) \end{aligned}$$

(56)

The condition holds for all sufficiently small $dW \in \mathbb {R}^{n \times n}$ satisfying $\left\| dW \right\| _{F} \le \epsilon '$ for some $\epsilon ' >0$ that depends on $\hat{W}$, and all $\Delta X \in \mathbb {R}^{n \times N}$ in the union of the following regions.

R1.:

The half-space $tr\left\{ (\hat{W}Z - \hat{X})\Delta X^{T} \right\} \le 0$.

R2.:

The local region defined by $\left\| \Delta X \right\| _{\infty } < \min _{i}\left\{ \beta _{s}(\hat{W} Z_{i}) : \left\| \hat{W} Z_{i} \right\| _{0}>s \right\}$.

Furthermore, if we have $\left\| \hat{W} Z_{i} \right\| _{0} \le s \, \forall \, i$, then $\Delta X$ can be arbitrary.

The following lemma is a slightly modified version of the one in (Ravishankar et al. 2014). We only state the minor modifications to the previous proof, for the following lemma to hold.

The lemma implies Lipschitz continuity (and therefore, continuity) of the function $u(B) \triangleq \left\| B y \,{-}\, H_{s}(By) \right\| _{2}^{2}$ on a bounded set.

Lemma 13

Given $c_{0}>0$, and $y \in \mathbb {R}^{n}$ satisfying $\left\| y \right\| _{2} \le c_{0}$, and a constant $c'>0$, the function $u(B) = \left\| B y \,{-}\, H_{s}(By) \right\| _{2}^{2}$ is uniformly Lipschitz with respect to $B$ on the bounded set $S \triangleq \left\{ B \in \mathbb {R}^{n \times n} : \left\| B \right\| _{2} \le c' \right\}$.

Proof

The proof is identical to that for Lemma 4 in (Ravishankar et al. 2014), except that the conditions $\left\| y \right\| _{2}=1$ and $\left\| B \right\| _{2} \le 1$ in (Ravishankar et al. 2014) are replaced by the conditions $\left\| y \right\| _{2} \le c_{0}$ and $\left\| B \right\| _{2} \le c'$ for the proof here. $\square$