Dictionary learning with the ${{\ell }_{1/2}}$-regularizer and the coherence penalty and its convergence analysis

Abstract

The ${{\ell }_{1/2}}$-regularizer has been studied widely in compressed sensing, but there have been few studies about dictionary learning problems. The dictionary learning method with the ${{\ell }_{1/2}}$-regularizer aims to learn a dictionary, which requires solving a very challenging nonconvex and nonsmooth optimization problem. In addition, the low mutual coherence of a dictionary is an important property that ensures the optimality of the sparse representation in the dictionary. In this paper, we address a dictionary learning problem involving the ${{\ell }_{1/2}}$-regularizer and the coherence penalty, which is difficult to solve quickly and efficiently. We employ a decomposition scheme and an alternating optimization, which transforms the overall problem into a set of minimizations of single-vector-variable subproblems. Although the subproblems are nonsmooth and even nonconvex, we propose the use of proximal operator technology to conquer them, which leads to a rapid and efficient dictionary learning algorithm. In a theoretical analysis, we establish the algorithm’s global convergence. Experiments were performed for dictionary learning using both synthetic data and real-world data. For the synthetic data, we demonstrated that our algorithm performed better than state-of-the-art algorithms. Using real-world data, the learned dictionaries were shown to be more efficient than algorithms using ${{\ell }_{1}}$-norm for sparsity.

Appendix A

In this appendix, we give a proof of Theorem 1. Let ${{\mathbf{Z}}^{\left( n \right)}}=({{\mathbf{W}}^{\left( n \right)}},~{{\mathbf{H}}^{\left( n \right)}})$ denote the sequence generated by Algorithm 1. The cost function $\phi \left( \mathbf{Z} \right)=\text{Q}\left( \mathbf{Z} \right)+\text{G}\left( \mathbf{W} \right)+\text{F}\left( \mathbf{H} \right)$ is a proper, lower semi-continuous function, where $\text{Q}\left( \mathbf{Z} \right),\text{G}\left( \mathbf{W} \right),$ and $\text{F}\left( \mathbf{H} \right)$ have been defined in Sect. 3.

Theorem 2

([28]) Assume 8 $\phi (z)$ is a proper and lower semicontinuous function with inf $\phi> - \infty.$ The sequence ${{\{{{z}^{(n)}}\}}_{n\in \mathbb{N}}}$ is a Cauchy sequence, and converges to the critical point of $\phi (z),$ if the following conditions hold:

• (V1) Sufficient decrease condition. There exists some positive constant ${{\rho }_{1}}$ such that:

  $$\phi \left( {{z}^{\left( n \right)}} \right)-\phi \left( {{z}^{\left( n+1 \right)}} \right)\ge {{\rho }_{1}}{{z}^{\left( n+1 \right)}}-{{z}^{\left( n \right)}}_{F}^{2},\text{ }\!\!~\!\!\text{ }\!\!~\!\!\text{ }\forall \text{ }\!\!~\!\!\text{ }n=1,\text{ }\!\!~\!\!\text{ }2,\text{ }\!\!~\!\!\text{ }\ldots$$

• (V2) Relative error condition. There exists some positive constant ${{\rho }_{2}}$ > 0 such that:

  $${{\epsilon }^{\left( n+1 \right)}}_{F}\text{ }\!\!~\!\!\text{ }\ge {{\rho }_{2}}{{z}^{\left( n+1 \right)}}-{{z}^{\left( n \right)}}_{F},\text{ }\!\!~\!\!\text{ }\!\!~\!\!\text{ }{{\epsilon }^{\left( n \right)}}\in \partial \phi \left( {{z}^{\left( n \right)}} \right),\text{ }\!\!~\!\!\text{ }\!\!~\!\!\text{ }\forall \text{ }\!\!~\!\!\text{ }n=1,\text{ }\!\!~\!\!\text{ }2,\text{ }\!\!~\!\!\text{ }\ldots$$

• (V3) Continuity condition. There exists a subsequence ${{\{{{z}^{({{n}_{k}})}}\}}_{k\in \mathbb{N}}}~$ and $z$ such that:

  $$z^{{(n_{k} )}} \to \bar{z},~~\phi \left( {z^{{\left( {n_{k} } \right)}} } \right) \to \phi \left( {\bar{z}} \right),~~{\text{as}}~k \to + \infty$$

• (V4) $\phi (z)$ is a KL function. $\phi (z)$ satisfies the KL property in its effective domain.

Lemma 1

The sequence ${{\{{{\mathbf{Z}}^{\left( n \right)}}\}}_{n\in \mathbb{N}}}$ satisfies:

$\left\{ \begin{matrix} \phi \left( \mathbf{T}_{k}^{\left( n+1 \right)},{{\mathbf{W}}^{\left( n \right)}} \right)\le \phi \left( \mathbf{T}_{k-1}^{\left( n+1 \right)},{{\mathbf{W}}^{\left( n \right)}} \right)-\frac{\mu _{k}^{\left( n \right)}}{2}||\mathbf{h}_{k}^{\left( n+1 \right)}-\mathbf{h}_{k}^{\left( n \right)}||_{F}^{2}, \\ \phi \left( {{\mathbf{H}}^{\left( n+1 \right)}},\mathbf{V}_{k}^{\left( n+1 \right)} \right)\le \phi \left( {{\mathbf{H}}^{\left( n+1 \right)}},\mathbf{V}_{k-1}^{\left( n+1 \right)} \right)-\frac{\gamma _{k}^{\left( n \right)}}{2}||\mathbf{w}_{k}^{\left( n+1 \right)}-\mathbf{w}_{k}^{\left( n \right)}||_{F}^{2}. \\\end{matrix} \right.$ for $1\le k\le r$, where $\left\{ {\begin{array}{*{20}c} {{\mathbf{T}}_{k}^{{\left( n \right)}} = ({\mathbf{h}}_{1}^{{\left( n \right){\text{T}}}} , \ldots ~{\mathbf{h}}_{k}^{{\left( n \right){\text{T}}}} ,~{\mathbf{h}}_{{k + 1}}^{{\left( {n - 1} \right){\text{T}}}} , \ldots {\mathbf{h}}_{r}^{{(n - 1){\text{T}}}} )^{{\text{T}}} ,{\mathbf{T}}_{0}^{{\left( n \right)}} = {\mathbf{H}}^{{(n - 1)}} ,~} \\ {{\mathbf{V}}_{k}^{{\left( n \right)}} = \left( {{\mathbf{w}}_{1}^{{\left( n \right)}} , \ldots ~{\mathbf{w}}_{k}^{{\left( n \right)}} ,~{\mathbf{w}}_{{k + 1}}^{{\left( {n - 1} \right)}} , \ldots {\mathbf{w}}_{r}^{{\left( {n - 1} \right)}} } \right),{\mathbf{V}}_{0}^{{\left( n \right)}} = {\mathbf{W}}^{{\left( {n - 1} \right)}} ,} \\ \end{array} } \right.$ and $\mu _{k}^{\left( n \right)}=\mathbf{w}_{k}^{T}{{\mathbf{w}}_{k}}>0$, $\gamma _{k}^{\left( n \right)}=\mathbf{h}_{k}^{T}{{\mathbf{h}}_{k}}>0.$.

Therefore, we can obtain:

$$\begin{gathered} \phi \left( {{\mathbf{H}}^{{\left( n \right)}} ,{\mathbf{W}}^{{\left( n \right)}} } \right) - \phi \left( {{\mathbf{H}}^{{\left( {n + 1} \right)}} ,{\mathbf{W}}^{{\left( {n + 1} \right)}} } \right)~~ \hfill \\ \quad \ge \mathop \sum \limits_{{k = 1}}^{r} (\frac{{\mu _{k}^{{\left( n \right)}} }}{2}||{\mathbf{h}}_{k}^{{\left( {n + 1} \right)}} - {\mathbf{h}}_{k}^{{\left( n \right)}} ||_{\user2{F}}^{2} + \frac{{\gamma _{k}^{{\left( n \right)}} }}{2}||{\mathbf{w}}_{k}^{{\left( {n + 1} \right)}} - {\mathbf{w}}_{k}^{{\left( n \right)}} ||_{\user2{F}}^{2} ) \hfill \\ \end{gathered}$$

Lemma 2

Define $\epsilon _{\mathbf{H}}^{\left( n \right)}={{(\epsilon _{\mathbf{H}}^{1\text{T}},\ldots ~\epsilon _{\mathbf{H}}^{r\text{T}})}^{\text{T}}}$ and $\epsilon _{\mathbf{W}}^{\left( n \right)}=(\epsilon _{\mathbf{W}}^{1},\ldots ~\epsilon _{\mathbf{W}}^{r})$, where $\left\{ \begin{matrix} \epsilon _{\mathbf{H}}^{k}={{\nabla }_{{{\mathbf{h}}_{k}}}}Q\left( {{\mathbf{Z}}^{\left( n \right)}} \right)-{{\nabla }_{{{\mathbf{h}}_{k}}}}Q\left( \mathbf{T}_{k}^{\left( n \right)},{{\mathbf{W}}^{\left( n-1 \right)}} \right)-\mu _{k}^{\left( n \right)}\left( \mathbf{h}_{k}^{\left( n \right)}-\mathbf{h}_{k}^{\left( n-1 \right)} \right), \\ \epsilon _{\mathbf{W}}^{k}={{\nabla }_{{{\mathbf{w}}_{k}}}}Q\left( {{\mathbf{Z}}^{\left( n \right)}} \right)-{{\nabla }_{{{\mathbf{w}}_{k}}}}Q\left( {{\mathbf{H}}^{\left( n \right)}},\mathbf{V}_{k}^{\left( n \right)} \right)-\gamma _{k}^{\left( n \right)}\left( \mathbf{w}_{k}^{\left( n \right)}-\mathbf{w}_{k}^{\left( n-1 \right)} \right). \\\end{matrix} \right.$

Then $\varepsilon ^{{(n)}}\,:= (\varepsilon _{\mathbf{H}}^{{(n)}} ,\varepsilon _{\mathbf{W}}^{{(n)}} ) \in \partial \phi \left( {\mathbf{Z}^{{(n)}} } \right)$ and there exists a constant $\rho$ > 0 such that: ${{\left\| {{\epsilon }^{\left( n \right)}} \right\|}_{F}}\ge \rho \,\left\| {{\mathbf{Z}}^{\left( n \right)}} \right.\,-{{\left. {{\mathbf{Z}}^{\left( n-1 \right)}} \right\|}_{F}}$.

Lemma 3

The sequence ${{\{{{\mathbf{Z}}^{\left( n \right)}}\}}_{n\in \mathbb{N}}}$ satisfies the continuity condition (V3).

Theorem 3

([28]) The function f is a proper and lower semicontinuous function. If f is semialgebraic then it satisfies the KL property at any point in dom f.

Lemma 4

$\text{Q}\left( \mathbf{Z} \right)$, $\text{G}\left( \mathbf{W} \right)$ and $\text{F}\left( \mathbf{H} \right)$ are semialgebraic functions. The cost function $\phi \left( \mathbf{Z} \right)=\text{Q}\left( \mathbf{Z} \right)+\text{G}\left( \mathbf{W} \right)+\text{F}\left( \mathbf{H} \right)$ is a semialgebraic function. Therefore, $\phi \,(\mathbf{Z})$ satisfies the KL property in its effective domain.