A Primal-Dual algorithm for nonnegative N-th order CP tensor decomposition: application to fluorescence spectroscopy data analysis

Abstract

This work concerns the resolution of inverse problems encountered in multidimensional signal processing problems. Here, we address the problem of tensor decomposition, more precisely the Canonical Polyadic (CP) Decomposition also known as Parafac. Yet, when the amount of data is very large, this inverse problem may become numerically ill-posed and consequently hard to solve. This requires to introduce a priori information about the system, often in the form of penalty functions. The difficulty that might arise is that the considered cost function may be non differentiable. It is the reason why we develop here a Primal-Dual Projected Gradient (PDPG) optimization algorithm to solve variational problems involving such cost functions. Moreover, despite its theoretical importance, most approaches of the literature developed for nonnegative CP decomposition, have not actually addressed the convergence issue. Furthermore, most methods with convergence rate guarantee require restrictive conditions for their theoretical analyses. Hence, a theoretical void still exists for the convergence rate problem under very mild conditions. Therefore, the two main aims of this work are (i) to design a CP-PDPG algorithm and then (ii) to prove, under mild conditions, its convergence to the set of minimizers at the rate O(1/k), k being the iteration number. The effectiveness and the robustness of the proposed approach are illustrated through numerical examples.

Appendices

Appendices

1.1 Appendix A

The steps of our proof proceed very much in the same way as the analysis provided by N. Angelia and O. Asuman in Nedić and Ozdaglar (2009) in the context of Lagrangian duality, where they consider a convex primal optimization problem and its Lagrangian dual. Then, before we prove the Theorem 2, we will state a simple following lemma:

Lemma 2

Let the sequences $\{ \mathbf{A }^{(n)}_k \}$ and $\{ \mathbf{Y} ^{(n)}_k \}$ be generated by the subgradient algorithm. Then, we have:

• 1. For any $\mathbf{A }^{(n)}\in U$ and for all $k \ge 0$,

  $$\begin{aligned} \parallel \mathbf{A }^{(n)}_{k+1}- \mathbf{A }^{(n)}\parallel ^2\le & {} \parallel \mathbf{A }^{(n)}_{k}- \mathbf{A }^{(n)}\parallel ^2-2\sigma ^{(n)}\left( g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k)-g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)}_k)\right) \\&+(\sigma ^{(n)})^2\parallel \partial _{\mathbf{A }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k)\parallel ^2 \end{aligned}$$

• 2. For any $\mathbf{Y }^{(n)}\in V^*$ and for all $k \ge 0$,

  $$\begin{aligned} \parallel \mathbf{Y }^{(n)}_{k+1}- \mathbf{Y }^{(n)}\parallel ^2\le & {} \parallel \mathbf{Y }^{(n)}_{k}- \mathbf{Y }^{(n)}\parallel ^2+2\tau ^{(n)}\left( g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k)-g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)})\right) \\&+(\tau ^{(n)})^2\parallel \partial _{\mathbf{Y }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k)\parallel ^2 \end{aligned}$$

Proof

• 1. Concerning the first inequality of the Lemma 2, for any $\mathbf{A }^{(n)}_k\in U$ and all $k \ge 0$

  $$\begin{aligned} \parallel \mathbf{A }^{(n)}_{k+1}- \mathbf{A }^{(n)}\parallel ^2&=\parallel P_{U} \{ \mathbf{A }^{(n)}_{k}-\sigma ^{(n)}\partial _{\mathbf{A }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k) \} - \mathbf{A }^{(n)}\parallel ^2\\&\le \parallel \mathbf{A }^{(n)}_{k}-\sigma ^{(n)}\partial _{\mathbf{A }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k) - \mathbf{A }^{(n)}\parallel ^2\\&= \parallel \mathbf{A }^{(n)}_{k}- \mathbf{A }^{(n)} \parallel ^2-2\sigma ^{(n)}(\mathbf{A }^{(n)}_{k}-\mathbf{A }^{(n)})^{T}\partial _{\mathbf{A }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k)\\&\quad +(\sigma ^{(n)})^2 \parallel \partial _{\mathbf{A }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k) \parallel ^2 \end{aligned}$$

  Since the function $g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)})$ is convex in $\mathbf{A }^{(n)}$ for each $\mathbf{Y }^{(n)}\in V^*$, and since $\partial _{\mathbf{A }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k)$ is a subgradient of $g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)}_k)$ with respect to $\mathbf{A }^{(n)}$ at $\mathbf{A }^{(n)}=\mathbf{A }^{(n)}_k$, we obtain for any $\mathbf{A }^{(n)}$

  $$\begin{aligned} \partial _{\mathbf{A }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k)^T(\mathbf{A }^{(n)}- \mathbf{A }^{(n)}_{k})\le g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)}_k)-g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k) \end{aligned}$$

  or equivalently

  $$\begin{aligned} -\partial _{\mathbf{A }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k)^T(\mathbf{A }^{(n)}_{k}- \mathbf{A }^{(n)})\le g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k)-g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)}_k) \end{aligned}$$

  Hence, for any $\mathbf{A }^{(n)}_k\in U$ and all $k \ge 0$,

  $$\begin{aligned} \parallel \mathbf{A }^{(n)}_{k+1}- \mathbf{A }^{(n)}\parallel ^2&\le \parallel \mathbf{A }^{(n)}_{k}- \mathbf{A }^{(n)} \parallel ^2-2\sigma ^{(n)}(g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)}_k)-g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k))\\&\quad +(\sigma ^{(n)})^2 \parallel \partial _{\mathbf{A }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k) \parallel ^2 \end{aligned}$$

• 2. In the same way, and by using the concavity of the function $g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)})$ with respect to $\mathbf{Y }^{(n)}$ for each $\mathbf{A }^{(n)}\in U$, we can establish the second inequality in the Lemma 2.

$\square$

Now, we are going to give the proof of Lemma 1

Proof

Using the first statement of Lemma 2 and assumption of boundedness of $\partial _{\mathbf{A }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k)$, we have for any $\mathbf{A }^{(n)}\in U$ and all $i \ge 0$,

$$\begin{aligned} \parallel \mathbf{A }^{(n)}_{i+1}- \mathbf{A }^{(n)}\parallel ^2 \le \parallel \mathbf{A }^{(n)}_{i}- \mathbf{A }^{(n)}\parallel ^2-2\sigma ^{(n)}\left( g(\mathbf{A }^{(n)}_i,\mathbf{Y }^{(n)}_i)-g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)}_i)\right) +(\sigma ^{(n)})^2 L^2 \end{aligned}$$

therefore

$$\begin{aligned} g(\mathbf{A }^{(n)}_i,\mathbf{Y }^{(n)}_i)-g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)}_i) \le \frac{1}{2\sigma ^{(n)}}\left( \parallel \mathbf{A }^{(n)}_{i}- \mathbf{A }^{(n)}\parallel ^2-\parallel \mathbf{A }^{(n)}_{i+1}- \mathbf{A }^{(n)}\parallel ^2 \right) +\frac{\sigma ^{(n)} L^2}{2} \end{aligned}$$

Let us now sum the last relation from $i = 0,\dots ,k-1$, we obtain for any $\mathbf{A }^{(n)}\in U$ and $i \ge 1$,

$$\begin{aligned}&\sum _{i=0}^{k-1}\left( g(\mathbf{A }^{(n)}_i,\mathbf{Y }^{(n)}_i)-g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)}_i)\right) \\&\qquad \le \frac{1}{2\sigma ^{(n)}}\left( \parallel \mathbf{A }^{(n)}_{0}- \mathbf{A }^{(n)}\parallel ^2-\parallel \mathbf{A }^{(n)}_{k}- \mathbf{A }^{(n)}\parallel ^2 \right) +\frac{k \sigma ^{(n)} L^2}{2} \end{aligned}$$

implying that

$$\begin{aligned} \frac{1}{k}\sum _{i=0}^{k-1}g(\mathbf{A }^{(n)}_i,\mathbf{Y }^{(n)}_i)-\frac{1}{k}\sum _{i=0}^{k-1}g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)}_i) \le \frac{\parallel \mathbf{A }^{(n)}_{0}- \mathbf{A} ^{(n)}\parallel ^2}{2\sigma ^{(n)} k}+\frac{\sigma ^{(n)} L^2}{2} \end{aligned}$$

Since the function $g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)})$ is concave in $\mathbf{Y }^{(n)}$ for any fixed $\mathbf{A }^{(n)}\in U$, there holds

$$\begin{aligned} \frac{1}{k}\sum _{i=0}^{k-1}g(\mathbf{A }^{(n)},\mathbf{Y }^{(n)}_i)\le g(\mathbf{A }^{(n)},\widehat{\mathbf{Y }}^{(n)}_i) \end{aligned}$$

Combining the preceding two relations, we obtain for any $\mathbf{A }^{(n)}\in U$ and $k \ge 1$,

$$\begin{aligned} \frac{1}{k}\sum _{i=0}^{k-1}g(\mathbf{A }^{(n)}_i,\mathbf{Y }^{(n)}_i)-g(\mathbf{A }^{(n)},\widehat{\mathbf{Y }}^{(n)}_k)\le \frac{\parallel \mathbf{A }^{(n)}_{0}- \mathbf{A} ^{(n)}\parallel ^2}{2\sigma ^{(n)} k}+\frac{\sigma ^{(n)} L^2}{2} \end{aligned}$$

thus establishing relation (32).

In the same way, using the second statement of Lemma 2, the convexity of the function g with respect to $A^{(n)}$ and the Assumption of boundedness of $\partial _{\mathbf{Y }^{(n)}} g(\mathbf{A }^{(n)}_k,\mathbf{Y }^{(n)}_k)$, we can establish the second relation (33) in the Lemma 1. $\square$

Now, we are going to establish the proof of the Theorem 2.

Proof

Using $\mathbf{A }^{(n)}=\mathbf{A }^{(n)}_{*}$ and $\mathbf{Y }^{(n)}=\mathbf{Y }^{(n)}_{*}$ in relations (32) and (33), respectively, we obtain for any $k \ge 1$:

$$\begin{aligned} \frac{1}{k} \underset{i=0}{\overset{k-1}{\sum }}g(\mathbf{A }^{(n)}_i,\mathbf{Y }^{(n)}_i)-g(\mathbf{A }^{(n)}_{*},\widehat{\mathbf{Y }}^{(n)}_k)\le & {} \frac{\parallel \mathbf{A }^{(n)}_0 -\mathbf{A }^{(n)}_{*}\parallel ^2}{2\sigma ^{(n)} k}+\frac{\sigma ^{(n)} L^2}{2}\\ -\frac{\parallel \mathbf{Y }^{(n)}_0 -\mathbf{Y }^{(n)}_{*}]\parallel ^2}{2\tau ^{(n)} k}-\frac{\tau ^{(n)} L^2}{2}\le & {} \frac{1}{k} \underset{i=0}{\overset{k-1}{\sum }}g(\mathbf{A }^{(n)}_i,\mathbf{Y }^{(n)}_i)-g(\widehat{\mathbf{A }}^{(n)}_k,\mathbf{Y }^{(n)}_{*}) \end{aligned}$$

since U and $V^*$ are convex sets, then $\widehat{\mathbf{A }}^{(n)}_{k}\in U$ and $\widehat{\mathbf{Y }}^{(n)}_{k}\in V^*$ for all $k \ge 1$. therefore, by the saddle-point relation (8), we have

$$\begin{aligned} g(\mathbf{A }^{(n)}_{*},\widehat{\mathbf{Y }}^{(n)}_k)\le g(\mathbf{A }^{(n)}_{*},\mathbf{Y }^{(n)}_{*})\le g(\widehat{\mathbf{A }}^{(n)}_k,\mathbf{Y }^{(n)}_{*}) \end{aligned}$$

Combining the preceding three relations, we obtain then

$$\begin{aligned} -\frac{\parallel \mathbf{Y }^{(n)}_0 -\mathbf{Y }^{(n)}_{*}\parallel ^2}{2\tau ^{(n)}k }-\frac{\tau ^{(n)} L^2}{2}\le & {} \frac{1}{k} \underset{i=0}{\overset{k-1}{\sum }}g(\mathbf{A }^{(n)}_i,\mathbf{Y }^{(n)}_i)-g(\mathbf{A }^{(n)}_{*},\mathbf{Y }^{(n)}_{*})\nonumber \\\le & {} \frac{\parallel \mathbf{A }^{(n)}_0 -\mathbf{A }^{(n)}_{*}\parallel ^2}{2\sigma ^{(n)} } +\frac{\sigma ^{(n)} L^2}{2} \end{aligned}$$

(34)

Moreover, since $\widehat{\mathbf{A }}^{(n)}\in U$ and $\widehat{\mathbf{Y }}^{(n)}\in V^*$ for all $k \ge 1$, using Lemma 2with $\mathbf{A }^{(n)}=\widehat{\mathbf{A }}^{(n)}$ and $\mathbf{Y }^{(n)}=\widehat{\mathbf{Y }}^{(n)}$, we obtain for all $k \ge 1$,

$$\begin{aligned} \frac{1}{k} \underset{i=0}{\overset{k-1}{\sum }}g(\mathbf{A }^{(n)}_i,\mathbf{Y }^{(n)}_i)-g(\widehat{\mathbf{A }}^{(n)}_k,\widehat{\mathbf{Y }}^{(n)}_k)\le & {} \frac{\parallel \mathbf{A }^{(n)}_0 -\widehat{\mathbf{A }}^{(n)}_k\parallel ^2}{2\sigma ^{(n)} k}+\frac{\sigma ^{(n)} L^2}{2}\\ -\frac{\parallel \mathbf{Y }^{(n)}_0 -\widehat{\mathbf{Y }}^{(n)}_k\parallel ^2}{2\tau ^{(n)} k}-\frac{\tau ^{(n)} L^2}{2}\le & {} \frac{1}{k} \underset{i=0}{\overset{k-1}{\sum }}g(\mathbf{A }^{(n)}_i,\mathbf{Y }^{(n)}_i)-g(\widehat{\mathbf{A }}^{(n)}_k,\widehat{\mathbf{Y }}^{(n)}_k) \end{aligned}$$

By multiplying by $-1$ the preceding relations and combining them, we finally obtain

$$\begin{aligned} -\frac{\parallel \mathbf{A }^{(n)}_0 -\widehat{\mathbf{A }}^{(n)}_k\parallel ^2}{2\sigma ^{(n)}k }-\frac{\sigma ^{(n)} L^2}{2}\le & {} g(\widehat{\mathbf{A }}^{(n)}_k,\widehat{\mathbf{Y }}^{(n)}_k)-\frac{1}{k} \underset{i=0}{\overset{k-1}{\sum }}g(\mathbf{A }^{(n)}_i,\mathbf{Y }^{(n)}_i)\nonumber \\\le & {} \frac{\parallel \mathbf{A }^{(n)}_0 -\widehat{\mathbf{A }}^{(n)}_k\parallel ^2}{2\sigma ^{(n)} } +\frac{\sigma ^{(n)} L^2}{2} \end{aligned}$$

(35)

Using this last relation together with (34), we finally demonstrate the result stated by the Theorem 2 of convergence. $\square$

1.2 Appendix B

In this appendix, we provide the proof of the Proposition 1. But, first, we need the following results:

Lemma 3

Given $\mathbf{z }\in {\mathbb {R}}^n$, a vector $\mathbf{x }\in Z$ is equal to $P_Z\{\mathbf{z }\}$ if and only if

$$\begin{aligned} <\mathbf{y }-\mathbf{x },\mathbf{z }-\mathbf{x }> \le 0\;\; \forall \mathbf{y }\in Z \end{aligned}$$

(36)

Proof

By definition of the orthogonal projection, $P_Z \{\mathbf{z }\}$ is the minimizer of $f(\mathbf{x })=|\mathbf{x }-\mathbf{z }|^2$ over all $\mathbf{x }\in Z$, Moreover, if a vector $\mathbf{z }\in Z$ minimizes $f : {\mathbb {R}}^n \longrightarrow {\mathbb {R}}$, then $<\mathbf{y }-\mathbf{z },\nabla f(\mathbf{z })> \ge 0,\; \forall \mathbf{y }\in Z$. We have

$$\begin{aligned} \nabla f(\mathbf{x })=2(\mathbf{x }-\mathbf{z }) \end{aligned}$$

(37)

and thus

$$\begin{aligned} <\mathbf{y }-\mathbf{x },\mathbf{z }-\mathbf{x }> \le 0\;\; \forall \mathbf{y }\in Z \end{aligned}$$

(38)

$\square$

Proof

(Proposition 1) Suppose that $\mathbf{z }^*=P_Z\{\mathbf{z }^* -r\mathbf{h }(\mathbf{z }^*)\}$, then, thanks to Lemma 3 we have:

$$\begin{aligned} <\mathbf{z }-\mathbf{z }^*,-r\mathbf{h }(\mathbf{z }^*)> \le 0\;\; \forall \mathbf{z }\in Z \end{aligned}$$

(39)

and since r is positive, it follows that $\mathbf{z }^*$ solves inequality (7). Conversely, suppose that $\mathbf{z }^*$ solves inequality (7), then

$$\begin{aligned} <\mathbf{z }-\mathbf{z }^*,r\mathbf{h }(\mathbf{z }^*)> \le 0\;\; \forall \mathbf{z }\in Z \end{aligned}$$

(40)

which can be rewritten as

$$\begin{aligned} <\mathbf{z }-\mathbf{z }^*,\mathbf{z }^* -(\mathbf{z }^*-r\mathbf{h }(\mathbf{z }^*))> \le 0\;\; \forall \mathbf{z }\in Z \end{aligned}$$

(41)

and by using the lemma, we get

$$\begin{aligned} \mathbf{z }^*=P_Z\{\mathbf{z }^* -r\mathbf{h }(\mathbf{z }^*)\} \end{aligned}$$

(42)

$\square$