SVD-based algorithms for fully-connected tensor network decomposition

Abstract

The popular fully-connected tensor network (FCTN) decomposition has achieved successful applications in many fields. A standard method to this decomposition is the alternating least squares. However, it often converges slowly and suffers from issues of numerical stability. In this work, we investigate the SVD-based algorithms for FCTN decomposition to tackle the aforementioned deficiencies. On the basis of a result about FCTN-ranks, a deterministic algorithm, namely FCTN-SVD, is first proposed, which can approximate the FCTN decomposition under a fixed accuracy. Then, we present the randomized version of the algorithm. Both synthetic and real data are used to test our algorithms. Numerical results show that they perform much better than the existing methods, and the randomized algorithm can indeed yield acceleration on FCTN-SVD. Moreover, we also apply our algorithms to tensor-on-vector regression and achieve quite decent performance.

Proof of Theorem 3

For convenience of expression, we let

$$\begin{aligned} \textrm{FCTN}(\{\mathcal{G}_k\}^{N}_{k=1})= \mathcal{G}_1 {\overline{\times }} \mathcal{G}_2 {\overline{\times }} \cdots {\overline{\times }} \mathcal{G}_N, \end{aligned}$$

where $\mathcal{G}_k$ with $k=1,\cdots ,N$ are from Algorithm 4 and ${\overline{\times }}$ denotes some contracted tensor product which now has no explicit definition. Further, considering the procedure of Algorithm 4 gives

$$\begin{aligned} \textrm{FCTN}(\{\mathcal{G}_k\}^{N}_{k=1})&=\left( \mathcal{X} {\overline{\times }} \mathcal{G}_1 {\overline{\times }} \mathcal{G}_2 {\overline{\times }} \cdots {\overline{\times }} \mathcal{G}_{N-1} \right) {\overline{\times }} \mathcal{G}_1 {\overline{\times }} \mathcal{G}_2 {\overline{\times }} \cdots {\overline{\times }} \mathcal{G}_{N-1} \\&=\mathcal{X} {\overline{\times }} \left( \left( \mathcal{G}_1 {\overline{\times }} \mathcal{G}_2 {\overline{\times }} \cdots {\overline{\times }} \mathcal{G}_{N-1} \right) {\overline{\times }} \left( \mathcal{G}_1 {\overline{\times }} \mathcal{G}_2 {\overline{\times }} \cdots {\overline{\times }} \mathcal{G}_{N-1} \right) \right) \\&= P_N(\mathcal{X}). \end{aligned}$$

Note that $\mathcal{G}_k$ with $k=1,\cdots ,N$ are reshaped from orthonormal matrices. Hence, $P_N(\mathcal{X})$ is actually an orthogonal projector. To find the bound between $\mathcal X$ and $P_N(\mathcal{X})$, we first present a definition.

Definition 7

(FCTN unfolding) The FCTN unfolding of a 3Nth-order tensor $\mathcal{Z} \in \mathbb {R}^{I_1 \times I_2 \cdots \times I_{3N}}$ is the matrix $\widehat{\textbf{Z}}$ of size $\prod _{j=1}^{N} I_{3j-1} \times \prod _{j=1}^{N} I_{3j-2}I_{3j}$ defined element-wise via

$$\begin{aligned} \widehat{\textbf{Z}}(\overline{i_2\cdots i_{3N-1}}, \overline{i_1i_3 \cdots i_{3N-2}i_{3N}})=\mathcal{Z}(i_1, \cdots , i_{3N}). \end{aligned}$$

Thus,

$$\begin{aligned} {\left\| \mathcal{X} - P_N(\mathcal{X}) \right\| _F^2}&= {\left\| \mathcal{X} \right\| _F^2} - {\left\| P_N(\mathcal{X}) \right\| _F^2} = {\left\| \mathcal{X} \right\| _F^2} - \left\langle {\widehat{\textbf{X}}^{\left( N-1 \right) }} , {\widehat{\textbf{X}}^{\left( N-1 \right) }} \right\rangle \nonumber \\&= {\left\| \mathcal{X} \right\| _F^2} - \left\langle ( (\textbf{Q}^{(N-1)})^T {\textbf{X}^{\left( N-2 \right) }} , ((\textbf{Q}^{(N-1)})^T{\textbf{X}^{\left( N-2 \right) }} \right\rangle \nonumber \\&= {\left\| \mathcal{X} \right\| _F^2} - \left\langle {\textbf{X}^{\left( N-2 \right) }} , \textbf{Q}^{(N-1)}(\textbf{Q}^{(N-1)})^T{\textbf{X}^{\left( N-2 \right) }} \right\rangle , \end{aligned}$$

(A1)

where $\widehat{\textbf{X}}^{\left( N-1 \right) } \in {\mathbb {R}}^{R_{N-1,N} \times R_{1,N}\cdots R_{N-2,N}I_N}$ is from $\mathcal{X}^{(N-1)}$, $\textbf{X}^{\left( N-2 \right) }$ is the matrix in line 8 in Algorithm 4 for $k=N-1$, and $\left\langle \cdot \right\rangle$ denotes the classical inner product of matrices.

Since $\textbf{Q}^{(k)}$ is orthonormal, for the matrix in line 8 in Algorithm 4, we have

$$\begin{aligned} \left\langle {\textbf{X}^{\left( k-1 \right) }} , \textbf{Q}^{(k)}(\textbf{Q}^{(k)})^T{\textbf{X}^{\left( k-1 \right) }} \right\rangle&= \left\langle {\textbf{X}^{\left( k-1 \right) }} , {\textbf{X}^{\left( k-1 \right) }} - \left( \textbf{I} - \textbf{Q}^{(k)}(\textbf{Q}^{(k)})^T \right) {\textbf{X}^{\left( k-1 \right) }} \right\rangle \\&= {\left\| {\textbf{X}^{\left( k-1 \right) }} \right\| _F^2} - {\left\| \left( \textbf{I} - \textbf{Q}^{(k)}(\textbf{Q}^{(k)})^T \right) {\textbf{X}^{\left( k-1 \right) }} \right\| _F^2} , \end{aligned}$$

where $\textbf{X}^{(0)} = \textbf{X}_{<1>}$. This result together with the fact

$$\begin{aligned} {\left\| {\textbf{X}^{\left( k-1 \right) }} \right\| _F^2} = {\left\| {\widehat{\textbf{X}}^{\left( k-1 \right) }} \right\| _F^2} \end{aligned}$$

leads to

$$\begin{aligned} \left\langle {\textbf{X}^{\left( k-1 \right) }} , \textbf{Q}^{(k)}(\textbf{Q}^{(k)})^T{\textbf{X}^{\left( k-1 \right) }} \right\rangle&= {\left\| {\widehat{\textbf{X}}^{\left( k-1 \right) }} \right\| _F^2} - {\left\| \left( \textbf{I} - \textbf{Q}^{(k)}(\textbf{Q}^{(k)})^T \right) {\textbf{X}^{\left( k-1 \right) }} \right\| _F^2} \\&= \left\langle {\textbf{X}^{\left( k-2 \right) }} , \textbf{Q}^{(k-1)}(\textbf{Q}^{(k-1)})^T{\textbf{X}^{\left( k-2 \right) }} \right\rangle -\\&\quad \ {\left\| \left( \textbf{I} - \textbf{Q}^{(k)}(\textbf{Q}^{(k)})^T \right) {\textbf{X}^{\left( k-1 \right) }} \right\| _F^2}. \end{aligned}$$

Thus, combining the above recursive formula with (A1) implies

$$\begin{aligned} {\left\| \mathcal{X} - P_N(\mathcal{X}) \right\| _F^2}&= {\left\| \mathcal{X} \right\| _F^2} - {\left\| {\textbf{X}_{ < 1 > }} \right\| _F^2} + \sum \limits _{k = 1}^{N - 1} {{\left\| \left( \textbf{I} - \textbf{Q}^{(k)}(\textbf{Q}^{(k)})^T \right) {\textbf{X}^{\left( {k-1} \right) }} \right\| _F^2}} \\&= \sum \limits _{k = 1}^{N - 1} {{\left\| \left( \textbf{I} - \textbf{Q}^{(k)}(\textbf{Q}^{(k)})^T \right) {\textbf{X}^{\left( {k-1} \right) }} \right\| _F^2}}. \end{aligned}$$

To continue, we need a lemma as follows.

Lemma 1

(Minster et al. (2020), Theorem 2.4) Let $\textbf{A} \in {\mathbb {R}}^{m\times n}$, and choose a target rank $r\ge 2$ and an oversampling parameter $p \ge 2$, where $r+p \le \min \left\{ {m,n} \right\}$. Draw a standard Gaussian matrix $\mathbf{\Omega } \in {\mathbb {R}}^{n\times {(r+p)}}$ and construct $\textbf{Y}=\textbf{A}{} \mathbf{\Omega }$. Assume $\textbf{Q}_Y$ and $\textbf{U}_{\textbf{Q}_Y^T\textbf{Y}}$ are the orthonormal basis matrices of ${{\,\textrm{range}\,}}(\textbf{Y})$ and ${{\,\textrm{range}\,}}(\textbf{Q}_Y^T\textbf{Y})$, respectively. Set $\textbf{Q}=\textbf{Q}_Y\textbf{U}_{\textbf{Q}_Y^T\textbf{Y}}(:,1:r)$. Then the following expected approximation error holds:

$$\begin{aligned} {\textbf{E}}_{\mathbf {\Omega }}\left\| {\textbf{A} - \textbf{Q}{\textbf{Q}^T}{} \textbf{A}} \right\| _F^2 \le {\left( {1 + \frac{r}{{p - 1}}} \right) }{\left( {\sum \limits _{l > r} {{\sigma ^2 _l}(\mathbf{{A}})} } \right) }, \end{aligned}$$

where $\sigma _{l }(A)$ is the lth singular value of $\textbf{A}$.

Hence,

$$\begin{aligned} {\textbf{E}}_{{\mathbf {\Omega }^{(k)}}}{{\left\| \left( \textbf{I} - \textbf{Q}^{(k)}(\textbf{Q}^{(k)})^T \right) {\textbf{X}^{\left( {k-1} \right) }} \right\| _F^2}} \le {\left( {1 + \frac{r^{N-k}}{{R_0 - 1}}} \right) }\sum \limits _{l > r^{N-k}} {\sigma ^2 _l\left( {{\textbf{X}^{\left( {k-1} \right) }} } \right) }. \end{aligned}$$

As a result,

$$\begin{aligned} {\textbf{E}}_{\{{\mathbf {\Omega }^{(k)}}\}^{N-1}_{k=1}}{\left\| \mathcal{X} - \textrm{FCTN}(\{\mathcal{G}_k\}^{N}_{k=1}) \right\| _F^2} \le \sum \limits _{k = 1}^{N - 1} {\left( {1 + \frac{{{r^{N - k}}}}{{R_0 - 1}}} \right) \sum \limits _{l > {r^{N - k}}} {{\sigma ^2 _l}\left( {{\mathbf{{X}}^{\left( {k - 1} \right) }}} \right) } } . \end{aligned}$$

Note that, by the procedure of Algorithm 4 and Theorem 1,

$$\begin{aligned} \sum \limits _{l> {r^{N - k}}}{{\sigma ^2 _l}\left( {{\mathbf{{X}}^{\left( {k - 1} \right) }}} \right) }&\le \sum \limits _{l> {r^{N - k}}}{{\sigma ^2 _l}\left( {{\mathbf{{X}}_{<k>}}} \right) } = \mathop {\min }\limits _{{{\,\textrm{rank}\,}}(\textbf{Y}_{<k>})\le {r^{N - k}}} \left\| {{\mathbf{{X}}_{<k>}}}-\textbf{Y}_{<k>}\right\| _F^2\\&\le \mathop {\min }\limits _{\mathrm{FCTN-ranks}(\mathcal{Y})\le \textbf{r}} \left\| {{\mathcal{{X}}}}-\mathcal Y\right\| _F^2, \end{aligned}$$

where $\textbf{r}=[r,\cdots ,r]$ are the FCTN-ranks of $\mathcal X$. Hence, the desired result holds.