Provable Stochastic Algorithm for Large-Scale Fully-Connected Tensor Network Decomposition

Abstract

The fully-connected tensor network (FCTN) decomposition is an emerging method for processing and analyzing higher-order tensors. For an Nth-order tensor, the standard deterministic algorithms, such as alternating least squares (FCTN-ALS) algorithm, need to store large coefficient matrices formed by contracting $N-1$ FCTN factor tensors. The memory cost of coefficient matrices grows exponentially with the size of the original tensor, which makes the algorithms memory-prohibitive for handling large-scale tensors. To enable FCTN decomposition to handle large-scale tensors effectively, we propose a stochastic gradient descent (FCTN-SGD) algorithm without sacrificing accuracy. The memory cost of FCTN-SGD algorithm grows linearly with the size of the original tensor and is significantly lower than that of the FCTN-ALS algorithm. The success of the FCTN-SGD algorithm lies in the suggested factor sampling operator, which cleverly avoids storing large coefficient matrices in the algorithm. By using the suggested operator, sampling on small factor tensors is equal to sampling on large coefficient matrices with a theoretical guarantee. Furthermore, we present an FCTN-VRSGD algorithm by introducing variance reduction into the FCTN-SGD algorithm, and theoretically prove the convergence of the FCTN-VRSGD algorithm under a mild assumption. Numerical experiments demonstrate the efficiency and accuracy of the proposed FCTN-SGD and FCTN-VRSGD algorithms, especially for real-world large-scale tensors.

Appendices

A. Proof of Lemma 1

Lemma 1

Under Assumptions 1.1–1.3, suppose parameters $\{\alpha ^s\}_{s\in \mathbb {N}}$ and $\{\gamma ^s\}_{s\in \mathbb {N}}$ satisfy the following:

$$\begin{aligned} \begin{aligned}&\frac{1}{4}\left( \frac{3}{4}-\beta ^sL\right) -\frac{2\beta ^s(1-\gamma ^s)^2}{15\beta ^{s+1}}>0\\&\quad \text {and}~~\frac{\beta ^s}{2}\left( \frac{19}{4}-\beta ^sL\right) -\frac{1}{30\beta ^sL^2}+\frac{(1-\gamma ^s)^2\big (1+4(\beta ^sL)^2\big )}{30\beta ^{s+1}L^2}\le 0. \end{aligned} \end{aligned}$$

(A.1)

Then the FCTN-VRSGD algorithm satisfies

$$\begin{aligned} \begin{aligned} \sum _{s=0}^{S-1}w_s\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^s}}f(\mathcal {G}^s)||_F^2\big ]\le \sum _{k=0}^{S-1}\frac{(\gamma ^s)^2\sigma ^2N}{15\beta ^{s+1}L^2}+\frac{\sigma ^2N}{30\beta ^0L^2}+f(\mathcal {G}^0)-f(\mathcal {G}^*), \end{aligned} \end{aligned}$$

(A.2)

where $w_k=\frac{\beta ^s}{4}(\frac{3}{4}-\beta ^sL)-\frac{2(\beta ^s(1-\gamma ^s))^2}{15\beta ^{s+1}}$.

Proof of Lemma 1

Before showing (A.2), let us introduce two inequalities as follows:

$$\begin{aligned} \begin{aligned} \mathbb {E}_{\mathcal {B}^{s+1}}\big [f(\mathcal {G}^{s+1})\big ]-\mathbb {E}_{\mathcal {B}^{s+1}}\big [f(\mathcal {G}^{s})\big ]\le&\frac{\beta ^s}{2}\left( \frac{19}{4}-\beta ^sL\right) \mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^s}^s||_F^2\big ]\\&-\frac{\beta ^s}{4}\left( \frac{3}{4}-\beta ^sL\right) \mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2\big ] \end{aligned} \end{aligned}$$

(A.3)

and

$$\begin{aligned} \mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^s}^s||_F^2\big ]&\le \frac{2(\gamma ^{s-1}\sigma )^2}{B}+\frac{4\big ((1-\gamma ^{s-1})\beta ^{s-1}L\big )^2}{N}\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2\big ]\nonumber \\&\quad +\frac{(1-\gamma ^{s-1})^2\big (1+4(\beta ^{s-1}L)^2\big )}{N}\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^{s-1}}^{s-1}||_F^2\big ], \end{aligned}$$

(A.4)

where $\mathcal {D}_n^s=\mathcal {R}_n^s-\nabla _{\mathcal {G}_n}f(\mathcal {G}^s)$.

The detailed proof of (A.3) is as follows. For a given $\xi ^s$, the update of $\mathcal {G}_{\xi ^s}$ in Algorithm 1 can be rewritten as

$$\begin{aligned} \begin{aligned} \mathcal {G}_{\xi ^s}^{s+1}=\mathop {\textrm{argmin}}_{\mathcal {G}_{\xi ^s}}<\mathcal {R}_{\xi ^s}^{s}, \mathcal {G}_{\xi ^s}-\mathcal {G}_{\xi ^s}^s>+\frac{1}{2\beta ^s}||\mathcal {G}_{\xi ^s}-\mathcal {G}_{\xi ^s}^s||_F^2. \end{aligned} \end{aligned}$$

(A.5)

Let $\mathcal {G}_{\xi ^s}=\mathcal {G}_{\xi ^s}^s$ and $\mathcal {G}_{\xi ^s}=\mathcal {G}_{\xi ^s}^{s+1}$ in (A.5) respectively, we have

$$\begin{aligned} \begin{aligned} <\mathcal {R}_{\xi ^s}^{s}, \mathcal {G}_{\xi ^s}^{s+1}-\mathcal {G}_{\xi ^s}^s>+\frac{1}{2\beta ^s}||\mathcal {G}_{\xi ^s}^{s+1}-\mathcal {G}_{\xi ^s}^s||_F^2\le 0. \end{aligned} \end{aligned}$$

(A.6)

By the block Lipschitz continuity of the quadratic function $f(\mathcal {G})$ [42], we can obtain

$$\begin{aligned} \begin{aligned} f(\mathcal {G}^{s+1})\le f(\mathcal {G}^s)+<\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s}),\mathcal {G}_{\xi ^s}^{s+1}-\mathcal {G}_{\xi ^s}^s>+\frac{L}{2}||\mathcal {G}_{\xi ^s}^{s+1}-\mathcal {G}_{\xi ^s}^s||_F^2, \end{aligned} \end{aligned}$$

(A.7)

where $L>0$ is a Lipschitz constant. The combination of (A.6) and (A.7) gives

$$\begin{aligned} f(\mathcal {G}^{s+1})&\le f(\mathcal {G}^s)-<\mathcal {R}_{\xi ^s}^{s}-\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s}),\mathcal {G}_{\xi ^s}^{s+1}-\mathcal {G}_{\xi ^s}^s>+\left( \frac{L}{2}-\frac{1}{2\beta ^s}\right) ||\mathcal {G}_{\xi ^s}^{s+1}-\mathcal {G}_{\xi ^s}^s||_F^2\nonumber \\&= f(\mathcal {G}^s)+\beta ^s<\mathcal {D}_{\xi ^s}^{s},\mathcal {P}_{\xi ^s}^{s}>+\left( \frac{(\beta ^s)^2L}{2}-\frac{\beta ^s}{2}\right) ||\mathcal {P}_{\xi ^s}^{s}||_F^2\nonumber \\&\overset{(a)}{\le }\ f(\mathcal {G}^s)+2\beta ^s||\mathcal {D}_{\xi ^s}^{s}||_F^2+\frac{\beta ^s}{8}||\mathcal {P}_{\xi ^s}^{s}||_F^2+\left( \frac{(\beta ^s)^2L}{2}-\frac{\beta ^s}{2}\right) ||\mathcal {P}_{\xi ^s}^{s}||_F^2\nonumber \\&=f(\mathcal {G}^s)+2\beta ^s||\mathcal {D}_{\xi ^s}^{s}||_F^2-\frac{\beta ^s}{2}\left( \frac{3}{4}-\beta ^sL\right) ||\mathcal {P}_{\xi ^s}^{s}||_F^2\nonumber \\&\overset{(b)}{\le }\ f(\mathcal {G}^s)+2\beta ^s||\mathcal {D}_{\xi ^s}^{s}||_F^2+\frac{\beta ^s}{2}\left( \frac{3}{4}-\beta ^sL\right) \times \left( -\frac{1}{2}||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2+||\mathcal {D}_{\xi ^s}^{s}||_F^2\right) \nonumber \\&=f(\mathcal {G}^s)-\frac{\beta ^s}{4}\left( \frac{3}{4}-\beta ^sL\right) ||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2+\frac{\beta ^s}{2}\left( \frac{19}{4}-\beta ^sL\right) ||\mathcal {D}_{\xi ^s}^{s}||_F^2, \end{aligned}$$

(A.8)

where $\mathcal {D}_{\xi ^s}^{s}=\mathcal {R}_{\xi ^s}^{s}-\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})$ and $\mathcal {P}_{\xi ^s}^{s}=\frac{1}{\beta ^s}(\mathcal {G}_{\xi ^s}^{s}-\mathcal {G}_{\xi ^s}^{s+1})$. (a) is obtained form $<\!\!\!A,B\!\!\!>\le 2||A||_F^2+\frac{1}{8}||B||_F^2$. When $0<\beta ^s\le \frac{3}{4L}$, (b) holds because

$$\begin{aligned} \begin{aligned} -||\mathcal {P}_{\xi ^s}^{s}||_F^2&\le -\frac{1}{2}||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2+||\mathcal {P}_{\xi ^s}^{s}-\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2\\&\le -\frac{1}{2}||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2+||\mathcal {D}_{\xi ^s}^{s}||_F^2. \end{aligned} \end{aligned}$$

(A.9)

The detailed proof of (A.4) is as follows.

$$\begin{aligned}&\mathbb {E}_{\zeta ^s}\big [||\mathcal {D}_{\xi ^s}^{s}||_F^2|\mathcal {B}^s,\xi ^s\big ]=\mathbb {E}_{\zeta ^s}\big [||\mathcal {R}_{\xi ^s}^{s}-\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&=\mathbb {E}_{\zeta ^s}\big [||\mathcal {Q}_{\xi ^s}^{s}\!\!-\!\!\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})\!-\!(1\!-\!\gamma ^{s-1})\big (\mathcal {Q}_{\xi ^s}^{s-1}\!-\!\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})\big )\!+\!(1\!-\!\gamma ^{s-1})\mathcal {D}_{\xi ^s}^{s-1}||_F^2|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&\overset{(a)}{=}\mathbb {E}_{\zeta ^s}\big [||\mathcal {Q}_{\xi ^s}^{s}\!-\!\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})\!-\!(1\!-\!\gamma ^{s-1})\big (\mathcal {Q}_{\xi ^s}^{s-1}\!-\!\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})\big )||_F^2|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&\quad +(1\!-\!\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\mathcal {D}_{\xi ^s}^{s-1}||_F^2|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&\overset{(b)}{\le }(1\!-\!\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\mathcal {D}_{\xi ^s}^{s-1}||_F^2|\mathcal {B}^s,\xi ^s\big ]+2(1-\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\mathcal {Q}_{\xi ^s}^{s}-\mathcal {Q}_{\xi ^s}^{s-1}||_F^2|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&\quad +2(\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}[||\mathcal {Q}_{\xi ^s}^{s-1}\!-\!\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2|\mathcal {B}^s,\xi ^s]. \end{aligned}$$

(A.10)

(a) is obtained from $\mathbb {E}_{\zeta ^s}\big [\mathcal {Q}_{\xi ^s}^{s}|\mathcal {B}^s,\xi ^s\big ]=\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})$, that is, the stochastic gradient $\mathcal {Q}_{\xi ^s}^{s}$ is an unbiased estimate for the full gradient for $\mathcal {G}_{\xi ^s}^{s}$, and thus $\mathbb {E}_{\zeta ^s}\big [<\mathcal {Q}_{\xi ^s}^{s}\!-\!\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s}),\mathcal {D}_{\xi ^s}^{s}>|\mathcal {B}^s,\xi ^s\big ]=0$. (b) is obtained from

$$\begin{aligned}&\mathbb {E}_{\zeta ^s}\big [||\mathcal {Q}_{\xi ^s}^{s}\!-\!\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})\!-\!(1\!-\!\gamma ^{s-1})\big (\mathcal {Q}_{\xi ^s}^{s-1}\!-\!\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})\big )||_F^2|\mathcal {B}^s,\xi ^s\big ] \end{aligned}$$

(A.11)

$$\begin{aligned}&=\mathbb {E}_{\zeta ^s}\big [||(1\!-\!\gamma ^{s-1})\big (\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})\!-\!\mathcal {Q}_{\xi ^s}^{s-1}+\mathcal {Q}_{\xi ^s}^{s}\!-\!\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})\big )\nonumber \\&\quad +\gamma ^{s-1}\big (\mathcal {Q}_{\xi ^s}^{s-1}\!-\!\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})\big )||_F^2|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&\le 2(1\!-\!\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})\!-\!\mathcal {Q}_{\xi ^s}^{s-1}+\mathcal {Q}_{\xi ^s}^{s}\!-\!\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&\quad +2(\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\mathcal {Q}_{\xi ^s}^{s-1}\!-\!\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&=2(1\!-\!\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2\!+\!||\mathcal {Q}_{\xi ^s}^{s-1}||_F^2+||\mathcal {Q}_{\xi ^s}^{s}||_F^2\!+\!||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2\nonumber \\&\quad -2<\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1}),\mathcal {Q}_{\xi ^s}^{s-1}>-2<\mathcal {Q}_{\xi ^s}^{s},\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})>\nonumber \\&\quad +2<\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1}),\mathcal {Q}_{\xi ^s}^{s}>+2<\mathcal {Q}_{\xi ^s}^{s-1},\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})>\nonumber \\&\quad -2<\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1}),\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})>-2<\mathcal {Q}_{\xi ^s}^{s-1},\mathcal {Q}_{\xi ^s}^{s}>|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&\quad +2(\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\mathcal {Q}_{\xi ^s}^{s-1}\!-\!\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&=2(1\!-\!\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2\!+\!||\mathcal {Q}_{\xi ^s}^{s-1}||_F^2+||\mathcal {Q}_{\xi ^s}^{s}||_F^2\!+\!||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2\nonumber \\&\quad -2||\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2-2||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2\nonumber \\&\quad -2<\mathcal {Q}_{\xi ^s}^{s-1},\mathcal {Q}_{\xi ^s}^{s}>+2<\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1}),\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})>|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&\quad +2(\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\mathcal {Q}_{\xi ^s}^{s-1}\!-\!\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2|\mathcal {B}^s,\xi ^s\big ]\end{aligned}$$

(A.12)

$$\begin{aligned}&=2(1\!-\!\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})-\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2+||\mathcal {Q}_{\xi ^s}^{s}-\mathcal {Q}_{\xi ^s}^{s-1}||_F^2\mathcal {B}^s,\xi ^s\big ]\nonumber \\&\quad +2(\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\mathcal {Q}_{\xi ^s}^{s-1}\!-\!\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&\le 2(1-\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\mathcal {Q}_{\xi ^s}^{s}-\mathcal {Q}_{\xi ^s}^{s-1}||_F^2|\mathcal {B}^s,\xi ^s\big ]\nonumber \\&\quad +2(\gamma ^{s-1})^2\mathbb {E}_{\zeta ^s}\big [||\mathcal {Q}_{\xi ^s}^{s-1}\!-\!\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2|\mathcal {B}^s,\xi ^s\big ]. \end{aligned}$$

(A.13)

Taking the total expectation of (A.10), we have the following inequality

$$\begin{aligned}{} & {} \mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^s}^{s}||_F^2\big ] \le (1\!-\!\gamma ^{s-1})^2\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^s}^{s-1}||_F^2\big ]+2(1-\gamma ^{s-1})^2\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {Q}_{\xi ^s}^{s}-\mathcal {Q}_{\xi ^s}^{s-1}||_F^2\big ]\nonumber \\{} & {} \quad +2(\gamma ^{s-1})^2\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {Q}_{\xi ^s}^{s-1}\!-\!\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2\big ]\nonumber \\{} & {} \overset{(a)}{\le }\frac{(1\!-\!\gamma ^{s-1})^2}{N}\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^{s-1}}^{s-1}||_F^2\big ]+2(\gamma ^{s-1}\sigma )^2+2(1-\gamma ^{s-1})^2L^2\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {G}_{\xi ^s}^{s}-\mathcal {G}_{\xi ^s}^{s-1}||_F^2\big ]\nonumber \\{} & {} \overset{(b)}{\le }\frac{(1\!-\!\gamma ^{s-1})^2}{N}\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^s}^{s-1}||_F^2\big ]+2(\gamma ^{s-1}\sigma )^2+\frac{2(1-\gamma ^{s-1})^2L^2}{N}\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {G}_{\xi ^{s-1}}^{s}-\mathcal {G}_{\xi ^{s-1}}^{s-1}||_F^2\big ]\nonumber \\{} & {} =\frac{(1\!-\!\gamma ^{s-1})^2}{N}\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^s}^{s-1}||_F^2\big ]+2(\gamma ^{s-1}\sigma )^2+\frac{2\big ((1-\gamma ^{s-1})\beta ^{s-1}L\big )^2}{N}\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {P}_{\xi ^{s-1}}^{s-1}||_F^2\big ]\nonumber \\{} & {} \overset{(c)}{\le }\frac{(1\!-\!\gamma ^{s-1})^2}{N}\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^s}^{s-1}||_F^2\big ]+2(\gamma ^{s-1}\sigma )^2+\frac{2\big ((1-\gamma ^{s-1})\beta ^{s-1}L\big )^2}{N}(2\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^{s-1}}^{s-1}||_F^2\big ]\nonumber \\{} & {} \quad +2\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2\big ])\nonumber \\{} & {} =2(\gamma ^{s-1}\sigma )^2+\frac{4\big ((1-\gamma ^{s-1})\beta ^{s-1}L\big )^2}{N}\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2\big ]\nonumber \\{} & {} \quad +\frac{(1-\gamma ^{s-1})^2\big (1+4(\beta ^{s-1}L)^2\big )}{N}\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^{s-1}}^{s-1}||_F^2\big ]. \end{aligned}$$

(A.14)

Now, we show that (A.2) holds. By setting $\phi (\mathcal {G}^s)=f(\mathcal {G}^s)+\frac{N}{30\beta ^sL^2}||\mathcal {D}_{\xi ^s}^s||_F^2$, we can obtain

$$\begin{aligned} \begin{aligned}&\mathbb {E}_{\mathcal {B}^{s+1}}\big [\phi (\mathcal {G}^{s+1})-\phi (\mathcal {G}^s)\big ]=\mathbb {E}_{\mathcal {B}^{s+1}} \left[ f(\mathcal {G}^{s+1})+\frac{N}{30\beta ^{s+1}L^2}||\mathcal {D}_{\xi ^{s+1}}^{s+1}||_F^2-f(\mathcal {G}^s)-\frac{N}{30\beta ^sL^2}||\mathcal {D}_{\xi ^s}^s||_F^2\right] \\&\le \frac{\beta ^s}{2}\left( \frac{19}{4}-\beta ^sL\right) \mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^s}^s||_F^2\big ]- \frac{\beta ^s}{4}\left( \frac{3}{4}-\beta ^sL\right) \mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^{s-1}}}f(\mathcal {G}^{s-1})||_F^2\big ]\\&\quad +\!\frac{N}{30\beta ^{s+\!1}L^2}\bigg (2(\gamma ^{s}\sigma )^2\!+\!\frac{4\big ((1-\gamma ^{s})\beta ^{s}L\big )^2}{N}\bigg )\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2\big ]\\&\quad +\!\frac{N}{30\beta ^{s+1}L^2}\times \frac{(1-\gamma ^{s})^2\big (1+4(\beta ^{s}L)^2\big )}{N}\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^{s}}^{s}||_F^2\big ]-\mathbb {E}_{\mathcal {B}^{s+1}}\big [\frac{N}{30\beta ^sL^2}||\mathcal {D}_{\xi ^s}^s||_F^2\big ]\\&\le \bigg (\frac{\beta ^s}{2}\left( \frac{19}{4}-\beta ^sL\right) +\frac{(1-\gamma ^{s})^2\big (1+4(\beta ^{s}L)^2\big )}{30\beta ^{s+1}L^2}-\frac{1}{30\beta ^sL^2}\bigg )\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\mathcal {D}_{\xi ^{s}}^{s}||_F^2\big ]\\&\quad -\bigg (\frac{\beta ^s}{4}\left( \frac{3}{4}-\beta ^sL\right) -\frac{4\big ((1-\gamma ^{s})\beta ^{s}\big )^2}{30\beta ^{s+\!1}}\bigg )\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2\big ]+\frac{N(\gamma ^{s}\sigma )^2}{15\beta ^{s+\!1}L^2}\\&\quad \overset{(a)}{\le }\frac{N(\gamma ^{s}\sigma )^2}{15\beta ^{s+\!1}L^2}-\bigg (\frac{\beta ^s}{4}\left( \frac{3}{4}-\beta ^sL\right) -\frac{4\big ((1-\gamma ^{s})\beta ^{s}\big )^2}{30\beta ^{s+\!1}}\bigg )\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2\big ], \end{aligned} \end{aligned}$$

(A.15)

where (a) holds when the second inequality of (A.1) is satisfied. Thus, we have

$$\begin{aligned} \begin{aligned} \mathbb {E}_{\mathcal {B}^{s+1}}\big [\phi (\mathcal {G}^{s+1})\!-\!\phi (\mathcal {G}^s)\big ]\!\le \!\frac{N(\gamma ^{s}\sigma )^2}{15\beta ^{s+\!1}L^2}-w_s\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2\big ], \end{aligned} \end{aligned}$$

(A.16)

where $w_s=\frac{\beta ^s}{4}(\frac{3}{4}-\beta ^sL)-\frac{4((1-\gamma ^{s})\beta ^{s})^2}{30\beta ^{s+\!1}}$. (A.16) can be rewritten as

$$\begin{aligned} \begin{aligned} w_s\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2\big ]\le \mathbb {E}_{\mathcal {B}^{s+1}}\big [\phi (\mathcal {G}^{s})\!-\!\phi (\mathcal {G}^{s+1})\big ]+\frac{N(\gamma ^{s}\sigma )^2}{15\beta ^{s+\!1}L^2}. \end{aligned} \end{aligned}$$

(A.17)

Summing up inequality (A.17) from $s=0$ to $s=S-1$, we have

$$\begin{aligned}{} & {} \sum _{s=0}^{S-1}w_s\mathbb {E}_{\mathcal {B}^{s+1}}\big [||\nabla _{\mathcal {G}_{\xi ^{s}}}f(\mathcal {G}^{s})||_F^2\big ]\le \mathbb {E}_{\mathcal {B}^{s+1}}\big [\phi (\mathcal {G}^{0})\!-\!\phi (\mathcal {G}^S)\big ]+\sum _{s=0}^{S-1}\frac{N(\gamma ^{s}\sigma )^2}{15\beta ^{s+\!1}L^2}\nonumber \\{} & {} \quad =f(\mathcal {G}^0)+\frac{N}{30\beta ^0L^2}||\mathcal {D}_{\xi ^0}^0||_F^2-f(\mathcal {G}^S)-\frac{N}{30\beta ^SL^2}||\mathcal {D}_{\xi ^S}^S||_F^2+\sum _{s=0}^{S-1}\frac{N(\gamma ^{s}\sigma )^2}{15\beta ^{s+\!1}L^2}\nonumber \\{} & {} \quad \le f(\mathcal {G}^0)+\frac{N}{30\beta ^0L^2}||\mathcal {Q}_{\xi ^0}^0||_F^2-f(\mathcal {G}^S)+\sum _{s=0}^{S-1}\frac{N(\gamma ^{s}\sigma )^2}{15\beta ^{s+\!1}L^2}\nonumber \\{} & {} \quad \le \sum _{s=0}^{S-1}\frac{N(\gamma ^{s}\sigma )^2}{15\beta ^{s+\!1}L^2}+\frac{\sigma ^2N}{30\beta ^0L^2}+f(\mathcal {G}^0)-f(\mathcal {G}^*). \end{aligned}$$

(A.18)

$\square$

B. Proof of Lemma 2

Lemma 2

Under Assumptions 1.1–1.3, suppose parameters $\{\beta ^s\}_{s\in \mathbb {N}}$ and $\{\gamma ^s\}_{s\in \mathbb {N}}$ satisfy the following:

$$\begin{aligned} \begin{aligned} \beta ^s=\frac{m}{(s+3)^{\frac{1}{3}}}~~~\text {and}~~~1>\gamma ^s\ge \frac{1+\big (4\beta ^s+15\beta ^{s+1}(\frac{19}{4}-\beta ^sL)\big )\beta ^sL^2-\frac{\beta ^{s+1}}{\beta ^s}}{1+4(\beta ^sL)^2}, \end{aligned} \end{aligned}$$

(B.1)

where $0<m\le \frac{3^{\frac{1}{3}}}{9L}$. Then the condition (A.1) in Lemma 1 holds.

Proof of Lemma 2

From $0<m\le \frac{3^{\frac{1}{3}}}{9L}$, we can obtain

$$\begin{aligned} \begin{aligned} \beta ^s=\frac{m}{(s+3)^{\frac{1}{3}}}\le \frac{3^{\frac{1}{3}}}{9L(s+3)^{\frac{1}{3}}}\le \frac{1}{9L} \end{aligned} \end{aligned}$$

(B.2)

and

$$\begin{aligned} \begin{aligned} \frac{\beta ^{s+1}}{\beta ^s}=\frac{(s+3)^{\frac{1}{3}}}{(s+4)^{\frac{1}{3}}}\ge (\frac{3}{4})^{\frac{1}{3}}. \end{aligned} \end{aligned}$$

(B.3)

The first inequality of (A.1) is equivalently expressed as

$$\begin{aligned} \begin{aligned} \frac{15\beta ^{s+1}}{8\beta ^s}\left( \frac{3}{4}-\beta ^sL\right) >(1-\gamma ^s)^2. \end{aligned} \end{aligned}$$

(B.4)

Combining (B.2) and (B.3), we can obtain that (B.4) holds by following:

$$\begin{aligned} \begin{aligned} \frac{15\beta ^{s+1}}{8\beta ^s}\left( \frac{3}{4}-\beta ^sL\right) \ge \frac{15}{8}\times \left( \frac{3}{4}\right) ^{\frac{1}{3}}\times \left( \frac{3}{4}-\frac{1}{9}\right) >1\ge 1-\gamma ^s\ge (1-\gamma ^s)^2, \end{aligned} \end{aligned}$$

(B.5)

where $\gamma ^s\in (0,1)$. The second inequality of (A.1) is equivalently expressed as

$$\begin{aligned} \begin{aligned} (1-\gamma ^s)^2\big (1+4(\beta ^sL)^2\big )\le \frac{\beta ^{s+1}}{\beta ^s}-15\beta ^s\beta ^{s+1}L^2\left( \frac{19}{4}-\beta ^sL\right) . \end{aligned} \end{aligned}$$

(B.6)

If

$$\begin{aligned} \begin{aligned} (1-\gamma ^s)\big (1+4(\beta ^sL)^2\big )\le \frac{\beta ^{s+1}}{\beta ^s}-15\beta ^s\beta ^{s+1}L^2\left( \frac{19}{4}-\beta ^sL\right) , \end{aligned} \end{aligned}$$

(B.7)

then (B.6) holds since $(1-\gamma ^s)^2\big (1+4(\beta ^sL)^2\big )\le (1-\gamma ^s)\big (1+4(\beta ^sL)^2\big )$. From (B.7), we have

$$\begin{aligned} \begin{aligned} \gamma ^s&\ge 1-{\frac{\frac{\beta ^{s+1}}{\beta ^s}-15\beta ^s\beta ^{s+1}L^2\left( \frac{19}{4}-\beta ^sL\right) }{1+4(\beta ^sL)^2}}\\&=\frac{1+\left( 4\beta ^s+15\beta ^{s+1}\left( \frac{19}{4}-\beta ^sL\right) \right) \beta ^sL^2-\frac{\beta ^{s+1}}{\beta ^s}}{1+4(\beta ^sL)^2}\in (0,1). \end{aligned} \end{aligned}$$

(B.8)

$\square$