A dynamic hypergraph regularized non-negative tucker decomposition framework for multiway data analysis

Abstract

Non-negative tensor decomposition has achieved significant success in machine learning due to its superiority in extracting the non-negative parts-based features and physically meaningful latent components from high-order data. To improve its representation ability, hypergraph has been incorporated into the tensor decomposition model to capture the nonlinear manifold structure of data. However, previous hypergraph regularized tensor decomposition methods rely on the original data space. This may result in inaccurate manifold structure and representation performance degeneration when original data suffer from noise corruption. To solve these problems, in this paper, we propose a dynamic hypergraph regularized non-negative Tucker decomposition (DHNTD) method for multiway data analysis. Specifically, to take full advantage of the multilinear structure and nonlinear manifold of tensor data, we learn the dynamic hypergraph and non-negative low-dimensional representation in a unified framework. Moreover, we develop a multiplicative update (MU) algorithm to solve our optimization problem and theoretically prove its convergence. Experimental results in clustering tasks using six image datasets demonstrate the superiority of our proposed method compared with the state-of-the-art methods.

Appendix

To prove the convergence of DHNTD-MU algorithm when we update $\{{\mathbf {A}}^{(n)}\}_{n=1}^{N}$ and $\varvec{{\mathcal {C}}}$, we use a similar procedure as in the literature [25]. We firstly consider the update of ${\mathbf {A}}^{(N)}$. The objective function $f_a({\mathbf {A}}^{(N)})$ of DHNTD can be rewritten as follows:

$$\begin{aligned} \begin{aligned}&f_a({\mathbf {A}}^{(N)})=\frac{1}{2}\left\| {\mathbf {X}}_{(N)} - {\mathbf {A}}^{(N)}{{\mathbf {B}}^{(N)}}^{\top } \right\| _F^2\\&\quad +\frac{\lambda }{2}{\text {Tr}}\left( {{\mathbf {A}}^{(N)}}^{\top } {\mathbf {L}}_{hyper}{\mathbf {A}}^{(N)}\right) .\\&\quad =\frac{1}{2}\sum _{i=1}^{I_N}\sum _{j=1}^{I_1\cdots I_{N-1}}{\left( {\left[ {\mathbf {X}}_{(N)}\right] }_{ij}-\sum _{k=1}^{R_N} {\left[ {\mathbf {A}}^{(N)}\right] }_{ik}{\left[ {\mathbf {B}}^{(N)}\right] }_{jk}\right) }^2\\&\quad +\frac{\lambda }{2}\sum _{k=1}^{R_N}\sum _{i=1}^{I_N}\sum _{l=1}^{I_N}\left( {\left[ {\mathbf {A}}^{(N)}\right] }_{ik} {\left[ {\mathbf {L}}_{hyper}\right] }_{il}{\left[ {\mathbf {A}}^{(N)}\right] }_{lk}\right) \end{aligned} \end{aligned}$$

(55)

For any element $a^{(N)}_{ij} (i.e., a^{(N)}_{ij} = {[{\mathbf {A}}^{(N)}]}_{ij})$ in ${\mathbf {A}}^{(N)}$, let $F_{ij}$ be the part of $f_a({\mathbf {A}}^{(N)})$ that is only relevant to ${\mathbf {A}}^{(N)}_{ij}$. We have

$$\begin{aligned} \begin{aligned}&F_{ij}^{\prime } = {\left[ \frac{\partial f_a({\mathbf {A}}^{(N)})}{\partial {\mathbf {A}}^{(N)}} \right] }_{ij}\\&\quad ={\left[ {\mathbf {A}}^{(N)}{{\mathbf {B}}^{(N)}}^{\top }{\mathbf {B}}^{(N)} -{\mathbf {X}}_{(N)}{\mathbf {B}}^{(N)}+\lambda {\mathbf {L}}_{hyper}{\mathbf {A}}^{(N)}\right] }_{ij}, \end{aligned} \end{aligned}$$

(56)

$$\begin{aligned} \begin{aligned}&F_{ij}^{\prime \prime } = {\left[ {{\mathbf {B}}^{(N)}}^{\top }{\mathbf {B}}^{(N)}\right] }_{jj} +\lambda \left[ {\mathbf {L}}_{hyper}\right] _{ii}. \end{aligned} \end{aligned}$$

(57)

Since our update rule for ${\mathbf {A}}^{(N)}$ is essentially element wise, it is only necessary to prove that each $F_{ij}$ is nonincreasing under the updating step of Eq. (24).

Proof of Lemma 1:

$$\begin{aligned} F\left( v^{t+1}\right) \le T\left( v^{t+1}, v^{t}\right) \le T\left( v^{t}, v^{t}\right) =F\left( v^{t}\right) . \end{aligned}$$

$\square$

Proof of Lemma 2:

It is obvious that $T\left( a,a\right) = F_{ij}\left( a\right)$. Therefore, we just need to prove $T\left( a,{a_{ij}^{(N)}}^{t}\right) \ge F_{ij}\left( a\right)$. We firstly perform Taylor series expansion of $F_{ij}\left( a\right)$,

$$\begin{aligned} \begin{aligned}&F_{i j}(v)= F_{i j}\left( {a_{ij}^{(N)}}^{t}\right) +F_{i j}^{\prime }\left( {a_{ij}^{(N)}}^{t}\right) \left( a-{a_{ij}^{(N)}}^{t}\right) \\&\quad +{\left( {\left[ {{\mathbf {B}}^{(N)}}^{\top }{\mathbf {B}}^{(N)}\right] }_{jj}+\lambda \left[ {\mathbf {L}}_{hyper}\right] _{ii}\right) }\left( a-{a_{ij}^{(N)}}^{t}\right) ^{2}. \end{aligned} \end{aligned}$$

(58)

According to Eq. (34), we can find that $T\left( a,{a_{ij}^{(N)}}^{t}\right) \ge F_{ij}\left( a\right)$ is equivalaent to

$$\begin{aligned} \begin{aligned}&\frac{{\left[ {\mathbf {A}}^{(N)}{{\mathbf {B}}^{(N)}}^{\top }{\mathbf {B}}^{(N)}\right] }_{ij}+\lambda {\left[ {\mathbf {I}} {\mathbf {A}}^{(N)}\right] }_{ij}}{{a_{ij}^{(N)}}^{t}} \\&\quad \ge {\left[ {{\mathbf {B}}^{(N)}}^{\top }{\mathbf {B}}^{(N)}\right] }_{jj}+\lambda \left[ {\mathbf {L}}_{hyper}\right] _{ii}. \end{aligned} \end{aligned}$$

(59)

We have

$$\begin{aligned} \begin{aligned}&{\left[ {\mathbf {A}}^{(N)}{{\mathbf {B}}^{(N)}}^{\top }{\mathbf {B}}^{(N)}\right] }_{ij} \\&\quad =\sum _{l=1}^{R_N}{a_{il}^{(N)}}^{t}\left[ {{\mathbf {B}}^{(N)}}^{\top }{\mathbf {B}}^{(N)}\right] _{lj}\ge {a_{ij}^{(N)}}^{t}\left[ {{\mathbf {B}}^{(N)}}^{\top }{\mathbf {B}}^{(N)}\right] _{jj}, \end{aligned} \end{aligned}$$

(60)

and

$$\begin{aligned} \begin{aligned} \lambda {\left[ {\mathbf {I}} {\mathbf {A}}^{(N)}\right] }_{ij}&=\lambda \sum _{m=1}^{I_N}I_{im}{a_{mj}^{(N)}}^{t} \ge \lambda I_{ii}{a_{ij}^{(N)}}^{t}\\&\ge \lambda \left[ {\mathbf {I}}-\varvec{\Theta }\right] _{ii}{a_{ij}^{(N)}}^{t} = \lambda \left[ {\mathbf {L}}_{hyper}\right] _{ii}{a_{ij}^{(N)}}^{t}. \end{aligned} \end{aligned}$$

(61)

Thus, $T\left( a,{a_{ij}^{(N)}}^{t}\right) \ge F_{ij}\left( a\right)$ holds and $T\left( a,{a_{ij}^{(N)}}^{t}\right)$ is an auxiliary function of $F_{ij}$. So the update rule of ${\mathbf {A}}^{(N)}$ results in a nonincreasing of the objective function $f_a({\mathbf {A}}^{(N)})$. As Eq. (34) is an auxiliary function, $F_{ij}$ is nonincreasing under this update rule, which leads to a nonincreasing of the the objective function $f_a({\mathbf {A}}^{(N)})$. $\square$

Similarly, if $\lambda = 0$, the above proof is also valid for updating ${\mathbf {A}}^{(n)}, \left( n\ne N\right)$. Here, we complete that the update rules of $\{{\mathbf {A}}^{(n)}\}_{n=1}^{N}$ result in a nonincreasing of the objective functions $f_a({\mathbf {A}}^{(n)}),n=1,2, \ldots ,N$. $\square$

Secondly, we consider the update rule of $\varvec{{\mathcal {C}}}$. The objective function $f_c({\text {vec}}(\varvec{{\mathcal {C}}}))$ of DHNTD can be rewritten as follow:

$$\begin{aligned} \begin{aligned} f_c({\text {vec}}(\varvec{{\mathcal {C}}}))=&\frac{1}{2}\left\| {\text {vec}} (\varvec{{\mathcal {X}}})-{\mathbf {Q}}{\text {vec}}(\varvec{{\mathcal {C}}}) \right\| _F^2\\ =&\frac{1}{2}\sum _{q=1}^{I_1\cdots I_N} {\left( \left[ {\text {vec}}(\varvec{{\mathcal {X}}})\right] _q-\sum _{l=1}^{R_1 \cdots R_N}\left[ Q\right] _{ql}\left[ {\text {vec}}\left( \varvec{{\mathcal {C}}}\right) \right] _{l}\right) }^{2}. \end{aligned} \end{aligned}$$

(62)

For any element $c_l (i.e., c_l=\left[ {\text {vec}}\left( \varvec{{\mathcal {C}}}\right) \right] _{l})$ in ${\text {vec}}\left( \varvec{{\mathcal {C}}}\right)$, let $C_l$ be the part of $f_c({\text {vec}}(\varvec{{\mathcal {C}}}))$, which is relevant to $c_l$. It is easy to check that

$$\begin{aligned} \begin{aligned} C_l^{\prime } = \left[ {\mathbf {Q}}^{\top }{\mathbf {Q}}{\text {vec}}(\varvec{{\mathcal {C}}}) -{\mathbf {Q}}^{\top }{\text {vec}}(\varvec{{\mathcal {X}}})\right] _{l}, \end{aligned} \end{aligned}$$

(63)

$$\begin{aligned} \begin{aligned} C_l^{\prime \prime } = \left[ {\mathbf {Q}}^{\top }{\mathbf {Q}}\right] _{ll}. \end{aligned} \end{aligned}$$

(64)

Since our update rule for $\varvec{{\mathcal {C}}}$ is essentially element wise, we want to show that $C_l$ is nonincreasing when updating $\varvec{{\mathcal {C}}}$ as Eq. (27).

Proof of Lemma 3:

It is obvious that $T(c,c) = C_l(c)$, and we only need to prove that $T(c,c_l^t) \ge C_l(c)$. Again, we perform Taylor expansion of $C_l(c)$,

$$\begin{aligned} \begin{aligned} C_l(c) = C_l\left( c_l^{t}\right) + C_l^{\prime }\left( c_l^{t}\right) (c-c_l^{t}) + \left[ {\mathbf {Q}}^{\top }{\mathbf {Q}}\right] _{ll}(c-c_l^{t})^2, \end{aligned} \end{aligned}$$

(65)

with Eq. (35) to find that $T(c,c_l^t) \ge C_l(c)$ is equivalent to

$$\begin{aligned} \begin{aligned} \left[ {\mathbf {Q}}^{\top }{\mathbf {Q}}{\text {vec}}(\varvec{{\mathcal {C}}})\right] _l \ge \left[ {\mathbf {Q}}^{\top }{\mathbf {Q}}\right] _{ll}. \end{aligned} \end{aligned}$$

(66)

We have

$$\begin{aligned} \begin{aligned} \left[ {\mathbf {Q}}^{\top }{\mathbf {Q}}{\text {vec}}(\varvec{{\mathcal {C}}})\right] _l = \sum _{r=1}^{R_1\cdots R_N} \left[ {\mathbf {Q}}^{\top }{\mathbf {Q}}\right] _{lr}c_r^t \ge \left[ {\mathbf {Q}}^{\top }{\mathbf {Q}}\right] _{ll}c_l^t. \end{aligned} \end{aligned}$$

(67)

Therefore, $T(c,c_l^t) \ge C_l(c)$ holds and $T(c,c_l^t)$ is an auxiliary function of $C_l$. Up to here, we can show that the update rule of $\varvec{{\mathcal {C}}}$ results in a nonincreasing of the objective function $f_c({\text {vec}}(\varvec{{\mathcal {C}}}))$. Since Eq. (35) is an auxiliary function, $C_l$ is nonincreasing under this update rule, which leads to a nonincreasing of the objective function $f_c({\text {vec}}(\varvec{{\mathcal {C}}}))$. $\square$