Ternary tree-based structural twin support tensor machine for clustering

Abstract

Most of the real-life applications usually involve complex data, e.g., grayscale images, where information is distributed spatially in the form of two-dimensional matrices (elements of second-order tensor space). Traditional vector-based clustering models such as k-means and support vector clustering rely on low-dimensional features representations for identifying patterns and are prone to loss of useful information which is present in spatial structure of the data. To overcome this limitation, tensor-based clustering models can be utilized for identifying relevant patterns in matrix data as they take advantage of structural information present in multi-dimensional framework and reduce computational overheads as well. However, despite these numerous advantages, tensor clustering has still remained relatively unexplored research area. In this paper, we propose a novel clustering framework, termed as Ternary Tree-based Structural Least Squares Support Tensor Clustering (TT-SLSTWSTC), that builds a cluster model as a hierarchical ternary tree, where at each node non-ambiguous data are dealt separately from ambiguous data points using the proposed Ternary Structural Least Squares Support Tensor Machine (TS-LSTWSTM). The TS-LSTWSTM classifier considers the structural risk minimization of data alongside a symmetrical L2-norm loss function. Also, initialization framework based on tensor k-means has been used in order to overcome the instability disseminated by random initialization. To validate the efficacy of the proposed framework, computational experiments have been performed on human activity recognition and image recognition problems. Experimental results show that our method is not only fast but yields significantly better generalization performance and is comparatively more robust in order to handle heteroscedastic noise and outliers when compared to related methods.

Appendices

Appendix 1: Finding the solution for the second problem

At iteration m, for any given non-zero vector $u_{2,m} \in \mathbb {R}^{n_1}$, let $a_i^{\rm T}={u_{2,m}}^{\rm T}A_i$, $b_i^{\rm T}={u_{2,m}}^{\rm T}B_i$ and ${c}_k^{\rm T}={u_{2,m}}^{\rm T}{X}_k$, we then solve for the following modified problem:

$$\begin{aligned}&\underset{v_{2,m},b_{2,m},\rho _{2,m}}{Min} \frac{\alpha }{2} \sum \limits _{j \in I_2}||b_jv_{2,m}+b_{2,m}||^2\nonumber \\&\quad + \frac{\beta }{2} (v_{2,m}^{\rm T}v_{2,m} + b_{2,m}^2) + \frac{\gamma }{2} \sum \limits _{i \in I_1}||a_iv_{2,m} +e_1b_{2,m} + (1-\rho _{2,m}) ||^2\nonumber \\&\quad + \frac{\delta }{2} \sum \limits _{k \in I_3}||c_kv_{2,m} + e_3b_{2,m} + (1-\rho _{2,m}-\epsilon )||^2. \end{aligned}$$

(16)

Considering Eq. (16) in vector form and differentiating it with respect to $v_m$ and $b_m$ leads to the following system of linear equations:

$$\begin{aligned} \frac{\partial L}{\partial v_{2,m}}&= \alpha B^{\rm T}(Bv_{2,m}+e_2b_{2,m}) \nonumber \\&\quad +\beta v_{2,m}+\gamma A^{\rm T}(Av_{2,m}+e_1b_{2,m}+e_1(1-\rho _{2,m})) \nonumber \\&\quad +\delta C^{\rm T}(Cv_{2,m}+e_3b_{2,m} +e_3(1-\rho _{2,m}-\epsilon ))=0 \end{aligned}$$

(17)

$$\begin{aligned} \frac{\partial L}{\partial b_{2,m}}&= \alpha e_2^{\rm T}(Bv_{2,m}+e_2b_{2,m})+\beta b_{2,m}\nonumber \\&\quad +\gamma e_1^{\rm T}(Av_{2,m}+e_1b_{2,m}+e_1(1-\rho _{2,m}))+\delta e_3^{\rm T}(Cv_{2,m}+e_3b_{2,m} \nonumber \\&\quad +e_3(1-\rho _{2,m}-\epsilon ))=0 \end{aligned}$$

(18)

$$\begin{aligned} \frac{\partial L}{\partial \rho _{1,m}}&= -\gamma (Av_{2,m}+e_1b_{2,m}+e_1-\rho _{2,m}) \nonumber \\&\quad -\delta (Cv_{2,m}+e_3b_{2,m}+e_3-\rho _{2,m}-\epsilon )=0 \nonumber \\&\Rightarrow -\gamma (H_1^{\rm T}z_{2,m}+e_1-e_1\rho _{2,m})-\delta (K_1^{\rm T}z_{2,m}+e_3-\rho _{2,m}-\epsilon ) =0 \nonumber \\&\Rightarrow -(\gamma H_1^{\rm T}+\delta K_1^{\rm T})z_{2,m}+(\gamma +\delta )\rho _{2,m} \nonumber \\&\quad - (\gamma e_2+\delta e_3-\delta \epsilon ) \nonumber \\&=0 \quad \text{ where } z_{2,m}=[v_{2,m}~~b_{2,m}]. \end{aligned}$$

(19)

Combining Eqs. (17) and (18) on the similar lines, we have

$$\begin{aligned}&(\alpha G_1^{\rm T}G_1+\beta I+\gamma H_1^{\rm T}H_1+\delta K_1^{\rm T}K_1)z+(-\gamma H_1^{\rm T}e_1-\delta K_1^{\rm T}e_3)\rho _{1,m}\nonumber \\&\quad =-(\gamma H_1^{\rm T}e_2+\delta K_1^{\rm T}e_3 (1-\epsilon )). \end{aligned}$$

(20)

$$\begin{aligned}&\left[ {\begin{array}{cc} z_{2,m} \\ \rho _{2,m} \end{array}}\right] \nonumber \\&\quad =\left[ {\begin{array}{cc} \alpha G_1^{\rm T}G_1+\beta I+\gamma H_1^{\rm T}H_1+\delta K_1^{\rm T}K_1 &{} -(\gamma H_1^{\rm T}e_1+\delta K_1^{\rm T}e_3) \\ -(\gamma e_2^{\rm T}G_1^{\rm T}+\delta e_3^{\rm T}K_1^{\rm T}) &{} \gamma e_1^{\rm T} +\delta e_3^{\rm T} \end{array}}\right] ^{-1} \nonumber \\&\left[ {\begin{array}{cc}-(\gamma H_1^{\rm T}e_2+\delta K_1^{\rm T} (1-\epsilon )) \\ (\gamma e_2+\delta e_3(1-\epsilon )) \end{array}}\right] , \end{aligned}$$

(21)

Once the solution to Eq. (21) are calculated, the optimal values of $v_{2,m}$, $b_{2,m}$ and $\rho _{2,m}$ are obtained. Thus, alternatively projecting obtained non-zero vector $v_{1,m} \in \mathbb {R}^{n_2}$, we have $\hat{a}_i^{\rm T}=A_i{v_{1,m}}$, $\hat{b}_j^{\rm T}={B}_j{v_{1,m}}$ and $\hat{c}_j^{\rm T}={C}_j{v_{1,m}}$ in Eq. (13). So, now, we solve for the following modified optimization problem:

$$\begin{aligned}&\underset{u_{2,m},b_{2,m},\rho _{2,m}}{Min} \frac{\alpha }{2} \sum \limits _{j \in I_2}||\hat{b}_ju_{2,m}+b_{2,m}||^2\nonumber \\&\quad + \frac{\beta }{2} (u_{2,m}^{\rm T}u_{2,m} + b_{2,m}^2) + \frac{\gamma }{2} \sum \limits _{i \in I_1}||\hat{a}_iu_{2,m} +b_{2,m} + (1-\rho _{2,m}) ||^2\nonumber \\&\quad + \frac{\delta }{2} \sum \limits _{k \in I_3}||\hat{c}_ku_{2,m} + b_{2,m} + (1-\rho _{2,m}-\epsilon )||^2. \end{aligned}$$

(22)

Working on the lines as above, and considering $\hat{z}=[u_{2,m} \quad b_{2,m}]$, we obtain $(u_{1,m}, b_{1,m}, \rho _{1,m})$ as follows

$$\begin{aligned}&\left[ {\begin{array}{cc} z_{2,m} \\ \rho _{2,m} \end{array}}\right] =\left[ {\begin{array}{cc} \alpha G_2^{\rm T}G_2+\beta I+\gamma H_2^{\rm T}H_2+\delta K_2^{\rm T}K_2 &{} -(\gamma H_2^{\rm T}e_1+\delta K_2^{\rm T}e_3) \\ -(\gamma e_2^{\rm T} H_2^{\rm T}+\delta e_3^{\rm T} K_2^{\rm T}) &{} \gamma e_1^{\rm T} +\delta e_3^{\rm T} \end{array}}\right] ^{-1} \nonumber \\&\left[ {\begin{array}{cc}-(\gamma H_2^{\rm T}e_2+\delta K_2^{\rm T} (1-\epsilon )) \\ (\gamma e_2+\delta e_3(1-\epsilon )) \end{array}}\right] , \end{aligned}$$

(23)

where $H_2$, $G_2$ and $K_2$ are matrices of points from classes +1, -1 and 0, respectively, augmented with a column of ones. Equations (21) and (23) are solved alternatively until $u_{2,m}$, $v_{2,m}$, $b_{2,m}$ and $\rho _{2,m}$ converges as per some predefined termination criteria.

Appendix 2: Finding the solution for the third problem

At iteration m, for any given non-zero vector $u_{3,m} \in \mathbb {R}^{n_1}$, let $a_i^{\rm T}={u_{3,m}}^{\rm T}A_i$, $b_i^{\rm T}={u_{3,m}}^{\rm T}B_i$ and ${c}_k^{\rm T}={u_{3,m}}^{\rm T}{X}_k$, we then solve for the following modified problem:

$$\begin{aligned}&\underset{v_{3,m},b_{3,m}}{Min} \frac{\alpha }{2} \sum \limits _{k \in I_3}||c_kv_{3,m}+b_{3,m}||^2\nonumber \\&\quad + \frac{\beta }{2} (v_{3,m}^{\rm T}v_{3,m} + b_{3,m}^2) + \frac{\gamma }{2} \sum \limits _{i \in I_1}||a_iv_{3,m} +b_{3,m} + (1-\rho _1-\epsilon ) ||^2\nonumber \\&\quad + \frac{\delta }{2} \sum \limits _{j \in I_2}||b_jv_{3,m} + b_{3,m} + (1-\rho _2-\epsilon )||^2. \end{aligned}$$

(24)

Considering Eq. (24) in vector form and differentiating it with respect to $v_m$ and $b_m$ leads to the following system of linear equations:

$$\begin{aligned} \frac{\partial L}{\partial v_{3,m}}& = {} \alpha C^{\rm T}(Cv_{3,m}+e_3b_{3,m})+\beta v_{3,m}\nonumber \\&\quad +\gamma A^{\rm T}(Av_{3,m}+e_1b_{3,m}+e_1(1-\rho _1-\epsilon )) \nonumber \\&\quad +\delta B^{\rm T}(Bv_{3,m}+e_2b_{3} +e_2(1-\rho _{2}-\epsilon ))=0 \end{aligned}$$

(25)

$$\begin{aligned} \frac{\partial L}{\partial b_{3,m}}& = {} \alpha e_3^{\rm T}(Cv_{3,m}+e_3b_{3,m})+\beta b_{3,m}\nonumber \\&\quad +\gamma e_1^{\rm T}(Av_{3,m}+e_1b_{3,m} +e_1(1-\rho _{1}-\epsilon ))\nonumber \\&\quad +\delta e_3^{\rm T}(Bv_{3,m}+e_2b_{3,m} +e_2(1-\rho _{2}-\epsilon ))=0 \end{aligned}$$

(26)

Combining Eqs. (25) and (26) on the similar lines, we have

$$\begin{aligned}&(\alpha K_1^{\rm T}K_1+\beta I+\gamma H_1^{\rm T}H_1+\delta G_1^{\rm T}G_1)z_{3,m}\nonumber \\&\quad =-(\gamma H_1^{\rm T}e_1+\delta H_1^{\rm T}e_1\rho _{1}+\gamma H_1^{\rm T}e_1\epsilon -\delta G_1^{\rm T}e_2 + \delta G_1^{\rm T}e_2 \delta + \delta G_1^{\rm T}e_2 \epsilon ) \end{aligned}$$

(27)

$$\begin{aligned}&z_{3,m}=-(\alpha K_1^{\rm T}K_1+\beta I+\gamma H_1^{\rm T}H_1+\delta G_1^{\rm T}G_1)^{-1} (\gamma H_1^{\rm T}e_1+\delta H_1^{\rm T}e_1\rho _{1}\nonumber \\&\qquad +\gamma H_1^{\rm T}e_1\epsilon -\delta G_1^{\rm T}e_2 + \delta G_1^{\rm T}e_2 \delta + \delta G_1^{\rm T}e_2 \epsilon ) \end{aligned}$$

(28)

Once the solution to Eq. (28) are calculated, the optimal values of $v_{3,m}$ and $b_{3,m}$ are obtained. Thus, alternatively projecting obtained non-zero vector $v_{3,m} \in \mathbb {R}^{n_2}$, we have $\hat{a}_i^{\rm T}=A_i{v_{3,m}}$, $\hat{b}_j^{\rm T}={B}_j{v_{3,m}}$ and $\hat{c}_j^{\rm T}={C}_j{v_{3,m}}$ in Eq. (14). So, now, we solve for the following modified optimization problem:

$$\begin{aligned}&\underset{u_{3,m},b_{3,m}}{Min} \frac{\alpha }{2} \sum \limits _{k \in I_3}||\hat{c}_ku_{3,m}+b_{3,m}||^2+ \frac{\beta }{2} (u_{3,m}^{\rm T}u_{3,m} + b_{3,m}^2) \nonumber \\&\quad + \frac{\gamma }{2} \sum \limits _{i \in I_1}||\hat{a}_iu_{3,m} +e_1b_{3,m} + (1-\rho _1-\epsilon ) ||^2\nonumber \\&\quad + \frac{\delta }{2} \sum \limits _{j \in I_2}||\hat{b}_ju_{3,m} + e_2b_{3,m} + (1-\rho _2-\epsilon )||^2. \end{aligned}$$

(29)

Working on the lines as above, and considering $\hat{z_{3,m}}=[u_{3,m} \quad b_{3,m}]$, we obtain $(u_{3,m}$ and $b_{3,m}$ as follows

$$\begin{aligned} z_{3,m}& = {} -(\alpha K_2^{\rm T}K_2+\beta I+\gamma H_2^{\rm T}H_2+\delta G_2^{\rm T}G_2)^{-1} (\gamma H_2^{\rm T}e_1+\delta H_2^{\rm T}e_1\rho _{1}+\gamma H_2^{\rm T}e_1\epsilon \nonumber \\ &\quad -\delta G_2^{\rm T}e_2 + \delta G_2^{\rm T}e_2 \delta + \delta G_2^{\rm T}e_2 \epsilon ) \end{aligned}$$

(30)

where $H_2$, $G_2$ and $K_2$ are matrices of points from classes + 1, − 1 and 0, respectively, augmented with a column of ones. Equations (28) and (30) are solved alternatively until $u_{3,m}$, $v_{3,m}$ and $b_{3,m}$ converges as per some predefined termination criteria.