Alternate algorithms to most referenced techniques of numerical optimization to solve the symmetric rank-R approximation problem of symmetric tensors

Abstract

The tensor low-rank approximation and tensor CANDECOMP/PARAFAC (CP) decomposition are useful in various fields such as machine learning, dimension reduction, tensor completion, data visualization etc. A symmetric tensor is a higher order generalization of a symmetric matrix. Comon et al. (2008) show that every symmetric tensor has a symmetric CP-decomposition. In this paper, we study numerical methods of real-valued symmetric CP-decomposition of symmetric tensors. We present an alternate gradient descent method, an alternate BFGS method and an alternate Levenberg–Marquardt (L–M) method for real-valued symmetric rank-R approximation of symmetric tensors. Moreover, we prove the convergence and effectiveness of the algorithms. Numerical examples show that the alternate gradient descent method costs more computing time than the other two methods and the latter two methods have high success rate and good stability.

Introduction

Notations: In the paper, we denote $\mathbb{F}=ℂ$ as the complex field or $\mathbb{R}$ as the real field, respectively. Let $s$ be a positive integer, denote $\left[s\right]=\{1,\dots ,s\}$ as an integer set from $1$ to $s$. High-order tensors are denoted by calligraphic letters as $\mathcal{A},\mathcal{B},\mathcal{C},\dots$, matrices are denoted by capital letters as $A,B,C,\dots$, and vectors are denoted by letters as $\mathbf{a},\mathbf{b},\mathbf{c},\dots$. Let $\mathbf{n}=(n_1,\dots ,n_m)$ be a positive integer vector, $T\left[m\right]\mathbb{F}\left[\mathbf{n}\right]$ denotes the set of order-$m$ dimension-$(n_1,\dots ,n_m)$ tensors. For positive integers $m$ and $n$, $T\left[m\right]\mathbb{F}\left[n\right]$ and $S\left[m\right]\mathbb{F}\left[n\right]$ denote the set of order-$m$ dimension-$n$ square tensors and the set of order-$m$ dimension-$n$ symmetric tensors, respectively.

Given matrix $U\in {\mathbb{R}}^{n\times R}$, we consider the following products:

$$U^{\otimes m}=\underset{m}{\underbrace{U\otimes U\otimes \cdots \otimes U}},U^{\odot m}=\underset{m}{\underbrace{U\odot U\odot \cdots \odot U}},$$$$U^{\ast m}=\underset{m}{\underbrace{U\ast U\ast \cdots \ast U}},U^{\circ m}=\underset{m}{\underbrace{U\circ U\circ \cdots \circ U}},$$

where $\otimes$, $\odot$, $\ast$, $\circ$ denote Matrix Kronecker product, Matrix Khatri–Rao product, Matrix Hadamard product and vector outer product, respectively.

A tensor is usually denoted as $\mathcal{T}=\left({\mathcal{T}}_{i_1,\dots ,i_m}\right)\in T\left[m\right]\mathbb{F}\left[\mathbf{n}\right]$ and represents a multi-array of entries ${\mathcal{T}}_{i_1,\dots ,i_m}\in \mathbb{F}$, where $i_j=1,\dots ,n_j$, $j=1,\dots ,m$. $\mathbb{F}=ℂ$ or $\mathbb{R}$. When $n_1=\cdots =n_m=n$, $\mathcal{T}$ is called an order-$m$ dimension-$n$ square tensor. For any order-$m$ dimension-$n$ square tensor $\mathcal{S}=\left({\mathcal{S}}_{i_1,\dots ,i_m}\right)\in T\left[m\right]\mathbb{F}\left[n\right]$, if its entries are invariant under any permutation of its indices, then $\mathcal{S}$ is called a symmetric tensor [1]. The CANDECOMP/PARAFAC(CP) decomposition can be considered as higher-order generalizations of the matrix singular value decomposition(SVD) and principal component analysis(PCA) [2]. The CP-decomposition factorizes a tensor into a sum of component rank-one tensors. For example, given a tensor $\mathcal{T}\in T\left[m\right]\mathbb{R}\left[\mathbf{n}\right]$, we wish to write it as

$$\mathcal{T}=\sum _{k=1}^R{\mathbf{u}}_k^{\left(1\right)}\circ {\mathbf{u}}_k^{\left(2\right)}\circ \cdots \circ {\mathbf{u}}_k^{\left(m\right)},$$

where $R$ is a positive integer and ${\mathbf{u}}_k^{\left(j\right)}\in {\mathbb{R}}^{n_j}$ for $k=1,\dots ,R$, $j=1,\dots ,m$. ‘$\circ$’ denotes the tensor product or the vector outer product.

Let $\mathcal{A}$ represent an order-$m$ dimension-$n$ symmetric tensor. Given a real-valued vector $\mathbf{u}$ of length $n$, we let ${\mathbf{u}}^{\circ m}$ denote the order-$m$ dimension-$n$ outer product rank-one symmetric tensor such that ${\left({\mathbf{u}}^{\circ m}\right)}_{i_1,\dots ,i_m}=u_{i_1}\cdots u_{i_m}$. Comon et al. [3] show that any real-valued symmetric tensor $\mathcal{A}$ can be decomposed as

$$\mathcal{A}=\sum _{k=1}^R{\lambda }_k{\mathbf{u}}_k^{\circ m},$$

with ${\lambda }_k\in \mathbb{R}$ and ${\mathbf{u}}_k\in {\mathbb{R}}^n$ [4].

The tensor low-rank approximation and tensor CP-decomposition are useful in various fields such as machine learning [5], dimension reduction [6], tensor completion [7], data visualization [8]. etc. The alternating least squares (ALS) algorithm is a common numerical algorithm solving the tensor decomposition problem. There are also some other algorithms such as the gradient-based optimization algorithm [9], the conjugate gradient algorithm for nonnegative 3-way tensor factorization [10], the damped Gauss–Newton algorithm for factorization of low-rank real- and complex-valued tensors by deriving a fast inverse for the approximate Hessian [11], randomized algorithms for the low multilinear rank approximations of tensors [12] and the second-order algorithm for fitting the canonical polyadic decomposition with non-least-squares cost [13].

In 2015, Kolda studies the problem of symmetric tensor real-valued decomposition with low-rank structure focusing on both unconstrained and nonnegative by computing the gradients [4]. In 2022, Liu proposes the alternate gradient descent method to solve the symmetric tensor decomposition problem for given orthogonal symmetric tensors or the orthogonal symmetric tensors with a small perturbations [14].

In this paper, we focus on the problem of general symmetric tensor real-valued decomposition. We first introduce an alternate algorithm framework for rank-R symmetric approximation of symmetric tensors, and then we design three numerical algorithms, which are alternate gradient descent algorithm, alternate BFGS algorithm and alternate Levenberg–Marquardt (L–M) algorithm. We also prove the convergence and effectiveness of these algorithms. Numerical experiments show the effectiveness of these algorithms.

The paper is structured as follows. In Section 2, we introduce the basic knowledge of tensor CP-decomposition. In Section 3, we deduce optimization formulation for symmetric rank-R approximation problem of symmetric tensors. In Section 4, we propose an alternate algorithm framework as well as an alternate gradient descent method, an alternate BFGS method and an alternate L–M method. Some numerical examples and experimental results are given in Section 5.