Improved recurrent neural networks for solving Moore-Penrose inverse of real-time full-rank matrix

doi:10.1016/j.neucom.2020.08.026

Neurocomputing

Volume 418, 22 December 2020, Pages 221-231

https://doi.org/10.1016/j.neucom.2020.08.026 Get rights and content

Abstract

Recently, motivated by Zhang neural network (ZNN) models, Lv et al. presented two novel neural network (NNN) models for solving Moore-Penrose inverse of a time-invariant full-rank matrix. The NNN models were established by introducing two new matrix factors in the ZNN models, which results in their higher convergence rates than those of the ZNN models. In this paper we extend the NNN models to the more general cases through introducing a “regularization” parameter and a power parameter in these two matrix factors. The new proposed models are named here as the improved recurrent neural networks (IMRNN) since their convergence performance can be much better than the NNN models by appropriate choices of the introduced parameters. Such convergence property is theoretically analyzed in detail. Some numerical experiments are also performed to validate the theoretical results, including the numerical comparisons with the existing gradient neural network (GNN), ZNN and NNN models. In particular, the proposed IMRNN models are successfully applied to the inverse kinematic control of a three-link redundant robot manipulator where the superiority of the IMRNN models to the GNN, ZNN and NNN models is also indicated.

Introduction

It is known that Moore-Penrose inverse (i.e., pseudoinverse) plays a key role in science and engineering, e.g., system modeling and design in a variety of applications involving robotics [1], signal processing [2], image restoration [3] and pattern recognition [4]. Therefore, how to efficiently compute Moore-Penrose inverse of a matrix has become a critical problem in real-time applications. Although there exist many numerical methods for computing Moore-Penrose inverse such as singular value decomposition (SVD) [5], Greville’s recursive method [6], Newton iteration method [7] and so on, and from the viewpoint of computation, their serial computational schemes are inefficient when they are applied to large-scale and real-time problems [8], [9]. In fact, the computational costs of these known methods increase significantly with the increase of the size of the matrices because of the complexity $O (n^{3})$ being usually required, which makes the solving process time-consuming. However, there are real-time matrix Moore-Penrose inversion problems in practice, for example, the inverse kinematic problem for online control of redundant robot manipulator needs the online solution of Moore-Penrose inverse [10], [11] (see also the Section 6 in this paper). Obviously, the existing serial algorithms are not suitable for the online solution of Moore-Penrose inverse of a matrix. For this reason, the parallel computing methods become the natural choices for reducing computational costs and increasing the computational efficiency, especially for the online solution of Moore-Penrose inverse.

Due to the parallel nature and convenience of hardware implementation, recurrent neural network (RNN), originated from Hopfield neural network, has been deemed as a powerful tool to solve online matrices inversion problems [12], [13], [7], [14], [15], [16]. Over the past decades, many recurrent neural network models for computing Moore-Penrose inverse of a full-rank matrix have been proposed and investigated [17], [18], [11], [19]. As a special recurrent neural network, gradient neural network (GNN) is designed intrinsically for solving static problems, and it has become an effective tool for solving the Moore-Penrose inverse of a constant matrix [18], [17], [20]. Nevertheless, many methods including GNN models designed for static problems may not be effective for solving time-varying ones [10], [9]. To overcome this shortcoming, Zhang neural network (ZNN), as another special case of RNN, has been proposed for both time-invariant and time-varying problems solving [21], [22], [23]. Meanwhile, ZNN models have been successfully applied to solve matrix Moore-Penrose inversion [10], [24], [25], [26]. Recently, based on the GNN and ZNN models, Lv et al. [8] presented two more efficient RNN models which were named as novel neural network (NNN) models. By the detailed analyses and comparisons to these neural network models, they theoretically and numerically proved that the proposed NNN models possess the better convergence performance than that of the GNN and ZNN models for the real-time matrix Moore-Penrose inverse solving [8]. As a continuation of their works, in this paper, we further extend the NNN models to the more general cases named as Improved Recurrent Neural Network (IMRNN) models, which can outperform the existing GNN, ZNN and NNN models under certain conditions. Similar to the other papers, here we also highlight the main contributions and novelties of this paper as follows.

(1) Two IMRNN models can be utilized to efficiently solve the Moore-Penrose inverse of an real-time full-rank matrix, and they can globally converge to the exact Moore-Penrose inverse with an arbitrary initial state and some special monotonically-increasing odd activation functions.
(2) The convergence rate of the proposed IMRNN models can be much faster than GNN, ZNN and NNN models through an appropriate choice of the parameters $λ$ and k involved in the models.
(3) The influence of the involved parameters on the convergence rate of IMRNN models is theoretically analyzed in detail and some numerical experiments are performed to further substantiated our theoretical results.
(4) The application of our IMRNN model to the inverse kinematic control problem of redundant robot manipulator also indicates the superiority to the known GNN, ZNN and NNN models.

The rest of this paper is organized as follows. In Section 2, we recall some basic notions and useful results on Moore-Penrose inverse. In Section 3, the existed GNN, ZNN and NNN models for computing left or right Moore-Penrose inverse of full-rank matrix are described briefly. Two improved recurrent neural network (i.e., IMRNN-L and IMRNN-R) models are proposed in Section 4, including the theoretical analyses of their superior convergence performance. In Section 5, some numerical experiments are performed to validate the superiority of IMRNN models. Finally, as a practical application and the testing of effects, our IMRNN models are applied to the kinematic control of redundant robot manipulator in Section 6.

Section snippets

Moore-Penrose inverse: notions, formulae and necessary properties

In this section, we briefly state the definition of Moore-Penrose inverse of a matrix and some formulae of a full-rank matrix. Some necessary properties related to the Moore-Penrose inverse are also mentioned.

Definition 2.1

[27], [11] Let $A \in R^{m \times n}$ . Then the existed and unique matrix $X \in R^{n \times m}$ satisfying the following four matrix equations $AXA = A, XAX = X, {(AX)}^{T} = AX, {(XA)}^{T} = XA,$ is called the Moore-Penrose inverse of A and denoted by $A^{†}$ , where the superscript T denotes the transpose of a matrix or vector.

Specially, if a given

Some existed neural network models

To describe our improved recurrent neural networks, in this section we first state some existed neural networks for solving the Moore-Penrose inverse of a full-rank matrix.

Improved recurrent neural network (IMRNN) models

Based on the NNN models, in this section we proposed two improved recurrent neural networks i.e., IMRNN-L and IMRNN-R models for solving the Moore-Penrose inverses of the full-rank matrices.

The IMRNN-L model for the left Moore-Penrose inverse could be formulated as $A^{T} A \dot{X} (t) = - γ {({(A^{T} A)}^{2} + λ I)}^{k} F (A^{T} AX (t) - A^{T}),$ while IMRNN-R model for the right Moore-Penrose inverse could be written as $\dot{X} (t) {AA}^{T} = - γ F (X (t) {AA}^{T} - A^{T}) {({({AA}^{T})}^{2} + λ I)}^{k},$ where the designed parameters $γ, λ, k$ satisfy $γ > 0, λ \geq 1$ and that $k ⩾ 1$ is integer, $F (\cdot) : R^{m \times n} \to R^{m \times n}$

Illustrative examples

In this section, we perform some numerical experiments on our proposed IMRNN models (16) and (17). However, the dynamic equations of the above neural network models are all described in matrix forms, which cannot be directly numerically simulated. Thus, we firstly transform the matrix-form differential equations into the vector-form differential equations via the Kronecker product and vectorization techniques [30], [31]. For eaxmple, the IMRNN-L model (16) can be transformed to the vector-form

Application to redundant-manipulator kinematic control

In this section, we apply the proposed IMRNN models to the minimum norm (MVN) scheme in the redundant-manipulator kinematic control with a three-link planar robot manipulator. The kinematic structure of such a robot manipulator is shown in Fig. 7.

Consider a three-link redundant robot manipulator of which the relationship between the end-effector position vector $r = [r_{x}; r_{y}] \in R^{2}$ and the joint angle vector $θ = {[θ_{1}, θ_{2}, θ_{3}]}^{T} \in R^{3}$ is embodied through the following forward kinematic equation $r (θ) = φ (θ),$

where the

Conclusions

In this paper we extend the known NNN models to the more general ones by introducing two parameters $λ$ and k which can greatly improve the convergence performance of the NNN models by increasing the parameter values of $γ, λ$ and k when computing the Moore-Penrose inverse of a full-rank matrix. Our proposed new models are named as the improved recurrent neural network (IMRNN) models, i.e., the IMRNN-L and IMRNN-R models for solving the left and right Moore-Penrose inverses, respectively. We analyze

CRediT authorship contribution statement

Wenqi Wu: Conceptualization, Methodology, Software, Writing - original draft. Bing Zheng: Supervision, Project administration, Resources, Validation, Formal analysis.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The authors thank the anonymous referees for their constructive suggestions and comments which greatly improve the presentation of this paper. This work is supported by the National Natural Science Foundation of China (Grant No. 11571004).

Wenqi Wu received her B.S. degree from School of Mathematics and Statistics, Heze University, Heze, China, in 2018. She is now pursuing his M.S. degree at School of Mathematics and Statistics, Lanzhou University, Lanzhou, China. Her research interests include neural networks, generalized inverse of matrix, and numerical linear algebra.

References (37)

M.D. Petkovic et al.
Iterative method for computing the moore-penrose inverse based on penrose equations
J. Comput. Appl. Math.
(2011)
S. Shrivastava et al.
Performance evaluation of feed-forward neural network with soft computing techniques for hand written english alphabets
Appl. Soft Comput.
(2011)
Z. Zhang et al.
A power-type varying gain discrete-time recurrent neural network for solving time-varying linear system
Neurocomputing
(2020)
G. Wu et al.
A recurrent neural network for computing pseudoinverse matrices
Math. Comput. Model.
(1994)
Y. Wei
Recurrent neural networks for computing weighted moore-penrose inverse 1
Appl. Math. Comput.
(2000)
L. Xiao et al.
A new noise-tolerant and predefined-time ZNN model for time-dependent matrix inversion
Neural Networks
(2019)
X. Xiao et al.
A parallel computing method based on zeroing neural networks for time-varying complex-valued matrix moore-penrose inversion
Inf. Sci.
(2020)
L. Jin et al.
Nonconvex projection activated zeroing neurodynamic models for time-varying matrix pseudoinversion with accelerated finite-time convergence
Appl. Soft Comput.
(2018)
D. Guo et al.
Li-function activated ZNN with finite-time convergence applied to redundant-manipulator kinematic control via time-varying Jacobian matrix pseudoinversion
Appl. Soft Comput.
(2014)
Y. Zhang et al.
Effective neural remedy for drift phenomenon of planar three-link robot arm using quadratic performance index
Electron. Lett.
(2008)

A.J.V.D. Veen et al.

A subspace approach to blind space-time signal processing for wireless communications

IEEE Trans. Signal Process.

(1997)

S. Chountasis, V. N. Katsikis, D. Pappas, Digital image reconstruction in the spectral domain utilizing the...

K.M. Olson, G.A. Ybarra, Performance comparison of neural network and statistical pattern recognition approaches to...

R.E. Hartwig

Singular value decomposition and the moore-penrose inverse of bordered matrices

Siam J. Appl. Math.

(1976)

J. Zhou et al.

Variants of the greville formula with applications to exact recursive least squares

Siam J. Matrix Anal. Appl.

(2002)

X. Lv et al.

Improved recurrent neural networks for online solution of moorepenrose inverse applied to redundant manipulator kinematic control

Asian J. Control

(2018)

Y. Zhang et al.

New discrete-solution model for solving future different-level linear inequality and equality with robot manipulator control

IEEE Trans. Ind. Inf.

(2019)

Y. Zhang et al.

Zhang neural network solving for time-varying full-rank matrix moore-penrose inverse

Computing

(2011)

Cited by (20)

Zeroing neural network approaches for computing time-varying minimal rank outer inverse
2024, Applied Mathematics and Computation
Generalized inverses are extremely effective in many areas of mathematics and engineering. The zeroing neural network (ZNN) technique, which is currently recognized as the state-of-the-art approach for calculating the time-varying Moore-Penrose matrix inverse, is investigated in this study as a solution to the problem of calculating the time-varying minimum rank outer inverse (TV-MROI) with prescribed range and/or TV-MROI with prescribed kernel. As a result, four novel ZNN models are introduced for computing the TV-MROI, and their efficiency is examined. Numerical tests examine and validate the effectiveness of the introduced ZNN models for calculating TV-MROI with prescribed range and/or prescribed kernel.
A new recurrent neural network based on direct discretization method for solving discrete time-variant matrix inversion with application
2024, Information Sciences
In recent years, many researchers have worked hard to find a better way for solving discrete time-variant problems in industrial control science and automation. For example, some researchers propose RNN models to deal with such problems. Typical discrete time-variant problems, such as discrete time-variant matrix inversion, are developed from continuous time-variant problems. In the present paper, an efficient and straightforward method is proposed to solve discrete time-variant matrix inversion, note that it can skip the solving procedures of continuous time-variant problem and solves matrix inversion directly in the discrete time-variant environment. Specifically, an innovative discrete time-variant recurrent neural network (I-DT-RNN) model for dealing with discrete time-variant matrix inversion is proposed, furthermore it is mathematically founded on the second-order Taylor expansion. The theoretical analysis results of I-DT-RNN model are also presented, which proves that the proposed I-DT-RNN model has a reasonable characteristic and also shows that the proposed I-DT-RNN model has an excellent computational performance. Moreover, in the numerical experiments part, we present three different matrices as numerical experiment examples and an application of two-link robot manipulator as an industrial example for validating the practicability of the I-DT-RNN model.
Discrete gradient-zeroing neural dynamics for future Moore–Penrose inverse with application to tracking control of manipulator
2023, Expert Systems with Applications
To handle the future Moore–Penrose inverse problem effectively, the continuous gradient-zeroing neural dynamics (CGZND) model without inverse computing is proposed via combining the advantages of the zeroing neural dynamics (ZND) model and the gradient neural dynamics (GND) model. In order to readily implement in computer and digital hardware, the order-6 discrete gradient-zeroing neural dynamics (O6-DGZND) algorithm is further proposed via the simultaneous combination of the order-6 Zhang time discretization (O6-ZTD) formula and the order-6 extrapolation (O6-E) formula. The O6-DGZND algorithm is not only highly accurate, but also utilized to solve the challenging Z-type future Moore–Penrose inverse problem. Besides, four other DGZND algorithms with low-precision are also presented to meet different precision requirements in actual engineering applications. The convergence properties of the DGZND algorithms, including the 0-stability, consistency, and truncation error, are theoretically analyzed. Numerical experiment results illustrate the efficacy and validity of the proposed DGZND algorithms, and the superiority of the O6-DGZND algorithm is also verified. In addition, with the aid of the O6-DGZND algorithm, an application on a UR10 robot manipulator tracking task is performed successfully, which implies that the practicability of the O6-DGZND algorithm in real-world applications.
Direct derivation scheme of DT-RNN algorithm for discrete time-variant matrix pseudo-inversion with application to robotic manipulator
2023, Applied Soft Computing
Citation Excerpt :
However, generally speaking, continuous time-variant algorithms may be more difficult to implement in industry than discrete time-variant algorithms, such as on the digital circuits and digital computers. Many researchers proposed improved RNN algorithms for solving discrete time-variant matrix pseudo-inversion after Zhang et al. introduced the algorithm [27], such as Wei et al. [28], Guo et al. [29], Liao et al. [30] and Petković et al. [31]. Besides, the study and processing of some mathematical problems can also be realized by RNNs [32–37].
The improvement of recurrent neural network (RNN) algorithms is one of targets of many researchers, and these algorithms are widely used to solve time-variant problems in a variety of domains. A novel direct derivation scheme of discrete time-variant RNN (DT-RNN) algorithm for addressing discrete time-variant matrix pseudo-inversion is discussed in this paper. To be more specific, firstly, a DT-RNN algorithm mathematically founded on the second-order Taylor expansion is proposed for dealing with discrete time-variant matrix pseudo-inversion, and it does not require the theoretical support of continuous time-variant RNN (CT-RNN) algorithm. Secondly, the results of theoretical analyses of the proposed DT-RNN algorithm are also presented in this paper. These results demonstrate that the novel DT-RNN algorithm has remarkable computing performance. The efficiency and applicability of the DT-RNN algorithm have been verified through one numerical experiment and two robotic manipulator experiments.
A novel hybrid Zhang neural network model for time-varying matrix inversion
2022, Engineering Science and Technology, an International Journal
Citation Excerpt :
Xiao et al. [22] proposed a complex valued RNN model for time-varying complex matrix inversion which converges faster than CVZNN [19]. An improved RNN model for solving Moore-Penrose inverse of real-time full-rank matrix was investigated in [23]. The remaining part of the paper is arranged as follows: The problem statement is described in Section 2.
A new hybrid Zhang neural network (HZNN) model is formulated to solve time-varying matrix inversion problem. The network is designed such that it encompasses the goodness of both gradient neural network (GNN) and Zhang neural network (ZNN). The stability of the network and convergence results are established theoretically. The rate of convergence of HZNN model is much faster compared to the classical GNN model and ZNN model for time-varying matrix inversion problem. Nonlinear activation functions for HZNN need not have to be odd and monotonically increasing for the network to converge, however with certain restrictions. Simulations are carried out to test the efficiency of HZNN over GNN and ZNN. Also, HZNN with odd and non-odd activation functions are examined for convergence. It is observed that the network converges much faster with the proposed non-odd function 3 $(nof 3)$ . Furthermore, the HZNN model with odd and non-odd functions are applied to the kinematic control of a two-link planar manipulator for tracking a circular path.
Solving quaternion nonsymmetric algebraic Riccati equations through zeroing neural networks
2024, AIMS Mathematics

View all citing articles on Scopus

Bing Zheng was born in 1963. He received the M.S. degree in fundamental Mathematics from the School of Mathematical Sciences, Anhui University, China, in 1989, and Ph.D. degree in computational mathematics from the College of Sciences, Shanghai University, China, in 2003. He is currently a full time professor of the School of Mathematics and Statistics at Lanzhou University, China. His research interests include numerical linear algebra and its applications, multi-linear algebra, optimization theory, total least squares problem, perturbation analysis of the matrix and so on. He has published more than 80 papers in scientific journals.

View full text

Improved recurrent neural networks for solving Moore-Penrose inverse of real-time full-rank matrix

Abstract

Introduction

Section snippets

Moore-Penrose inverse: notions, formulae and necessary properties

Some existed neural network models

Improved recurrent neural network (IMRNN) models

Illustrative examples

Application to redundant-manipulator kinematic control

Conclusions

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgements

J. Comput. Appl. Math.

Appl. Soft Comput.

Neurocomputing

Math. Comput. Model.

Appl. Math. Comput.

Neural Networks

Inf. Sci.

Appl. Soft Comput.

Appl. Soft Comput.

Effective neural remedy for drift phenomenon of planar three-link robot arm using quadratic performance index

Electron. Lett.

A subspace approach to blind space-time signal processing for wireless communications

IEEE Trans. Signal Process.

Singular value decomposition and the moore-penrose inverse of bordered matrices

Siam J. Appl. Math.

Variants of the greville formula with applications to exact recursive least squares

Siam J. Matrix Anal. Appl.

Improved recurrent neural networks for online solution of moorepenrose inverse applied to redundant manipulator kinematic control

Asian J. Control

New discrete-solution model for solving future different-level linear inequality and equality with robot manipulator control

IEEE Trans. Ind. Inf.

Zhang neural network solving for time-varying full-rank matrix moore-penrose inverse

Computing