Adaptive Coefficient Designs for Nonlinear Activation Function and Its Application to Zeroing Neural Network for Solving Time-Varying Sylvester Equation

https://doi.org/10.1016/j.jfranklin.2020.06.029Get rights and content

Abstract

For obtaining better convergence performance when zeroing neural network (ZNN) is applied to solve time-varying Sylvester equation, three different types of adaptive design coefficients for the sign-bi-power nonlinear activation function are developed and investigated in this paper. Based on these adaptive coefficients, three new adaptive ZNN models are proposed to improve the convergence performance of the standard ZNN model. For better analysis, the effect of two parts constituting the sign-bi-power activation function on the convergence of the standard ZNN model is first discussed. Then, detailed theoretical derivations and proofs are provided to verify excellent performance of the proposed adaptive ZNN models. Finally, illustrative comparison experiments are presented to show the enhanced finite-time convergence performance of the proposed adaptive ZNN models for solving time-varying Sylvester equation.

Introduction

The Sylvester equation is a complicated computational problem often encountered in scientific research and engineering applications [1], [2], [3], [4], [5], [6], [7]. Especially in the field of control theory, the Sylvester equation is often used to analyze the stability of the dynamics system, such as in robotic application [5], [8] and robot manipulator control [6], [9].

In general, the effective research results regarding solving the Sylvester equation can be divided into two categories: one is the numerical solution based on serial processing which consists of a series of linear transformations and matrix decompositions [2], [4], [7], [10], [11], [12], and the other is the neural computation solution based on parallel processing which consists of recurrent neural networks [5], [6], [13], [14], [15], [16]. As well known, the classical algorithm for the numerical solution of the Sylvester equation is the Bartels-Stewart algorithm [10], which uses the QR algorithm to transform the coefficient matrix into a Schur form, and then invert the triangular matrix by the back-substitution method. However, such an algorithm exists a cubic time complexity about the dimension of the coefficient matrix. It means that this algorithm is not very effective for real-time application if it cannot complete the execution in every sampling period. Furthermore, the high complexity of the serial numerical solution makes it impossible to complete calculations in large-scale data calculations, not to mention the more challenging scenario to solve the equation continuously in real-time.

In contrast to conventional numerical methods, the dynamic solution based on the neural network can be regarded as a parallel processing technique that is exceedingly competitive in solving such complex computing problems. It is worth pointing out that gradient-based neural network (GNN) [15], [16], a classic kind of neural dynamic methods, has been reported for finding the solution of the Sylvester equation. The first step in the design and implementation of the solver using GNN is to define a scalar-valued energy function that is used to evaluate the computational error of GNN. Then, a well-designed evolutionary formula is proposed to force the error to decrease along the negative gradient-based descent direction. At last, GNN completes the solution to the Sylvester equation when the error function converges to zero. Although the neural dynamic approach can cope with the increased amount of the computational complexity, It does not perform well in the online solution for the time-varying problem. This is because the GNN solver ignores the varying error information over time during the implementation process [13], [17], [18].

Because of the above defects arisen in GNN for solving time-varying Sylvester equation, zeroing neural network (ZNN) as a type of recurrent neural network is presented as a novel effective method to crack this issue [13]. During the design and implementation of ZNN, the first-order derivatives of the time-varying coefficients are used as the velocity compensation of time-varying parameters. Therefore, the study of ZNN has exerted a tremendous fascination on many researchers due to its strong ability to handle real-time calculations. Moreover, in Refs. [6], [18], [19], [20], [21], [22], varieties of ZNN models have been reported, studied and designed for solving different time-varying calculation problems, including the theme discussed in this paper, the Sylvester equation.

To better demonstrate the contribution of this paper and explain the development of ZNN, some related works are introduced and summarized as follows. The authors in [13] delicately designed a recurrent neural network for solving the Sylvester equation with time-varying coefficient matrices. The implicit dynamical equation was introduced to ensure the computational error decrease to zero exponentially. By following this successful research direction, a variety of extended neural dynamical models are proposed to address different zeroing problems. This type of isomorphic neural network with the same design architecture is called zeroing neural network (ZNN). Although ZNN’s design architecture provides a guarantee for exponential convergence of the neural computational model, the later research finds that the convergence speed can be further accelerated by designing different activation functions. Based on this idea, many studies on activation functions have been developed to enable ZNN to achieve superior convergence performance [18], [23], [24], [25], [26]. But still the ZNN model activated by these nonlinear functions has an imperfection. That is, the neural model needs infinite time to converge to zero. This situation has continued until the emergence of a technological breakthrough when Li et al. presented the sign-bi-power activation function to accelerate ZNN to finite-time convergence in 2013 [14]. After that, varieties of finite-time ZNN models are designed and studied for solving different time-varying problems [20], [27], [28], [29], [30]. Furthermore, some studies on modifying the sign-bi-power function to obtain the superior convergence speed and the accurate upper bound have also been reported in [31], [32].

Inspired by the above existing results, the motivation of this paper is to improve the convergence performance of ZNN by adding adaptive coefficients during the design process of the model. After analyzing the sign-bi-power function, we discover that the sign-bi-power function can be regarded as a combination of two specially constructed parts which act differently but have the same coefficient as 1/2. For this reason, the ratio of two parts that are activated by these two functions respectively is fixed from beginning to end, despite the constantly changing over time of the error of neuron in ZNN. However, the role of each part played in the sign-bi-power function greatly depends on the computational error of ZNN. In light of this deficiency of the sign-bi-power function, we believe that designing a proper adaptive coefficient correspondingly with the variation of the computational error can accelerate the convergence speed of ZNN. Keep this idea in mind, in this paper, three different types of adaptive design coefficients for the sign-bi-power nonlinear activation function are developed and investigated in this paper for solving the time-varying Sylvester equation. Based on these adaptive coefficients, three new ZNN models are proposed to improve the convergence performance of the standard ZNN model. Then the improved finite-time convergence is investigated by analytical theorems and demonstrated by illustrative experiments. Therefore, the validity of the newly proposed models is completely verified. To the best of our knowledge, this is the first time to design adaptive coefficients to adjust the ZNN model to achieve superior convergence performance. The main contributions of this paper are listed as follows.

  • The role of two components of the sign-bi-power function is first studied to show how to impact the convergence speed of the standard ZNN model.

  • By designing three different adaptive coefficients for the sign-bi-power function, three new adaptive ZNN models are proposed for solving time-varying Sylvester equation.

  • The superior finite-time convergence of the proposed adaptive ZNN models is theoretically proved and the corresponding convergence upper bounds are derived to be less conservative.

  • Numerical comparative experiments have been conducted to show that the proposed adaptive ZNN models have faster convergence speed and shorter convergence time.

The rest of this paper is organized as follows. Section 2 introduces some recently published work related to the topic of this paper. In Section 3, the time-varying Sylvester equation is proposed and the standard ZNN model design procedures are given. Besides, the role of the two parts of the sign-bi-power function in accelerating model convergence is theoretically analyzed. Section 4 presents three new ZNN models with adaptive coefficients and the theoretical results are provided correspondingly. In Section 5, an illustrative example is given to demonstrate the better convergence performance and verify the theoretical theorems. Section 6 is the conclusion of this paper.

Section snippets

Related Work

In this section, many studies related to the topic of this work are introduced to enhance the research background and make a comparison. These studies can be summarized in the following aspects. First, adaptive design ideas have been widely applied in the programming of the control system. By adding adaptive coefficients to the model design of the control system, its automatic adjustment performance is effectively improved. In [33], an adaptive fuzzy stochastic switched control scheme was

ZNN and Activation Function

This work concentrates on calculating the following time-varying Sylvester equation of the form [5], [6] :A(t)X(t)X(t)B(t)=C(t),where t[0,+) denotes time, A(t)Rn×n, B(t)Rn×n and C(t)Rn×n are given coefficient matrices, and X(t) represents an unknown matrix to be found to obey Eq. (1). For the discussion to make sense, we assume there are no common eigenvalues between A(t) and B(t) at any time t. Therefore the Sylvester Eq. (1) has a unique solution for X(t) exactly.

New ZNN Models with Adaptive Coefficients

On the basis of discussion in the above section, three new ZNN models are first proposed to improve the convergence performance of the standard ZNN model (4) by designing three adaptive coefficients in the sign-bi-power function (5). Then, the improved finite-time convergence of new ZNN models is theoretically proven, and the corresponding theoretical upper bounds are analytically calculated. For concise expression, three new ZNN models are respectively called the static adaptive ZNN (SA-ZNN)

Numerical Verification Example

In the previous sections, three ZNN models with adaptive coefficients [i.e., SA-ZNN model (22), PA-ZNN model (29) and EA-ZNN model (37)] are proposed and analyzed for solving Sylvester Eq. (1). To further confirm better convergence performance and verify the theoretical results, an illustrative example is given, solved and explained in this section. In addition, the proposed three new models and the standard ZNN model activated by the sign-bi-power function are simulated by digital computer for

Conclusion

Three novel zeroing neural network (ZNN) models with different adaptive coefficients have been proposed for solving time-varying Sylvester equation in this work. For designing better adaptive coefficients, the role of two components of the sign-bi-power function has been first studied to impact the convergence speed of the standard ZNN model. Based on the theoretical conclusions, three different adaptive coefficients have been designed to improve the convergence performance of the standard ZNN

Declaration of Competing Interest

I confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.

Acknowledgment

This work was supported by the National Natural Science Foundation of China under grants 61866013, 61976089 and 61966014; the Natural Science Foundation of Hunan Province of China under grants 2019JJ50478 and 18A289; and the Hunan Provincial Science and Technology Project Foundation of China under grants 2018TP1018 and 2018RS3065.

References (43)

  • L. Xiao et al.

    From different Zhang functions to various ZNN models accelerated to finite-time convergence for time-varying linear matrix equation

    Neural Process. Lett.

    (2014)
  • W. Li

    Design and analysis of a novel finite-time convergent and noise-tolerant recurrent neural network for time-variant matrix inversion

    IEEE Trans. Syst. Man Cybern. -Syst.

    (2018)
  • Q. Wei et al.

    Fast fusion of multi-band images based on solving a Sylvester equation

    IEEE Trans. Image Process

    (2015)
  • G. Sangalli et al.

    Isogeometric preconditioners based on fast solvers for the Sylvester equation

    SIAM J. Sci. Comput.

    (2016)
  • Q. Wei et al.

    R-FUSE: Robust fast fusion of multiband images based on solving a Sylvester equation

    IEEE Signal Process. Lett.

    (2016)
  • D. Sorensen et al.

    The Sylvester equation and approximate balanced reduction

    Linear Alg. Appl.

    (2002)
  • B. Huang et al.

    Gradient-based iterative algorithms for generalized coupled Sylvester-conjugate matrix equations

    Comput. Math. Appl.

    (2018)
  • X. Sheng

    A relaxed gradient based algorithm for solving generalized coupled Sylvester matrix equations

    J. Frankl. Inst.

    (2018)
  • S. Li et al.

    Accelerating a recurrent neural network to finite-time convergence for solving time-varying Sylvester equation by using a sign-bi-power activation function

    Neural Process. Lett.

    (2013)
  • Y. Zhang et al.

    MATLAB simulation of gradient-based neural network for Sylvester equation solving

    Journal of System Simulation.

    (2009)
  • Y. Zhang et al.

    Performance analysis of gradient neural network exploited for online time-varying matrix inversion

    IEEE Trans. Autom. Control.

    (2009)
  • Cited by (0)

    View full text