Incremental Quaternion Random Neural Networks

Abstract

Quaternion, as a hypercomplex number with three imaginary elements, is effective in characterizing three- and four-dimensional vector signals. Quaternion neural networks with randomly generated quaternions as the hidden node parameters become attractive for the good learning capability and generalization performance. In this paper, a novel incremental quaternion random neural network trained by extreme learning machine (IQ-ELM) is proposed. To fully exploit the second-order Q-properness statistic of quaternion random variables, the augmented quaternion vector is further applied in IQ-ELM (IAQ-ELM) for hypercomplex data learning. The network is constructed by gradually adding the hidden neuron one-by-one, where the output weight is optimized by minimizing the residual error based on the fundamental of the generalized HR calculus (GHR) of quaternion variable function. Experiments on multidimensional chaotic system regression, aircraft trajectory tracking, face and image recognition are conducted to show the effectiveness of IQ-ELM and IAQ-ELM. Comparisons to two popular quaternion RNNs, Schmidt NN (SNN) and random vector functional-link net (RVFL), are also provided to show the feasibility and superiority of using quaternions in RNN for incremental learning.

This work was supported by the National Natural Science Foundation of China (U1909209), the National Key Research and Development Program of China (2021YFE0100100, 2021YFE0205400), the Natural Science Key Foundation of Zhejiang Province (LZ22F030002), and the Research Funding of Education of Zhejiang Province (GK228810299201).

A Appendices

1.1 A.1 Proof of Proposition 1

With the well established GHR of quaternion variable function and Lemmas 2${\sim } 4$, the optimization of (7) can be solved by finding the gradient of the objective to $\beta _n$. It is observed that the norm of the objective residual error $||e_n||^2$ can be equivalently calculated as $e_n e_n^*$ or $e_n^* e_n$, where $e_n^*$ is the conjugate of $e_n$. With the left- and right-derivatives, and the product rule in quaternion derivative, one can readily find that the same results will be obtained either on $e_n e_n^*$ or $e_n^* e_n$. We take the derivative to $e_n^* e_n$ as an example in this paper, with Lemma 3, which can be expressed as

$$\begin{aligned} \begin{aligned} \frac{\partial ||e_n||^2}{\partial {\beta _n}}&=e^*_n \frac{\partial e_n}{\partial {\beta _n}}+\frac{\partial e^*_n}{\partial {\beta _n}^{e_n}}e_n\\&=e^*_n \frac{\partial (e_{n-1}-\beta _n \sigma _n)}{\partial {\beta _n}}+\frac{\partial (e_{n-1}-\beta _n \sigma _n)^*}{\partial {\beta _n}^{e_n}}e_n \end{aligned} \end{aligned}$$

(15)

Since $e_{n-1}$ and $\beta _n$ are not related, $\frac{\partial e_{n-1}}{\beta _n}=0$. With (4), we have

$$\begin{aligned} \frac{\partial ||e_n||^2}{\partial {\beta _n}}&=e^*_n \frac{\partial e_n}{\partial {\beta _n}}+\frac{\partial e^*_n}{\partial {\beta _n}^{e_n}}e_n\\&=e^*_n \frac{\partial (e_{n-1}-\beta _n \sigma _n)}{\partial {\beta _n}}+\frac{\partial (e_{n-1}-\beta _n \sigma _n)^*}{\partial {\beta _n}^{e_n}}e_n\\&=-e^*_n S(\sigma _n)+\frac{\partial (-\sigma ^*_n \beta ^*_n)}{\partial {\beta _n}^{e_n}}e_n\\&=-e^*_n S(\sigma _n)+\frac{1}{2}\sigma ^*_n e^*_n\\&=-e^*_n S(\sigma _n)+\frac{1}{2}S(\sigma _n)e^*_n-\frac{1}{2}V(\sigma _n)e^*_n\\&=-\frac{1}{2}\sigma _n({e^*_{n-1}}-{\sigma ^*_n} {\beta ^*_n}). \end{aligned}$$

Here, $e_{n-1}$ denotes the residual error of previous $n-1$ nodes, $\sigma _n$ represents the n-th hidden node output, $S(\sigma _n)$ and $V(\sigma _n)$ are the real (scalar) and imaginary parts of the output $\sigma _n$, respectively. When $\frac{\partial ||e_n||^2}{\partial \beta _n}=0$, the objective (7) reaches the minimum, so we have

$$ -\frac{1}{2}\sigma _n({e^*_{n-1}}-{\sigma ^*_n} {\beta ^*_n})=0$$

$$\sigma _n e^*_{n-1}=\sigma _n {\sigma ^*_n} \beta ^*_n$$

$$\beta ^*_n=\frac{\sigma _n e^*_{n-1}}{||\sigma _n||^2}$$

If and only if $\beta _n=\frac{e_{n-1}{\sigma ^*_n}}{||\sigma _n||^2}$, the objective (7) reaches the minimum. That finishes the proof.