Multilayer Perceptrons to Approximate Quaternion Valued Functions

doi:10.1016/S0893-6080(96)00048-2

Neural Networks

Volume 10, Issue 2, March 1997, Pages 335-342

https://doi.org/10.1016/S0893-6080(96)00048-2 Get rights and content

Abstract

In this paper a new type of multilayer feedforward neural network is introduced. Such a structure, called hypercomplex multilayer perceptron (HMLP), is developed in quaternion algebra and allows quaternionic input and output signals to be dealt with, requiring a lower number of neurons than the real MLP, thus providing a reduced computational complexity. The structure introduced represents a generalization of the multilayer perceptron in the complex space (CMLP) reported in the literature. The fundamental result reported in the paper is a new density theorem which makes HMLPs universal interpolators of quaternion valued continuous functions. Moreover the proof of the density theorem can be restricted in order to formulate a density theorem in the complex space. Due to the identity between the quaternion and the four-dimensional real space, such a structure is also useful to approximate multidimensional real valued functions with a lower number of real parameters, decreasing the probability of being trapped in local minima during the learning phase. A numerical example is also reported in order to show the efficiency of the proposed structure. © 1997 Elsevier Science Ltd. All Rights Reserved.

Section snippets

INTRODUCTION

In the last decade the use of artificial neural networks has received a growing interest in a wide variety of scientific fields. A great number of neural structures have been proposed in the literature, in order to deal with different kinds of applications. Among these structures, our interest in this paper focuses on multilayer perceptron based structures, able to deal with quaternionic input and output signals.

Quaternion algebra was invented in 1843 by the Irish mathematician W.R. Hamilton (

MLPS IN QUATERNION ALGEBRA

Let us define an HMLP (hypercomplex multilayer perceptron) as a multilayer perceptron in which input and output values, weights and biases are quaternions, and the activation functions in the hidden layer neurons are quaternion-valued sigmoidal functions. In the following, only HMLPs with one hidden layer will be considered, since it will be proven that this structure is a universal interpolator in H. However, the learning algorithm has been also developed for a multi-hidden layer structure (

DENSITY THEOREMS FOR HMLPS

In this section a density theorem for continuous functions f: X → H, where X is a compact subset of Hⁿ will be proven. As for the real MLP (Cybenko, 1989) and the CMLP (Arena et al., 1993) such a result states that HMLPs with the activation function (2) are universal interpolators of continuous quaternion valued functions. In order to prove the theorem the following notation is introduced: let σ: $R$ → $R$ be the real valued sigmoidal function: $σ(t)= 1 1+ exp (−t)$ and let q = q₀ + iq₁ + jq₂ + kq₃ ∈ H.

A NUMERICAL EXAMPLE

In this section a numerical example is reported to remark the efficiency of the HMLP. The considered task is the short term prediction of a time series generated from a chaotic electronic circuit, namely the hyperchaotic Saito circuit (Mitsubori and Saito, 1994). The system is characterized by four state variables; the state equations, numerically simulated to generate the learning patterns, are the following: $x ̇_{1} y ̇_{1} = −1 1 −α_{1} α_{1} β_{1} x_{1} −ηp_{1} h(z) y_{1} −η p_{1} β_{1} h(z)$ $x ̇_{2} y ̇_{2} = −1 1 −α_{2} α_{2} β_{2} x_{2} −ηp_{2} h(z) y_{2} −η p_{2} β_{2} h(z)$ where

CONCLUSIONS

The approximation capabilities of a new type of multilayer feedforward neural network defined in the quaternion algebra (HMLP) are theoretically analyzed in this paper. For this structure a new density theorem is proven in the quaternion set. It is therefore possible to state that HMLPs are able to approximate quaternionic continuous functions with the desired degree of accuracy. Moreover HMLP approximation capabilities are useful also for real and complex functions: in such cases HMLPs allow

References (37)

K.I. Funahashi
On the approximate realization of continuous mappings by neural networks
Neural Networks
(1989)
D. Hestenes
Invariant body kinematics: I. Saccadic and compensatory eye movements
Neural Networks
(1994)
K. Hornik et al.
Multilayer feedforward networks are universal approximators
Neural Networks
(1989)
K.W. Spring
Euler parameters and the use of quaternion algebra in the manipulation of finite rotations: a review
Mechanism Machine Theory
(1986)
Arena, P. (1994). Neural networks for modelling and identifiction in circuits and systems. PhD Thesis, DEES, University...
Arena, P., Fortuna, L., Re, R., & Xibilia, M.G. (1993). On the capability of neural networks with complex neurons in...
Arena, P., Fortuna, L., & Xibilia, M.G. (1994a). Predicting complex chaotic time series via complex valued MLPs IEEE...
Arena, P., Fortuna, L., Occhipinti, L., & Xibilia, M.G. (1994b). Neural Networks for quaternion valued function...
Arena, P., Baglio, S., Fortuna, L., & Xibilia, M.G. (1995). Chaotic time-series prediction via quaternionic MLPs....
P. Arena et al.
Neural networks in multidimensional domains
Journal of System Engineering
(1996)

Arena, P., Fortuna, L., Re, R., & Xibilia, M.G. (1996b). Multilayer perceptrons to approximate complex valued...

Benvenuto, N., & Piazza, F. (1992). On the complex backpropagation algorithm. IEEE Transactions On signal Processing 40...

Benvenuto, N., Marchesi, M., Piazza, F., & Uncini, A. (1991a). Nonlinear satellite radio links equalized using blind...

Benvenuto, N., Marchesi, M., Piazza, F., & Uncini, A. (1991b). A comparison between real and complex valued neural...

J. Chou

Quaternion kinematic and dynamic differential equation

IEEE Transactions on Robotics and Automation

(1992)

J. Chou et al.

Finding the position and orientation of a sensor on a robot manipulator using quaternions

International Journal of Robotics Research

(1991)

Cui, X., & Shin, K. (1991). Intelligent coordination of multiple systems with neural networks. IEEE Transactions on...

G. Cybenko

Approximation by superposition of a sigmoidal function

Mathematics of Control Signals and Systems

(1989)

Cited by (140)

A comparison of quaternion neural network backpropagation algorithms
2023, Expert Systems with Applications
This research paper focuses on quaternion neural networks (QNNs) - a type of neural network wherein the weights, biases, and input values are all represented as quaternion numbers. Previous studies have shown that QNNs outperform real-valued neural networks in basic tasks and have potential in high-dimensional problem spaces. However, research on QNNs has been fragmented, with contributions from different mathematical and engineering domains leading to unintentional overlap in QNN literature. This work aims to unify existing research by evaluating four distinct QNN backpropagation algorithms, including the novel GHR-calculus backpropagation algorithm, and providing concise, scalable implementations of each algorithm using a modern compiled programming language. Additionally, the authors apply a robust Design of Experiments (DoE) methodology to compare the accuracy and runtime of each algorithm. The experiments demonstrate that the Clifford Multilayer Perceptron (CMLP) learning algorithm results in statistically significant improvements in network test set accuracy while maintaining comparable runtime performance to the other three algorithms in four distinct regression tasks. By unifying existing research and comparing different QNN training algorithms, this work develops a state-of-the-art baseline and provides important insights into the potential of QNNs for solving high-dimensional problems.
Hypercomplex-valued recurrent correlation neural networks
2021, Neurocomputing
Recurrent correlation neural networks (RCNNs), introduced by Chiueh and Goodman as an improved version of the bipolar correlation-based Hopfield neural network, can be used to implement high-capacity associative memories. In this paper, we extend the bipolar RCNNs for processing hypercomplex-valued data. Precisely, we present the mathematical background for a broad class of hypercomplex-valued RCNNs. Then, we address the stability of the new hypercomplex-valued RCNNs using synchronous and asynchronous update modes. Examples with bipolar, complex, hyperbolic, quaternion, and octonion-valued RCNNs are given to illustrate the theoretical results. Finally, computational experiments confirm the potential application of hypercomplex-valued RCNNs as associative memories designed for the storage and recall of gray-scale images.
Information geometry of hyperbolic-valued Boltzmann machines
2021, Neurocomputing
Citation Excerpt :
Nitta showed that quaternion-valued MLPs accelerated the learning speed by computer simulations [17]. Arena et al. applied the quaternion-valued MLPs to the function approximation and robotics [18,19]. Many versions of hypercomplex-valued Hopfield networks have been proposed.
Information geometry is a useful tool for analysis of neural networks. In this work, information geometry is introduced to analysis of hyperbolic-valued neural networks. First, hyperbolic-valued Boltzmann machines (HBMs) are organized. Next, the HBMs are analyzed as a neuro manifold from standpoint of information geometry. We prove that the HBMs form an exponential family and provide the natural and mixture parameters. Moreover, the Fisher metric is determined. In addition, we prove the existence of mixed parameters for all the distributions, which are useful for learning algorithms.
Universal Approximation Theorem for Vector- and Hypercomplex-Valued Neural Networks
2024, arXiv
Clifford Convolutional Neural Networks for Lymphoblast Image Classification
2024, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Quaternion Quantum Neural Network for Classification
2023, Advances in Applied Clifford Algebras

View all citing articles on Scopus

View full text

Multilayer Perceptrons to Approximate Quaternion Valued Functions

Abstract

Section snippets

INTRODUCTION

MLPS IN QUATERNION ALGEBRA

DENSITY THEOREMS FOR HMLPS

A NUMERICAL EXAMPLE

CONCLUSIONS

Neural Networks

Neural Networks

Neural Networks

Mechanism Machine Theory

Neural networks in multidimensional domains

Journal of System Engineering

Quaternion kinematic and dynamic differential equation

IEEE Transactions on Robotics and Automation

Finding the position and orientation of a sensor on a robot manipulator using quaternions

International Journal of Robotics Research

Approximation by superposition of a sigmoidal function

Mathematics of Control Signals and Systems