Optimization of quantum-inspired neural network using memetic algorithm for function approximation and chaotic time series prediction

Highlights

• A novel memetic algorithm based on hybrid genetic algorithm and gradient descent is proposed.

• We develop a new and efficient type of quantum-inspired neural networks model.

• The accuracy of the approach is investigated for function approximation and time series prediction problems.

• Numerical experiments show the excellent effectiveness and efficiency of the proposed approach.

Abstract

Heuristic and deterministic optimization methods are extensively applied for the training of artificial neural networks. Both of these methods have their own advantages and disadvantages. Heuristic stochastic optimization methods like genetic algorithm perform global search, but they suffer from the problem of slow convergence rate near global optimum. On the other hand deterministic methods like gradient descent exhibit a fast convergence rate around global optimum but may get stuck in a local optimum. Motivated by these problems, a hybrid learning algorithm combining genetic algorithm (GA) with gradient descent (GD), called HGAGD, is proposed in this paper. The new algorithm combines the global exploration ability of GA with the accurate local exploitation ability of GD to achieve a faster convergence and also a better accuracy of final solution. The HGAGD is then employed as a new training method to optimize the parameters of a quantum-inspired neural network (QINN) for two different applications. Firstly, two benchmark functions are chosen to demonstrate the potential of the proposed QINN with the HGAGD algorithm in dealing with function approximation problems. Next, the performance of the proposed method in forecasting Mackey–Glass time series and Lorenz attractor is studied. The results of these studies show the superiority of the introduced approach over other published approaches.

Introduction

A variety of learning algorithms have been developed for artificial neural networks (ANNs). Stochastic and deterministic optimization techniques are among the most popular learning algorithms for ANNs [1], [2], [3], [4], [5]. Both techniques have their own advantages and disadvantage. Stochastic search algorithms such as genetic algorithm (GA) [1], [2], shuffled frog-leaping algorithm (SFLA) [3] and particle swarm optimization (PSO) [4] perform global search but they suffer from the problem of slow convergence rate near global optimum. On the other hand deterministic search algorithms like gradient descent [5] exhibit a fast convergence rate around global optimum but may get stuck in a local optimum. Furthermore, the weight initialization of the back-propagation neural network can significantly affect the speed of convergence, the probability of convergence and the generalization. From the point of speed of convergence, the number of iterations of the training algorithm and the convergence time will vary depending on the weight initialization. If the initial values of the weights are close to the solution, the algorithm needs a small number of iterations to achieve an acceptable solution. But, if the initial values of the weights are far from the solution, searching the best solution takes a very long time during the training process of networks.

The memetic algorithm (MA) is a combination of global search and local search algorithms that exploit the advantages of both algorithms [6]. MAs are also called genetic local search, Lamarckian evolutionary algorithms, cultural algorithms, and hybrid evolutionary algorithms. In recent years, memetic algorithms have been proven to be powerful in solving the complex optimization problems in wide areas of engineering fields [7], [8], [9], [10], [11]. For example, in [7] the computation of global structural balance was solved as an optimization problem by using the memetic algorithm. In their work, the optimization algorithm combines genetic algorithm and a greedy strategy as the local search procedure. A memetic algorithm based on genetic algorithm with two different local search strategies was proposed to maximize the modularity density in [8]. In the proposed methodology, one local search strategy was simulated annealing (SA), and the other one was tightness greedy optimization (TGO). A fractional particle swarm optimization-based memetic algorithm (FPSOMA) to solve optimization problem using fractional calculus concept was introduced in [9]. In the proposed algorithm the FPSOMA accomplishes global search over the whole search space through PSO whereas local search was performed by PSO with fractional order velocity to alter the memory of best location of the particles. To optimize the urban transit network a memetic algorithm was proposed in [10]. In the proposed algorithm four types of local search operators were embedded in the classical genetic algorithm with the aim of improving the computational performance. In [11] a comprehensive learning particle swarm optimization (CLPSO)-based memetic algorithm (CLPSO-MA) was developed for short-term load forecasting. In the proposed CLPSO-MA algorithm, CLPSO was applied to explore the solution space, while a problem-specific local search was used for conducting individual learning.

MAs have recently gained much importance for learning of artificial neural network (ANN) because they offer more advantages over the conventional learning algorithms used individually. Many researchers such as the authors in [12], [13], [14], [15] have applied the MAs to examine their proposed algorithms in learning of ANN. In [12] an automatic search methodology for the optimization of the parameters and performance of neural networks was proposed. In the proposed methodology, a combination of evolution strategies, particle swarm optimization, and genetic algorithm was performed to build a hybrid method capable of seeking near-optimal or even optimal neural networks. A hybrid of PSO and gravitational search algorithm (GSA) was introduced as new training method for feed-forward neural networks in [13]. The objective of the study was to investigate the efficiencies of the GSA and PSOGSA in reducing the problems of trapping in local minima and the slow convergence rate of current evolutionary learning algorithms. In [14] a combined genetic algorithm and gradient descent was proposed for the complex permittivity determination of arbitrary shaped materials. In the proposed algorithm the final solution obtained by the GA was the start point for the GD. A two-step learning scheme for radial basis function (RBF) neural networks, which combines the genetic algorithm (GA) with the hybrid learning algorithm (HLA), was proposed in [15]. In their work, the parameters of an RBF neural network were first optimized by a genetic algorithm. Then a hybrid learning process which combines the gradient paradigm and the linear least square (LLS) paradigm was used to modify the weights and continue adjusting the parameters.

In recent years, several researches have been looking for better ways to design neural networks for chaotic time series prediction [16], [17]. A new methodology to model and predict chaotic time series based on a new recurrent predictor neural network (RPNN) was proposed in [16]. In this study a long-term prediction by making accurate multistep predictions was realized. In [17] by combining the advantage of the Bayesian framework and the echo state mechanisms, a novel echo state network (ESN) was proposed for chaotic time series prediction.

Quantum-inspired neural networks (QINNs) model is based on the combination of quantum computing with the properties of neural networks that can be implemented on a classical computer. Quantum computing based on the quantum mechanical nature of physics [18] is a good candidate for enhancing the computational efficiency of the classical neural networks. It should be noted that the quantum-inspired neural networks mistakenly called as quantum neural networks (QNNs) in some literatures. More complete details on this problem can be found in [19], [20]. Several previous studies have considered a QINNs model to solve problems such as image compression, approximation, time series prediction, identification and system controlling [21], [22], [23], [24], [25]. The performance of the quantum-inspired neural network of large size in image compression problems was evaluated in [21]. The results showed that the proposed model has same or more efficient processing abilities for image compression compared with conventional neural network. In [22] was proved that a universal approximation theorem can be uniformly approximated by a quantum based artificial neural network. A complex quantum neuron was developed to conduct time series predictions in [23]. All network parameters were trained by the Levenberg–Marquardt algorithm. In [24] an attempt was made to develop a QINN model and used as identifier to estimate the ammonia in humid air. The results showed that the identification performed by the presented model were quiet well and overcome the inference of humidity caused. In [25] a direct neural network controller using the multi-layer quantum-inspired neural network was proposed, while a real-coded genetic algorithm was used for training. The computational experiments were conducted for controlling a discrete-time nonlinear system and a two-wheeled robot.

Based on this observation, this work proposes to combine the genetic algorithm (GA) with gradient descent (GD), called HGAGD, to form a hybrid learning algorithm. The proposed HGAGD algorithm includes two major phases, namely, global search and local search. In the first phase, a global search is performed by a GA optimization method which starts with an initial population. In the second phase, the algorithm switches to a GD optimization method. At this point the GD algorithm is used as a local search algorithm around the global best individuals (elites) found by GA in each generation. By using the proposed HGAGD method, a full use of the strong global search ability of the GA and the strong local search ability of the GD algorithm, can be achieved. Therefore, they can hopefully remedy the weaknesses of each other. The resulting hybrid algorithm is then used to train the parameters of a quantum-inspired neural network (QINN). Two application studies are done in this paper, including the function approximation and chaotic time series prediction. In the first application study, the proposed approach is used to approximate the functions with one and two variables. The results obtained confirm the potential and effectiveness of the proposed HGAGD compared to other optimization algorithms. The second application study confirms the superiority of the proposed approach in prediction of Mackey–Glass chaotic time series and Lorenz attractor when compared to other prediction approaches reported in the literature.

In summary, the novel contributions of this paper are as follows:

• A novel memetic algorithm based on hybrid genetic algorithm and gradient descent is proposed.

• We develop a new and efficient type of quantum-inspired neural networks model.

• The accuracy of the approach is investigated for function approximation and time series prediction problems.

• Numerical experiments show the excellent effectiveness and efficiency of the proposed approach.

The rest of the paper is organized as follows. Section 2 introduces the architecture of the quantum-inspired neural network. The gradient descent method for the QINN is described in Section 3. The proposed hybrid learning algorithm based on genetic algorithm and gradient descent is discussed in Section 4. In Section 5, experimental result and discussion are presented. Finally, the conclusions and future work are given in Section 6.