Swarm intelligence based approach for efficient training of regressive neural networks

Abstract

This work proposes an efficient approach to solve the problem of training a regressive neural network efficiently. Regressive networks are characterized by delay lines possibly in both the input and the output feedback. Each delay line is connected to the network with synaptic weights and thus increases the number of parameters that must be optimized by the training algorithm. Training algorithms such as the Levenberg–Marquardt, normally used to train neural networks, are prone to local minima entrapment, and for this reason, a strategy to initialize the training procedure correctly is needed. To solve this problem, the continuous flock of starling optimization algorithm, a highly explorative optimizer based on swarm intelligence, is used. The proposed approach is tested and validated on an experimental benchmark featuring a second-order nonlinear dynamic system.

Appendix: Comparison against classic GA and PSO

The present appendix reports a brief comparison against classic global optimization techniques, in place of the CFSO, for the initialization of the search space to be investigated by the LM. In particular, the genetic algorithm (GA) and the classic PSO were used. Both the PSO and the GA were used considering a set of 10 agents (i.e., 10 particles for the PSO, 10 individuals for the GA). For the GA, an intermediate crossover rule was used with a 10% proportional mutation probability. No elitism was implemented. The PSO was directly derived from the FSO by forcing to 0 the H terms. Analogously to the tests described in Sect. 6.2, 300 iterations were considered: 200 for the global optimization algorithm (PSO or GA) and 100 for the refinement via LM. The test was done only on the optimal sizing parameters of the NARX; thus, N = 7, DU = 3 and DX = 1. For both algorithms GA + LM and PSO + LM, the test was repeated 150 times. Average RMSE for the GA + LM is 0.05981, and average RMSE for the PSO + LM is 0.07005. Both the methods outperform random initialization of the LM, with a distinct advantage of the GA over the PSO. Still, the PSO used is the simplest version ever formulated, and several variations [40,41,42] exist that can give much better optimization capabilities. Results are summarized in Table 5, and a time response of the system is shown in Fig. 6. As can be seen from the figure, GA and CFSO have very similar performance, and PSO shows a small accuracy degradation.

Table 5 Comparative analysis against classic meta-heuristics

Fig. 6

Time response for the double-tank dynamic system (bottom, black), the optimally trained neural network by CFSO + LM (bottom, red), the optimally trained neural network by GA + LM (bottom, blue), the optimally trained neural network by PSO + LM (bottom, magenta), given a time-variable input (top, pale blue). Simulations are performed on the validation set (color figure online)