Abstract
Echo state network belongs to a kind of recurrent neural networks that have been extensively employed to model time-series datasets. The function of reservoir in echo state network is expected to extract the feature context from time-series datasets. However, generalization of echo state networks is limited in real-world application because the architectures of the network are fixed and the hyper-parameters are hard to be automatically determined. In the present study, the ensemble Bayesian deep echo network (EBDEN) model with deep and flexible architecture is proposed. Such networks with deep architecture progressively extract more dynamic echo states through multiple reservoirs than those with the shallow reservoir. To enhance the flexibility of the configuration for the network, this study investigates the Bayesian optimization procedure of hyper-parameters and ensures the suitable hyper-parameters to activate the network. In addition, when dealing with more complex time-series datasets, ensemble mechanism of EBDEN can measure the redundancy for the channels of the time series without sacrificing the algorithm’s performance. In this paper, the deep, optimization and ensemble architectures of EBDEN are verified by experiments benchmarked on multivariate time-series repositories and realistic tasks such as chaotic series representation and Dansgaard–Oeschger estimation tasks. According to the results, EBDEN achieves high level of the goodness-of-fit and classification performance in comparison with state-of-the-art models.
Similar content being viewed by others
References
Lin P, Chang S, Wang H, Huang Q, He J (2018) SpikeCD: a parameter-insensitive spiking neural network with clustering degeneracy strategy. Neural Comput Appl 5786:1–13. https://doi.org/10.1007/s00521-017-3336-6
Hu R, Chang S, Wang H, He J, Huang Q (2018) Efficient multi-spike learning for spiking neural networks using probability-modulated timing method. IEEE Trans Neural Netw Learn Syst 99:1–14. https://doi.org/10.1109/TNNLS.2018.2875471
Sheng P, Han J, Hua W, Hathal A, Yu Z, Mazrouei SM (2018) Modulation classification based on signal constellation diagrams and deep learning. IEEE Trans Neural Netw Learn Syst 30:718–727. https://doi.org/10.1109/TNNLS.2018.2850703
Tang ZR, Chang S, Ma QM, Zhu RH, He J, Wang H, Huang QJ (2018) A hardware friendly unsupervised memristive neural network with weight sharing mechanism. Neurocomputing 332:193–202. https://doi.org/10.1016/j.neucom.2018.12.049
Wang Z, Yan W, Oates T (2017) Time-series classification from scratch with deep neural networks: a strong baseline. In: Proceedings IJCNN, pp 2161–2161-8
Serra J, Pascual S, Karatzoglou A (2018) Towards a universal neural network encoder for time series. In: International conference of the Catalan Association for Artificial Intelligence, pp 120–129
Zheng Y, Liu Q, Chen E, Ge Y, Zhao JL (2016) Exploiting multi-channels deep convolutional neural networks for multivariate time series classification. Front Comput Sci 10:96–112. https://doi.org/10.1007/s11704-015-4478-2
Zhao B, Lu H, Chen S, Liu J, Wu D (2017) Convolutional neural networks for time series classification. Syst Eng Electron 28:162–169. https://doi.org/10.1007/978-3-319-59060-8_57
Karim F, Majumdar S, Darabi H, Chen S (2018) LSTM fully convolutional networks for time series classification. IEEE Access 6:1662–1669. https://doi.org/10.1109/ACCESS.2017.2779939
Ibrahim AO, Shamsuddin SM, Abraham A (2012) Adaptive memetic method of multi-objective genetic evolutionary algorithm for backpropagation neural network. Neural Comput Appl 31:4945–4962. https://doi.org/10.1007/s00521-018-03990-0
He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: Proceedings ECCV, pp 630–645
Yang C, Qiao J, Wang L (2018) Dynamical regularized echo state network for time series prediction. Neural Comput Appl 31:6781–6794. https://doi.org/10.1007/s00521-018-3488-z
Hu R, Huang Q, Wang H, Chang S (2019) Monitor-based spiking recurrent network for the representation of complex dynamic patterns. Int J Neural Syst 29:1950006–1950023. https://doi.org/10.1142/s0129065719500060
Jaeger H, Haas H (2004) Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304:78–80. https://doi.org/10.1126/science.1091277
Li Q, Wu Z, Zhang H (2020) Spatio-temporal modeling with enhanced flexibility and robustness of solar irradiance prediction: a chain-structure echo state network approach. J Clean Prod 261:1–10. https://doi.org/10.1016/j.jclepro.2020.121151
Wu Z, Li Q, Xia X (2020) Multi-timescale forecast of solar irradiance based on multi-task learning and echo state network approaches. IEEE Trans Ind Inf. https://doi.org/10.1109/TII.2020.2987096
Gallicchio C, Micheli A (2017) Echo state property of deep reservoir computing networks. Cognit Comput 9:337–350. https://doi.org/10.1007/s12559-017-9461-9
Chen S, Chen M (2013) Addressing the advantages of using ensemble probabilistic models in estimation of distribution algorithms for scheduling problems. Int J Prod Econ 141:24–33. https://doi.org/10.1016/j.ijpe.2012.05.010
Qiao J, Li F, Han H, Li W (2017) Growing echo-state network with multiple subreservoirs. IEEE Trans Neural Netw Learn Syst 28:391–404. https://doi.org/10.1109/TNNLS.2016.2514275
Li Z, Zheng Z, Outbib R (2019) Adaptive prognostic of fuel cells by implementing ensemble echo state networks in time-varying model space. IEEE Trans Ind Electron 67:379–389. https://doi.org/10.1109/TIE.2019.2893827
Bacic B (2016) Echo state network ensemble for human motion data temporal phasing: a case study on tennis phasing: a case study on tennis forehands. Int Conf Neural Inf Process. https://doi.org/10.1007/978-3-319-46681-1_2
IbanezSoria D, SoriaFrisch A, GarciaOjalvo J, Ruffini G (2018) Echo state networks ensemble for SSVEP dynamical online detection. https://doi.org/10.1101/268581
Jaeger H, Lukoševičius M, Popovici D, Siewert U (2007) Optimization and applications of echo state networks with leaky-integrator neurons. Neural Netw 20:335–352. https://doi.org/10.1016/j.neunet.2007.04.016
Xiang K, Nan LB, Zhang L, Pang M, Wang M, Li X (2009) Regularized Taylor echo state networks for predictive control of partially observed system. IEEE Access 4:3300–3309. https://doi.org/10.1109/ACCESS.2016.2582478
Chatzis SP, Demiris Y (2011) Echo state Gaussian process. IEEE Trans Neural Netw 22(9):1435–1445. https://doi.org/10.1109/TNN.2011.2162109
Rodan A, Faris H (2015) Echo state network with SVM-readout for customer churn prediction. IEEE Jordan Conf Appl Electr Eng Comput Technol. https://doi.org/10.1109/AEECT.2015.7360579
Graves A, Schmidhuber J (2005) Frame phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw 18:602–610. https://doi.org/10.1016/j.neunet.2005.06.042
Pan WJ, Dibeklioglu H, Tax D, Maaten L (2018) Multivariate time series classification using the hidden unit logistic model. IEEE Trans Neural Netw Learn Syst 29:920–931. https://doi.org/10.1109/TNNLS.2017.2651018
Hu R, Huang Q, Chang S, Wang H (2019) The MBPEP: a deep ensemble pruning algorithm providing high quality uncertainty prediction. Appl Intell 49:2942–2955. https://doi.org/10.1007/s10489-019-01421-8
Baydogan MG. Multivariate time series classification datasets. www.mustafabaydogan.com. Accessed 2015
Weigend S, Morgan M, Srivastava AN (1995) Nonlinear gated experts for time series: discovering regimes and avoiding overfitting. Int J Neural Syst 6:373–399. https://doi.org/10.1142/s0129065795000251
Yue Y, Cheng X, Gao S (2017) Data driven identification and control of nonlinear systems using multiple NARMA-L2 models. Int J Robust Nonlinear Control. https://doi.org/10.1002/rnc.3818
Vishik IM (2001) Attractors for equations of mathematical physics. Am Math Soc Colloq Publ Am Math Soc 49:363. https://doi.org/10.1007/s10489-019-01421-8
Gallicchio C, Micheli A (2011) Architectural and markovian factors of echo state networks. Neural Netw 24:440–456. https://doi.org/10.1016/j.neunet.2011.02.002
Gallicchio C, Micheli A, Pedrelli L (2017) Deep reservoir computing: a critical experimental analysis. Neurocomputing 268:87–99. https://doi.org/10.1016/j.neucom.2016.12.089
Grigoryeva L, Henriques J, Larger L, Ortega JP (2016) Nonlinear memory capacity of parallel time-delay reservoir computers in the processing of multidimensional signals 28:1411–1451. https://doi.org/10.1162/NECO_a_00845
Demsar J (2006) Statistical comparisons of classifiers over multiple datasets. J Mach Learn Res 7:1–30. https://doi.org/10.1007/s10846-005-9016-2
Petropoulous A, Chatzis SP, Xanthopoulous S (2017) A hidden markov model with dependence jumps for predictive modeling of multidimensional time-series. Inf Sci 412:50–66. https://doi.org/10.1016/j.ins.2017.05.038
Yu P, Li W, Ng F (2017) The generalized conditional autoregressive Wishart model for multivariate realized volatility. J Bus Econ Stat 35:1–41. https://doi.org/10.1080/07350015.2015.1096788
Kate RJ (2016) Using dynamic time warping distances as features for improved time series classification. Data Min Knowl Disc 30:283–312. https://doi.org/10.1007/s10618-015-0418-x
Baydogan M, Runger G (2016) Time series representation and similarity based on local autopatterns. Data Min Knowl Disc 30:476–509. https://doi.org/10.1007/s10618-015-0425-y
Cui Z, Chen W, Chen Y (2016) Multi-scale convolutional neural network for time series classification. arXiv: 1603.06995
Tuncel KS, Baydogan MG (2018) Autoregressive forests for time series modeling. Pattern Recognit 73:202–215. https://doi.org/10.1016/j.patcog.2017.08.016
Hills J, Lines J, Baranauskas E, Mapp J, Bagnall A (2014) Classification of time series by shapelet transformation. Data Min Knowl Disc 28(4):851–881. https://doi.org/10.1007/s10618-013-0322-1
Baydogan M, Runger G (2015) Learning a symbolic representation for multivariate time series classification. Data Min Knowl Disc 29:400–422. https://doi.org/10.1007/s10618-014-0349-y
Shen D, Zhang LR, Liu X, Liu N (2013) A novel method of using chaotic sequences in MIMI radar for multiple targets detection. In: Proceedings IEEE ICCT, pp 1–5. https://doi.org/10.1109/ICCT.2012.6511328
Sawyers DR, Sen M, Chang HC (1996) Effect of chaotic interfacial stretching on bimolecular chemical reaction in helical-coil reactors. Chem Eng J 64:129–139. https://doi.org/10.1016/S0923-0467(96)03132-6
Liu WH, Huang QJ, Chang S, Wang H, He J (2018) Multiple feature branch convolutional neural network for myocardial infarction diagnosis using electrocardiogram. Biomed Signal Process Control 45:22–32. https://doi.org/10.1016/j.bspc.2018.05.013
Lohmann J, Ditlevsen PD (2018) Random and externally controlled occurrences of Dansgaard–Oeschger events. Clim Past 14:609–617. https://doi.org/10.5194/cp-14-609-2018
Acknowledgements
This work was supported by the National Science Foundation for Young Scientists of China (61801338), the National Natural Science Foundation of China (61874079 and 61574102), the Wuhan Research Program of Application Foundation (2018010401011289) and the Luojia Young Scholars Program.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Hu, R., Tang, ZR., Song, X. et al. Ensemble echo network with deep architecture for time-series modeling. Neural Comput & Applic 33, 4997–5010 (2021). https://doi.org/10.1007/s00521-020-05286-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-020-05286-8