Skip to main content
Log in

Ensemble echo network with deep architecture for time-series modeling

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Echo state network belongs to a kind of recurrent neural networks that have been extensively employed to model time-series datasets. The function of reservoir in echo state network is expected to extract the feature context from time-series datasets. However, generalization of echo state networks is limited in real-world application because the architectures of the network are fixed and the hyper-parameters are hard to be automatically determined. In the present study, the ensemble Bayesian deep echo network (EBDEN) model with deep and flexible architecture is proposed. Such networks with deep architecture progressively extract more dynamic echo states through multiple reservoirs than those with the shallow reservoir. To enhance the flexibility of the configuration for the network, this study investigates the Bayesian optimization procedure of hyper-parameters and ensures the suitable hyper-parameters to activate the network. In addition, when dealing with more complex time-series datasets, ensemble mechanism of EBDEN can measure the redundancy for the channels of the time series without sacrificing the algorithm’s performance. In this paper, the deep, optimization and ensemble architectures of EBDEN are verified by experiments benchmarked on multivariate time-series repositories and realistic tasks such as chaotic series representation and Dansgaard–Oeschger estimation tasks. According to the results, EBDEN achieves high level of the goodness-of-fit and classification performance in comparison with state-of-the-art models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Lin P, Chang S, Wang H, Huang Q, He J (2018) SpikeCD: a parameter-insensitive spiking neural network with clustering degeneracy strategy. Neural Comput Appl 5786:1–13. https://doi.org/10.1007/s00521-017-3336-6

    Article  Google Scholar 

  2. Hu R, Chang S, Wang H, He J, Huang Q (2018) Efficient multi-spike learning for spiking neural networks using probability-modulated timing method. IEEE Trans Neural Netw Learn Syst 99:1–14. https://doi.org/10.1109/TNNLS.2018.2875471

    Article  Google Scholar 

  3. Sheng P, Han J, Hua W, Hathal A, Yu Z, Mazrouei SM (2018) Modulation classification based on signal constellation diagrams and deep learning. IEEE Trans Neural Netw Learn Syst 30:718–727. https://doi.org/10.1109/TNNLS.2018.2850703

    Article  Google Scholar 

  4. Tang ZR, Chang S, Ma QM, Zhu RH, He J, Wang H, Huang QJ (2018) A hardware friendly unsupervised memristive neural network with weight sharing mechanism. Neurocomputing 332:193–202. https://doi.org/10.1016/j.neucom.2018.12.049

    Article  Google Scholar 

  5. Wang Z, Yan W, Oates T (2017) Time-series classification from scratch with deep neural networks: a strong baseline. In: Proceedings IJCNN, pp 2161–2161-8

  6. Serra J, Pascual S, Karatzoglou A (2018) Towards a universal neural network encoder for time series. In: International conference of the Catalan Association for Artificial Intelligence, pp 120–129

  7. Zheng Y, Liu Q, Chen E, Ge Y, Zhao JL (2016) Exploiting multi-channels deep convolutional neural networks for multivariate time series classification. Front Comput Sci 10:96–112. https://doi.org/10.1007/s11704-015-4478-2

    Article  Google Scholar 

  8. Zhao B, Lu H, Chen S, Liu J, Wu D (2017) Convolutional neural networks for time series classification. Syst Eng Electron 28:162–169. https://doi.org/10.1007/978-3-319-59060-8_57

    Article  Google Scholar 

  9. Karim F, Majumdar S, Darabi H, Chen S (2018) LSTM fully convolutional networks for time series classification. IEEE Access 6:1662–1669. https://doi.org/10.1109/ACCESS.2017.2779939

    Article  Google Scholar 

  10. Ibrahim AO, Shamsuddin SM, Abraham A (2012) Adaptive memetic method of multi-objective genetic evolutionary algorithm for backpropagation neural network. Neural Comput Appl 31:4945–4962. https://doi.org/10.1007/s00521-018-03990-0

    Article  Google Scholar 

  11. He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: Proceedings ECCV, pp 630–645

  12. Yang C, Qiao J, Wang L (2018) Dynamical regularized echo state network for time series prediction. Neural Comput Appl 31:6781–6794. https://doi.org/10.1007/s00521-018-3488-z

    Article  Google Scholar 

  13. Hu R, Huang Q, Wang H, Chang S (2019) Monitor-based spiking recurrent network for the representation of complex dynamic patterns. Int J Neural Syst 29:1950006–1950023. https://doi.org/10.1142/s0129065719500060

    Article  Google Scholar 

  14. Jaeger H, Haas H (2004) Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304:78–80. https://doi.org/10.1126/science.1091277

    Article  Google Scholar 

  15. Li Q, Wu Z, Zhang H (2020) Spatio-temporal modeling with enhanced flexibility and robustness of solar irradiance prediction: a chain-structure echo state network approach. J Clean Prod 261:1–10. https://doi.org/10.1016/j.jclepro.2020.121151

    Article  Google Scholar 

  16. Wu Z, Li Q, Xia X (2020) Multi-timescale forecast of solar irradiance based on multi-task learning and echo state network approaches. IEEE Trans Ind Inf. https://doi.org/10.1109/TII.2020.2987096

    Article  Google Scholar 

  17. Gallicchio C, Micheli A (2017) Echo state property of deep reservoir computing networks. Cognit Comput 9:337–350. https://doi.org/10.1007/s12559-017-9461-9

    Article  Google Scholar 

  18. Chen S, Chen M (2013) Addressing the advantages of using ensemble probabilistic models in estimation of distribution algorithms for scheduling problems. Int J Prod Econ 141:24–33. https://doi.org/10.1016/j.ijpe.2012.05.010

    Article  Google Scholar 

  19. Qiao J, Li F, Han H, Li W (2017) Growing echo-state network with multiple subreservoirs. IEEE Trans Neural Netw Learn Syst 28:391–404. https://doi.org/10.1109/TNNLS.2016.2514275

    Article  Google Scholar 

  20. Li Z, Zheng Z, Outbib R (2019) Adaptive prognostic of fuel cells by implementing ensemble echo state networks in time-varying model space. IEEE Trans Ind Electron 67:379–389. https://doi.org/10.1109/TIE.2019.2893827

    Article  Google Scholar 

  21. Bacic B (2016) Echo state network ensemble for human motion data temporal phasing: a case study on tennis phasing: a case study on tennis forehands. Int Conf Neural Inf Process. https://doi.org/10.1007/978-3-319-46681-1_2

    Article  Google Scholar 

  22. IbanezSoria D, SoriaFrisch A, GarciaOjalvo J, Ruffini G (2018) Echo state networks ensemble for SSVEP dynamical online detection. https://doi.org/10.1101/268581

  23. Jaeger H, Lukoševičius M, Popovici D, Siewert U (2007) Optimization and applications of echo state networks with leaky-integrator neurons. Neural Netw 20:335–352. https://doi.org/10.1016/j.neunet.2007.04.016

    Article  MATH  Google Scholar 

  24. Xiang K, Nan LB, Zhang L, Pang M, Wang M, Li X (2009) Regularized Taylor echo state networks for predictive control of partially observed system. IEEE Access 4:3300–3309. https://doi.org/10.1109/ACCESS.2016.2582478

    Article  Google Scholar 

  25. Chatzis SP, Demiris Y (2011) Echo state Gaussian process. IEEE Trans Neural Netw 22(9):1435–1445. https://doi.org/10.1109/TNN.2011.2162109

    Article  Google Scholar 

  26. Rodan A, Faris H (2015) Echo state network with SVM-readout for customer churn prediction. IEEE Jordan Conf Appl Electr Eng Comput Technol. https://doi.org/10.1109/AEECT.2015.7360579

    Article  Google Scholar 

  27. Graves A, Schmidhuber J (2005) Frame phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw 18:602–610. https://doi.org/10.1016/j.neunet.2005.06.042

    Article  Google Scholar 

  28. Pan WJ, Dibeklioglu H, Tax D, Maaten L (2018) Multivariate time series classification using the hidden unit logistic model. IEEE Trans Neural Netw Learn Syst 29:920–931. https://doi.org/10.1109/TNNLS.2017.2651018

    Article  Google Scholar 

  29. Hu R, Huang Q, Chang S, Wang H (2019) The MBPEP: a deep ensemble pruning algorithm providing high quality uncertainty prediction. Appl Intell 49:2942–2955. https://doi.org/10.1007/s10489-019-01421-8

    Article  Google Scholar 

  30. Baydogan MG. Multivariate time series classification datasets. www.mustafabaydogan.com. Accessed 2015

  31. Weigend S, Morgan M, Srivastava AN (1995) Nonlinear gated experts for time series: discovering regimes and avoiding overfitting. Int J Neural Syst 6:373–399. https://doi.org/10.1142/s0129065795000251

    Article  Google Scholar 

  32. Yue Y, Cheng X, Gao S (2017) Data driven identification and control of nonlinear systems using multiple NARMA-L2 models. Int J Robust Nonlinear Control. https://doi.org/10.1002/rnc.3818

    Article  MATH  Google Scholar 

  33. Vishik IM (2001) Attractors for equations of mathematical physics. Am Math Soc Colloq Publ Am Math Soc 49:363. https://doi.org/10.1007/s10489-019-01421-8

    Article  MathSciNet  MATH  Google Scholar 

  34. Gallicchio C, Micheli A (2011) Architectural and markovian factors of echo state networks. Neural Netw 24:440–456. https://doi.org/10.1016/j.neunet.2011.02.002

    Article  Google Scholar 

  35. Gallicchio C, Micheli A, Pedrelli L (2017) Deep reservoir computing: a critical experimental analysis. Neurocomputing 268:87–99. https://doi.org/10.1016/j.neucom.2016.12.089

    Article  Google Scholar 

  36. Grigoryeva L, Henriques J, Larger L, Ortega JP (2016) Nonlinear memory capacity of parallel time-delay reservoir computers in the processing of multidimensional signals 28:1411–1451. https://doi.org/10.1162/NECO_a_00845

    Article  Google Scholar 

  37. Demsar J (2006) Statistical comparisons of classifiers over multiple datasets. J Mach Learn Res 7:1–30. https://doi.org/10.1007/s10846-005-9016-2

    Article  MathSciNet  MATH  Google Scholar 

  38. Petropoulous A, Chatzis SP, Xanthopoulous S (2017) A hidden markov model with dependence jumps for predictive modeling of multidimensional time-series. Inf Sci 412:50–66. https://doi.org/10.1016/j.ins.2017.05.038

    Article  Google Scholar 

  39. Yu P, Li W, Ng F (2017) The generalized conditional autoregressive Wishart model for multivariate realized volatility. J Bus Econ Stat 35:1–41. https://doi.org/10.1080/07350015.2015.1096788

    Article  MathSciNet  Google Scholar 

  40. Kate RJ (2016) Using dynamic time warping distances as features for improved time series classification. Data Min Knowl Disc 30:283–312. https://doi.org/10.1007/s10618-015-0418-x

    Article  MathSciNet  MATH  Google Scholar 

  41. Baydogan M, Runger G (2016) Time series representation and similarity based on local autopatterns. Data Min Knowl Disc 30:476–509. https://doi.org/10.1007/s10618-015-0425-y

    Article  MathSciNet  MATH  Google Scholar 

  42. Cui Z, Chen W, Chen Y (2016) Multi-scale convolutional neural network for time series classification. arXiv: 1603.06995

  43. Tuncel KS, Baydogan MG (2018) Autoregressive forests for time series modeling. Pattern Recognit 73:202–215. https://doi.org/10.1016/j.patcog.2017.08.016

    Article  Google Scholar 

  44. Hills J, Lines J, Baranauskas E, Mapp J, Bagnall A (2014) Classification of time series by shapelet transformation. Data Min Knowl Disc 28(4):851–881. https://doi.org/10.1007/s10618-013-0322-1

    Article  MathSciNet  MATH  Google Scholar 

  45. Baydogan M, Runger G (2015) Learning a symbolic representation for multivariate time series classification. Data Min Knowl Disc 29:400–422. https://doi.org/10.1007/s10618-014-0349-y

    Article  MathSciNet  MATH  Google Scholar 

  46. Shen D, Zhang LR, Liu X, Liu N (2013) A novel method of using chaotic sequences in MIMI radar for multiple targets detection. In: Proceedings IEEE ICCT, pp 1–5. https://doi.org/10.1109/ICCT.2012.6511328

  47. Sawyers DR, Sen M, Chang HC (1996) Effect of chaotic interfacial stretching on bimolecular chemical reaction in helical-coil reactors. Chem Eng J 64:129–139. https://doi.org/10.1016/S0923-0467(96)03132-6

    Article  Google Scholar 

  48. Liu WH, Huang QJ, Chang S, Wang H, He J (2018) Multiple feature branch convolutional neural network for myocardial infarction diagnosis using electrocardiogram. Biomed Signal Process Control 45:22–32. https://doi.org/10.1016/j.bspc.2018.05.013

    Article  Google Scholar 

  49. Lohmann J, Ditlevsen PD (2018) Random and externally controlled occurrences of Dansgaard–Oeschger events. Clim Past 14:609–617. https://doi.org/10.5194/cp-14-609-2018

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Science Foundation for Young Scientists of China (61801338), the National Natural Science Foundation of China (61874079 and 61574102), the Wuhan Research Program of Application Foundation (2018010401011289) and the Luojia Young Scholars Program.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sheng Chang.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, R., Tang, ZR., Song, X. et al. Ensemble echo network with deep architecture for time-series modeling. Neural Comput & Applic 33, 4997–5010 (2021). https://doi.org/10.1007/s00521-020-05286-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05286-8

Keywords

Navigation