Skip to main content
SpringerLink
Log in

Deep learning frameworks to learn prediction and simulation focused control system models

  • Published:
Applied Intelligence Aims and scope Submit manuscript

A Correction to this article was published on 22 May 2021

This article has been updated

Abstract

Deep learning (DL) methods have brought world-shattering breakthroughs, especially in computer vision and classification problems. Yet, the design and deployment of DL methods in time series prediction and nonlinear system identification applications still need more progress. In this paper, we present DL frameworks that are developed to provide novel approaches as solutions to the aforementioned engineering problems. The proposed DL frameworks leverage the advantages of autoencoders and long-short term memory network, which are known being data compression and recurrent structures, respectively, to design Deep Neural Networks (DNN) for modeling time series and nonlinear systems with high performance. We provide recommendations on how deep AEs and LSTMs should be utilized to end up with efficient Prediction-focused (Pf) and Simulation-focused (Sf) DNNs for time series and system identification problems. We present systematic learning methods for the DL frameworks that allow straightforward learning of Pf-DNN and Sf-DNN models in detail. To demonstrate the efficiency of the developed DNNs, we present various comparative results conducted on the benchmark and real-world datasets in comparison with their conventional, shallow, and deep neural network counterparts. The results clearly show that the deployment of the proposed DL frameworks results with DNNs that have high accuracy, even with a low dimensional feature vector.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Change history

References

  1. Courville A, Goodfellow I, Bengio Y (2016) Deep learning. MIT press, Cambridge

    MATH  Google Scholar 

  2. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Google Scholar 

  3. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117

    Google Scholar 

  4. Janssens O, Van De Walle R, Loccufier M, Van Hoecke S (2018) Deep learning for infrared thermal image based machine health monitoring. IEEE/ASME Trans Mechatron 23:151–159

    Google Scholar 

  5. Deng L, Hinton G, Kingsbury B (2013) New types of deep neural network learning for speech recognition and related applications: an overview. In: ICASSP, IEEE Int Conf Acoust, Speech and Signal Process – Proc :8599–8603

  6. Deng L, Yu D (2013) Deep learning: methods and applications. Found Trends in Signal Process 7:197–387

    MathSciNet  MATH  Google Scholar 

  7. Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26

    Google Scholar 

  8. Wang J, Fu P, Zhang L, Gao RX, Zhao R (2019) Multilevel information fusion for induction motor fault diagnosis. IEEE/ASME Trans Mechatron 24:2139–2150

    Google Scholar 

  9. Kim D, Kwon J, Han S, Park YL, Jo S (2019) Deep full-body motion network for a soft wearable motion sensing suit. IEEE/ASME Trans Mechatron 24:56–66

    Google Scholar 

  10. Terzi M, Susto GA, Chaudhari P (2020) Directional adversarial training for cost sensitive deep learning classification applications. Eng Appl Artif Intell 91:103550

    Google Scholar 

  11. Masti D, Bemporad A (2019) Learning nonlinear state-space models using deep autoencoders. In: Proc IEEE Conf Decis Control :3862–3867

  12. Bansal S, Akametalu AK, Jiang FJ, Laine F, Tomlin CJ (2016) Learning quadrotor dynamics using neural network for flight control. In: 2016 IEEE 55th Conf Decis control, CDC :4653–4660

  13. Gensler A, Henze J, Sick B, Raabe N (2016) Deep Learning for solar power forecasting - An approach using AutoEncoder and LSTM Neural Networks. In: 2016 IEEE Int Conf Syst, Man, and Cybern :2858–2865

  14. Lusch B, Kutz JN, Brunton SL (2018) Deep learning for universal linear embeddings of nonlinear dynamics. Nat Commun 9:1–10

    Google Scholar 

  15. Qiao J, Wang G, Li W, Li X (2018) A deep belief network with PLSR for nonlinear system modeling. Neural Netw 104:68–79

    MATH  Google Scholar 

  16. de la Rosa E, Yu W (2016) Randomized algorithms for nonlinear system identification with deep learning modification. Inf Sci 364:197–212

    MATH  Google Scholar 

  17. Qiu X, Zhang L, Ren Y, Suganthan P, Amaratunga G (2014) Ensemble deep learning for regression and time series forecasting. In: 2014 IEEE Sym Comput Int Ensemble Learn, Proc :1–6

  18. Hirose N, Tajima R (2017) Modeling of rolling friction by recurrent neural network using LSTM. In: Proc - IEEE Int Conf Robot Autom :6471–6478

  19. Sagheer A, Kotb M (2019) Time series forecasting of petroleum production using deep LSTM recurrent networks. Neurocomputing 323:203–213

    Google Scholar 

  20. Liu C, Jin Z, Gu J, Qiu C (2017) Short-term load forecasting using a long short-term memory network. In: 2017 IEEE PES Innov smart grid Technol Conf Eur Proc :1–6

  21. Zhao Z, Chen W, Wu X, Chen PCV, Liu J (2017) LSTM network: a deep learning approach for short-term traffic forecast. IET Image Process 11:68–75

    Google Scholar 

  22. Qu X, Kang X, Chao Z, Shuai J, Ma X (2016) Short-term prediction of wind power based on deep long short-term memory. In: Asia-Pacific Power and Energy Eng Conf :1148–1152

  23. Fischer T, Krauss C (2018) Deep learning with long short-term memory networks for financial market predictions. Eur J Oper Res 270:654–669

    MathSciNet  MATH  Google Scholar 

  24. Hsu D (2017) Time series forecasting based on augmented long short-term memory. arXiv preprint arXiv: 170700666 [csNE]

  25. Laptev N, Yosinski J, Erran Li L et al (2017) Time-series extreme event forecasting with neural networks at Uber. Int Conf Mach Learn - Time Ser Workshop 34:1–5

    Google Scholar 

  26. How DNT, Loo CK, Sahari KSM (2016) Behavior recognition for humanoid robots using long short-term memory. Int J Adv Robot Syst 13(6):1729881416663369

    Google Scholar 

  27. Li Y, Zhu Z, Kong D, Han H, Zhao Y (2019) EA-LSTM: evolutionary attention-based LSTM for time series prediction. Knowl-Based Syst 181:104785

    Google Scholar 

  28. Hua Y, Zhao Z, Li R, Chen X, Liu Z, Zhang H (2019) Deep learning with long short-term memory for time series prediction. IEEE Commun Mag 57:114–119

    Google Scholar 

  29. Jin X, Yu X, Wang X, et al (2020) Prediction for time series with CNN and LSTM. In: Lect Notes Electrical Eng 582:631–641

  30. Elsheikh A, Yacout S, Ouali MS (2019) Bidirectional handshaking LSTM for remaining useful life prediction. Neurocomputing 323:148–156

    Google Scholar 

  31. George K, Mutalik P (2019) A multiple model approach to time-series prediction using an online sequential learning algorithm. IEEE Trans Syst Man Cybern Syst 49:976–990

    Google Scholar 

  32. Kuremoto T, Obayashi M, Kobayashi K, Hirata T, Mabu S (2014) Forecast chaotic time series data by DBNs. In: Proc - 2014 7th Int Congr image and signal process :1130–1135

  33. Qin M, Li Z, Du Z (2017) Red tide time series forecasting by combining ARIMA and deep belief network. Knowl-Based Syst 151:39–52

    Google Scholar 

  34. Choi JY, Lee B (2018) Combining LSTM network ensemble via adaptive weighting for improved time series forecasting. Math Probl Eng 2018:1–8

    Google Scholar 

  35. Zarkias KS, Passalis N, Tsantekidis A, Tefas A (2019) Deep reinforcement learning for financial trading using Price trailing. In: Proc IEEE Int Conf Acoust, Speech and Signal Process :3067–3071

  36. Wee CK, Nayak R (2019) Adaptive load forecasting using reinforcement learning with database technology. J Inf Telecommun 3:381–399

    Google Scholar 

  37. Hirata T, Kuremoto T, Obayashi M, Mabu S, Kobayashi K (2016) Deep belief network using reinforcement learning and its applications to time series forecasting. In: Lect notes Comput Sci (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) :30–37

  38. Längkvist M, Karlsson L, Loutfi A (2014) A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recogn Lett 42:11–24

    Google Scholar 

  39. Du S, Li T, Yang Y, Horng SJ (2020) Multivariate time series forecasting via attention-based encoder–decoder framework. Neurocomputing 388:269–279

    Google Scholar 

  40. Zhang CY, Chen CLP, Gan M, Chen L (2015) Predictive deep Boltzmann machine for multiperiod wind speed forecasting. IEEE Trans Sustain Energy 6:1416–1425

    Google Scholar 

  41. Bao W, Yue J, Rao Y (2017) A deep learning framework for financial time series using stacked autoencoders and long-short term memory. PLoS One 12:e0180944

    Google Scholar 

  42. Kuremoto T, Kimura S, Kobayashi K, Obayashi M (2014) Time series forecasting using a deep belief network with restricted Boltzmann machines. Neurocomputing 137:47–56

    Google Scholar 

  43. Lin T, Guo T, Aberer K (2017) Hybrid neural networks for learning the trend in time series. In: IJCAI Int Joint Conf Artif Intell :2273–2279

  44. Wang Y (2017) A new concept using LSTM neural networks for dynamic system identification. In: Proc Am Control Conf :5324–5329

  45. Ogunmolu O, Gu X, Jiang, S, Gans N (2016). Nonlinear systems identification using deep dynamic neural networks. arXiv preprint arXiv:1610.01439

  46. Donon B, Donnot B, Guyon I, Liu Z, Marot A, Panciatici P, Schoenauer M (2020) LEAP nets for system identification and application to power systems. Neurocomputing. 416:316–327. https://doi.org/10.1016/j.neucom.2019.12.135

    Article  Google Scholar 

  47. Kashima K (2016) Nonlinear model reduction by deep autoencoder of noise response data. In: 2016 IEEE 55th Conf Decis control :5750–5755

  48. Punjani A, Abbeel P (2015) Deep learning helicopter dynamics models. In: Proc - IEEE Int Conf Robot Automa :3223–3230

  49. Woo J, Park J, Yu C, Kim N (2018) Dynamic model identification of unmanned surface vehicles using deep learning network. Appl Ocean Res 78:123–133

    Google Scholar 

  50. Genc S (2017) Parametric system identification using deep convolutional neural networks. In: Proc Int Joint Conf Neural Netw :2112–2119

  51. Kashihara K (2018) Nonlinear system identification based on convolutional neural networks for multiple drug interactions. In: Proc Annu Int Conf IEEE Eng Med Biol Soc :1–4

  52. Wu RT, Jahanshahi MR (2019) Deep convolutional neural network for structural dynamic response estimation and system identification. J Eng Mech 145:04018125

    Google Scholar 

  53. Zhu W, Li X, Liu C, Xue F, Han Y (2020) An STFT-LSTM system for P-wave identification. IEEE Geosci Remote Sens Lett 17:519–523

    Google Scholar 

  54. Feng C, Chang L, Li C, Ding T, Mai Z (2019) Controller optimization approach using LSTM-based identification model for pumped-storage units. IEEE Access 7:32714–32727

    Google Scholar 

  55. Wang G, Qiao J, Bi J, Zhou M (2019) An Efficient Deep Belief Network with Fuzzy Learning for Nonlinear System Modeling. In: 2019 IEEE Int Conf Syst, Man and Cybern :3549–3554

  56. Yeung E, Kundu S, Hodas N (2019) Learning deep neural network representations for Koopman operators of nonlinear dynamical systems. In: 2019 am control Conf :4832–4839

  57. Mattos CLC, Dai Z, Damianou A, Barreto GA, Lawrence ND (2017) Deep recurrent Gaussian processes for outlier-robust system identification. J Process Control 60:82–94

    Google Scholar 

  58. Gonzalez J, Yu W (2018) Non-linear system modeling using LSTM neural networks. IFAC-PapersOnLine 51:485–489

    Google Scholar 

  59. Jiang M, Jin Q (2019) Multivariable system identification method based on continuous action reinforcement learning automata. Processes 7:546

    Google Scholar 

  60. Yu W, de la Rosa E (2019) Deep Boltzmann machine for nonlinear system modelling. Int J Mach Learn Cybern 10:1705–1716

    Google Scholar 

  61. Zhou H, Ibrahim C, Pan W (2019). A sparse Bayesian deep learning approach for identification of cascaded tanks benchmark. arXiv preprint arXiv:1911.06847

  62. Greff K, Srivastava RK, Koutnik J, Steunebrink BR, Schmidhuber J (2017) LSTM: a search space odyssey. IEEE Trans Neural Netw Learn Syst 28:2222–2232

    MathSciNet  Google Scholar 

  63. Mestres AC, Bochkina N, Mayer C (2018) Selection of the regularization parameter in graphical models using network characteristics. J Comput Graph Stat 27:323–333

    MathSciNet  Google Scholar 

  64. Liu Z, Hansson A, Vandenberghe L (2013) Nuclear norm system identification with missing inputs and outputs. Syst Control Lett 62:605–612

    MathSciNet  MATH  Google Scholar 

  65. Hansson A, Liu Z, Vandenberghe L (2012) Subspace system identification via weighted nuclear norm optimization. In: Proceedings of the IEEE Conf Decis Control :3439–3444

  66. McNames Portland State University. http://web.cecs.pdx.edu/~mcnames/DataSets/index.html. Accessed 28 June 2020

  67. Eyoh I, John R, De Maere G (2017) Time series forecasting with interval type-2 intuitionistic fuzzy logic systems. In: IEEE Int Conf Fuzzy Sys :1–6

  68. Juang CF, Huang RB, Cheng WY (2010) An interval type-2 fuzzy-neural network with support-vector regression for noisy regression problems. IEEE Trans Fuzzy Syst 18:686–699

    Google Scholar 

  69. Chiang JH, Hao PY (2004) Support vector learning mechanism for fuzzy rule-based modeling: a new approach. IEEE Trans Fuzzy Syst 12:1–12

    Google Scholar 

  70. Australian Energy Market Operator (2013). http://www.aemo.com.au. Accessed 28 June 2020

  71. Box GEP, Jenkins GM, Reinsel GC, Ljung GM (2015) Time series analysis: forecasting and control. John Wiley & Sons, Hoboken, New Jersey

  72. Chithra Chakra N, Song KY, Gupta MM, Saraf DN (2013) An innovative neural forecast of cumulative oil production from a petroleum reservoir employing higher-order neural networks (HONNs). J Pet Sci Eng 106:18–33

    Google Scholar 

  73. Schoukens M, Noël JP (2017) Three benchmarks addressing open challenges in nonlinear system identification. IFAC-PapersOnLine 50:446–451

    Google Scholar 

  74. Svensson A, Schön TB (2017) A flexible state–space model for learning nonlinear dynamical systems. Automatica 80:189–199

    MathSciNet  MATH  Google Scholar 

  75. Mattsson P, Zachariah D, Stoica P (2018) Identification of cascade water tanks using a PWARX model. Mech Syst Signal Process 106:40–48

    Google Scholar 

  76. Brunot M, Janot A, Carrillo F (2017) Continuous-time nonlinear systems identification with output error method based on derivative-free optimisation. IFAC-PapersOnLine 50:464–469

    Google Scholar 

  77. Schoukens M, Scheiwe FG (2016) Modeling nonlinear systems using a Volterra feedback model, workshop on nonlinear system identification benchmarks, Brussels, Belgium

  78. Relan R, Tiels K, Marconato A, Schoukens J (2017) An unstructured flexible nonlinear model for the cascaded water-tanks benchmark. IFAC-PapersOnLine 50:452–457

    Google Scholar 

  79. De Moor BLR (1997) DaISy: Database for the Identification of Systems http://homes.esat.kuleuven.be/~smc/daisy/. Accessed 28 June 2020

  80. Hong X, Mitchell R, Di Fatta G (2019) Simplex basis function based sparse least squares support vector regression. Neurocomputing 330:394–402

    Google Scholar 

  81. MATLAB and System Identification Toolbox Release (2019a), The MathWorks, Inc., Natick, Massachusetts, United States

Download references

Author information

Authors and Affiliations

  1. Department of Control and Automation Engineering, Istanbul Technical University, Istanbul, Turkey

    Turcan Tuna, Aykut Beke & Tufan Kumbasar

Authors
  1. Turcan Tuna
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aykut Beke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tufan Kumbasar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tufan Kumbasar.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tuna, T., Beke, A. & Kumbasar, T. Deep learning frameworks to learn prediction and simulation focused control system models. Appl Intell 52, 662–679 (2022). https://doi.org/10.1007/s10489-021-02416-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02416-0

Keywords

Search

Navigation