Skip to main content
SpringerLink
Log in

Efficient Strategies of Static Features Incorporation into the Recurrent Neural Network

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Recurrent neural networks (RNNs) have evolved to become one of the most powerful tools for making predictions on sequenced data, such as time series, textual data, signals, music etc. In many real-life cases, however, sequenced data are additionally characterized by static features which, due to their non-sequential nature, cannot be transferred directly into RNNs. In this paper, we discuss a method which incorporates static features into RNNs in order to influence and generalize the learning process. Furthermore, we will demonstrate that our approach significantly enhances the performance of RNNs, enabling the networks to learn the sequenced data exhibiting varying characteristics and then distinguish between them through the use of static supplementary information. Finally, we will evaluate our model against real energy consumption measurements of energy time series and verify that high-accuracy demand forecasts for different types of customers can be achieved only by way of incorporation of static features.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Bengio Y, Frasconi P, Simard P (1993) The problem of learning long-term dependencies in recurrent networks. In: IEEE international conference on neural networks, IEEE. https://doi.org/10.1109/ICNN.1993.298725, https://ieeexplore.ieee.org/document/298725

  2. Breiman L (2001) Random forests. Mach Learn 45(1):5–32. https://doi.org/10.1023/A:1010933404324

    Article  MATH  Google Scholar 

  3. Chen D, Li S, Lin FJ (2019) New super-twisting zeroing neural-dynamics model for tracking control of parallel robots: a finite-time and robust solution. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2019.2930662

    Article  Google Scholar 

  4. Chen D, Li S, Lin FJ, Wu Q (2019b) Rejecting chaotic disturbances using a super-exponential-zeroing neurodynamic approach for synchronization of chaotic sensor systems. IEEE Trans Cybern 19(1):74. https://doi.org/10.3390/s19010074

    Article  Google Scholar 

  5. Chen D, Li S, Wu Q, Luo X (2019) New disturbance rejection constraint for redundant robot manipulators: an optimization perspective. IEEE Trans Ind Inf. https://doi.org/10.1109/TII.2019.2930685

    Article  Google Scholar 

  6. Dudek G (2015) Short-term load forecasting using random forests, vol 323. Springer, Berlin, pp 821–828

    Google Scholar 

  7. Eck D, Schmidhuber J (2002) Learning the long-term structure of the blues, vol 2415. Springer, Berlin, pp 284–289

    MATH  Google Scholar 

  8. Esteban C, Staeck O, Yang Y, Tresp V (2016) Predicting clinical events by combining static and dynamic information using recurrent neural networks. arXiv: 1602.02685 [cs], arXiv:1602.02685

  9. Feilat EA, Bouzguenda M (2011) Medium-term load forecasting using neural network approach. In: 2011 IEEE PES conference on innovative smart grid technologies—Middle East, IEEE, pp 1–5. https://doi.org/10.1109/ISGT-MidEast.2011.6220810, http://ieeexplore.ieee.org/document/6220810/

  10. Gers FA, Schmidhuber J, Cummins F (2000) Learning to forget: continual prediction with LSTM. Neural Comput 12(10):2451–2471. https://doi.org/10.1162/089976600300015015

    Article  Google Scholar 

  11. González-Romera E, Jaramillo-Morán M, Carmona-Fernández D (2008) Monthly electric energy demand forecasting with neural networks and fourier series. Energy Convers Manag 49(11):3135–3142. https://doi.org/10.1016/j.enconman.2008.06.004

    Article  Google Scholar 

  12. Graves A, Fernández S, Liwicki M, Bunke H, Schmidhuber J (2007) Unconstrained online handwriting recognition with recurrent neural networks. In: NIPS’07 Proceedings of the 20th international conference on neural information processing systems, Curran Associates Inc., https://dl.acm.org/citation.cfm?id=2981562.2981635

  13. Graves A, Mohamed A, Hinton GE (2013) Speech recognition with deep recurrent neural networks. In: 2013 IEEE international conference on acoustics, speech and signal processing, pp 6645–6649

  14. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  15. Kaytez F, Taplamacioglu MC, Cam E, Hardalac F (2015) Forecasting electricity consumption: a comparison of regression analysis, neural networks and least squares support vector machines. Int J Electr Power Energy Syst 67:431–438. https://doi.org/10.1016/j.ijepes.2014.12.036

    Article  Google Scholar 

  16. Khosravani H, Castilla M, Berenguel M, Ruano A, Ferreira P (2016) A comparison of energy consumption prediction models based on neural networks of a bioclimatic building. Energies 9(1):57. https://doi.org/10.3390/en9010057

    Article  Google Scholar 

  17. Kuo PH, Huang CJ (2018) A high precision artificial neural networks model for short-term energy load forecasting. Energies 11(1):213. https://doi.org/10.3390/en11010213

    Article  Google Scholar 

  18. Leontjeva A, Kuzovkin I (2016) Combining static and dynamic features for multivariate sequence classification. In: 2016 IEEE international conference on data science and advanced analytics (DSAA) pp. 21–30. https://doi.org/10.1109/DSAA.2016.10, arXiv: 1712.08160

  19. Liang Y, Niu D, Ye M, Hong WC (2016) Short-term load forecasting based on wavelet transform and least squares support vector machine optimized by improved cuckoo search. Energies 9(10):827. https://doi.org/10.3390/en9100827

    Article  Google Scholar 

  20. Liu N, Tang Q, Zhang J, Fan W, Liu J (2014) A hybrid forecasting model with parameter optimization for short-term load forecasting of micro-grids. Appl Energy 129:336–345. https://doi.org/10.1016/j.apenergy.2014.05.023

    Article  Google Scholar 

  21. Mayer H, Gome F, Wierstra D, Nagy I, Knoll A, Schmidhuber J (2006) A system for robotic heart surgery that learns to tie knots using recurrent neural networks. In: 2006 IEEE/RSJ international conference on intelligent robots and systems. https://doi.org/10.1109/IROS.2006.282190

  22. Niu D, Dai S (2017) A short-term load forecasting model with a modified particle swarm optimization algorithm and least squares support vector machine based on the denoising method of empirical mode decomposition and grey relational analysis. Energies 10(3):408. https://doi.org/10.3390/en10030408

    Article  Google Scholar 

  23. Niu D, Shi H, Wu DD (2012) Short-term load forecasting using bayesian neural networks learned by hybrid Monte Carlo algorithm. Appl Soft Comput 12(6):1822–1827. https://doi.org/10.1016/j.asoc.2011.07.001

    Article  Google Scholar 

  24. Ringwood JV, Bofelli D, Murray FT (2001) Forecasting electricity demand on short, medium and long time scales using neural networks. J Intell Rob Syst 31(1):129–147. https://doi.org/10.1023/A:1012046824237

    Article  MATH  Google Scholar 

  25. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323:533–536. https://doi.org/10.1038/323533a0

    Article  MATH  Google Scholar 

  26. Salehinejad H, Sankar S, Barfett J, Colak E, Valaee S (2018) Recent advances in recurrent neural networks. arXiv:1801.01078

  27. Selakov A, Cvijetinović D, Milović L, Mellon S, Bekut D (2014) Hybrid pso-svm method for short-term load forecasting during periods with significant temperature variations in city of burbank. Appl Soft Comput 16:80–88. https://doi.org/10.1016/j.asoc.2013.12.001

    Article  Google Scholar 

  28. Suganthi L, Iniyan S, Samuel AA (2015) Applications of fuzzy logic in renewable energy systems—a review. Renew Sustain Energy Rev 48:585–607. https://doi.org/10.1016/j.rser.2015.04.037

    Article  Google Scholar 

  29. Tanti M, Gatt A, Camilleri KP (2018) Where to put the image in an image caption generator. Nat Lang Eng 24(3):467–489. https://doi.org/10.1017/S1351324918000098

    Article  Google Scholar 

  30. Vinyals O, Toshev A, Bengio S, Erhan D (2014) Show and tell: a neural image caption generator. arXiv:1411.4555

  31. Wu Y, Schuster M, Chen Z, Le QW, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K, Klingner J, Shah A, Johnson M, Liu X, Kaiser L, Gouws S, Kato Y, Kudo T, Kazawa H, Stevens K, Kurian G, Patil N, Wang W, Young C, Smith J, Riesa J, Rudnick A, Vinyals O, Corrado GS, Hughes M, Dean J (2016) Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144

  32. Yang Y, Fasching PA, Tresp V (2017) Predictive modeling of therapy decisions in metastatic breast cancer with recurrent neural network encoder and multinomial hierarchical regression decoder. In: 2017 IEEE international conference on healthcare informatics (ICHI), IEEE, pp 46–55. https://doi.org/10.1109/ICHI.2017.51, http://ieeexplore.ieee.org/document/8031131/

  33. Zheng H, Yuan J, Chen L (2017) Short-term load forecasting using EMD-LSTM neural networks with a Xgboost algorithm for feature importance evaluation. Energies 10(8):1168. https://doi.org/10.3390/en10081168

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Advanced Analytics Team, PSI Poland, Towarowa 35, 61-896, Poznań, Poland

    Grzegorz Miebs, Małgorzata Mochol-Grzelak, Adam Karaszewski & Rafał A. Bachorz

Authors
  1. Grzegorz Miebs
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Małgorzata Mochol-Grzelak
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adam Karaszewski
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rafał A. Bachorz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rafał A. Bachorz.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 204 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Miebs, G., Mochol-Grzelak, M., Karaszewski, A. et al. Efficient Strategies of Static Features Incorporation into the Recurrent Neural Network. Neural Process Lett 51, 2301–2316 (2020). https://doi.org/10.1007/s11063-020-10195-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-020-10195-x

Keywords

Search

Navigation