Skip to main content
SpringerLink
Log in

tofee-tree: automatic feature engineering framework for modeling trend-cycle in time series forecasting

  • S.I. : ‘Babel Fish’ for Feature-driven Machine Learning to Maximise Societal Value
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Most time series forecasting tasks using Artificial Neural Networks (ANNs) relegate trend-cycle modeling to a simple preprocessing step. In this work, we propose an automatic feature engineering framework for modeling the trend-cycle (tofee-tree) in time series forecasting. The first stage of the framework automatically creates over 286 deterministic linear and nonlinear engineered features to model the trend-cycle. These features are based only on the time of observation and length of the time series, making them domain-agnostic. In the second stage of the framework, a SHapley Additive exPlanations (SHAP)—based feature selection procedure using Light Gradient Boosted Machine (LightGBM) selects the most relevant features. These relevant features can be used for forecasting with ANNs in addition to the auto-regressive lags. Two popular ANNs—Multi-Layer Perceptron (MLP) and Long Short Term Memory network (LSTM) are used to evaluate our proposed tofee-tree framework. Comparisons against two empirical studies using the M3 competition dataset show that the proposed framework improved the overall Symmetric Mean Absolute Percentage Error (SMAPE) in the one-step, medium- and long-term. The relative improvement in one-step SMAPE is 3% for MLP and 23% for LSTM. We also show that the residual seasonality left after deseasonalization can be modeled using the tofee-tree framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Ahmed NK, Atiya AF, Gayar NE, El-Shishiny H (2010) An empirical comparison of machine learning models for time series forecasting. Economet Rev 29(5–6):594–621

    Article  MathSciNet  Google Scholar 

  2. Ali A, Zhu Y, Chen Q, Yu J, Cai H (2019, December). Leveraging spatio-temporal patterns for predicting citywide traffic crowd flows using deep hybrid neural networks. In 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS) (pp. 125–132). IEEE

  3. Allaire JJ, Chollet F (2021). keras: R Interface to “Keras.” Retrieved from https://cran.r-project.org/package=keras

  4. Balkin SD, Ord JK (2000) Automatic neural network modeling for univariate time series. Int J Forecast 16(4):509–515

    Article  Google Scholar 

  5. Bergmeir CN, Benítez Sánchez JM (2012). Neural networks in R using the Stuttgart neural network simulator: RSNNS. J Statistic Softw, 46(7)

  6. Bojer CS, Meldgaard JP (2021) Kaggle forecasting competitions: An overlooked learning opportunity. Int J Forecast 37(2):587–603. https://doi.org/10.1016/j.ijforecast.2020.07.007

    Article  Google Scholar 

  7. Cabrera D, Sancho F, Li C, Cerrada M, Sánchez R-V, Pacheco F, de Oliveira JV (2017) Automatic feature extraction of time-series applied to fault severity assessment of helical gearbox in stationary and non-stationary speed operation. Appl Soft Comput 58:53–64

    Article  Google Scholar 

  8. Carbonneau R, Laframboise K, Vahidov R (2008) Application of machine learning techniques for supply chain demand forecasting. Eur J Oper Res 184(3):1140–1154

    Article  MATH  Google Scholar 

  9. Cerqueira V, Moniz N, Soares C (2021). Vest: Automatic feature engineering for forecasting. Machine Learning, 1–23

  10. Christ M, Braun N, Neuffer J, Kempa-Liehr AW (2018) Time series feature extraction on basis of scalable hypothesis tests (tsfresh–a python package). Neurocomputing 307:72–77

    Article  Google Scholar 

  11. Christ M, Kempa-Liehr AW, Feindt M (2016). Distributed and parallel time series feature extraction for industrial big data applications. ACML Workshop on Learning on Big Data WLBD:1–17, (November), 1–17. Retrieved from http://arxiv.org/abs/1610.07717

  12. Chung H, Shin K (2020) Genetic algorithm-optimized multi-channel convolutional neural network for stock market prediction. Neural Comput Appl 32(12):7897–7914

    Article  Google Scholar 

  13. Crone SF, Hibon M, Nikolopoulos K (2011) Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction. Int J Forecast 27(3):635–660

    Article  Google Scholar 

  14. do Nascimento Reis GF (2019). Automated feature engineering for classification problems. Retrieved from https://repositorio-aberto.up.pt/handle/10216/122592

  15. Dokumentov A, Hyndman RJ (2020). STR: A seasonal-trend decomposition procedure based on regression. ArXiv Preprint https://arxiv.org/abs/2009.05894

  16. Domingos P (2012) A few useful things to know about machine learning. Commun ACM 55(10):78–87

    Article  Google Scholar 

  17. Dudek G (2020) Multilayer perceptron for short-term load forecasting: from global to local approach. Neural Comput Appl 32(8):3695–3707

    Article  Google Scholar 

  18. Fulcher BD (2018) Feature-based time-series analysis. Feature Eng Machine Learn Data Analyt. https://doi.org/10.1201/9781315181080-4

    Article  Google Scholar 

  19. Fulcher BD, Jones NS (2017) hctsa: a computational framework for automated time-series phenotyping using massive feature extraction. Cell Syst 5(5):527–531

    Article  Google Scholar 

  20. Guyon I, Elisseeff A (2006). Feature Extraction, Foundations and Applications: An introduction to feature extraction. Studies in Fuzziness and Soft Computing, 207, 1–25. Retrieved from http://eprints.pascal-network.org/archive/00002475/

  21. Hastie T, Tibshirani R (2003). Expression arrays and the p≫ n problem. See< http://Www-Stat.Stanford.Edu/~Hastie/Papers/Pgtn.Pdf, 1–14

  22. Hewamalage H, Bergmeir C, Bandara K (2021) Recurrent neural networks for time series forecasting: current status and future directions. Int J Forecast 37(1):388–427. https://doi.org/10.1016/j.ijforecast.2020.06.008

    Article  Google Scholar 

  23. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  24. Hong T, Pinson P, Fan S (2014). Global energy forecasting competition 2012. International Journal of Forecasting, 30(2)

  25. Horn F, Pack R, Rieger M (2019). The autofeat python library for automated feature engineering and selection. ArXiv Preprint https://arxiv.org/abs/1901.07329, 1167 CCIS, 111–120. https://doi.org/10.1007/978-3-030-43823-4_10

  26. Hyndman RJ, Athanasopoulos G (2018). Forecasting: principles and practice. OTexts

  27. Hyndman RJ, Athanasopoulos G, Bergmeir C, Caceres G, Chhay L, O’Hara-Wild M, et al. Wang E (2020). Package ‘forecast.’ Online] https://Cran.r-Project.Org/Web/Packages/Forecast/Forecast.Pdf

  28. Hyndman RJ, Koehler AB (2006) Another look at measures of forecast accuracy. Int J Forecast 22(4):679–688

    Article  Google Scholar 

  29. Johnstone IM, Titterington DM (2009) Statistical challenges of high-dimensional data. Philosophic Trans Royal Soc A: Mathematic, Physic Eng Sci 367(1906):4237–4253. https://doi.org/10.1098/rsta.2009.0159

    Article  MathSciNet  MATH  Google Scholar 

  30. Kamalov F (2020). Forecasting significant stock price changes using neural networks. Neural Computing and Applications, 1–13

  31. Kanter JM, Veeramachaneni K (2015). Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE international conference on data science and advanced analytics (DSAA) (pp. 1–10)

  32. Katz G, Shin EC R, Song D (2016). Explorekit: Automatic feature generation and selection. In 2016 IEEE 16th International Conference on Data Mining (ICDM) (pp. 979–984)

  33. Kaul A, Maheshwary S, Pudi V (2017). Autolearn—Automated feature generation and selection. In 2017 IEEE International Conference on data mining (ICDM) (pp. 217–226)

  34. Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, Liu T-Y (2017) Lightgbm: a highly efficient gradient boosting decision tree. Adv Neural Inf Process Syst 30:3146–3154

    Google Scholar 

  35. Ke G, Soukhavong D, Lamb J, Meng Q, Finley T, Wang T, … Liu T-Y (2021). lightgbm: Light Gradient Boosting Machine. Retrieved from https://cran.r-project.org/package=lightgbm

  36. Kendall MG, others. (1946). The advanced theory of statistics. The Advanced Theory of Statistics., (2nd Ed)

  37. Khurana U, Turaga D, Samulowitz H, Parthasrathy S (2016). Cognito: Automated feature engineering for supervised learning. In 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW) (pp. 1304–1307)

  38. Komisarczyk K, Kozminski P, Maksymiuk S, Biecek P (2021). treeshap: Fast SHAP values computation for ensemble models. Retrieved from https://github.com/ModelOriented/treeshap

  39. Krollner B, Vanstone BJ, Finnie,GR (2010). Financial time series forecasting with machine learning techniques: a survey. In Esann

  40. Lam HT, Thiebaut JM, Sinn M, Chen B, Mai T, Alkan O (2017). One button machine for automating feature engineering in relational databases. ArXiv Preprint https://arxiv.org/abs/1706.00327

  41. Lapedes A, Farber R (1987). Nonlinear signal processing using neural networks: Prediction and system modelling

  42. Lawrence R (1997) Using neural networks to forecast stock market prices. University of Manitoba 333:2006–2013

    Google Scholar 

  43. Lippmann R (1987) An introduction to computing with neural nets. IEEE ASSP Mag 4(2):4–22

    Article  Google Scholar 

  44. Livieris IE, Pintelas E, Pintelas P (2020). A CNN–LSTM model for gold price time-series forecasting. Neural Computing and Applications, 1–10

  45. Lundberg SM, Erion GG, Lee SI (2018). Consistent individualized feature attribution for tree ensembles. ArXiv Preprint (2). Retrieved from http://arxiv.org/abs/1802.03888

  46. Makridakis S, Hibon M (2000) The M3-Competition: results, conclusions and implications. Int J Forecast 16(4):451–476

    Article  Google Scholar 

  47. Makridakis S, Spiliotis E, Assimakopoulos V (2018) The M4 Competition: Results, findings, conclusion and way forward. Int J Forecast 34(4):802–808. https://doi.org/10.1016/j.ijforecast.2018.06.001

    Article  Google Scholar 

  48. Makridakis S, Spiliotis E, Assimakopoulos V (2018) Statistical and Machine Learning forecasting methods: Concerns and ways forward. PloS One, 13(3), e0194889

  49. Makridakis S, Spiliotis, E, Assimakopoulos V (2020). The M5 accuracy competition: Results, findings and conclusions. Int J Forecast, (October), 1–44. Retrieved from https://www.researchgate.net/publication/344487258

  50. McMenamin JS, Monforte FA (1998). Short term energy forecasting with neural networks. The Energy Journal, 19(4)

  51. Mierswa I, Morik K (2005) Automatic feature extraction for classifying audio data. Mach Learn 58(2):127–149

    Article  MATH  Google Scholar 

  52. Miller DM, Williams D (2004) Damping seasonal factors: Shrinkage estimators for the X-12-ARIMA program. Int J Forecast 20(4):529–549

    Article  Google Scholar 

  53. Møller MF (1993) A scaled conjugate gradient algorithm for fast supervised learning. Neural Netw 6(4):525–533

    Article  Google Scholar 

  54. Montero-Manso P, Athanasopoulos G, Hyndman RJ, Talagala TS (2020) FFORMA: feature-based forecast model averaging. Int J Forecast 36(1):86–92. https://doi.org/10.1016/j.ijforecast.2019.02.011

    Article  Google Scholar 

  55. Oliveira M, Torgo L (2015). Ensembles for time series forecasting. In Asian Conference on Machine Learning (pp. 360–370)

  56. Smyl S (2020) A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. Int J Forecast 36(1):75–85. https://doi.org/10.1016/j.ijforecast.2019.03.017

    Article  Google Scholar 

  57. Takens F (1981). Detecting strange attractors in turbulence. In Dynamical systems and turbulence, Warwick 1980 (pp. 366–381). Springer

  58. Wong C, Versace M (2012) CARTMAP: a neural network method for automated feature selection in financial time series forecasting. Neural Comput Appl 21(5):969–977

    Article  Google Scholar 

  59. Zhang GP, Qi M (2005) Neural network forecasting for seasonal and trend time series. Eur J Oper Res 160(2):501–514

    Article  MATH  Google Scholar 

  60. Zhang GP (2012). Neural Networks for Time-Series Forecasting. In Handbook of Natural Computing (pp. 461–477)

Download references

Acknowledgements

We are grateful to the four reviewers and the editors for their constructive suggestions and insightful comments that have significantly helped us improve this article's earlier version. We also thank Prerit Jain and Kaushal Kumar Dewangan, MBA alumni of IIT Madras, for their support in compiling the code for the different engineered features proposed in this work.

Author information

Authors and Affiliations

  1. Department of Management Studies, Indian Institute of Technology Madras, Chennai, 600036, India

    Santhosh Kumar Selvam & Chandrasekharan Rajendran

Authors
  1. Santhosh Kumar Selvam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chandrasekharan Rajendran
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Santhosh Kumar Selvam.

Ethics declarations

Conflict of interest

The authors have received no funding for this particular work. The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Selvam, S.K., Rajendran, C. tofee-tree: automatic feature engineering framework for modeling trend-cycle in time series forecasting. Neural Comput & Applic 35, 11563–11582 (2023). https://doi.org/10.1007/s00521-021-06438-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06438-0

Keywords

Search

Navigation