Abstract
Meta-learning has been successfully applied to time series forecasting. For such, it uses meta-datasets created by previous machine learning applications. Each row in a meta-dataset represents a time series dataset. Each row, apart from the last, is meta-feature describing aspects of the related dataset. The last column is a target value, a meta-label. Here, the meta-label is the forecasting model with the best predictive performance for a specific error metric. In the previous studies applying meta-learning to time series forecasting, error metrics have been arbitrarily chosen. We believe that the error metric used can affect the results obtained by meta-learning. This study presents an experimental analysis of the predictive performance obtained by using different error metrics for the definition of the meta-label value. The experiments performed used 100 time series collected from the ICMC time series prediction open access repository, which has time series from a large variety of application domains. A traditional meta-learning framework for time series forecasting was used in this work. According to the experimental results, the mean absolute error can be the best metric for meta-label definition.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Datasets are available here.
References
Ali, A.R., Gabrys, B., Budka, M.: Cross-domain meta-learning for time-series forecasting. Procedia Comput. Sci. 126, 9–18 (2018)
Armstrong, J.S.: Combining forecasts. In: Armstrong J.S. (eds) Principles of Forecasting. International Series in Operations Research & Management Science, vol 30. Springer, Boston, MA (2001) https://doi.org/10.1007/978-0-306-47630-3_19
Barak, S., Nasiri, M., Rostamzadeh, M.: Time series model selection with a meta-learning approach: evidence from a pool of forecasting algorithms. arXiv preprint arXiv:1908.08489 (2019)
Bergmeir, C.N., et al.: Neural networks in r using the stuttgart neural network simulator: RSNNS. J. Stat. Softw. (2012)
Box, G.E., Jenkins, G.M., Reinsel, G.C., Ljung, G.M.: Time Series Analysis: Forecasting and Control. John Wiley & Sons (2015)
Brazdil, P., Carrier, C.G., Soares, C., Vilalta, R.: Metalearning: Applications to data mining. Cognitive Technologies. Springer Science & Business Media, Springer-Verlag Berlin Heidelberg (2008). https://doi.org/10.1007/978-3-540-73263-1
Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and Regression Trees. CRC Press Book, Monterey, CA (1984)
Cryer, J.D., Chan, K.S.: Time Series Analysis: With Applications in R, 2nd edn. Springer Texts in Statistics, Springer (2008). https://doi.org/10.1007/978-0-387-75959-3
Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice Hall PTR (1994)
Hyndman, R., Koehler, A.B., Ord, J.K., Snyder, R.D.: Forecasting with exponential smoothing: the state space approach. Springer Series in Statistics. Springer Science & Business Media, Springer-Verlag Berlin Heidelberg (2008). https://doi.org/10.1007/978-3-540-71918-2
Hyndman, R.J., Athanasopoulos, G.: Forecasting: Principles and Practice. OTexts (2018)
Hyndman, R.J., et al.: Automatic Time Series for Forecasting: The Forecast Package for R. Monash University, Department of Econometrics and Business Statistics. (2007)
Hyndman, R.J., et al.: Another look at forecast-accuracy metrics for intermittent demand. Foresight Int. J. Appl. Forecast. 4(4), 43–46 (2006)
Kuhn, M.: The caret package. R Found. Stat. Comput. Vienna, Austria. https://cran.r-project.org/package=caret (2012)
Kück, M., Crone, S.F., Freitag, M.: Meta-learning with neural networks and landmarking for forecasting model selection an empirical evaluation of different feature sets applied to industry data. In: 2016 International Joint Conference on Neural Networks (IJCNN) (2016)
Lemke, C., Gabrys, B.: Meta-learning for time series forecasting and forecast combination. Neurocomput. 73(10–12), 2006–2016 (2010)
Liaw, A., Wiener, M.: Classification and regression by randomforest. R News. 2(3), 18–22 (2002). https://CRAN.R-project.org/doc/Rnews/
Makridakis, S., Spiliotis, E., Assimakopoulos, V.: Statistical and machine learning forecasting methods: concerns and ways forward. PloS one 13(3), e0194889 (2018)
Mantovani, R.G., Rossi, A.L.D., Alcobaça, E., Vanschoren, J., de Carvalho, A.C.P.L.F.: A meta-learning recommender system for hyperparameter tuning: Predicting when tuning improves SVM classifiers. Inf. Sci. 501, 193–221 (2019)
Marsland, S.: Machine Learning: An Algorithmic Perspective. Chapman and Hall/CRC (2014)
Meade, N.: Evidence for the selection of forecasting methods. J. Forecast. 19(6), 515–535 (2000)
Medeiros, M.C., Teräsvirta, T., Rech, G.: Building neural network models for time series: a statistical approach. J. Forecast. 25(1), 49–75 (2006)
Mersmann, O., et al.: Microbenchmark: accurate timing functions. R Package Version 1(4–2), 1 (2015)
Montgomery, D.C., Johnson, L.A., Gardiner, J.S.: Forecasting and Time Series Analysis. McGraw-Hill Companies (1990)
Parmezan, A.R.S., Batista, G.E.A.P.A.: ICMC-USP time series prediction repository. Instituto de Ciências Matemáticas e de Computa cao, Universidade de Sao Paulo, Sao Carlos, Brasil (2014) https://goo.gl/uzxGZJ
Pimentel, B.A., de Carvalho, A.C.P.L.F.: A new data characterization for selecting clustering algorithms using meta-learning. Inf. Sci. 477, 203–219 (2019)
Prudêncio, R.B., Ludermir, T.B.: Meta-learning approaches to selecting time series models. Neurocomput. 61, 121–137 (2004)
Rice, J.R.: The algorithm selection problem. Adv. Comput. 15, 65–118 (1976)
Shah, C.: Model selection in univariate time series forecasting using discriminant analysis. Int. J. Forecast. 13(4), 489–500 (1997)
Feres de Souza, B., Soares, C., de Carvalho, A.C.: Meta-learning approach to gene expression data classification. Int. J. Intell. Comput. Cybern. 2(2), 285–303 (2009)
Tsay, R.S.: Multivariate Time Series Analysis: with R and Financial Applications. John Wiley & Sons (2013)
Vilalta, R., Drissi, Y.: A perspective view and survey of meta-learning. Artif. Intell. Rev. 18(2), 77–95 (2002)
Widodo, A., Budi, I.: Model selection using dimensionality reduction of time series characteristics. In: International Symposium on Forecasting, Seoul, South Korea (2013)
Wolpert, D.H.: The lack of a priori distinctions between learning algorithms. Neural Comput. 8(7), 1341–1390 (1996)
Acknowledgments
This study was partially funded by the Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) - Process 2019/10012-2 and Intel Inc.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Santos, M.R., Mundim, L.R., Carvalho, A.C.P.L.F. (2020). Evaluation of Error Metrics for Meta-learning Label Definition in the Forecasting Task. In: de la Cal, E.A., Villar Flecha, J.R., Quintián, H., Corchado, E. (eds) Hybrid Artificial Intelligent Systems. HAIS 2020. Lecture Notes in Computer Science(), vol 12344. Springer, Cham. https://doi.org/10.1007/978-3-030-61705-9_33
Download citation
DOI: https://doi.org/10.1007/978-3-030-61705-9_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61704-2
Online ISBN: 978-3-030-61705-9
eBook Packages: Computer ScienceComputer Science (R0)