Exploiting resampling techniques for model selection in forecasting: an empirical evaluation using out-of-sample tests

Abstract

Model selection is a complex task widely examined in the literature due to the major gains in forecasting accuracy when performed successfully. To do so, many approaches have been proposed exploiting the available historical data in different ways. In-sample testing is the most common approach but is highly affected by the data and parameter estimation uncertainty. Out-of-sample tests, which use part of the data to evaluate the performance of the forecasting methods, are also well-known alternatives which usually lead to improvements. However, these are still vulnerable to data uncertainty such as noise and outliers. On the other hand, resampling techniques can be used to produce multiple clones of a time series with the same characteristics but a different component of randomness. In this paper, a model selection technique is proposed which takes advantage of the bootstrapping process to mitigate the effect of noise in the original data and then applies out-of-sample tests to the generated series to evaluate the forecasting performance of different methods. The approach is assessed across a large dataset of diverse time series and benchmarked versus other traditional approaches leading to promising results.