Abstract
When deep learning models are used to predict the probability distribution of future values, a task called probabilistic forecasting, they need to handle the epistemic and the aleatoric uncertainties. For the former, some are based on Monte Carlo dropout and have focused on prediction interval estimation assuming a normal distribution for the aleatoric uncertainty, without looking into general probabilistic forecasting tasks such as quantile regression or scenario generation. Time series in practice are usually not normally distributed, usually having a non-symmetrical distribution. So, these models need to be adapted with pre- and post-processing ad-hoc transformations to recover a normal distribution. We propose a supervised deep model based on Monte Carlo dropout that handles both sources of uncertainty and can be applied for multi-step probabilistic forecasting, including prediction interval estimation, quantile regression, and scenario generation. For the aleatoric uncertainty, our model allows changing the normality assumption easily to other families of distributions with the same number of parameters, such as Gamma, Weibull, and log-normal. For multi-step prediction, we use a procedure of re-injection of intermediate samples. To validate our proposal, we examine the behavior of the model on wind speed, wind power and electrical load forecasting, important tasks for the energy sector, as electrical grids present more uncertainty due to the penetration of variable renewable sources. We show that generating scenarios with our recurrent approach works better than directly estimating the distribution of all future values, also that considering the epistemic uncertainty makes the model more robust, and that changing the output distribution of the model is a property that may improve the metrics for specific datasets.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The datasets analyzed during the current study are available in the following repositories:
\(\bullet \) UC Irvine Machine Learning Repository, by Dua and Graff (2017): Individual household electric power consumption https://archive-beta.ics.uci.edu/ml/datasets/individual+household+electric+power+consumption.
\(\bullet \) Global Energy Forecasting Competition 2014: Load forecasting datahttp://blog.drhongtao.com/2017/03/gefcom2014-load-forecasting-data.html.
\(\bullet \) Wind and Solar resource measurement campaign (Chilean Energy Ministry): http://walker.dgf.uchile.cl/Mediciones/.
\(\bullet \) Chilean National Electrical Coordinator Operational Data: Real operation and real generation: https://www.coordinador.cl/operacion/graficos/operacion-real/generacion-real/.
Code availability
To improve the reproducibility and transparency of the experiments described in this paper, the authors have shared the code publicly in the following GitHub repository: https://github.com/cserpell/param_prob_forec.
References
Bergmeir C, Hyndman RJ, Koo B (2018) A note on the validity of cross-validation for evaluating autoregressive time series prediction. Comput Stat Data Anal 120:70–83. https://doi.org/10.1016/j.csda.2017.11.003
Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13(10):281–305
Bilgili M, Ozbek A, Sahin B et al (2015) An overview of renewable electric power capacity and progress in new technologies in the world. Renew Sustain Energy Rev 49:323–334. https://doi.org/10.1016/j.rser.2015.04.148
Black JD, Hofmann A, Hong T et al (2018) Weather data for energy analytics: from modeling outages and reliability indices to simulating distributed photovoltaic fleets. Power Energy Mag 16(3):43–53. https://doi.org/10.1109/MPE.2018.2801442
Cerqueira V, Torgo L, Smailović J et al (2017) A comparative study of performance estimation methods for time series forecasting. In: 2017 IEEE international conference on data science and advanced analytics (DSAA), Tokyo, pp 529–538. https://doi.org/10.1109/DSAA.2017.7
Darji MP, Dabhi VK, Prajapati HB (2015) Rainfall forecasting using neural network: a survey. In: 2015 international conference on advances in computer engineering and applications, Ghaziabad, India, pp 706–713. https://doi.org/10.1109/ICACEA.2015.7164782
Dua D, Graff C (2017) UCI machine learning repository. http://archive.ics.uci.edu/ml
Eapen RR, Simon SP (2019) Performance analysis of combined similar day and day ahead short term electrical load forecasting using sequential hybrid neural networks. IETE J Res 65(2):216–226. https://doi.org/10.1080/03772063.2017.1417749
Egil Martinsson (2016) WTTE-RNN: Weibull time to event recurrent neural network. Master’s thesis, University of Gothenburg
Gal Y, Ghahramani Z (2016) Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In: Balcan MF, Weinberger KQ (eds) Proceedings of the 33rd international conference on machine learning, proceedings of machine learning research, vol 48. PMLR, New York, pp 1050–1059, https://proceedings.mlr.press/v48/gal16.html
Gneiting T, Katzfuss M (2014) Probab Forecast. Ann Rev Stat Appl 1:125–151. https://doi.org/10.1146/annurev-statistics-062713-085831
Graves A (2013) Generating sequences with recurrent neural networks. https://doi.org/10.48550/arXiv.1308.0850
Graves A (2011) Practical variational inference for neural networks. In: Shawe-Taylor J, Zemel RS, Bartlett PL et al (eds) Advances in neural information processing systems, vol 24. Curran Associates Inc., Granada, pp 2348–2356
Hanif M, Shahzad MK, Mehmood V et al (2021) EPFG: electricity price forecasting with enhanced Gans neural network. IETE J Res 10(1080/03772063):2000510
Haseli G, Ranjbarzadeh R, Hajiaghaei-Keshteli M et al (2023) Hecon: weight assessment of the product loyalty criteria considering the customer decision’s halo effect using the convolutional neural networks. Inf Sci 623:184–205. https://doi.org/10.1016/j.ins.2022.12.027
Hong T, Fan S (2016) Probabilistic electric load forecasting: a tutorial review. Int J Forecast 32(3):914–938. https://doi.org/10.1016/j.ijforecast.2015.11.011
Hong T, Pinson P, Fan S et al (2016) Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond. Int J Forecast 32(3):896–913. https://doi.org/10.1016/j.ijforecast.2016.02.001
Hz Wang, Gq Li, Gb Wang et al (2017) Deep learning based ensemble approach for probabilistic wind power forecasting. Appl Energy 188:56–70. https://doi.org/10.1016/j.apenergy.2016.11.111
Jung C, Schindler D (2019) Wind speed distribution selection—a review of recent development and progress. Renew Sustain Energy Rev 114(109):290. https://doi.org/10.1016/j.rser.2019.109290
Kabir HMD, Khosravi A, Hosen MA et al (2018) Neural network-based uncertainty quantification: a survey of methodologies and applications. IEEE Access 6:36218–36234. https://doi.org/10.1109/ACCESS.2018.2836917
Kendall A, Gal Y (2017) What uncertainties do we need in Bayesian deep learning for computer vision? In: Guyon I, Luxburg UV, Bengio S et al (eds) Advances in neural information processing systems, vol 30. Curran Associates Inc, Red Hook, pp 5574–5584
Khosravi A, Nahavandi S (2014) An optimized mean variance estimation method for uncertainty quantification of wind power forecasts. Int J Electr Power Energy Syst 61:446–454. https://doi.org/10.1016/j.ijepes.2014.03.060
Khosravi A, Nahavandi S, Creighton D (2013) Prediction intervals for short-term wind farm power generation forecasts. IEEE Trans Sustain Energy 4(3):602–610. https://doi.org/10.1109/TSTE.2012.2232944
Kingma DP, Salimans T, Welling M (2015) Variational Dropout and the local reparameterization trick. In: Cortes C, Lawrence ND, Lee DD et al (eds) Advances in neural information processing systems, vol 28. Curran Associates Inc, Montreal, pp 2575–2583
López E, Valle C, Allende H et al (2018) Wind power forecasting based on Echo State networks and long short-term memory. Energies 11(3):1–22. https://doi.org/10.3390/en11030526
Lu KD, Wu ZG, Huang T (2022) Differential evolution-based three stage dynamic cyber-attack of cyber-physical power systems. IEEE/ASME Trans Mechatron. https://doi.org/10.1109/TMECH.2022.3214314
Madsen H, Pinson P, Kariniotakis G et al (2005) Standardizing the performance evaluation of short-term wind power prediction models. Wind Eng 29(6):475–489. https://doi.org/10.1260/030952405776234599
Men Z, Yee E, Lien FS et al (2016) Short-term wind speed and power forecasting using an ensemble of mixture density neural networks. Renew Energy 87:203–211. https://doi.org/10.1016/j.renene.2015.10.014
Nix DA, Weigend AS (1994) Estimating the mean and variance of the target probability distribution. In: Proceedings of 1994 IEEE international conference on neural networks (ICNN’94), IEEE, Orlando, FL. vol 1, pp 55–60. https://doi.org/10.1109/ICNN.1994.374138
Saadi SB, Sarshar NT, Sadeghi S et al (2022) Investigation of effectiveness of shuffled frog-leaping optimizer in training a convolution neural network. J Healthc Eng 4703682:11. https://doi.org/10.1155/2022/4703682
Salinas D, Flunkert V, Gasthaus J et al (2020) DeepAR: probabilistic forecasting with autoregressive recurrent networks. Int J Forecast 36(3):1181–1191. https://doi.org/10.1016/j.ijforecast.2019.07.001
Serpell C, Araya IA, Valle C et al (2020) Addressing model uncertainty in probabilistic forecasting using Monte Carlo dropout. Intell Data Anal 24(S1):185–205. https://doi.org/10.3233/IDA-200015
Shahroudi N (2019) Probabilistic forecasting with Monte–Carlo Dropout in neural networks. Master’s thesis, University of Tartu
Singh U, Rizwan M (2021) A systematic review on selected applications and approaches of wind energy forecasting and integration. J Inst Eng (India) Ser B 102(5):1061–1078. https://doi.org/10.1007/s40031-021-00618-1
Srivastava N, Hinton G, Krizhevsky A et al (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(56):1929–1958
Toubeau JF, Bottieau J, Vallée F et al (2019) Deep learning-based multivariate probabilistic forecasting for short-term scheduling in power markets. IEEE Trans Power Syst 34(2):1203–1215. https://doi.org/10.1109/TPWRS.2018.2870041
van Amersfoort J, Smith L, Teh YW et al (2020) Uncertainty estimation using a single deep deterministic neural network. In: III HD, Singh A (eds) Proceedings of the 37th international conference on machine learning, proceedings of machine learning research, vol 119. PMLR, pp 9690–9700. https://proceedings.mlr.press/v119/van-amersfoort20a.html
Van Houdt G, Mosquera C, Nápoles G (2020) A review on the long short-term memory model. Artif Intell Rev 53(8):5929–5955. https://doi.org/10.1007/s10462-020-09838-1
Wan C, Xu Z, Pinson P et al (2014) Probabilistic forecasting of wind power generation using extreme learning machine. IEEE Trans Power Syst 29(3):1033–1044. https://doi.org/10.1109/TPWRS.2013.2287871
Zhu L, Laptev N (2017) Deep and confident prediction for time series at Uber. In: 2017 IEEE international conference on data mining workshops (ICDMW), New Orleans, LA, pp 103–110 https://doi.org/10.1109/ICDMW.2017.19
Acknowledgements
Powered@NLHPC: This research was supported in part by the supercomputing infrastructure of the NLHPC (ECM-02).
Funding
This work was supported in part by the Agencia Nacional de Investigación y Desarrollo (doctoral scholarship 2017-21170109, PIA/APOYO AFB220004, FONDECYT Iniciación project 11230351) and Universidad Técnica Federico Santa María (PIIC Grant 030/2019).
Author information
Authors and Affiliations
Contributions
All authors contributed to the conception and design of the study. Cristián Serpell is the principal author. He performed material preparation, data collection and analysis, and wrote the first draft of the manuscript. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Serpell, C., Valle, C. & Allende, H. Multi-step probabilistic forecasting model using deep learning parametrized distributions. Soft Comput 27, 9479–9500 (2023). https://doi.org/10.1007/s00500-023-08444-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-023-08444-x