Abstract
Probabilistic regression models trained with maximum likelihood estimation (MLE), can sometimes overestimate variance to an unacceptable degree. This is mostly problematic in the multivariate domain. While univariate models often optimize the popular Continuous Ranked Probability Score (CRPS), in the multivariate domain, no such alternative to MLE has yet been widely accepted. The Energy Score – the most investigated alternative – notoriously lacks closed-form expressions and sensitivity to the correlation between target variables. In this paper, we propose Conditional CRPS: a multivariate strictly proper scoring rule that extends CRPS. We show that closed-form expressions exist for popular distributions and illustrate their sensitivity to correlation. We then show in a variety of experiments on both synthetic and real data, that Conditional CRPS often outperforms MLE, and produces results comparable to state-of-the-art non-parametric models, such as Distributional Random Forest (DRF).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
I.e. distributions P for which a countable set \(\Omega \subset \mathbbm {R}^d\) exists such that \(\mathbbm {P}_{Y \sim P}(Y \in \Omega ) = 1\).
- 2.
I.e. distributions P for which a Lebesgue integratable function \(f_P: \mathbbm {R}^d \rightarrow [0, \infty )\) exists, such that for all measurable sets \(U \subseteq \mathbbm {R}^d\), we have \(\mathbbm {P}_{Y \sim P}(Y \in U) = \int _U f_P(u)du\).
- 3.
Support for backpropagation through matrix inversions is offered in packages such as Tensorflow. However, for larger matrices, gradients can become increasingly unstable.
- 4.
References
Aggarwal, K., Kirchmeyer, M., Yadav, P., Keerthi, S.S., Gallinari, P.: Regression with conditional gan (2019). http://arxiv.org/abs/1905.12868
Alexander, C., Coulon, M., Han, Y., Meng, X.: Evaluating the discrimination ability of proper multi-variate scoring rules. Ann. Oper. Res. (C) (2022). https://doi.org/10.1016/j.apenergy.2011.1. https://ideas.repec.org/a/eee/appene/v96y2012icp12-20.html
Avati, A., Duan, T., Zhou, S., Jung, K., Shah, N.H., Ng, A.Y.: Countdown regression: sharp and calibrated survival predictions. In: Adams, R.P., Gogate, V. (eds.) Proceedings of The 35th Uncertainty in Artificial Intelligence Conference. Proceedings of Machine Learning Research, vol. 115, pp. 145–155. PMLR (2020). https://proceedings.mlr.press/v115/avati20a.html
Bjerregård, M.B., Møller, J.K., Madsen, H.: An introduction to multivariate probabilistic forecast evaluation. Energy AI 4, 100058 (2021). https://doi.org/10.1016/j.egyai.2021.100058. https://www.sciencedirect.com/science/article/pii/S2666546821000124
Canadian Meteorological Centre: Gem, the global environmental multiscale model (2020). https://collaboration.cmc.ec.gc.ca/science/rpn/gef_html_public/index.html. Accessed 03 May 2023
Canadian Meteorological Centre: Geps, the global ensemble prediction system (2021). https://weather.gc.ca/grib/grib2_ens_geps_e.html. Accessed 13 May 2023
Carney, M., Cunningham, P., Dowling, J., Lee, C.: Predicting probability distributions for surf height using an ensemble of mixture density networks. In: Proceedings of the 22nd International Conference on Machine Learning - ICML 2005. ACM Press (2005). https://doi.org/10.1145/1102351.1102366
DWD Climate Data Center (CDC): Historical hourly station observations of solar incoming (total/diffuse) and longwave downward radiation for germany (1981–2021)
Gebetsberger, M., Messner, J., Mayr, G., Zeileis, A.: Estimation methods for nonhomogeneous regression models: minimum continuous ranked probability score versus maximum likelihood. Monthly Weather Rev. 146 (2018). https://doi.org/10.1175/MWR-D-17-0364.1
Gneiting, T., Balabdaoui, F., Raftery, A.E.: Probabilistic forecasts, calibration and sharpness. J. Royal Stat. Soc. Series B (Stat. Methodol.) 69(2), 243–268 (2007). https://doi.org/10.1111/j.1467-9868.2007.00587.x. https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-9868.2007.00587.x
Gneiting, T., Katzfuss, M.: Probabilistic forecasting. Ann. Rev. Stat. Appl. 1(1), 125–151 (2014). https://doi.org/10.1146/annurev-statistics-062713-085831
Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102(477), 359–378 (2007). https://doi.org/10.1198/016214506000001437
Grimit, E.P., Gneiting, T., Berrocal, V.J., Johnson, N.A.: The continuous ranked probability score for circular variables and its application to mesoscale forecast ensemble verification. Q. J. Royal Meteorol. Soc. 132(621C), 2925–2942 (2006). https://doi.org/10.1256/qj.05.235. https://rmets.onlinelibrary.wiley.com/doi/abs/10.1256/qj.05.235
Gurney, K.: An Introduction to Neural Networks. Taylor & Francis Inc., Boston (1997)
Haynes, W.: Encyclopedia of Systems Biology, pp. 1190–1191. Springer, New York (2013). https://doi.org/10.1007/978-1-4419-9863-7_1235
Jiao, Y., Sharma, A., Ben Abdallah, A., Maddox, T.M., Kannampallil, T.: Probabilistic forecasting of surgical case duration using machine learning: model development and validation. J. Am. Med. Inf. Assoc. 27(12), 1885–1893 (2020)
Jordan, A., Krüger, F., Lerch, S.: Evaluating probabilistic forecasts with scoringrules. J. Stat. Softw. 90(12), 1–37 (2019). https://doi.org/10.18637/jss.v090.i12, https://www.jstatsoft.org/index.php/jss/article/view/v090i12
Kanazawa, T., Gupta, C.: Sample-based uncertainty quantification with a single deterministic neural network (2022). https://doi.org/10.48550/ARXIV.2209.08418
Koninklijk Nederlands Meteorologisch Instituut: Uurgegevens van het weer in nederland (2008–2020). http://projects.knmi.nl/klimatologie/uurgegevens/. Accessed 03 May 2023
Matheson, J.E., Winkler, R.L.: Scoring rules for continuous probability distributions. Manag. Sci. 22(10), 1087–1096 (1976). http://www.jstor.org/stable/2629907
Murad, A., Kraemer, F.A., Bach, K., Taylor, G.: Probabilistic deep learning to quantify uncertainty in air quality forecasting. Sensors (Basel) 21(23) (2021)
Muschinski, T., Mayr, G.J., Simon, T., Umlauf, N., Zeileis, A.: Cholesky-based multivariate gaussian regression. Econometrics Stat. (2022). https://doi.org/10.1016/j.ecosta.2022.03.001
National Centers for Environmental Information: Global forecast system (gfs)l(2020). https://www.ncei.noaa.gov/products/weather-climate-models. Accessed 03 May 2023
Nowotarski, J., Weron, R.: Computing electricity spot price prediction intervals using quantile regression and forecast averaging. Comput. Stat. 30(3), 791–803 (2014). https://doi.org/10.1007/s00180-014-0523-0
Pinson, P., Tastu, J.: Discrimination ability of the Energy score. No. 15 in DTU Compute-Technical Report-2013, Technical University of Denmark (2013)
Rasp, S., Lerch, S.: Neural networks for postprocessing ensemble weather forecasts. Monthly Weather Rev. 146(11), 3885–3900 (2018). https://doi.org/10.1175/MWR-D-18-0187.1
Scheuerer, M., Hamill, T.: Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities*. Monthly Weather Rev. 143, 1321–1334 (2015). https://doi.org/10.1175/MWR-D-14-00269.1
Viroli, C., McLachlan, G.J.: Deep gaussian mixture models (2017). https://arxiv.org/abs/1711.06929, ArXiv-preprint:1711.06929
Zhu, Y., Toth, Z., Wobus, R., Richardson, D., Mylne, K.: The economic value of ensemble-based weather forecasts. Bull. Am. Meteorol. Soc. 83(1), 73–83 (2002). http://www.jstor.org/stable/26215325
Önkal, D., Muradoǧlu, G.: Evaluating probabilistic forecasts of stock prices in a developing stock market. Eur. J. Oper. Res. 74(2), 350–358 (1994). https://doi.org/10.1016/0377-2217(94)90102-3. https://www.sciencedirect.com/science/article/pii/0377221794901023, financial Modelling
Ćevid, D., Michel, L., Näf, J., Meinshausen, N., Bühlmann, P.: Distributional random forests: heterogeneity adjustment and multivariate distributional regression (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Roordink, D., Hess, S. (2023). Scoring Rule Nets: Beyond Mean Target Prediction in Multivariate Regression. In: Koutra, D., Plant, C., Gomez Rodriguez, M., Baralis, E., Bonchi, F. (eds) Machine Learning and Knowledge Discovery in Databases: Research Track. ECML PKDD 2023. Lecture Notes in Computer Science(), vol 14170. Springer, Cham. https://doi.org/10.1007/978-3-031-43415-0_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-43415-0_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43414-3
Online ISBN: 978-3-031-43415-0
eBook Packages: Computer ScienceComputer Science (R0)
