Abstract
This paper describes the GRAPES Evaluation Tools based on Python (GetPy), a community verification and diagnostic tool for the evaluation of numerical models. The traditional statistical verification with confidence level test, the comprehensive scorecard, the precipitation skill score such as TS, ETS, diagnostic score SEEPS and the spatial verification techniques are used as verification modules. The Error tracing techniques conducted on the performance with different scales by wavelet analysis. The diurnal cycle of precipitation can also be calculated by Precipitation frequency-intensity method. Based on simple script architecture GetPy also includes a revised and simplified installation procedure and interactive display system. Users can easily access graphic products and carry out evaluation applications.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ahijevych, D., Gilleland, E., Brown, B.G., et al.: Application of spatial verification methods to idealized and NWP-gridded precipitation forecasts. Wea. Forecast. 24, 1485–1497 (2009)
Atger, F.: Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlinear Proc. Geophys. 8, 401–417 (2001)
Baldwin, M.E., Kain, J.S.: Sensitivity of several performance measures to displacement error, bias, and event frequency. Wea. Forecast. 21, 636–648 (2006)
Bannon, P.R.: Atmospheric available energy. J. Atmos. Sci. 69(12), 3745–3762 (2012)
Brady, R.X., Spring, A.: climpred: verification of weather and climate forecasts. J. Open Source Software 6(59), 2781 (2021)
Brill, K.F., Mesinger, F.: Applying a general analytic, method for assessing bias sensitivity to bias-adjusted threat, and equitable threat scores. Wea. Forecast. 24, 1748–1754 (2009)
Brown, J.D., Demargne, J., Seo, D.-J., Liu, Y.: The ensemble verification system (EVS): a software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations. Environ. Model. Softw. 25, 854–872 (2010)
Davison, A., Hinkley, D.: Bootstrap methods and their application. Cambridge University Press (1997)
Demargne, J., Brown, J.D., Y. Liu Y., D-J. Seo, L. Wu, Z. Toth, and Y. Zhu,: Diagnostic verification of hydrometeorological and hydrologic ensembles. Atmos. Sci. Lett. 11, 114–122 (2010)
DiCiccio, T., Efron, B.: Bootstrap confidence intervals. Stat. Sci. 11, 189–228 (1996)
Eyring, V., Bock, L., Lauer, A., Righi, M., Schlund, et al.: Earth system model evaluation tool (ESMValTool) v2.0 - an extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP. Geosci. Model Dev. 13, 3383–3438 (2020). https://doi.org/10.5194/gmd-13-3383-2020
Ferranti, L., Molteni, F., et al.: Diagnosis of extratropical variability in seasonal integrations of the ECMWF model. J. Clim. 7(6), 849–868 (1994)
Fraley, C., Raftery, A. E., Gneiting, T., Sloughter J. M.: ensembleBMA: An R package for probabilistic forecasting using ensembles and bayesian model averaging, technical report No. 516R, Department of Statistics, University of Washington (2007)
Gilleland, E. Confidence intervals for forecast verification. NCAR Technical Note NCAR/TN-479+STR, 71pp (2010)
Haiden, T.M., Rodwell, M.J., Richardson, D.S.: Intercomparison of global model precipitation forecast skill in 2010/11 using the SEEPS score. Mon Wea Rev 140, 2720–2733 (2012)
Hall, P., Horowitz, J., Jing, B.: On blocking rules for the bootstrap with dependent data. Biometrika 82, 561–574 (1995)
Klinker, E., Sardeshmukh, P.D.: The diagnosis of mechanical dissipation in the atmosphere from large-scale balance requirements. J. Atm Sci. 49(7), 608–627 (1992)
Lahiri, S.: Theoretical comparisons of block bootstrap methods. Ann. Stat. 27, 386–404 (1999)
Laio, F., Tamea, S.: Verification tools for probabilistic forecasts of continuous hydrological variables. Hydrol. Earth Syst. Sci. 11, 1267–1277 (2007)
Mass, C.F., Ovens, D., Westrick, K., et al.: Does increasing horizontal resolution produce more skillful forecasts? Bull. Amer. Meteor. Soc. 83, 407–430 (2002)
Murphy, A.H.: The Finley affair: a signal event in the history of forecast verification. Wea. Forecast. 11, 3–20 (1996)
Murphy, A.H., Winkler, R.L.: A general framework for forecast verification. Mon. Wea. Rev. 115, 1330–1338 (1987)
Palmer, T.N., Brankovic, C., et al.: The European centre for medium-range weather forecasts (ECMWF) program on extended-range prediction. Bull. Am. Meteor. Soc. 71(9), 1317–1330 (1990)
Paul, K., Mickelson, S., Dennis, J.M.: Light-weight parallel Python tools for earth system modeling workflows. IEEE Int. Confer. Big Data (big Data). 2015, 1985–1994 (2015)
Pulkkinen, S., Nerini, D., Pérez Hortal, A.A.: Pysteps: an open-source Python library for probabilistic precipitation nowcasting (v.10). Geosci. Model Develop. Copernic. GmbH 12(10), 4185–4219 (2019)
Roberts, N.M., Lean, H.W.: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev. 136, 78–97 (2008)
Robin, X., Turck, N., Sanchez, J. Muller M.: pROC: Tools for visualizing, smoothing and comparing receiver operating characteristic (ROC curves). R package version 1.3.2 (http://CRAN.R-project.org/package=pROC) (2010)
Rocklin, M.” Dask: Parallel computation with blocked algorithms and task scheduling. Proceedings of the 14th python in science conference. 126–132 (2015)
Rodwell, M.J., Richardson, D.S., Hewson, T.D., et al.: A new equitable score suitable for verifying precipitation in numerical weather prediction. Quart. J. Roy. Meteor. Soc. 136, 1344–1363 (2010)
Roebber, P.J.: Visualizing multiple measures of forecast quality. Wea. Forecast. 24, 601–608 (2009)
Skok, G., Roberts, N.: Analysis of fractions skill score properties for random precipitation fields and ECMWF forecasts. Q.J.R. Meteorol. Soc. 142, 2599–2610 (2016)
i Ventura, J. F., Lainer, M., Schauwecker, Z.: Pyrad: a real-time weather radar data processing framework based on Py-ART. J. Open Res. Softw. Ubiquity Press. 8(1): 28 (2000)
Weisman, M.L., Davis, C., Wang, W., et al.: Experiences with 0–36-h explicit convective forecasts with the WRF-ARW model. Weather Forecast. 23, 407–437 (2008)
Zhao, B., Zhang, B.: Assessing hourly precipitation forecast skill with the fractions skill score. J. Meteor. Res. 32(1), 135–145 (2018)
Funding
This study was funded by the National Key Technologies Research and Development Program of Anhui Province of China grant number (2017YFA0604502).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Bin Zhao, Ph.D., primarily undertaking research on numerical weather prediction.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhao, B., Hu, J., Wang, D. et al. The GRAPES evaluation tools based on Python (GetPy). CCF Trans. HPC 5, 347–359 (2023). https://doi.org/10.1007/s42514-022-00127-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42514-022-00127-7