Skip to main content
Springer Nature Link
Log in

Efficient parallel implementation of DDDAS inference using an ensemble Kalman filter with shrinkage covariance matrix estimation

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

This paper develops an efficient and parallel implementation of dynamically data-driven application systems inference using an ensemble Kalman filter based on shrinkage covariance matrix estimation. The proposed implementation works as follows: each model component is surrounded by a local box of radius size r and then, local assimilation steps are carried out in parallel at the different local boxes. Once local analyses are obtained, they are mapped back onto the global domain from which the global analysis state is obtained. Local background error correlations are estimated using the Rao–Blackwell Ledoit and Wolf estimator in order to mitigate the impact of spurious correlations whenever the number of local model components is larger than the ensemble size. The numerical atmospheric general circulation model (SPEEDY) is utilized for the numerical experiments with the T-63 resolution on the BlueRidge cluster at Virginia Tech. The number of processors ranges from 96 to 2048. The proposed implementation outperforms in terms of accuracy the well-known local ensemble transform Kalman filter (LETKF) for all the model variables. The computational time of the proposed implementation is similar to that of the parallel LETKF method (where no covariance estimation is performed) for the largest number of processors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Anderson, J.L.: Localization and sampling error correction in ensemble Kalman filter data assimilation. Mon. Weather Rev. 140(7), 2359–2371 (2012)

    Article  Google Scholar 

  2. Anderson, J.L., Anderson, S.L.: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Weather Rev. 127(12), 2741–2758 (1999)

    Article  Google Scholar 

  3. Anderson, E., Bai, Z., Dongarra, J., Greenbaum, A., McKenney, A., Du Croz, J., Hammerling, S., Demmel, J., Bischof, C., Sorensen, D.: LAPACK: a portable linear algebra library for high-performance computers. In: Proceedings of the 1990 ACM/IEEE Conference on Supercomputing, Supercomputing ’90, pp. 2–11. IEEE Computer Society Press, Los Alamitos (1990)

  4. Aved, A., Darema, F., Blasch, E.: Dynamic data driven application systems. www.1dddas.org (2014)

  5. Blackford, L.S., Demmel, J., Dongarra, J., Duff, I., Hammarling, S., Henry, G., Heroux, M., Kaufman, L., Lumsdaine, A., Petitet, A., Pozo, R., Remington, K., Whaley, R.C.: An updated set of basic linear algebra subprograms (BLAS). ACM Trans. Math. Softw. 28, 135–151 (2001)

    Article  MathSciNet  Google Scholar 

  6. Blasch, E., Seetharaman, G., Reinhardt, K.: Dynamic data driven applications system concept for information fusion. Proc. Comput. Sci. 18, 1999–2007 (2013). 2013 International Conference on Computational Science

  7. Chen, Y., Wiesel, A., Eldar, Y.C., Hero, A.O.: Shrinkage algorithms for MMSE covariance estimation. IEEE Trans. Signal Process. 58(10), 5016–5029 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  8. Cheng, H., Jardak, M., Alexe, M., Sandu, A.: A hybrid approach to estimating error covariances in variational data assimilation. Tellus A 62(3), 288–297 (2010)

    Article  Google Scholar 

  9. Cheng, H., Jardak, M., Alexe, M., Sandu, A.: A hybrid approach to estimating error covariances in variational data assimilation. Tellus A 62(3), 288–297 (2010)

    Article  Google Scholar 

  10. Couillet, R., McKay, M.: Large dimensional analysis and optimization of robust shrinkage covariance matrix estimators. J. Multivar. Anal. 131, 99–120 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  11. Daniels, M.J., Kass, R.E.: Shrinkage estimators for covariance matrices. Biometrics 57(4), 1173–1184 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  12. Evensen, G.: Data assimilation: the ensemble Kalman filter. Springer, Secaucus (2006)

    MATH  Google Scholar 

  13. Evensen, G.: EnKF—the ensemble Kalman filter. http://enkf.nersc.no/ (2015). Accessed 24 Apr 2015

  14. Godinez, H.C., Moulton, J.D.: An efficient matrix-free algorithm for the ensemble Kalman filter. Comput. Geosci. 16(3), 565–575 (2012)

    Article  Google Scholar 

  15. Jonathan, P., Fuqing, Z., Weng, Y.: The effects of sampling errors on the EnKF assimilation of inner-core hurricane observations. Mon. Weather Rev. 142(4), 1609–1630 (2014)

    Article  Google Scholar 

  16. Keppenne, C.L.: Data assimilation into a primitive-equation model with a parallel ensemble Kalman filter. Mon. Weather Rev. 128(6), 1971–1981 (2000)

    Article  Google Scholar 

  17. Kucharski, F., Molteni, F., Bracco, A.: Decadal interactions between the western tropical Pacific and the North Atlantic oscillation. Clim. Dynam. 26(1), 79–91 (2006)

    Article  Google Scholar 

  18. Ledoit, O., Wolf, M.: Honey, i shrunk the sample covariance matrix. UPF economics and business working paper (691) (2003)

  19. Ledoit, O., Wolf, M.: A well-conditioned estimator for large-dimensional covariance matrices. J. Multivar. Anal. 88(2), 365–411 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  20. Lorenc, A.C.: Analysis methods for numerical weather prediction. Q. J. R. Meteorol. Soc. 112(474), 1177–1194 (1986)

    Article  Google Scholar 

  21. Molteni, F.: Atmospheric simulations using a GCM with simplified physical parametrizations. I: model climatology and variability in multi-decadal experiments. Clim. Dynam. 20(2–3), 175–191 (2003)

    Article  Google Scholar 

  22. Nino-Ruiz, E.D., Sandu, A.: An efficient parallel implementation of the ensemble Kalman filter based on shrinkage covariance matrix estimation. In: Proceedings of the 2015 IEEE 22nd International Conference on High Performance Computing Workshops (HiPCW). IEEE Computer Society (2015)

  23. Nino-Ruiz, E.D., Sandu, A.: Ensemble Kalman filter implementations based on shrinkage covariance matrix estimation. Ocean Dynam. 65(11), 1423–1439 (2015)

    Article  Google Scholar 

  24. Nino-Ruiz, E.D., Sandu, A., Anderson, J.: An efficient implementation of the ensemble Kalman filter based on an iterative Sherman-Morrison formula. Stat. Comput. 25(3), 561–577 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  25. Ott, E., Hunt, B.R., Szunyogh, I., Zimin, A.V., Kostelich, Eric J, Corazza, Matteo, Kalnay, Eugenia, Patil, D .J., Yorke, James A: A local ensemble Kalman filter for atmospheric data assimilation. Tellus A 56(5), 415–428 (2004)

    Article  Google Scholar 

  26. Ott, E., Hunt, B., Szunyogh, I., Zimin, A.V., Kostelich, Eic J, Corazza, Matteo, Kalnay, Eugenia, Patil, D .J., Yorke, James A: A local ensemble transform Kalman filter data assimilation system for the NCEP global model. Tellus A 60(1), 113–130 (2008)

    Article  Google Scholar 

  27. Petra, C.G., Zavala, V.M., Nino-Ruiz, E.D., Anitescu, M.: A high-performance computing framework for analyzing the economic impacts of wind correlation. Electr. Power Syst. Res. 141, 372–380 (2016)

    Article  Google Scholar 

  28. Rao, V., Sandu, A.: A posteriori error estimates for DDDAS inference problems. In: Proceedings of the International Conference on Computational Science (ICCS-2014), vol. 29, pp. 1256–1265 (2014)

  29. Rao, V., Sandu, A.: A posteriori error estimates for inverse problems. SIAM/ASA J. Uncertain. Quantif. 3(1), 737–761 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  30. Sakov, P., Bertino, L.: Relation between two common localisation methods for the ENKF. Comput. Geosci. 15(2), 225–237 (2011)

    Article  MATH  Google Scholar 

  31. Sandu, A., Constantinescu, E.M., Carmichael, G.R., Chai, T., Daescu, D., Seinfeld, J.H.: Ensemble methods for dynamic data assimilation of chemical observations in atmospheric models. J. Algorithms Comput. Technol. 5(4), 667–692 (2011)

    Article  MathSciNet  Google Scholar 

  32. Schäfer, J., Strimmer, K., et al.: A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Stat. Appl. Genet. Mol. Biol. 4(1), 32 (2005)

    Article  MathSciNet  Google Scholar 

  33. Xiaohui, C., Wang, Z.J., McKeown, M.J.: Shrinkage-to-tapering estimation of large covariance matrices. IEEE Trans. Signal Process. 60(11), 5640–5656 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  34. Zupanski, M.: Theoretical and practical issues of ensemble data assimilation in weather and climate. In: Park, S.K., Xu, L. (eds.) Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications, pp. 67–84. Springer, Berlin, Heidelberg (2009)

    Chapter  Google Scholar 

Download references

Acknowledgements

This work was supported in part by awards NSF CCF-1218454, AFOSR FA9550-12-1-0293-DEF, and by the Computational Science Laboratory at Virginia Tech.

Author information

Authors and Affiliations

  1. Department of Computer Science, Universidad del Norte, Barranquilla, ATL, Colombia

    Elias D. Nino-Ruiz

  2. Computational Science Laboratory, Department of Computer Science, Virginia Tech, Blacksburg, VA, 24060, USA

    Adrian Sandu

Authors
  1. Elias D. Nino-Ruiz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adrian Sandu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elias D. Nino-Ruiz.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nino-Ruiz, E.D., Sandu, A. Efficient parallel implementation of DDDAS inference using an ensemble Kalman filter with shrinkage covariance matrix estimation. Cluster Comput 22 (Suppl 1), 2211–2221 (2019). https://doi.org/10.1007/s10586-017-1407-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-017-1407-1

Keywords