skip to main content
10.1145/1646468.1646482acmconferencesArticle/Chapter ViewAbstractPublication PagesscConference Proceedingsconference-collections
research-article

Many task computing for multidisciplinary ocean sciences: real-time uncertainty prediction and data assimilation

Published: 16 November 2009 Publication History

Abstract

Error Subspace Statistical Estimation (ESSE), an uncertainty prediction and data assimilation methodology employed for real-time ocean forecasts, is based on a characterization and prediction of the largest uncertainties. This is carried out by evolving an error subspace of variable size. We use an ensemble of stochastic model simulations, initialized based on an estimate of the dominant initial uncertainties, to predict the error subspace of the model fields. The dominant error covariance (generated via an SVD of the ensemble-generated error covariance matrix) is used for data assimilation. The resulting ocean fields are provided as the input to acoustic modeling, allowing for the prediction and study of the spatiotemporal variations in acoustic propagation and their uncertainties.
The ESSE procedure is a classic case of Many Task Computing: These codes are managed based on dynamic workflows for the: (i) perturbation of the initial mean state, (ii) subsequent ensemble of stochastic PE model runs, (iii) continuous generation of the covariance matrix, (iv) successive computations of the SVD of the ensemble spread until a convergence criterion is satisfied, and (v) data assimilation. Its ensemble nature makes it a many task data intensive application and its dynamic workflow gives it heterogeneity. Subsequent acoustics propagation modeling involves a very large ensemble of short-in-duration acoustics runs.
We study the execution characteristics and challenges of a distributed ESSE workflow on a large dedicated cluster and the usability of enhancing this with runs on Amazon EC2 and the Teragrid and the I/O challenges faced.

References

[1]
Evangelinos, C., Lermusiaux, P. F. J., Geiger, S., Chang, R. C. and Patrikalakis, N. M. 2006. Web-Enabled Configuration and Control of Legacy Codes: An Application to Ocean Modeling. Ocean Modeling, 13, 197--220.
[2]
Gardiner, C. W. 1983. Handbook of Stochastic Methods for Physics, Chemistry and the Natural Sciences, Springer-Verlag, 442 pages.
[3]
Gelb, A. (ed.) 1974. Applied Optimal Estimation, MIT Press, Cambridge, MA.
[4]
Gentzsch, W. 2001. Sun Grid Engine: Towards Creating a Compute Power Grid. In Proceedings of 1st IEEE International Symposium on Cluster Computing and the Grid (May 15--18, 2001). CCGRID. IEEE Computer Society, Washington, DC, 35.
[5]
Heaney, K. D., Gawarkiewicz, G., Duda, T. F. and Lermusiaux, P. F. J. 2007. Non-linear Optimization of Autonomous Undersea Vehicle Sampling Strategies for Oceanographic Data-Assimilation. In special issue on "Underwater Robotics", Journal of Field Robotics, 24(6), 437--448.
[6]
Hoskins, M. E. 2006. SSHFS: super easy file access over SSH. Linux J., 146 (Jun. 2006), 4.
[7]
Ide, K., Courtier, P., Ghil, M., and Lorenc A. C. 1997. Unified notation for data assimilation: operational, sequential and variational. J. of the Meteorol. Soc. of Japan, 75(1B), 181--189.
[8]
Jazwinski, A. H. 1970. Stochastic Processes and Filtering Theory, Academic Press.
[9]
Lam, F. P, Haley Jr., P. J., Janmaat, J., Lermusiaux, P. F. J., Leslie, W. G., Schouten, M. W., te Raa, L. A. and Rixen, M. 2009. At-sea Real-time Coupled Four-dimensional Oceanographic and Acoustic Forecasts during Battlespace Preparation 2007. In special issue of the Journal of Marine Systems on "Coastal processes: Challenges for Monitoring and Prediction", J. W. Book, M. Orlic and M. Rixen, Eds.
[10]
Lermusiaux, P. F. J. 2006. Uncertainty Estimation and Prediction for Interdisciplinary Ocean Dynamics. In special issue on "Uncertainty Quantification". J. Glimm and G. Karniadakis, Eds., Journal of Computational Physics, 176--199.
[11]
Lermusiaux, P. F. J. 2007. Adaptive Modeling, Adaptive Data Assimilation and Adaptive Sampling. In special issue on "Mathematical Issues and Challenges in Data Assimilation for Geophysical Systems: Interdisciplinary Perspectives", Physica D, C. K. R. T. Jones and K. Ide Eds., 230, 172--196.
[12]
Lermusiaux, P. F. J. and Chiu, C.-S. 2002. Four-dimensional data assimilation for coupled physical-acoustical fields. In Acoustic Variability, N. G. Pace and F. B. Jensen, Eds., SACLANTCEN. Kluwer Academic Press, 417--424.
[13]
Lermusiaux, P. F. J., Chiu, C.-S., Gawarkiewicz, G. G., Abbot, P., Robinson, A. R., Miller, R. N., Haley Jr., P. J., Leslie, W. G., Majumdar, S. J., Pang, A. and Lekien, F. 2006. Quantifying Uncertainties in Ocean Predictions. In special issue on "Advances in Computational Oceanography", Oceanography, T. Paluszkiewicz and S. Harper, Eds., 19(1), 92--105.
[14]
Lermusiaux, P. F. J., Haley Jr., P. J., and Yilmaz, N. K. 2007. Environmental Prediction, Path Planning and Adaptive Sampling: Sensing and Modeling for Efficient Ocean Monitoring, Management and Pollution Control. Sea Technology, 48(9), 35--38.
[15]
Lermusiaux, P. F. J. and Robinson, A. R. 1999. Data assimilation via error subspace statistical estimation, Part I: theory and schemes. Mon. Wea. Rev. 127(7), 1385--1407.
[16]
Lermusiaux, P. F. J., Robinson, A. R., Haley Jr., P. J., and Leslie, W. G. 2002. Advanced interdisciplinary data assimilation: Filtering and smoothing via Error Subspace Statistical Estimation. In Proceedings of "The OCEANS 2002 MTS/IEEE" conference, IEEE, Holland Publications, 795--802.
[17]
Raicu, I., Foster, I., and Zhao Y. 2008. Many-Task Computing for Grids and Supercomputers, Proceedings of "IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS08)".
[18]
Sapsis, T. P. and Lermusiaux, P. F. J. 2009. Dynamically orthogonal field equations for continuous stochastic dynamical systems. Physica D.
[19]
Thain D., Tannenbaum T. and Livny M. 2005. Distributed computing in practice: the Condor experience. Research Articles. Concurr. Comput.: Pract. Exper. 17, 2--4 (Feb. 2005), 323--356.
[20]
Walker, E. 2006. A distributed file system for a wide-area high performance computing infrastructure. In Proceedings of the 3rd conference on USENIX Workshop on Real, Large Distributed Systems - Volume 3 (Seattle, WA). USENIX Association, Berkeley, CA, pp. 9
[21]
Walker, E., Gardner, P. J., Litvin, V., and Turner, E. 2007. Personal Adaptive Clusters as Containers for Scientific Jobs. Cluster Computing, 10(3), Sept. 2007.
[22]
Wang, D., Lermusiaux, P. F. J., Haley Jr., P. J., Eickstedt, D., Leslie, W. G. and Schmidt, H. 2009. Acoustically Focused Adaptive Sampling and On-board Routing for Marine Rapid Environmental Assessment. In special issue of the Journal of Marine Systems on "Coastal Processes: Challenges for Monitoring and Prediction", J. W. Book, M. Orlic and M. Rixen, Eds.
[23]
Xu, J., Lermusiaux, P. F. J., Haley Jr., P. J., Leslie, W. G. and Logutov, O. G. 2008. Spatial and Temporal Variations in Acoustic propagation during the PLUSNet07 Exercise in Dabob Bay. In Proceedings of Meetings on Acoustics (POMA), 4, 155th Meeting Acoustical Society of America.
[24]
Yilmaz N. K., Evangelinos, C., Lermusiaux, P. F. J. and Patrikalakis, N. M. 2008. Path Planning of Autonomous Underwater Vehicles for Adaptive Sampling Using Mixed Integer Linear Programming. IEEE Transactions, Journal of Oceanic Engineering. 33 (4), 522--537.

Cited By

View all
  • (2012)In Cloud, Can Scientific Communities Benefit from the Economies of Scale?IEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2011.14423:2(296-303)Online publication date: 1-Feb-2012
  • (2011)Design and implementation of "many parallel task" hybrid subsurface modelProceedings of the 2011 ACM international workshop on Many task computing on grids and supercomputers10.1145/2132876.2132884(25-32)Online publication date: 14-Nov-2011
  • (2011)MPI Performance Analysis of Amazon EC2 Cloud Services for High Performance ComputingAdvances in Computing and Communications10.1007/978-3-642-22709-7_38(371-381)Online publication date: 2011
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MTAGS '09: Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers
November 2009
131 pages
ISBN:9781605587141
DOI:10.1145/1646468
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 November 2009

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. MTC
  2. assimilation
  3. data-intensive
  4. ensemble

Qualifiers

  • Research-article

Funding Sources

Conference

SC '09
Sponsor:

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)1
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2012)In Cloud, Can Scientific Communities Benefit from the Economies of Scale?IEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2011.14423:2(296-303)Online publication date: 1-Feb-2012
  • (2011)Design and implementation of "many parallel task" hybrid subsurface modelProceedings of the 2011 ACM international workshop on Many task computing on grids and supercomputers10.1145/2132876.2132884(25-32)Online publication date: 14-Nov-2011
  • (2011)MPI Performance Analysis of Amazon EC2 Cloud Services for High Performance ComputingAdvances in Computing and Communications10.1007/978-3-642-22709-7_38(371-381)Online publication date: 2011
  • (2010)Many-task applications in the Integrated Plasma Simulator2010 3rd Workshop on Many-Task Computing on Grids and Supercomputers10.1109/MTAGS.2010.5699425(1-10)Online publication date: Nov-2010
  • (2010)Developing a Cloud Computing Charging Model for High-Performance Computing ResourcesProceedings of the 2010 10th IEEE International Conference on Computer and Information Technology10.1109/CIT.2010.72(210-217)Online publication date: 29-Jun-2010

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media