Abstract
The goal of this study is design of a database holding archives of weather forecast systems. The detailed description of the project and analysis of the database performance is presented, along with the experimental performance tests of key algorithms. A standard mode of data access is optimal for a sequential access to individual 2D grids (time-local and spatially-global), whereas new applications require reading long time series of localised data (spatially-local and time-global). The design goal is to increase a performance of access to archival spatially-local-time-global data, without visible degradation of performance of the standard access mode. The database is designed as two separate layers. The Format Translation Layer (FTL) is an interface between the database and the file-based output of the simulation and analysis programs. The Distributed Data Storage Layer (DDSL) is responsible for a secure data storage and an efficient access. The FTL reads output forecasts and converts them to the spatially local format. It splits large 2D arrays of data into small patches and forms 3D arrays, using time as the third dimension. The theoretical analysis of the performance shows that four orders of magnitudes improvement in comparison with the standard serial access and two orders of magnitude in comparison with parallelised version of the spatially-global access can be achieved.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Allen R. Retrieval from facet spaces,. Electronic Publishing, 8:247-257, 1996.
Becla J, Lim K-T. Report from the 1st Workshop on Extremely Large Databases. Data Science Journal, 7:1-13, 2008.
Becla J, Lim K-T. Report from the 2nd Workshop on Extremely Large Databases. Data Science Journal, 7:196-208, 2008.
Benson DA, Boguski MS, Lipman DJ, Ostell J, Ouellette BFF. GenBank. Nucleic Acids Research, 26(1):1-7, 1998.
Berman H, Westbrook J, Feng Z, Gilliland G, Bhat T, Weissig H, Shindyalov I, Bourne P. The Protein Data Bank. Nucleic Acids Research, 28:235,-242, 2000.
Cudre-Mauroux P, Kimura H, Lim K-T, Rogers J, Simakov R, Soroush E, Velikhov P, Wang DL, Balazinska M, Becla J, DeWitt D, Heath B, Maier D, Madden S, Patel J, Stonebraker M, Zdonik S. A demonstration of SciDB:a science-oriented DBMS. Proceedings of the VLDB Endowment, 2(2), 2009.
Davies T, Cullen MJP, Malcolm AJ, Mawson MH, Staniforth A, White AA, Wood N. A new dynamical core for the Met Office’s global and regional modelling of the atmosphere. Quarterly Journal of the Royal Meteorological Society, 131(608):1759-1782, 2005.
Davis G, Rew R, Hartnett E, Caron J, Emmerson S, Davies H. NETCDF Users Guide, 2011.
Folk M, Cheng A, Yates K. HDF5: A _le format and I/O library for high performance computing. In Proceedings of Supercomputing’99. National Center for Supercomputing Applications (NCSA), 1999.
Folk M, Pourmal E, Balancing Performance and Preservation Lessons learned with HDF5, 2010.
Hodur RM. The Naval Research Laboratory’s Coupled Ocean/Atmosphere Mesoscale Prediction System (COAMPS). Monthly Weather Review, 125(7):1414, 1997.
Kantor J, Axelrod T, Becla J, Cook K, Nikolaev S, Gray J, Plante R, Nieto-Santisteban M, Szalay A, Thakar A. Designing for Peta-Scale in the LSST Database. ASP Conference Series, Vol. XXX. In R. A. Shaw, F. Hill, and D. J. Bell, editors, Astronomical Data Analysis Software and Systems XVI, 2007.
Loebel-Carpenter L, Lueking L, Moore C, Pordes R, Trumbo J, Veseli S, Terekhov I, Vranicar M, White S, White V. SAM and the particle physics data Grid. In: Computing in High-Energy and Nuclear Physics, pages 765-769, 2001.
Pop M, Salzberg SL. Bioinformatics challenges of new sequencing technology. Trends in genetics : TIG, 24(3):142-9, Mar. 2008.
Slezak D, Wróblewski J, Eastwood V, Synak P. Brighthouse: an analytic data warehouse for ad-hoc queries. Proc. VLDB Endow, 1(2):13371345, 2008.
Terekhov I. Meta-computing at D0. Nuclear Instruments and Methods in Physics Research, Section A, 502:402-406, 2002.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Walkowiak, S., Ligowski, L., Wawruch, K., Rudnicki, W.R. (2011). Design of the Efficient Archival Warehouse for Weather Forecast System. In: Golinska, P., Fertsch, M., Marx-Gómez, J. (eds) Information Technologies in Environmental Engineering. Environmental Science and Engineering(), vol 3. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-19536-5_58
Download citation
DOI: https://doi.org/10.1007/978-3-642-19536-5_58
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-19535-8
Online ISBN: 978-3-642-19536-5
eBook Packages: Earth and Environmental ScienceEarth and Environmental Science (R0)