Kailash Kotwani
Terrence McLaren
ABSTRACT
The Dark Energy Survey (DES) collaboration is a multi-national science effort to understand cosmic acceleration and the nature of 'dark energy' responsible for this phenomenon. Dark Energy Survey Data Management (DESDM) system is a new observational astronomy processing pipeline and data management system that will be used to: process raw images obtained from a survey with the new DES field camera (DECam) covering 5000 sq degree of southern sky; archive intermediate and final co-added images; extract catalogs of celestial objects from every image and deliver data products to the astronomy community through portals and services. DESDM has been designed as a data intensive Science Gateway coupling use of shared computational resources (e.g. Teragrid) with project-owned databases and file systems for storage distributed across three continents. DESDM system over the next six years time will perform over 10 million CPU-hours (SUs) of image processing and serve over 4 Petabytes of images and 14 billion cataloged objects to the international DES collaboration. When delivered for operations in 2011, it will be one of, if not the, most scalable and powerful systems for processing telescope images, creating co-added deep images, and generating detailed star and galaxy catalogs in existence. The project's software components consist of a processing framework, an ensemble of astronomy codes, an integrated archive, a data-access framework and a portal infrastructure. This paper provides an overview of the DESDM scope and highlights, the architectural features developed and planned to be able to support Gateway-style management peta-scale intensive continuous processing and on-demand user queries for analysis.
- Schmidt, B. et al., "The High-Z Supernova Search: Measuring Cosmic Deceleration and Global Curvature of the Universe Using Type IA Supernovae," The Astronomical Journal, 507, 46--63 (1998).Google Scholar
- Perlmutter, S. et al., "Measurements of Omega and Lambda from 42 High-redshift Supernovae," The Astrophysical Journal, 517, 565--586 (1999).Google ScholarCross Ref
- Bahcall, N. A., Ostriker, J. P., Perlmutter, S. and Steinhardt, P. J., "The Cosmic Triangle: Revealing the State of the Universe," Science, 284, 1481 (1999).Google ScholarCross Ref
- Spergel, D. et al. "First-Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Determination of Cosmological Parameters," The Astrophysical Journal, 148, 175--194 (2003).Google ScholarCross Ref
- The DES collaboration, "Dark Energy Survey Science Program," http://www.darkenergysurvey.org/reports/proposal-standalone.pdfGoogle Scholar
- The DES collaboration, "Memorandum of Understanding for the Dark Energy Survey," http://www.darkenergysurvey.org/reports/DES_MOU_as_executed.pdfGoogle Scholar
- Joseph Mohr et al. "Dark Energy Survey Data Management Requirements and Technical Specifications," http://cosmology.illinois.edu/DES/documents/DESDM_Requirements.pdfGoogle Scholar
- Joseph Mohr et al. "Dark Energy Survey Data Management System," Marseille Meeting, SPIE 2008.Google Scholar
- Jim Gray et al. "Designing and Mining Multi-Terabyte Astronomy Archives: The Sloan Digital Sky Survey," Microsoft Research Technical Report, 1999, http://research.microsoft.com/en-us/um/people/gray/papers/ms_tr_99_30_sloan_digital_sky_survey.pdfGoogle Scholar
- Ani Thaker et al. "The Sloan Digital Sky Survey Drinking from the Fire Hose," Computing Science and Engineering, Vol. 10, No. 1 Google ScholarDigital Library
- Alex Szalay et al. "The SDSS DR1 SkyServer Public Access to a Terabyte of Astronomical Data," SDSS survey draft paper, http://cas.sdss.org/dr7/en/skyserver/paper/Google Scholar
- Robert Luption et al. "The SDSS imaging pipelines," Astronomical Data Analysis Software and Systems X ASP Conference Series, Vol. 238, 2001Google Scholar
- Charged Couple Device, http://en.wikipedia.org/wiki/Charge-coupled_deviceGoogle Scholar
- Joseph Mohr. et al., "Blanco Cosmology Survey (BCS," see http://cosmology.uiuc.edu/BCSGoogle Scholar
- Carlstrom, J. E. et al., "South Pole Telescope (SPT)," see http://spt.uchicago.eduGoogle Scholar
- NVO Science Definition Team, "Towards a National Virtual Observatory," Science Reports April 2002, http://www.us-vo.org/pubs/files/sdt-final.pdfGoogle Scholar
- Albert Rossi et al., "Elf/Ogrescript agents and containers," https://wiki.ncsa.illinois.edu/display/MRDPUB/Elf, https://wiki.ncsa.illinois.edu/display/MRDPUB/OgrescriptGoogle Scholar
- Michelle Gower, Joseph Mohr et al., "The Dark Energy Survey Data Management System: The Processing Framework," Astronomical Data Analysis Software and Systems, ASP Conference Series, Vol XXX, 2008Google Scholar
Index Terms
- The dark energy survey data management system as a data intensive science gateway
Recommendations
Enabling dark energy survey science analysis with simulations on XSEDE resources
XSEDE '13: Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to DiscoveryUpcoming wide-area sky surveys offer the power to test the source of cosmic acceleration by placing extremely precise constraints on existing cosmological model parameters. These observational surveys will employ multiple tests based on statistical ...
A Survey of Data-Intensive Scientific Workflow Management
Nowadays, more and more computer-based scientific experiments need to handle massive amounts of data. Their data processing consists of multiple computational steps and dependencies within them. A data-intensive scientific workflow is useful for ...
Comments