Abstract
Multi-wavelength data cross-match among multiple catalogs is a basic and unavoidable step to make distributed digital archives accessible and interoperable. As current catalogs often contain millions or billions objects, it is a typical data-intensive computation problem. In this paper, a high-efficient parallel approach of astronomical cross-match is introduced. We issue our partitioning and parallelization approach, after that we address some problems introduced by task partition and give the solutions correspondingly, including a sky splitting function HEALPix we selected which play a key role on both the task partitioning and the database indexing, and a quick bit-operation algorithm we advanced to resolve the block-edge problem. Our experiments prove that the function has a marked performance superiority comparing with the previous functions and is fully applicable to large-scale cross-match.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Nieto-Santisteban, M.A., Thakar, A.R., Szalay, A.S.: Cross-Matching Very Large Datasets. Johns Hopkins University, Baltimore (2006)
Nieto-Santisteban, M.A., Thakar, A.R., Szalay, A.S., Gray, J.: Large-Scale Query and XMatch, Entering the Parallel Zone. In: Astronomical Data Analysis Software and Systems XV ASP Conference Series, El Escoreal, Spain (2005)
Gray, J., Szalay, A., Budavri, T., Thakar, A.R., Nieto-Santisteban, M.A., Thakar, A.: Cross-Matching Multiple Spatial Observations and Dealing with Missing Data. Microsoft Technical Report, MSR-TR-2006-175, Redmond, WA (2006)
Gray, J., Szalay, A., Fekete, G.: Using Table Valued Functions in SQL Server 2005 To Implement a Spatial Data Library. MSR-TR-2005-122. Microsoft Technical Report, Redmond, WA (2005)
Gray, J., Nieto-Santisteban, M.A., Szalay, A.S.: The Zones Algorithm for Finding Points-Near-a-Point or Cross-Matching Spatial Datasets. MSR-TR-2006-52. Microsoft Technical Report, Redmond, WA (2006)
Gray, J., Szalay, A.S., Nieto-Santisteban, M.A., Heber, G., Rots, A.H.: There Goes the Neighborhood: Relational Algebra for Spatial Data Search. MSR-TR-2004-32. Microsoft Technical Report, Redmond, WA (2004)
Report on Cross Matching Catalogues. Technical report, AstroGrid, http://wiki.astrogrid.org/pub/Astrogrid/DataFederationandDataMining/cross.htm
Spatial Joins and Spatial Indexing Revisted. Technical report, AstroGrid, http://wiki.astrogrid.org/bin/view/Astrogrid/SpatialIndexing
Clive Page: Indexing the Sky. Technical Report, AstroGrid (2002)
Clive Page: Comments on the XMATCH function in ADQL. Technical report, AstroGrid (2004)
Dan, G., Zhang, Y.X., Zhao, Y.H.: Implementation of Cross-Matching on Very Large Multi-Wavelength catalogs. In: Astronomical Research and Technology. publications of National Astronomical Observatories of China (2005)
Dan, G.: Very Large Astronomical Data Sets Fusion System’s Development and Data Mining Algorithms’ Research. Doctor’s degree dissertation, National Astronomical Observatories of Chinese Academy of Sciences (2008)
Dan, G.: A System Integrated with Query, Cross-matching and Visualization. In: Lewis, H., Bridger, A. (eds.) Advanced Software and Control for Astronomy, vol. 6274. SPIE (2006)
Dan, G., Zhang, Y.X.: The Application of kd-tree in Astronomy. In: Astronomical Data Analysis Software and Systems XV2, ASP Conference Series (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhao, Q., Sun, J., Yu, C., Cui, C., Lv, L., Xiao, J. (2009). A Paralleled Large-Scale Astronomical Cross-Matching Function. In: Hua, A., Chang, SL. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2009. Lecture Notes in Computer Science, vol 5574. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03095-6_57
Download citation
DOI: https://doi.org/10.1007/978-3-642-03095-6_57
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-03094-9
Online ISBN: 978-3-642-03095-6
eBook Packages: Computer ScienceComputer Science (R0)