skip to main content
10.1145/3017611.3017627acmconferencesArticle/Chapter ViewAbstractPublication PagesgisConference Proceedingsconference-collections
short-paper

On storing and retrieving geospatial big-data in cloud

Published:31 October 2016Publication History

ABSTRACT

Cloud storage is a kind of external storage which can provide by unlimited storage space with high availability and low cost on maintenance. On the other side, the size of geospatial data is too large and is increasing dramatically which makes such data is hard to be stored in the local data warehouse. Hence following the benefits of Cloud storage, such geospatial data is suitable to be stored in Cloud storage and managed by local data warehouse. However, there is a gap between Cloud storages and data warehouses built on traditional infrastructures, such as the mostly adopted massive parallel processing (MPP) based data warehouse. Therefore, in this paper, we propose a middleware-like architecture to connect MPP data warehouse and Cloud storage. It supports traditional geospatial data retrieving while integrating the Cloud storage lineage by a set of technical designs. Based on the prototype system and practical data, we demonstrate the appreciable performance and the flexibility for other third parties' development. Another major contribution of this paper is that we implement the prototype on open-source data warehouse and we make it open-sourced to public.

References

  1. B. Sanou, "The world in 2013: ICT facts and figures," in International Communication Union, United Nations, 2013.Google ScholarGoogle Scholar
  2. Open Street Map: http://www.openstreetmap.org.Google ScholarGoogle Scholar
  3. J. Pateletal. Building a Scalable Geo-Spatial DBMS: Technology, Implementation, and Evaluation. In: ACM SIGMOD, pp. 336--347, 1997.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. In: OSDI, pp. 137--150, 2004.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Michael Stonebraker, Daniel Abadi, David J. DeWitt, Sam Madden, Erik Paulson, Andrew Pavlo, and Alexander Rasin. 2010. MapReduce and parallel DBMSs: friends or foes? Communications of the ACM, 53(1):64--71, January 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Abouzeid, A., Bajda-Pawlikowski, K., Abadi, D.J., Silberschatz, A., and Rasin, A. HadoopDB: An architectural hybrid of MapReduce and DBMS technologies for analytical workloads. In: Proc. VLDB Endow. 2(1):922--933, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. A. Floratou, U. F. Minhas, and F. Özcan. SQL-on-Hadoop: full circle back to shared-nothing database architectures. In: Proc. VLDB Endow. 7(12): 1295--1306, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Yueguo Chen, Xiongpai Qin, Haoqiong Bian, et al. A Study of SQL-on-Hadoop Systems. Big Data Benchmarks, Performance Optimization, and Emerging Hardware, 2014. Google ScholarGoogle ScholarCross RefCross Ref
  9. B.Zhao, B.I.P.Rubinstein, J.Gemmell, and J.Han. A bayesian approach to discovering truth from conflicting sources for data integration. PVLDB, 5(6):550--561, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Chang, L., Wang, Z., Ma, T., et al, M.: Hawq: a massively parallel processing sql engine in hadoop. In: SIGMOD Conference, pp. 1223--1234 (2014) Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Iu, M.-Y., Zwaenepoel, W.: Hadooptosql: a mapreduce query optimizer. In: EuroSys Conference, pp. 251--264 (2010) Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. PostgreSQL: http://www.postgresql.org.Google ScholarGoogle Scholar
  13. S3_fdw: https://github.com/umitanuki/s3_fdw.Google ScholarGoogle Scholar
  14. Floratou, A., Teletia, N., DeWitt, D.J., Patel, J.M., Zhang, D.: Can the elephants handle the nosql onslaught? PVLDB 5(12), 1712--1723 (2012). Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. S. Li, S. Hu, R. Ganti, M. Srivatsa, and T. Abdelzaher. Pyro: A Spatial-Temporal Big-Data Storage System, In: ACM USENIX ATC Conference, 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Kuien Liu, Yandong Yao, Danhuai Guo. On managing geospatial big-data in emergency management: some perspectives, In Proc. of ACM GIS, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Greenplum: http://greenplum.org/.Google ScholarGoogle Scholar
  18. Amazon S3: https://aws.amazon.com/s3/.Google ScholarGoogle Scholar
  19. PostGIS: http://postgis.net/.Google ScholarGoogle Scholar
  20. Microsoft Azure: https://azure.microsoft.com/en-us/services/.Google ScholarGoogle Scholar
  21. Google Cloud Storage: https://cloud.google.com/storage/.Google ScholarGoogle Scholar
  22. Prototype source-code: http://github.com/greenplum-db/gpdb/tree/master/gpAux/extensions/gps3ext/.Google ScholarGoogle Scholar

Index Terms

  1. On storing and retrieving geospatial big-data in cloud

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in
        • Published in

          cover image ACM Conferences
          EM-GIS '16: Proceedings of the Second ACM SIGSPATIALInternational Workshop on the Use of GIS in Emergency Management
          October 2016
          101 pages
          ISBN:9781450345804
          DOI:10.1145/3017611

          Copyright © 2016 ACM

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 31 October 2016

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • short-paper

          Acceptance Rates

          EM-GIS '16 Paper Acceptance Rate16of26submissions,62%Overall Acceptance Rate30of54submissions,56%

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader