skip to main content
10.1145/1878500.1878511acmconferencesArticle/Chapter ViewAbstractPublication PagesgisConference Proceedingsconference-collections
research-article

A programming framework for integrating web-based spatiotemporal sensor data with MapReduce capabilities

Published: 02 November 2010 Publication History

Abstract

Web-based sensor data, provided by organizations such as the National Oceanographic and Atmospheric Administration, provide a valuable service to the public and scientific communities. However, much of this data is locked in a variety of presentation formats and is computationally inaccessible. In addition, although these data have a spatiotemporal context, both the spatial and temporal data are usually only implicitly defined. Although storing this data in a consistent database can partially resolve this problem, a data-driven programming model coupled with MapReduce capabilities is a more flexible and extensible solution. Our implementation of this programming model allows users to parse a wide array of sensor data and express complex computation in a simple, scalable manner. In addition, our framework uses a simple key-value storage mechanism and provides convenient geospatial output mechanisms. In this paper, we discuss some early results of our programming model within the context of our current Java-oriented implementation, and demonstrate how the system can be used to create many different applications. We also discuss and evaluate our system with respect to memory usage and scalability.

References

[1]
T. Abdelzaher, B. Blum, Q. Cao, Y. Chen, D. Evans, J. George, S. George, L. Gu, T. He, S. Krishnamurthy, L. Luo, S. Son, J. Stankovic, R. Stoleru, and A. Wood. Envirotrack: Towards an environmental computing paradigm for distributed sensor networks. In International Conference on Distributed Computing Systems (ICDCS), 2004.
[2]
B. Beran, D. Fay, and C. van Ingen. Sciscope: Using virtual globes for environmental data discovery. In American Geophysical Union, 2008.
[3]
F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A distributed storage system for structured data. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2006.
[4]
K. Chang, N. Yau, M. Hansen, and D. Estrin. Sensorbase.org - a centralized repository to slog sensor network data. In Euro-American Workshop on Middleware for Sensor Networks (EAWMS - DCOSS), 2006.
[5]
E. Cheong, E. A. Lee, and Y. Zhao. Viptos: a graphical development and simulation environment for tinyos-based wireless sensor networks. In ACM Conference on Embedded Networked Sensor Systems (SenSys), 2005.
[6]
C. J. Date. A guide to the SQL standard. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1986.
[7]
J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. 2004.
[8]
D. Gay, P. Levis, R. von Behren, M. Welsh, E. Brewer, and D. Culler. The nesc language: A holistic approach to networked embedded systems. In Programming Language Design and Implementation (PLDI), 2003.
[9]
O. Gnawali, B. Greenstein, K.-Y. Jang, A. Joki, J. Paek, M. Vieira, D. Estrin, R. Govindan, and E. Kohler. The tenet architecture for tiered sensor networks. In ACM Conference on Embedded Networked Sensor Systems (SenSys), 2006.
[10]
B. L. Gorman, D. R. Resseguie, and C. H. Tomkins-Tinch. Sensorpedia: Information sharing across incompatible sensor systems. In International Symposium on Collaborative Technologies and Systems, 2009.
[11]
R. Gummadi, O. Gnawali, and R. Govindan. Macro-programming wireless sensor networks using kairos. In International Conference on Distributed Computing in Sensor Systems (DCOSS), 2005.
[12]
B. He, W. Fang, Q. Luo, N. K. Govindaraju, and T. Wang. Mars: a mapreduce framework on graphics processors. In International Conference on Parallel Architectures and Compilation Techniques (PACT), 2008.
[13]
J. Horey, A. Kilzer, J.-C. Tournier, P. Widener, and A. B. Maccabe. A filesystem interface for sensor networks. Technical report, University of New Mexico, 2008.
[14]
J. Horey, A. B. Maccabe, and A. Mielke. Kensho: A dynamic tasking architecture for sensor networks. In Workshop for Wireless Sensor Network Architectures - IPSN, 2007.
[15]
P. Levis and D. Culler. Maté: A Tiny Virtual Machine for Sensor Networks. In Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2002.
[16]
J. Liu and F. Zhao. Towards semantic services for sensor-richinformation systems. In International Conference on Broadband Networks (BROADNETS), 2005.
[17]
S. R. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong. Tinydb: an acquisitional query processing system for sensor networks. ACM Transaction Database Systems, pages 122--173, 2005.
[18]
G. Mainland and M. Welsh. Programming sensor networks using abstract regions. In Symposium on Networked Systems Design and Implementation (NSDI), 2004.
[19]
W. P. McCartney and N. Sridhar. Tosdev: a rapid development environment for tinyos. In ACM Conference on Embedded Networked Sensor Systems (SenSys), 2006.
[20]
S. A. Mcilraith, T. C. Son, and H. Zeng. Semantic web services. IEEE Intelligent Systems, 16:46--53, 2001.
[21]
E. Meijer, B. Beckman, and G. Bierman. Linq: reconciling object, relations and xml in the .net framework. In ACM SIGMOD International Conference on Management of Data, 2006.
[22]
D. Mills. Network time protocol rfc (version 3, march 1992).
[23]
S. Nath, J. Liu, and F. Zhao. Sensormap for wide-area sensor webs. IEEE Computer Magazine, 40(7):90--93, 2007.
[24]
R. R. Newton, L. D. Girod, J. G. Morrisett, M. B. Craig, and S. R. Madden. Design and evaluation of a compiler for embedded stream programs. In ACM SIGPLAN/SIGBED Conference on Languages, Compilers, and Tools for Embedded Systems (LCTES), 2008.
[25]
R. R. Newton, J. G. Morrisett, and M. Welsh. The regiment macroprogramming system. In Information Processing in Sensor Networks (IPSN), 2007.
[26]
C. Ranger, R. Raghuraman, A. Penmetsa, G. Bradski, and C. Kozyrakis. Evaluating mapreduce for multi-core and multiprocessor systems. In IEEE International Symposium on High Performance Computer Architecture (HPCA), 2007.
[27]
K. Whitehouse, C. Sharp, E. Brewer, and D. Culler. Hood: a neighborhood abstraction for sensor networks. In International Conference on Mobile Systems, Applications, and Services (MobiSys), 2004.
[28]
H.-c. Yang, A. Dasdan, R.-L. Hsiao, and D. S. Parker. Mapreduce-merge: simplified relational data processing on large clusters. In ACM SIGMOD international conference on Management of data (SIGMOD), 2007.
[29]
Y. Yao and J. Gehrke. The Cougar Approach to In-Network Query Processing in Sensor Networks. In ACM SIGMOD Conference, 2002.
[30]
Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. Kumar, and G. J. Currey. Dryadlinq: A system for general-purpose distributed data-parallel computing using a high-level language. 2008.

Cited By

View all
  • (2013)The QUASIT Model and Framework for Scalable Data Stream Processing with Quality of ServiceMobile Wireless Middleware, Operating Systems, and Applications10.1007/978-3-642-36660-4_7(92-107)Online publication date: 2013
  • (2013)MapReduce Performance in Federated Cloud Computing EnvironmentsHigh Performance Cloud Auditing and Applications10.1007/978-1-4614-3296-8_12(301-322)Online publication date: 1-Aug-2013
  • (2012)Design and Implementation of a Scalable and QoS-aware Stream Processing FrameworkProceedings of the 2012 IEEE International Conference on Green Computing and Communications10.1109/GreenCom.2012.54(458-467)Online publication date: 20-Nov-2012

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
IWGS '10: Proceedings of the ACM SIGSPATIAL International Workshop on GeoStreaming
November 2010
67 pages
ISBN:9781450304313
DOI:10.1145/1878500
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2010

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. MapReduce
  2. programming
  3. sensor data

Qualifiers

  • Research-article

Conference

GIS '10
Sponsor:

Acceptance Rates

Overall Acceptance Rate 7 of 9 submissions, 78%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1
  • Downloads (Last 6 weeks)0
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2013)The QUASIT Model and Framework for Scalable Data Stream Processing with Quality of ServiceMobile Wireless Middleware, Operating Systems, and Applications10.1007/978-3-642-36660-4_7(92-107)Online publication date: 2013
  • (2013)MapReduce Performance in Federated Cloud Computing EnvironmentsHigh Performance Cloud Auditing and Applications10.1007/978-1-4614-3296-8_12(301-322)Online publication date: 1-Aug-2013
  • (2012)Design and Implementation of a Scalable and QoS-aware Stream Processing FrameworkProceedings of the 2012 IEEE International Conference on Green Computing and Communications10.1109/GreenCom.2012.54(458-467)Online publication date: 20-Nov-2012

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media