Article

Operator placement for in-network stream query processing

Authors:

Utkarsh Srivastava,

Kamesh Munagala,

Jennifer WidomAuthors Info & Claims

PODS '05: Proceedings of the twenty-fourth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

Pages 250 - 258

https://doi.org/10.1145/1065167.1065199

Published: 13 June 2005 Publication History

Abstract

In sensor networks, data acquisition frequently takes place at low-capability devices. The acquired data is then transmitted through a hierarchy of nodes having progressively increasing network band-width and computational power. We consider the problem of executing queries over these data streams, posed at the root of the hierarchy. To minimize data transmission, it is desirable to perform "in-network" query processing: do some part of the work at intermediate nodes as the data travels to the root. Most previous work on in-network query processing has focused on aggregation and inexpensive filters. In this paper, we address in-network processing for queries involving possibly expensive conjunctive filters, and joins. We consider the problem of placing operators along the nodes of the hierarchy so that the overall cost of computation and data transmission is minimized. We show that the problem is tractable, give an optimal algorithm, and demonstrate that a simpler greedy operator placement algorithm can fail to find the optimal solution. Finally we define a number of interesting variations of the basic operator placement problem and demonstrate their hardness.

References

[1]

Y. Ahmad and U. Cetintemel. Network-aware query processing for stream-based applications. In Proc. of the 2004 Intl. Conf. on Very Large Data Bases, pages 456--467, Sept. 2004.

Digital Library

[2]

B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom. Models and issues in data stream systems. In Proc. of the 2002 ACM Symp. on Principles of Database Systems, pages 1--16, June 2002.

Digital Library

[3]

S. Chaudhuri and K. Shim. Optimization of queries with user-defined predicates. ACM Trans. on Database Systems, 24(2):177--228, 1999.

Digital Library

[4]

J. Chen, D. J. DeWitt, F. Tian, and Y. Wang. NiagaraCQ: A scalable continuous query system for internet databases. In Proc. of the 2000 ACM SIGMOD Intl. Conf. on Management of Data, pages 379--390, May 2000.

Digital Library

[5]

M. Cherniack et al. Scalable distributed stream processing. In Proc. First Biennial Conf. on Innovative Data Systems Research (CIDR), Jan. 2003.

[6]

C. Cranor et al. Gigascope: high performance network monitoring with an SQL interface. In Proc. of the 2002 ACM SIGMOD Intl. Conf. on Management of Data, page 623, May 2002.

Digital Library

[7]

A. Deshpande, S. Nath, P. Gibbons, and S. Seshan. Cache-and-query for wide area sensor databases. In Proc. of the 2003 ACM SIGMOD Intl. Conf. on Management of Data, pages 503--514, 2003.

Digital Library

[8]

R. Epstein, M. Stonebraker, and E. Wong. Distributed query processing in a relational data base system. In Proc. of the 1978 ACM SIGMOD Intl. Conf. on Management of Data, pages 169--180, May 1978.

Digital Library

[9]

U. Feige, L. Lovász, and P. Tetali. Approximating min-sum set cover. Algorithmica, 2004.

Digital Library

[10]

M. Franklin et al. Design Considerations for High Fan-in Systems: The HiFi Approach. In Proc. Second Biennial Conf. on Innovative Data Systems Research (CIDR), Jan. 2005.

[11]

M. Garey and D. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman & Co., 1979.

Digital Library

[12]

L. Haas et al. R*: A Research Project on Distributed Relational DBMS. IEEE Data Engineering Bulletin, 5(4):28--32, 1982.

[13]

J. Hellerstein and M. Stonebraker. Predicate migration: Optimizing queries with expensive predicates. In Proc. of the 1993 ACM SIGMOD Intl. Conf. on Management of Data, pages 267--276, 1993.

Digital Library

[14]

S. Madden, M. Franklin, J. Hellerstein, and W. Hong. TAG: A tiny aggregation service for ad-hoc sensor networks. In Proceedings of the 5th USENIX Symposium on OSDI, Dec. 2002.

Digital Library

[15]

S. Madden, M. Franklin, J. Hellerstein, and W. Hong. The design of an acquisitional query processor for sensor networks. In Proc. of the 2003 ACM SIGMOD Intl. Conf. on Management of Data, pages 491--502, 2003.

Digital Library

[16]

K. Munagala, S. Babu, R. Motwani, and J. Widom. The pipelined set cover problem. Proc. Intl. Conf. Database Theory, 2005. (To appear).

Digital Library

[17]

P. Pietzuch et al. Path optimization in stream-based overlay networks. Technical report, Harvard University, 2004.

[18]

S. Viglas, J. F. Naughton, and J. Burger. Maximizing the output rate of multi-join queries over streaming information sources. In Proc. of the 2003 Intl. Conf. on Very Large Data Bases, pages 285--296, Sept. 2003.

Digital Library

Cited By

Tang XZhang FZhang SLiu YHe BHe BDu XDu X(2024)Enabling Adaptive Sampling for Intra-Window Join: Simultaneously Optimizing Quantity and QualityProceedings of the ACM on Management of Data10.1145/36771342:4(1-31)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3677134
Helmer SWood PStuefer M(2024)Optimism and Pessimism in Database Query Optimisation2024 IEEE 18th International Conference on Semantic Computing (ICSC)10.1109/ICSC59802.2024.00049(269-276)Online publication date: 5-Feb-2024
https://doi.org/10.1109/ICSC59802.2024.00049
Volnes EPlagemann TGoebel V(2024)To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream ProcessingIEEE Communications Surveys & Tutorials10.1109/COMST.2023.333095326:1(670-705)Online publication date: Sep-2025
https://doi.org/10.1109/COMST.2023.3330953
Show More Cited By

Recommendations

Migrating operator placement for compositional stream graphs
MSWiM '12: Proceedings of the 15th ACM international conference on Modeling, analysis and simulation of wireless and mobile systems

Wireless sensor networks (WSN) and mobile clouds are composed of sensor nodes that have limited energy resources. For wireless sensor networks, query processing is the state-of-the-art for data gathering and processing applications to avoid low-level ...
Adaptive and Decentralized Operator Placement for In-Network Query Processing

In-network query processing is critical for reducing network traffic when accessing and manipulating sensor data. It requires placing a tree of query operators such as filters and aggregations but also correlations onto sensor nodes in order to minimize ...
Distributed stream join query processing with semijoins

This paper addresses the distributed stream processing of window-based multi-way join queries considering the semijoin as a key join operator. In distributed stream processing, data streams arriving at remote sites need to be shipped to the processing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

PODS '05: Proceedings of the twenty-fourth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

June 2005

388 pages

ISBN:1595930620

DOI:10.1145/1065167

General Chair:
Georg Gottlob
Vienna University of Technology, Austria
,
Program Chair:
Foto Afrati
National Technical University of Athens, Greece

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SIGMOD/PODS05

Sponsor:

SIGMOD/PODS05: International Conference on Management of Data and Symposium on Principles Database and Systems

June 13 - 15, 2005

Maryland, Baltimore

Acceptance Rates

Overall Acceptance Rate 642 of 2,707 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

136
Total Citations
View Citations
1,026
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)2

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tang XZhang FZhang SLiu YHe BHe BDu XDu X(2024)Enabling Adaptive Sampling for Intra-Window Join: Simultaneously Optimizing Quantity and QualityProceedings of the ACM on Management of Data10.1145/36771342:4(1-31)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3677134
Helmer SWood PStuefer M(2024)Optimism and Pessimism in Database Query Optimisation2024 IEEE 18th International Conference on Semantic Computing (ICSC)10.1109/ICSC59802.2024.00049(269-276)Online publication date: 5-Feb-2024
https://doi.org/10.1109/ICSC59802.2024.00049
Volnes EPlagemann TGoebel V(2024)To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream ProcessingIEEE Communications Surveys & Tutorials10.1109/COMST.2023.333095326:1(670-705)Online publication date: Sep-2025
https://doi.org/10.1109/COMST.2023.3330953
Ntumba PGeorgantas NChristophides V(2024)Adaptive Scheduling of Continuous Operators for IoT Edge AnalyticsFuture Generation Computer Systems10.1016/j.future.2024.04.029158(277-293)Online publication date: Sep-2024
https://doi.org/10.1016/j.future.2024.04.029
Tang BHan HYang QXu W(2024)Operator placement for data stream processing based on publisher/subscriber in hybrid cloud-fog-edge infrastructureCluster Computing10.1007/s10586-023-04065-z27:3(2741-2759)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10586-023-04065-z
Rong KBudiu MSkiadopoulos ASuresh LTai A(2023)Scaling a Declarative Cluster Manager Architecture with Query Optimization TechniquesProceedings of the VLDB Endowment10.14778/3603581.360359916:10(2618-2631)Online publication date: 8-Aug-2023
https://dl.acm.org/doi/10.14778/3603581.3603599
Akili SPurtzel SWeidlich M(2023)INEv: In-Network Evaluation for Event Stream ProcessingProceedings of the ACM on Management of Data10.1145/35889551:1(1-26)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1145/3588955
Kiehn FSchmidt MGlake DPanse FWingerath WWollmer BPoppinga MRitter N(2022)Polyglot data managementProceedings of the VLDB Endowment10.14778/3554821.355489115:12(3750-3753)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.14778/3554821.3554891
Sunkel MElmamooz GGrawunder MHoang PRauch ESchmeling LThurner SNicklas D(2022)Resource-Aware Classification via Model Management Enabled Data Stream Optimization2022 IEEE International Conference on Pervasive Computing and Communications (PerCom)10.1109/PerCom53586.2022.9762391(23-33)Online publication date: 21-Mar-2022
https://doi.org/10.1109/PerCom53586.2022.9762391
Akili SWeidlich MLi GLi ZIdreos SSrivastava D(2021)MuSE Graphs for Flexible Distribution of Event Stream Processing in NetworksProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3457318(10-22)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3457318
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten