skip to main content
research-article
Open access

INEv: In-Network Evaluation for Event Stream Processing

Published: 30 May 2023 Publication History

Abstract

Complex event processing (CEP) detects situations of interest by evaluating queries over event streams. Once CEP is used in networked applications, the distribution of query evaluation among the event sources enables performance optimization. Instead of collecting all events at one location for query evaluation, sub-queries are placed at network nodes to reduce the data transmission overhead. Yet, existing techniques either place such sub-queries at exactly one node in the network, which neglects the benefits of truly distributed evaluation, or are agnostic to the network structure, which ignores transmission costs due to the absence of direct network links.
To overcome the above limitations, we propose INEV graphs for in-network evaluation of CEP queries with rich semantics, including Kleene closure and negation. Our idea is to introduce fine-granular routing of partial results of sub-queries as an additional degree of freedom in query evaluation: We exploit events already disseminated in the network as part of one sub-query, when evaluating another one. We show how to instantiate INEv graphs by splitting a query workload into sub-queries, placing them at network nodes, and forwarding of their results to other nodes. Also, we characterize INEv graphs that guarantee correct and complete query evaluation, and discuss their construction based on a cost model that unifies transmission and processing latency. Our experimental results indicate that INEv graphs can reduce transmission costs for distributed CEP by up to eight orders of magnitude compared to baseline strategies.

Supplemental Material

MP4 File
Presentation video for SIGMOD 2023
GZ File
Source Code
PDF File
Read me

References

[1]
Jagrati Agrawal, Yanlei Diao, Daniel Gyllstrom, and Neil Immerman. 2008. Efficient pattern matching over event streams. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10--12, 2008, Jason Tsong-Li Wang (Ed.). ACM, 147--160. https://doi.org/10.1145/1376616.1376634
[2]
Yanif Ahmad and Ugur Cetintemel. 2004. Network-aware query processing for stream-based applications. In Proceedings of the International Conference on Very Large Data Bases. 456--467.
[3]
Mert Akdere, U?ur cC etintemel, and Nesime Tatbul. 2008. Plan-based complex event detection across distributed sources. Proceedings of the VLDB Endowment, Vol. 1, 1 (2008), 66--77.
[4]
Akili et al. 2022. INEv: In-Network Evaluation for Event Stream Processing -- Extended Version. https://github.com/samieze/INEv/blob/main/INEv_TR.pdf.
[5]
Samira Akili and Matthias Weidlich. 2021a. MuSE Graphs for Flexible Distribution of Event Stream Processing in Networks. In SIGMOD '21: International Conference on Management of Data, Virtual Event, China, June 20--25, 2021, Guoliang Li, Zhanhuai Li, Stratos Idreos, and Divesh Srivastava (Eds.). ACM, 10--22. https://doi.org/10.1145/3448016.3457318
[6]
Samira Akili and Matthias Weidlich. 2021b. Reasoning on the Efficiency of Distributed Complex Event Processing. Fundam. Informaticae, Vol. 179, 2 (2021), 113--134. https://doi.org/10.3233/FI-2021--2017
[7]
Peter M. G. Apers, Alan R. Hevner, and S. Bing Yao. 1983. Optimization algorithms for distributed queries. IEEE transactions on software engineering 1 (1983), 57--68.
[8]
Arvind Arasu, Brian Babcock, Shivnath Babu, John Cieslewicz, Mayur Datar, Keith Ito, Rajeev Motwani, Utkarsh Srivastava, and Jennifer Widom. 2016. STREAM: The Stanford Data Stream Management System. In Data Stream Management - Processing High-Speed Data Streams, Minos N. Garofalakis, Johannes Gehrke, and Rajeev Rastogi (Eds.). Springer, 317--336. https://doi.org/10.1007/978--3--540--28608-0_16
[9]
Alexander Artikis, Alessandro Margara, Mart'i n Ugarte, Stijn Vansummeren, and Matthias Weidlich. 2017. Complex Event Recognition Languages: Tutorial. In Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems, DEBS 2017, Barcelona, Spain, June 19--23, 2017. ACM, 7--10. https://doi.org/10.1145/3093742.3095106
[10]
Alexander Artikis, Matthias Weidlich, Francc ois Schnitzler, Ioannis Boutsis, Thomas Liebig, Nico Piatkowski, Christian Bockermann, Katharina Morik, Vana Kalogeraki, Jakub Marecek, Avigdor Gal, Shie Mannor, Dimitrios Gunopulos, and Dermot Kinane. 2014. Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management. In Proceedings of the 17th International Conference on Extending Database Technology, EDBT 2014, Athens, Greece, March 24--28, 2014, Sihem Amer-Yahia, Vassilis Christophides, Anastasios Kementsietsidis, Minos N. Garofalakis, Stratos Idreos, and Vincent Leroy (Eds.). OpenProceedings.org, 712--723. https://doi.org/10.5441/002/edbt.2014.77
[11]
Cagri Balkesen, Nihal Dindar, Matthias Wetter, and Nesime Tatbul. 2013. RIP: run-based intra-query parallelism for scalable complex event processing. In The 7th ACM International Conference on Distributed Event-Based Systems, DEBS '13, Arlington, TX, USA - June 29 - July 03, 2013, Sharma Chakravarthy, Susan Darling Urban, Peter R. Pietzuch, and Elke A. Rundensteiner (Eds.). ACM, 3--14. https://doi.org/10.1145/2488222.2488257
[12]
Shahid H. Bokhari. 1981. A Shortest Tree Algorithm for Optimal Assignments Across Space and Time in a Distributed Processor System. IEEE Trans. Software Eng., Vol. 7, 6 (1981), 583--589. https://doi.org/10.1109/TSE.1981.226469
[13]
Lei Cao, Jiayuan Wang, and Elke A. Rundensteiner. 2016. Sharing-Aware Outlier Analytics over High-Volume Data Streams. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016, San Francisco, CA, USA, June 26 - July 01, 2016, Fatma Ö zcan, Georgia Koutrika, and Sam Madden (Eds.). ACM, 527--540. https://doi.org/10.1145/2882903.2882920
[14]
Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache Flink#8482;: Stream and Batch Processing in a Single Engine. IEEE Data Eng. Bull., Vol. 38, 4 (2015), 28--38. http://sites.computer.org/debull/A15dec/p28.pdf
[15]
Sharma Chakravarthy and D. Mishra. 1994. Snoop: An Expressive Event Specification Language for Active Databases. Data Knowl. Eng., Vol. 14, 1 (1994), 1--26. https://doi.org/10.1016/0169-023X(94)90006-X
[16]
Koral Chapnik, Ilya Kolchinsky, and Assaf Schuster. 2021. DARLING: Data-Aware Load Shedding in Complex Event Processing Systems. Proc. VLDB Endow., Vol. 15, 3 (2021), 541--554. http://www.vldb.org/pvldb/vol15/p541-chapnik.pdf
[17]
Georgios Chatzimilioudis, Alfredo Cuzzocrea, Dimitrios Gunopulos, and Nikos Mamoulis. 2013. A novel distributed framework for optimizing query routing trees in wireless sensor networks via optimal operator placement. J. Comput. System Sci., Vol. 79, 3 (2013), 349--368.
[18]
Jianxia Chen, Lakshmish Ramaswamy, David K Lowenthal, and Shivkumar Kalyanaraman. 2012. Comet: Decentralized complex event detection in mobile delay tolerant networks. In 2012 IEEE 13th International Conference on Mobile Data Management. IEEE, 131--136.
[19]
Ming-Syan Chen and Philip S Yu. 1992. Interleaving a join sequence with semijoins in distributed query processing. IEEE Transactions on Parallel and Distributed Systems, Vol. 3, 5 (1992), 611--621.
[20]
M-S Chen and Philip S. Yu. 1993. Combining joint and semi-join operations for distributed query processing. IEEE Transactions on Knowledge and Data Engineering, Vol. 5, 3 (1993), 534--542.
[21]
citi Bike. 2022. http://www.citibikenyc.com/system-data. .
[22]
Gianpaolo Cugola and Alessandro Margara. 2013. Deployment strategies for distributed complex event processing. Computing, Vol. 95, 2 (2013), 129--156.
[23]
Raul Castro Fernandez, Matthias Weidlich, Peter R. Pietzuch, and Avigdor Gal. 2014. Scalable stateful stream processing for smart grids. In The 8th ACM International Conference on Distributed Event-Based Systems, DEBS '14, Mumbai, India, May 26--29, 2014, Umesh Bellur and Ravi Kothari (Eds.). ACM, 276--281. https://doi.org/10.1145/2611286.2611326
[24]
Ioannis Flouris, Nikos Giatrakos, Antonios Deligiannakis, and Minos N. Garofalakis. 2020. Network-wide complex event processing over geographically distributed data sources. Inf. Syst., Vol. 88 (2020). https://doi.org/10.1016/j.is.2019.101442
[25]
M. R. Garey and David S. Johnson. 1977. The Rectilinear Steiner Tree Problem in NP Complete. SIAM Journal of Applied Mathematics, Vol. 32 (1977), 826--834.
[26]
Nikos Giatrakos, Elias Alevizos, Alexander Artikis, Antonios Deligiannakis, and Minos N. Garofalakis. 2020. Complex event recognition in the Big Data era: a survey. VLDB J., Vol. 29, 1 (2020), 313--352. https://doi.org/10.1007/s00778-019-00557-w
[27]
Mohit Jain, Vikas Chandan, Marilena Minou, George A. Thanos, Tri Kurniawan Wijaya, Achim Lindt, and Arne Gylling. 2015. Methodologies for effective demand response messaging. In 2015 IEEE International Conference on Smart Grid Communications, SmartGridComm 2015, Miami, FL, USA, November 2--5, 2015. IEEE, 453--458. https://doi.org/10.1109/SmartGridComm.2015.7436342
[28]
Ilya Kolchinsky and Assaf Schuster. 2019. Real-Time Multi-Pattern Detection over Event Streams. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD Conference 2019, Amsterdam, The Netherlands, June 30 - July 5, 2019, Peter A. Boncz, Stefan Manegold, Anastasia Ailamaki, Amol Deshpande, and Tim Kraska (Eds.). ACM, 589--606. https://doi.org/10.1145/3299869.3319869
[29]
Lawrence T. Kou, George Markowsky, and Leonard Berman. 1981. A Fast Algorithm for Steiner Trees. Acta Informatica, Vol. 15 (1981), 141--145. https://doi.org/10.1007/BF00288961
[30]
Qian Lin, Beng Chin Ooi, Zhengkui Wang, and Cui Yu. 2015. Scalable distributed stream join processing. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. 811--825.
[31]
Manisha Luthra, Boris Koldehofe, Niels Danger, Pascal Weisenburger, Guido Salvaneschi, and Ioannis Stavrakakis. 2021. TCEP: Transitions in operator placement to adapt to dynamic network environments. J. Comput. Syst. Sci., Vol. 122 (2021), 94--125. https://doi.org/10.1016/j.jcss.2021.05.003
[32]
Yuan Mei and Samuel Madden. 2009. ZStream: a cost-based query processor for adaptively detecting composite events. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2009, Providence, Rhode Island, USA, June 29 - July 2, 2009, Ugur cC etintemel, Stanley B. Zdonik, Donald Kossmann, and Nesime Tatbul (Eds.). ACM, 193--206. https://doi.org/10.1145/1559845.1559867
[33]
Matteo Nardelli, Valeria Cardellini, Vincenzo Grassi, and Francesco LO PRESTI. 2019. Efficient Operator Placement for Distributed Data Stream Processing Applications. IEEE Transactions on Parallel and Distributed Systems (2019).
[34]
M Tamer Özsu and Patrick Valduriez. 1996. Distributed and parallel database systems. ACM Computing Surveys (CSUR), Vol. 28, 1 (1996), 125--128.
[35]
Peter R. Pietzuch, Jonathan Ledlie, Jeffrey Shneidman, Mema Roussopoulos, Matt Welsh, and Margo I. Seltzer. 2006. Network-Aware Operator Placement for Stream-Processing Systems. In Proceedings of the 22nd International Conference on Data Engineering, ICDE 2006, 3--8 April 2006, Atlanta, GA, USA, Ling Liu, Andreas Reuter, Kyu-Young Whang, and Jianjun Zhang (Eds.). IEEE Computer Society, 49. https://doi.org/10.1109/ICDE.2006.105
[36]
Orestis Polychroniou, Rajkumar Sen, and Kenneth A Ross. 2014. Track join: distributed joins with minimal network traffic. In Proceedings of the 2014 ACM SIGMOD international conference on Management of data. 1483--1494.
[37]
Olga Poppe, Chuan Lei, Lei Ma, Allison Rozet, and Elke A. Rundensteiner. 2021. To Share, or not to Share Online Event Trend Aggregation Over Bursty Event Streams. In SIGMOD '21: International Conference on Management of Data, Virtual Event, China, June 20--25, 2021, Guoliang Li, Zhanhuai Li, Stratos Idreos, and Divesh Srivastava (Eds.). ACM, 1452--1464. https://doi.org/10.1145/3448016.3452785
[38]
Gururaghav Raman, Jimmy Chih-Hsien Peng, Bo Zhao, and Matthias Weidlich. 2019. Dynamic Decision Making for Demand Response through Adaptive Event Stream Monitoring. In 2019 IEEE Power & Energy Society General Meeting (PESGM). IEEE, 1--5.
[39]
Medhabi Ray, Chuan Lei, and Elke A. Rundensteiner. 2016. Scalable Pattern Sharing on Event Streams. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016, San Francisco, CA, USA, June 26 - July 01, 2016, Fatma Ö zcan, Georgia Koutrika, and Sam Madden (Eds.). ACM, 495--510. https://doi.org/10.1145/2882903.2882947
[40]
Wolf Rödiger, Sam Idicula, Alfons Kemper, and Thomas Neumann. 2016. Flow-join: Adaptive skew handling for distributed joins over high-speed networks. In 2016 IEEE 32nd International Conference on Data Engineering (ICDE). IEEE, 1194--1205.
[41]
Lukas Rupprecht, William Culhane, and Peter Pietzuch. 2017. SquirrelJoin: Network-aware distributed join processing with lazy partitioning. Proceedings of the VLDB Endowment, Vol. 10, 11 (2017), 1250--1261.
[42]
Nicholas Poul Schultz-Møller, Matteo Migliavacca, and Peter R. Pietzuch. 2009. Distributed complex event processing with query rewriting. In Proceedings of the Third ACM International Conference on Distributed Event-Based Systems, DEBS 2009, Nashville, Tennessee, USA, July 6--9, 2009, Aniruddha S. Gokhale and Douglas C. Schmidt (Eds.). ACM. https://doi.org/10.1145/1619258.1619264
[43]
Utkarsh Srivastava, Kamesh Munagala, and Jennifer Widom. 2005. Operator placement for in-network stream query processing. In Proceedings of the twenty-fourth ACM SIGMOD-SIGACT-SIGART sym posium on Principles of database systems. ACM, 250--258.
[44]
Fabrice Starks, Vera Goebel, Stein Kristiansen, and Thomas Plagemann. 2018. Mobile Distributed Complex Event Processing - Ubi Sumus? Quo Vadimus? In Mobile Big Data, A Roadmap from Models to Technologies, Georgios Skourletopoulos, George Mastorakis, Constandinos X. Mavromoustakis, Ciprian Dobre, and Evangelos Pallis (Eds.). Lecture Notes on Data Engineering and Communications Technologies, Vol. 10. Springer, 147--180. https://doi.org/10.1007/978--3--319--67925--9_7
[45]
Fabrice Starks and Thomas Peter Plagemann. 2015. Operator placement for efficient distributed complex event processing in MANETs. In 11th IEEE International Conference on Wireless and Mobile Computing, Networking and Communications, WiMob 2015, Abu Dhabi, United Arab Emirates, October 19--21, 2015. IEEE Computer Society, 83--90. https://doi.org/10.1109/WiMOB.2015.7347944
[46]
Kia Teymourian, Malte Rohde, and Adrian Paschke. 2012. Knowledge-based processing of complex stock market events. In 15th International Conference on Extending Database Technology, EDBT '12, Berlin, Germany, March 27--30, 2012, Proceedings, Elke A. Rundensteiner, Volker Markl, Ioana Manolescu, Sihem Amer-Yahia, Felix Naumann, and Ismail Ari (Eds.). ACM, 594--597. https://doi.org/10.1145/2247596.2247674
[47]
David Waitzman, Craig Partridge, and Stephen E Deering. 1988. Distance vector multicast routing protocol. Technical Report.
[48]
John Wilkes. 2020. Yet more Google compute cluster trace data. Google research blog. Posted at https://ai.googleblog.com/2020/04/yet-more-google-compute-cluster-trace.html. .
[49]
Bo Zhao, Nguyen Quoc Viet Hung, and Matthias Weidlich. 2020. Load Shedding for Complex Event Processing: Input-based and State-based Techniques. In 36th IEEE International Conference on Data Engineering, ICDE 2020, Dallas, TX, USA, April 20--24, 2020. IEEE, 1093--1104. https://doi.org/10.1109/ICDE48307.2020.00099
[50]
Bo Zhao, Han van der Aa, Thanh Tam Nguyen, Quoc Viet Hung Nguyen, and Matthias Weidlich. 2021. EIRES: Efficient Integration of Remote Data in Event Stream Processing. In SIGMOD '21: International Conference on Management of Data, Virtual Event, China, June 20--25, 2021, Guoliang Li, Zhanhuai Li, Stratos Idreos, and Divesh Srivastava (Eds.). ACM, 2128--2141. https://doi.org/10.1145/3448016.3457304
[51]
Yongluan Zhou, Ying Yan, Feng Yu, and Aoying Zhou. 2006. Pmjoin: Optimizing distributed multi-way stream joins by stream partitioning. In International Conference on Database Systems for Advanced Applications. Springer, 325--341.

Cited By

View all
  • (2024)On-Demand Pattern Aggregation in Event NetworksProceedings of the International Workshop on Big Data in Emergent Distributed Environments10.1145/3663741.3664781(1-6)Online publication date: 9-Jun-2024
  • (2024)DecoPa: Query Decomposition for Parallel Complex Event ProcessingProceedings of the ACM on Management of Data10.1145/36549352:3(1-26)Online publication date: 30-May-2024
  • (2024)ACER: Accelerating Complex Event Recognition via Two-Phase Filtering under Range Bitmap-Based IndexesProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671814(1933-1943)Online publication date: 25-Aug-2024
  • Show More Cited By

Index Terms

  1. INEv: In-Network Evaluation for Event Stream Processing

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Management of Data
    Proceedings of the ACM on Management of Data  Volume 1, Issue 1
    PACMMOD
    May 2023
    2807 pages
    EISSN:2836-6573
    DOI:10.1145/3603164
    Issue’s Table of Contents
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 30 May 2023
    Published in PACMMOD Volume 1, Issue 1

    Badges

    Author Tags

    1. event stream processing
    2. in-network evaluation
    3. operator placement

    Qualifiers

    • Research-article

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)245
    • Downloads (Last 6 weeks)54
    Reflects downloads up to 25 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)On-Demand Pattern Aggregation in Event NetworksProceedings of the International Workshop on Big Data in Emergent Distributed Environments10.1145/3663741.3664781(1-6)Online publication date: 9-Jun-2024
    • (2024)DecoPa: Query Decomposition for Parallel Complex Event ProcessingProceedings of the ACM on Management of Data10.1145/36549352:3(1-26)Online publication date: 30-May-2024
    • (2024)ACER: Accelerating Complex Event Recognition via Two-Phase Filtering under Range Bitmap-Based IndexesProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671814(1933-1943)Online publication date: 25-Aug-2024
    • (2024)Local Self-Adaptation for Distributed Complex Event ProcessingProceedings of the 18th ACM International Conference on Distributed and Event-based Systems10.1145/3629104.3672432(189-190)Online publication date: 24-Jun-2024
    • (2024)Stream Data Model and ArchitectureData Analytics and Machine Learning10.1007/978-981-97-0448-4_5(81-104)Online publication date: 20-Mar-2024

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Full Access

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media