Samira Akili
ABSTRACT
Complex event processing (CEP) enables reactive and predictive applications through the continuous evaluation of queries over streams of event data. In a network of event sources, efficient query evaluation is achieved through distribution: Queries are split into operators (query decomposition), which are then assigned to some of the nodes (operator placement). Yet, existing solutions limit the decomposition to the operator hierarchy of a query, ignoring possible rewritings of it, and place each operator at exactly one node in the network. That neglects optimizations based on pattern composition through multiple queries as results are always gathered at a single sink node.
In this paper, we propose a new evaluation model for CEP, coined Multi-Sink Evaluation (MuSE) graphs. It incorporates arbitrary projections of queries for distribution and assigns them to potentially many nodes. We prove correctness of query evaluation with MuSE graphs and provide a cost model to assess its efficiency. Since the construction of cost-optimal MuSE graphs is intractable, we present an approximation algorithm and several pruning trategies. Our evaluation results show that MuSE graphs reduce network transmission costs by up to three orders of magnitude over baseline strategies.
Supplemental Material
- Jagrati Agrawal, Yanlei Diao, Daniel Gyllstrom, and Neil Immerman. 2008. Efficient pattern matching over event streams. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10--12, 2008, Jason Tsong-Li Wang (Ed.). ACM, 147--160. https://doi.org/10.1145/1376616.1376634Google Scholar
Digital Library
- Mert Akdere and Nesime Tatbul. 2008. Plan-based complex event detection across distributed sources. Proceedings of the VLDB Endowment, Vol. 1, 1 (2008), 66--77.Google ScholarDigital Library
- Arvind Arasu, Brian Babcock, Shivnath Babu, John Cieslewicz, Mayur Datar, Keith Ito, Rajeev Motwani, Utkarsh Srivastava, and Jennifer Widom. 2016. STREAM: The Stanford Data Stream Management System. In Data Stream Management - Processing High-Speed Data Streams, Minos N. Garofalakis, Johannes Gehrke, and Rajeev Rastogi (Eds.). Springer, 317--336. https://doi.org/10.1007/978--3--540--28608-0_16Google Scholar
- Alexander Artikis, Alessandro Margara, Martin Ugarte, Stijn Vansummeren, and Matthias Weidlich. 2017. Complex Event Recognition Languages: Tutorial. In Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems, DEBS 2017, Barcelona, Spain, June 19--23, 2017. ACM, 7--10. https://doi.org/10.1145/3093742.3095106Google ScholarDigital Library
- Alexander Artikis, Matthias Weidlich, Francc ois Schnitzler, Ioannis Boutsis, Thomas Liebig, Nico Piatkowski, Christian Bockermann, Katharina Morik, Vana Kalogeraki, Jakub Marecek, Avigdor Gal, Shie Mannor, Dimitrios Gunopulos, and Dermot Kinane. 2014. Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management. In Proceedings of the 17th International Conference on Extending Database Technology, EDBT 2014, Athens, Greece, March 24--28, 2014, Sihem Amer-Yahia, Vassilis Christophides, Anastasios Kementsietsidis, Minos N. Garofalakis, Stratos Idreos, and Vincent Leroy (Eds.). OpenProceedings.org, 712--723. https://doi.org/10.5441/002/edbt.2014.77Google Scholar
- Roger S. Barga, Jonathan Goldstein, Mohamed H. Ali, and Mingsheng Hong. 2007. Consistent Streaming Through Time: A Vision for Event Stream Processing. In CIDR 2007, Third Biennial Conference on Innovative Data Systems Research, Asilomar, CA, USA, January 7--10, 2007, Online Proceedings. www.cidrdb.org, 363--374. http://cidrdb.org/cidr2007/papers/cidr07p42.pdfGoogle Scholar
- Shahid H. Bokhari. 1981. A Shortest Tree Algorithm for Optimal Assignments Across Space and Time in a Distributed Processor System. IEEE Trans. Software Eng., Vol. 7, 6 (1981), 583--589. https://doi.org/10.1109/TSE.1981.226469Google ScholarDigital Library
- Georgios Chatzimilioudis, Alfredo Cuzzocrea, Dimitrios Gunopulos, and Nikos Mamoulis. 2013. A novel distributed framework for optimizing query routing trees in wireless sensor networks via optimal operator placement. J. Comput. System Sci., Vol. 79, 3 (2013), 349--368.Google ScholarDigital Library
- Jianxia Chen, Lakshmish Ramaswamy, David K. Lowenthal, and Shivkumar Kalyanaraman. 2012a. Comet: Decentralized Complex Event Detection in Mobile Delay Tolerant Networks. In 13th IEEE International Conference on Mobile Data Management, MDM 2012, Bengaluru, India, July 23--26, 2012, Karl Aberer, Anupam Joshi, Sougata Mukherjea, Dipanjan Chakraborty, Hua Lu, Nalini Venkatasubramanian, and Salil S. Kanhere (Eds.). IEEE Computer Society, 131--136. https://doi.org/10.1109/MDM.2012.18Google ScholarDigital Library
- Jianxia Chen, Lakshmish Ramaswamy, David K Lowenthal, and Shivkumar Kalyanaraman. 2012b. Comet: Decentralized complex event detection in mobile delay tolerant networks. In 2012 IEEE 13th International Conference on Mobile Data Management. IEEE, 131--136.Google ScholarDigital Library
- Gianpaolo Cugola and Alessandro Margara. 2013. Deployment strategies for distributed complex event processing. Computing, Vol. 95, 2 (2013), 129--156.Google ScholarCross Ref
- Raul Castro Fernandez, Matthias Weidlich, Peter R. Pietzuch, and Avigdor Gal. 2014. Scalable stateful stream processing for smart grids. In The 8th ACM International Conference on Distributed Event-Based Systems, DEBS '14, Mumbai, India, May 26--29, 2014, Umesh Bellur and Ravi Kothari (Eds.). ACM, 276--281. https://doi.org/10.1145/2611286.2611326Google ScholarDigital Library
- Ioannis Flouris, Nikos Giatrakos, Antonios Deligiannakis, and Minos N. Garofalakis. 2020. Network-wide complex event processing over geographically distributed data sources. Inf. Syst., Vol. 88 (2020). https://doi.org/10.1016/j.is.2019.101442Google ScholarDigital Library
- Nikos Giatrakos, Elias Alevizos, Alexander Artikis, Antonios Deligiannakis, and Minos N. Garofalakis. 2020. Complex event recognition in the Big Data era: a survey. VLDB J., Vol. 29, 1 (2020), 313--352. https://doi.org/10.1007/s00778-019-00557-wGoogle ScholarDigital Library
- Jonathan Goldstein, Ahmed S. Abdelhamid, Mike Barnett, Sebastian Burckhardt, Badrish Chandramouli, Darren Gehring, Niel Lebeck, Christopher Meiklejohn, Umar Farooq Minhas, Ryan Newton, Rahee Peshawaria, Tal Zaccai, and Irene Zhang. 2020. A.M.B.R.O.S.I.A: Providing Performant Virtual Resiliency for Distributed Applications. Proc. VLDB Endow., Vol. 13, 5 (2020), 588--601. http://www.vldb.org/pvldb/vol13/p588-goldstein.pdfGoogle ScholarDigital Library
- InSystems. 2021. proANT Transport Robots. http://www.insystems.de/en/produkte/proant-transport-roboter/.Google Scholar
- Matteo Nardelli, Valeria Cardellini, Vincenzo Grassi, and Francesco LO PRESTI. 2019. Efficient Operator Placement for Distributed Data Stream Processing Applications. IEEE Transactions on Parallel and Distributed Systems (2019).Google ScholarCross Ref
- Peter R. Pietzuch, Jonathan Ledlie, Jeffrey Shneidman, Mema Roussopoulos, Matt Welsh, and Margo I. Seltzer. 2006. Network-Aware Operator Placement for Stream-Processing Systems. In Proceedings of the 22nd International Conference on Data Engineering, ICDE 2006, 3--8 April 2006, Atlanta, GA, USA, Ling Liu, Andreas Reuter, Kyu-Young Whang, and Jianjun Zhang (Eds.). IEEE Computer Society, 49. https://doi.org/10.1109/ICDE.2006.105Google ScholarDigital Library
- Medhabi Ray, Chuan Lei, and Elke A. Rundensteiner. 2016. Scalable Pattern Sharing on Event Streams. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016, San Francisco, CA, USA, June 26 - July 01, 2016, Fatma Ö zcan, Georgia Koutrika, and Sam Madden (Eds.). ACM, 495--510. https://doi.org/10.1145/2882903.2882947Google ScholarDigital Library
- Samira Akili and Matthias Weidlich. 2021. MuSE Graphs for Flexible Distribution of Event Stream Processing in Networks -- Technical Report. https://github.com/samieze/aMuSE .Google Scholar
- Nicholas Poul Schultz-Møller, Matteo Migliavacca, and Peter R. Pietzuch. 2009. Distributed complex event processing with query rewriting. In Proceedings of the Third ACM International Conference on Distributed Event-Based Systems, DEBS 2009, Nashville, Tennessee, USA, July 6--9, 2009, Aniruddha S. Gokhale and Douglas C. Schmidt (Eds.). ACM. https://doi.org/10.1145/1619258.1619264Google ScholarDigital Library
- Utkarsh Srivastava, Kamesh Munagala, and Jennifer Widom. 2005. Operator placement for in-network stream query processing. In Proceedings of the twenty-fourth ACM SIGMOD-SIGACT-SIGART sym posium on Principles of database systems. ACM, 250--258.Google ScholarDigital Library
- Fabrice Starks, Vera Goebel, Stein Kristiansen, and Thomas Plagemann. 2018. Mobile Distributed Complex Event Processing - Ubi Sumus? Quo Vadimus? In Mobile Big Data, A Roadmap from Models to Technologies, Georgios Skourletopoulos, George Mastorakis, Constandinos X. Mavromoustakis, Ciprian Dobre, and Evangelos Pallis (Eds.). Lecture Notes on Data Engineering and Communications Technologies, Vol. 10. Springer, 147--180. https://doi.org/10.1007/978--3--319--67925--9_7Google Scholar
- Kia Teymourian, Malte Rohde, and Adrian Paschke. 2012. Knowledge-based processing of complex stock market events. In 15th International Conference on Extending Database Technology, EDBT '12, Berlin, Germany, March 27--30, 2012, Proceedings, Elke A. Rundensteiner, Volker Markl, Ioana Manolescu, Sihem Amer-Yahia, Felix Naumann, and Ismail Ari (Eds.). ACM, 594--597. https://doi.org/10.1145/2247596.2247674Google ScholarDigital Library
- John Wilkes. 2020. Yet more Google compute cluster trace data. Google research blog. Posted at https://ai.googleblog.com/2020/04/yet-more-google-compute-cluster-trace.html.Google Scholar
- Lei Ying, Zhen Liu, Don Towsley, and Cathy H Xia. 2008. Distributed operator placement and data caching in large-scale sensor networks. In IEEE INFOCOM 2008-The 27th Conference on Computer Communications. IEEE, 977--985.Google ScholarCross Ref
- Haopeng Zhang, Yanlei Diao, and Neil Immerman. 2014. On complexity and optimization of expensive queries in complex event processing. In International Conference on Management of Data, SIGMOD 2014, Snowbird, UT, USA, June 22--27, 2014, Curtis E. Dyreson, Feifei Li, and M. TamerÖzsu (Eds.). ACM, 217--228. https://doi.org/10.1145/2588555.2593671Google ScholarDigital Library
Index Terms
- MuSE Graphs for Flexible Distribution of Event Stream Processing in Networks
Recommendations
EIRES: Efficient Integration of Remote Data in Event Stream Processing
SIGMOD '21: Proceedings of the 2021 International Conference on Management of DataTo support reactive and predictive applications, complex event processing (CEP) systems detect patterns in event streams based on predefined queries. To determine the events that constitute a query match, their payload data may need to be assessed ...
Reasoning on the Efficiency of Distributed Complex Event Processing
Concurrency, Specification, and Programming: Special Issue of Selected Papers of CS&P 2018Complex event processing (CEP) evaluates queries over streams of event data to detect situations of interest. If the event data are produced by geographically distributed sources, CEP may exploit in-network processing that distributes the evaluation of a ...
INEv: In-Network Evaluation for Event Stream Processing
PACMMODComplex event processing (CEP) detects situations of interest by evaluating queries over event streams. Once CEP is used in networked applications, the distribution of query evaluation among the event sources enables performance optimization. Instead of ...
Comments