skip to main content
10.1145/3459637.3482094acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
short-paper

EasyFlinkCEP: Big Event Data Analytics for Everyone

Published: 30 October 2021 Publication History

Abstract

FlinkCEP is the Complex Event Processing (CEP) API of the Flink Big Data platform. The high expressive power of the language of FlinkCEP comes at the cost of cumbersome parameterization of the queried patterns, acting as a barrier for FlinkCEP's adoption. Moreover, properly configuring a FlinkCEP program to run over a computer cluster requires advanced skills on modern hardware administration which non-expert programmers do not possess. In this work (i) we build a novel, logical CEP operator that receives CEP pattern queries in the form of extended regular expressions and seamlessly re-writes them to FlinkCEP programs, (ii) we build a CEP Optimizer that automatically decides good job configurations for these FlinkCEP programs. We also present an experimental evaluation which demonstrates the significant benefits of our approach.

Supplementary Material

MP4 File (CIKM21-rgsp2638.mp4)
We present EasyFlinkCEP, a prototype CEP component operating on top of a state-of-the-art parallel CEP engine, namely FlinkCEP. EasyFlinkCEP offers (i) the EasyFlinkCEP Operator which abstracts the details of coding directly on FlinkCEP, allowing non-programmer event analysts to focus exclusively on rapidly defining the business rules they need to monitor, and (ii) the EasyFlinkCEP Optimizer to optimize FlinkCEP jobs executed in a Flink cluster of choice. Thus, business event analysts can exploit the processing capacity of modern hardware without needing any cluster administration background knowledge.

References

[1]
[n.d.]. Esper. http://esper.espertech.com/release-5.3.0/esper-reference/html/eploperator.html#epl-operator-ref-keyword-regexp. [Online; accessed 06-June2021].
[2]
[n.d.]. Oracle CEP EPL Language Reference. https://docs.oracle.com/cd/E12839_ 01/apirefs.1111/e14304/operators.htm#EPLLR206. [Online; accessed 06-June2021].
[3]
Jagrati Agrawal, Yanlei Diao, Daniel Gyllstrom, and Neil Immerman. 2008. Efficient pattern matching over event streams. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10--12, 2008, Jason Tsong-Li Wang (Ed.). ACM, 147--160. https://doi.org/10.1145/1376616.1376634
[4]
Manolis Pitsikalis, Alexander Artikis, Richard Dreo, Cyril Ray, Elena Camossi, and Anne-Laure Jousselme. 2019. Composite Event Recognition for Maritime Monitoring. In Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems, DEBS 2019, Darmstadt, Germany, June 24--28, 2019. ACM, 163--174. https://doi.org/10.1145/3328905.3329762
[5]
Alexandros Troupiotis-Kapeliaris, Konstantinos Chatzikokolakis, Dimitris Zissis, and Elias Alevizos. 2020. Experimental Comparison of Complex Event Processing Systems in the Maritime Domain. In 21st IEEE International Conference on Mobile Data Management, MDM 2020, Versailles, France, June 30 - July 3, 2020. IEEE, 293--298. https://doi.org/10.1109/MDM48529.2020.00066
[6]
Elias Alevizos, Alexander Artikis, and Georgios Paliouras. 2018. Wayeb: a Tool for Complex Event Forecasting. In LPAR-22. 22nd International Conference on Logic for Programming, Artificial Intelligence and Reasoning, Awassa, Ethiopia, 16--21 November 2018 (EPiC Series in Computing, Vol. 57), Gilles Barthe, Geoff Sutcliffe, and Margus Veanes (Eds.). EasyChair, 26--35. https://doi.org/10.29007/2s9t
[7]
Alexander Artikis, Alessandro Margara, Martín Ugarte, Stijn Vansummeren, and Matthias Weidlich. 2017. Complex Event Recognition Languages: Tutorial. In Proceedings of the 11th ACM International Conference on Distributed and Eventbased Systems, DEBS 2017, Barcelona, Spain, June 19--23, 2017. ACM, 7--10. https://doi.org/10.1145/3093742.3095106
[8]
Antonios Deligiannakis, Nikos Giatrakos, Yannis Kotidis, Vasilis Samoladas, and Alkis Simitsis. 2021. Extreme-Scale Interactive Cross-Platform Streaming Analytics -- The INFORE Approach. In Proceedings of the 2nd Workshop on Search, Exploration, and Analysis in Heterogeneous Datastores, SEA Data 2021, Copenhagen, DenMark, August 20, 2021. http://ceur-ws.org/, 7--13. http://ceur-ws.org/Vol2929/paper2.pdf
[9]
Flink. [n.d.]. https://flink.apache.org/. [Online; accessed 06-June-2021].
[10]
FlinkCEP. [n.d.]. https://ci.apache.org/projects/flink/flink-docs-release-1.13/dev/libs/cep.html. [Online; accessed 06-June-2021].
[11]
Ioannis Flouris, Nikos Giatrakos, Minos N. Garofalakis, and Antonios Deligiannakis. 2015. Issues in Complex Event Processing Systems. In 2015 IEEE TrustCom/BigDataSE/ISPA, Helsinki, Finland, August 20--22, 2015, Volume 2. IEEE, 241-- 246. https://doi.org/10.1109/Trustcom.2015.590
[12]
Nikos Giatrakos, Elias Alevizos, Alexander Artikis, Antonios Deligiannakis, and Minos N. Garofalakis. 2020. Complex event recognition in the Big Data era: a survey. VLDB J. 29, 1 (2020), 313--352. https://doi.org/10.1007/s00778-019-00557- w
[13]
Nikos Giatrakos, David Arnu, Theodoros Bitsakis, Antonios Deligiannakis, Minos N. Garofalakis, Ralf Klinkenberg, Aris Konidaris, Antonis Kontaxakis, Yannis Kotidis, Vasilis Samoladas, Alkis Simitsis, George Stamatakis, Fabian Temme, Mate Torok, Edwin Yaqub, Arnau Montagud, Miguel Ponce de Leon, Holger Arndt, and Stefan Burkard. 2020. INforE: Interactive Cross-platform Analytics for Everyone. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19--23, 2020, Mathieu d'Aquin, Stefan Dietze, Claudia Hauff, Edward Curry, and Philippe CudréMauroux (Eds.). ACM, 3389--3392. https://doi.org/10.1145/3340531.3417435
[14]
Nikos Giatrakos, Alexander Artikis, Antonios Deligiannakis, and Minos N. Garofalakis. 2017. Complex Event Recognition in the Big Data Era. Proc. VLDB Endow. 10, 12 (2017), 1996--1999. https://doi.org/10.14778/3137765.3137829
[15]
Alejandro Grez, Cristian Riveros, Martín Ugarte, and Stijn Vansummeren. 2020. On the Expressiveness of Languages for Complex Event Recognition. In 23rd International Conference on Database Theory, ICDT 2020, March 30-April 2, 2020, Copenhagen, Denmark (LIPIcs, Vol. 155), Carsten Lutz and Jean Christoph Jung (Eds.). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 15:1--15:17. https://doi.org/10.4230/LIPIcs.ICDT.2020.15
[16]
J Hopcroft, R Motwani, and J Ullman. 2007. Introduction to automata theory, languages, and computation. Pearson/Addison Wesley.
[17]
Antonios Kontaxakis, Antonios Deligiannakis, Holger Arndt, Stefan Burkard, Claus-Peter Kettner, Elke Pelikan, and Kathleen Noack. 2021. Real-time processing of geo-distributed financial data. In DEBS '21: The 15th ACM International Conference on Distributed and Event-based Systems, Virtual Event, Italy, June 28 - July 2, 2021, Alessandro Margara, Emanuele Della Valle, Alexander Artikis, Nesime Tatbul, and Helge Parzyjegla (Eds.). ACM, 190--191. https://doi.org/10.1145/3465480.3467842
[18]
D. Luckham. 2008. Complex Event Processing in Financial Services. Journal of Financial Services Technology 2, 1 (2008), 13--19.
[19]
Ruben Mayer. 2018. Window-based data parallelization in complex event processing. Ph.D. Dissertation. University of Stuttgart, Germany. https://nbn-resolving.org/urn:nbn:de:bsz:93-opus-ds-97440
[20]
Ruben Mayer, Ahmad Slo, Muhammad Adnan Tariq, Kurt Rothermel, Manuel Gräber, and Umakishore Ramachandran. 2017. SPECTRE: supporting consumption policies in window-based parallel complex event processing. In Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference, Las Vegas, NV, USA, December 11 - 15, 2017, K. R. Jayaram, Anshul Gandhi, Bettina Kemme, and Peter R. Pietzuch (Eds.). ACM, 161--173. https://doi.org/10.1145/3135974.3135983
[21]
Ruben Mayer, Muhammad Adnan Tariq, and Kurt Rothermel. 2017. Minimizing Communication Overhead in Window-Based Parallel Complex Event Processing. In Proceedings of the 11th ACM International Conference on Distributed and Eventbased Systems, DEBS 2017, Barcelona, Spain, June 19--23, 2017. ACM, 54--65. https://doi.org/10.1145/3093742.3093914
[22]
Yuan Mei and Samuel Madden. 2009. ZStream: a cost-based query processor for adaptively detecting composite events. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2009, Providence, Rhode Island, USA, June 29 - July 2, 2009, Ugur Çetintemel, Stanley B. Zdonik, Donald Kossmann, and Nesime Tatbul (Eds.). ACM, 193--206. https://doi.org/10.1145/1559845.1559867
[23]
Scikit optimize Library. [n.d.]. https://scikit-optimize.github.io/stable/. [Online; accessed 06-June-2021].
[24]
Kostas Patroumpas and Timos K. Sellis. 2006. Window Specification over Data Streams. In Current Trends in Database Technology - EDBT 2006, EDBT 2006 Workshops PhD, DataX, IIDB, IIHA, ICSNW, QLQP, PIM, PaRMA, and Reactivity on the Web, Munich, Germany, March 26--31, 2006, Revised Selected Papers (Lecture Notes in Computer Science, Vol. 4254), Torsten Grust, Hagen Höpfner, Arantza Illarramendi, Stefan Jablonski, Marco Mesiti, Sascha Müller, Paula-Lavinia Patranjan, Kai-Uwe Sattler, Myra Spiliopoulou, and Jef Wijsen (Eds.). Springer, 445--464. https://doi.org/10.1007/11896548_35
[25]
Srinath Perera and Sriskandarajah Suhothayan. 2015. Solution patterns for realtime streaming analytics. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, DEBS '15, Oslo, Norway, June 29 - July 3, 2015, Frank Eliassen and Roman Vitenberg (Eds.). ACM, 247--255. https://doi.org/10.1145/2675743.2774214
[26]
Streaming Extension RapidMiner. [n.d.]. https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_streaming. [Online; accessed 06-June-2021].
[27]
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. 2016. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proc. IEEE 104, 1 (2016), 148--175. https://doi.org/10.1109/JPROC.2015.2494218
[28]
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms., 2960--2968 pages. https://proceedings.neurips.cc/paper/2012/hash/05311655a15b75fab86956663e1819cd-Abstract.html
[29]
Eleni Kougioumtzi, Antonios Kontaxakis, Antonios Deligiannakis, and Yannis Kotidis. 2021. Towards creating a generalized complex event processing operator using FlinkCEP: architecture & benchmark. In DEBS '21: The 15th ACM International Conference on Distributed and Event-based Systems, Virtual Event, Italy, June 28 - July 2, 2021, Alessandro Margara, Emanuele Della Valle, Alexander Artikis, Nesime Tatbul, and Helge Parzyjegla (Eds.). ACM, 188--189. https://doi.org/10.1145/3465480.3467841
[30]
Oracle JMX Technology. [n.d.]. https://docs.oracle.com/javase/tutorial/jmx/overview/. [Online; accessed 06-June-2021].
[31]
Haopeng Zhang, Yanlei Diao, and Neil Immerman. 2014. On complexity and optimization of expensive queries in complex event processing. In International Conference on Management of Data, SIGMOD 2014, Snowbird, UT, USA, June 22--27, 2014, Curtis E. Dyreson, Feifei Li, and M. Tamer Özsu (Eds.). ACM, 217--228. https://doi.org/10.1145/2588555.2593671

Cited By

View all
  • (2024)An Intelligent Retrieval Method for Audio and Video Content: Deep Learning Technology Based on Artificial IntelligenceIEEE Access10.1109/ACCESS.2024.345092012(123430-123446)Online publication date: 2024
  • (2023)Optimizing Resource Allocation for Tumor Simulations over HPC Infrastructures2023 IEEE 10th International Conference on Data Science and Advanced Analytics (DSAA)10.1109/DSAA60987.2023.10302484(1-10)Online publication date: 9-Oct-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management
October 2021
4966 pages
ISBN:9781450384469
DOI:10.1145/3459637
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 October 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. big data
  2. complex event processing
  3. flinkcep
  4. optimizer

Qualifiers

  • Short-paper

Funding Sources

Conference

CIKM '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)10
  • Downloads (Last 6 weeks)0
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)An Intelligent Retrieval Method for Audio and Video Content: Deep Learning Technology Based on Artificial IntelligenceIEEE Access10.1109/ACCESS.2024.345092012(123430-123446)Online publication date: 2024
  • (2023)Optimizing Resource Allocation for Tumor Simulations over HPC Infrastructures2023 IEEE 10th International Conference on Data Science and Advanced Analytics (DSAA)10.1109/DSAA60987.2023.10302484(1-10)Online publication date: 9-Oct-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media