skip to main content
10.1145/3210284.3210293acmconferencesArticle/Chapter ViewAbstractPublication PagesdebsConference Proceedingsconference-collections
research-article

Probabilistic Management of Late Arrival of Events

Published: 25 June 2018 Publication History

Abstract

In a networked world, events are transmitted from multiple distributed sources into CEP systems, where events are related to one another along multiple dimensions, e.g., temporal and spatial, to create complex events. The big data era brought with it an increase in the scale and frequency of event reporting. Internet of Things adds another layer of complexity with multiple, continuously changing event sources, not all of which are perfectly reliable, often suffering from late arrivals. In this work we propose a probabilistic model to deal with the problem of reduced reliability of event arrival time. We use statistical theories to fit the distributions of inter-generation at the source and network delays per event type. Equipped with these distributions we propose a predictive method for determining whether an event belonging to a window has yet to arrive. Given some user-defined tolerance levels (on quality and timeliness), we propose an algorithm for dynamically determining the amount of time a complex event time-window should remain open. Using a thorough empirical analysis, we compare the proposed algorithm against state-of-the-art mechanisms for delayed arrival of events and show the superiority of our proposed method.

References

[1]
Daniel J Abadi, Don Carney, Ugur Çetintemel, Mitch Cherniack, Christian Convey, Sangdon Lee, Michael Stonebraker, Nesime Tatbul, and Stan Zdonik. 2003. Aurora: a new model and architecture for data stream management. The VLDB Journal -The International Journal on Very Large Data Bases 12, 2 (2003), 120--139.
[2]
Alain Biem, Eric Bouillet, Hanhua Feng, Anand Ranganathan, Anton Riabov, Olivier Verscheure, Haris Koutsopoulos, and Carlos Moran. 2010. IBM Infosphere Streams for Scalable, Real-time, Intelligent Transportation Services. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data (SIGMOD). ACM, 1093--1104.
[3]
Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 38, 4 (2015), 28--38.
[4]
Tathagata Das, Yuan Zhong, Ion Stoica, and Scott Shenker. 2014. Adaptive Stream Processing Using Dynamic Batch Sizing. In Proceedings of the ACM Symposium on Cloud Computing (SOCC). ACM, 1--13.
[5]
Yuanzhen Ji, Anisoara Nica, Zbigniew Jerzak, Gregor Hackenbroich, and Christof Fetzer. 2016. Quality-driven Disorder Handling for Concurrent Windowed Stream Queries with Shared Operators. In Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems (DEBS). ACM, 25--36.
[6]
Yuanzhen Ji, J. Sun, Anisoara Nica, Zbigniew Jerzak, Gregor Hackenbroich, and Christof Fetzer. 2016. Quality-driven disorder handling for m-way sliding window stream joins. In Proceedings of the 32nd International Conference on Data Engineering (ICDE). IEEE, 493--504.
[7]
Yuanzhen Ji, Hongjin Zhou, Zbigniew Jerzak, Anisoara Nica, Gregor Hackenbroich, and Christof Fetzer. 2015. Quality-Driven Continuous Query Execution over Out-of-Order Data Streams. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD). ACM, 889--894.
[8]
Yuanzhen Ji, Hongjin Zhou, Zbigniew Jerzak, Anisoara Nica, Gregor Hackenbroich, and Christof Fetzer. 2015. Quality-driven Processing of Sliding Window Aggregates over Out-of-order Data Streams. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems (DEBS). ACM, 68--79.
[9]
Chuan-Wen Li, Yu Gu, Ge Yu, and Bonghee Hong. 2011. Aggressive complex event processing with confidence over out-of-order streams. Journal of Computer Science and Technology 26, 4 (2011), 685--696.
[10]
Jin Li, Kristin Tufte, Vladislav Shkapenyuk, Vassilis Papadimos, Theodore Johnson, and David Maier. 2008. Out-of-order Processing: A New Architecture for High-performance Stream Systems. Proceedings of the VLDB Endowment 1, 1 (2008), 274--288.
[11]
Christopher Mutschler and Michael Philippsen. 2013. Reliable speculative processing of out-of-order event streams in generic publish/subscribe middlewares. In Proceedings of the 7th ACM international conference on Distributed Event-Based Systems (DEBS). ACM, 147--158.
[12]
Nicolo Rivetti, Yann Busnel, and Achour Mostéfaoui. 2015. Efficiently Summarizing Data Streams over Sliding Windows. In Proceedings of the IEEE 14th International Symposium on Network Computing and Applications (NCA). IEEE, 151--158.
[13]
Stuart J. Russell and Peter Norvig. 2003. Artificial Intelligence: A Modern Approach. Pearson Education.
[14]
Esther Ryvkina, Anurag S Maskey, Mitch Cherniack, and Stan Zdonik. 2006. Revision processing in a stream processing engine: A high-level design. In Proceedings of the 22nd International Conference on Data Engineering (ICDE). IEEE, 141--143.
[15]
Utkarsh Srivastava and Jennifer Widom. 2004. Flexible Time Management in Data Stream Systems. In Proceedings of the 23rd ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems (PODS). ACM, 263--274.
[16]
The Apache Software Foundation. Apache Spark. https://spark.apache.org.
[17]
The Apache Software Foundation. Apache Flink. https://flink.apache.org/.
[18]
Srikanta Tirthapura, Bojian Xu, and Costas Busch. 2006. Sketching Asynchronous Streams over a Sliding Window. In Proceedings of the 25th Annual ACM Symposium on Principles of Distributed Computing (PODC). ACM, 82--91.
[19]
Peter A. Tucker, David Maier, Tim Sheard, and Leonidas Fegaras. 2003. Exploiting punctuation semantics in continuous data streams. IEEE Transactions on Knowledge and Data Engineering 15, 3 (2003), 555--568.
[20]
Nikos Zacheilas and Vana Kalogeraki. 2016. Chess: Cost-effective scheduling across multiple heterogeneous mapreduce clusters. In Proceedings of the 2016 IEEE International Conference on Autonomic Computing Autonomic Computing (ICAC). IEEE, 65--74.
[21]
Nikos Zacheilas, Vana Kalogeraki, Yiannis Nikolakopoulos, Vincenzo Gulisano, Marina Papatriantafilou, and Philippas Tsigas. 2017. Maximizing Determinism in Stream Processing Under Latency Constraints. In Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems (DEBS). ACM, 112--123.
[22]
Quan Zhang, Yang Song, Ramani R Routray, and Weisong Shi. 2016. Adaptive block and batch sizing for batched stream processing system. In Proceedings of the 2016 IEEE International Conference on Autonomic Computing Autonomic Computing (ICAC). IEEE, 35--44.

Cited By

View all
  • (2024)Efficient Pattern Matching over Out-of-Order Event Streams Using Sliding BufferJournal of Information Processing10.2197/ipsjjip.32.96332(963-972)Online publication date: 2024
  • (2022)Edge-Based Runtime Verification for the Internet of ThingsIEEE Transactions on Services Computing10.1109/TSC.2021.307495615:5(2713-2727)Online publication date: 1-Sep-2022
  • (2021)Klink: Progress-Aware Scheduling for Streaming Data SystemsProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3452794(485-498)Online publication date: 9-Jun-2021
  • Show More Cited By

Index Terms

  1. Probabilistic Management of Late Arrival of Events

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    DEBS '18: Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems
    June 2018
    289 pages
    ISBN:9781450357821
    DOI:10.1145/3210284
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 25 June 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Complex Event Processing
    2. Late arrivals
    3. Probabilistic Prediction
    4. Sliding Window

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    DEBS '18

    Acceptance Rates

    DEBS '18 Paper Acceptance Rate 12 of 31 submissions, 39%;
    Overall Acceptance Rate 145 of 583 submissions, 25%

    Upcoming Conference

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)16
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Efficient Pattern Matching over Out-of-Order Event Streams Using Sliding BufferJournal of Information Processing10.2197/ipsjjip.32.96332(963-972)Online publication date: 2024
    • (2022)Edge-Based Runtime Verification for the Internet of ThingsIEEE Transactions on Services Computing10.1109/TSC.2021.307495615:5(2713-2727)Online publication date: 1-Sep-2022
    • (2021)Klink: Progress-Aware Scheduling for Streaming Data SystemsProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3452794(485-498)Online publication date: 9-Jun-2021
    • (2021)Ermis: a middleware for bridging data collection and data processing in IoT streaming applications2021 17th International Conference on Distributed Computing in Sensor Systems (DCOSS)10.1109/DCOSS52077.2021.00050(259-266)Online publication date: Jul-2021
    • (2021)Reactive Programming Extensions for Time-Sensitive IoT Applications2021 17th International Conference on Distributed Computing in Sensor Systems (DCOSS)10.1109/DCOSS52077.2021.00048(244-251)Online publication date: Jul-2021
    • (2020)Kairos: a self-configuring approach for short and accurate event timeouts in IoTMobiQuitous 2020 - 17th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services10.1145/3448891.3448917(347-356)Online publication date: 7-Dec-2020
    • (2020)The role of event-time order in data streaming analysisProceedings of the 14th ACM International Conference on Distributed and Event-based Systems10.1145/3401025.3404088(214-217)Online publication date: 13-Jul-2020
    • (2020)Simplifying CPS Application Development through Fine-grained, Automatic Timeout PredictionsACM Transactions on Internet of Things10.1145/33859601:3(1-30)Online publication date: 1-Jun-2020
    • (2019)KhronosProceedings of the 13th ACM International Conference on Distributed and Event-based Systems10.1145/3328905.3329507(127-138)Online publication date: 24-Jun-2019
    • (2019)Complex event recognition in the Big Data era: a surveyThe VLDB Journal10.1007/s00778-019-00557-wOnline publication date: 25-Jul-2019

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media