ABSTRACT
In surveillance, sports, finances, etc., distributed event-based systems are used to detect meaningful events with low latency in high data rate event streams. Both known approaches to deal with the predominant out-of-order event arrival at the distributed detectors have their shortcomings: buffering approaches introduce latencies for event ordering and stream revision approaches may result in system overloads due to unbounded retraction cascades. This paper presents a speculative processing technique for out-of-order event streams that enhances typical buffering approaches. In contrast to other stream revision approaches our novel technique encapsulates the event detector, uses the buffering technique to delay events but also speculatively processes a portion of it, and adapts the degree of speculation at runtime to fit the available system resources so that detection latency becomes minimal.
Our technique outperforms known approaches on both synthetical data and real sensor data from a Realtime Locating System (RTLS) with several thousands of out-of-order sensor events per second. Speculative buffering exploits system resources and reduces latency by 40% on average.
- M. Stonebraker, U. Çetintemel, and S. Zdonik, "The 8 requirements of real-time stream processing," SIGMOD Rec., vol. 34, no. 4, pp. 42--47, 2005. Google ScholarDigital Library
- D. O'Keeffe and J. Bacon, "Reliable complex event detection for pervasive computing," in Proc. 4th Intl. Conf. Distributed Event-Based Systems, (Cambridge, UK), pp. 73--84, 2010. Google ScholarDigital Library
- P. A. Tucker, D. Maier, T. Sheard, and L. Fegaras, "Exploiting punctuation semantics in continuous data streams," IEEE Trans. Knowledge and Data Engineering, vol. 15, no. 3, pp. 555--568, 2003. Google ScholarDigital Library
- J. Li, K. Tufte, V. Shkapenyuk, V. Papadimos, T. Johnson, and D. Maier, "Out-of-order processing: a new architecture for high-performance stream systems," in Proc. VLDB Endow., vol. 1, (Auckland, NZ), pp. 274--288, 2008. Google ScholarDigital Library
- U. Srivastava and J. Widom, "Flexible time management in data stream systems," in Proc. 23rd ACM Symp. Principles Database Systems, (Paris, France), pp. 263--274, 2004. Google ScholarDigital Library
- E. Wu, Y. Diao, and S. Rizvi, "High-performance complex event processing over streams," in Proc. ACM Intl. Conf. Management of Data, (Chicago, IL), pp. 407--418, 2006. Google ScholarDigital Library
- M. Li, M. Liu, L. Ding, E. Rundensteiner, and M. Mani, "Event stream processing with out-of-order data arrival," in Proc. 27th Intl. Conf. Distrib. Comp. Systems Workshops, (Toronto, CAN), pp. 67--74, 2007. Google ScholarDigital Library
- J. Agrawal, Y. Diao, D. Gyllstrom, and N. Immerman, "Efficient pattern matching over event streams," in Proc. ACM Intl. Conf. Management of Data, (Vancouver, CAN), pp. 147--160, 2008. Google ScholarDigital Library
- A. Demers, J. Gehrke, M. Hong, M. Riedewald, and W. White, "Towards expressive publish/subscribe systems," in Proc. 10th Intl. Conf. Extending Database Technology, (Munich, Germany), pp. 627--644, 2006. Google ScholarDigital Library
- A. Demers, J. Gehrke, B. Panda, M. Riedewald, V. Sharma, and W. White, "Cayuga: a general purpose event monitoring system," in Proc. 3rd Biennial Conf. Innovative Data Systems Research, (Pacific Grove, CA), pp. 412--422, 2007.Google Scholar
- L. Brenna, A. Demers, J. Gehrke, M. Hong, J. Ossher, B. Panda, M. Riedewald, M. Thatte, and W. White, "Cayuga: a high-performance event processing engine," in Proc. ACM Intl. Conf. Management of Data, (Beijing, China), pp. 1100--1102, 2007. Google ScholarDigital Library
- L. Brenna, J. Gehrke, M. Hong, and D. Johansen, "Distributed event stream processing with non-deterministic finite automata," in Proc. 3rd Intl. Conf. Distributed Event-Based Systems, (Nashville, TN), pp. 3:1--3:12, 2009. Google ScholarDigital Library
- M. Balazinska, Y. Kwon, N. Kuchta, and D. Lee, "Moirae: history-enhanced monitoring," in Proc. 3rd Biennial Conf. Innovative Data Systems Research, (Pacific Grove, CA), pp. 375--386, 2007.Google Scholar
- A. S. Maskey and M. Cherniack, "Replay-based approaches to revision processing in stream query engines," in Proc. 2nd Intl. Workshop Scalable Stream Processing Systems, (Nantes, France), pp. 3--12, 2008. Google ScholarDigital Library
- D. Anicic, S. Rudolph, P. Fodor, and N. Stojanovic, "Retractable complex event processing and stream reasoning," in Proc. 5th Intl. Conf. Rule-based Reasoning, Programming, and Applications, (Fort Lauderdale, FL), pp. 122--137, 2011. Google ScholarDigital Library
- C.-W. Li, Y. Gu, G. Yu, and B. Hong, "Aggressive complex event processing with confidence over out-of-order streams," Comp. Science and Technol., vol. 26, no. 4, pp. 685--696, 2011.Google ScholarCross Ref
- E. Ryvkina, A. S. Maskey, M. Cherniack, and S. Zdonik, "Revision processing in a stream processing engine: a high-level design," in Proc. 22nd Intl. Conf. Data Engineering, (Atlanta, GA), pp. 141--143, 2006. Google ScholarDigital Library
- A. Brito, C. Fetzer, H. Sturzrehm, and P. Felber, "Speculative out-of-order event processing with software transaction memory," in Proc. 2nd Intl. Conf. Distributed Event-Based Systems, (Rome, Italy), pp. 265--275, 2008. Google ScholarDigital Library
- B. Wester, J. Cowling, E. B. Nightingale, P. M. Chen, J. Flinn, and B. Liskov, "Tolerating latency in replicated state machines through client speculation," in Proc. 6th USENIX Symp. Networked Systems Design and Implementation, (Boston, MA), pp. 245--260, 2009. Google ScholarDigital Library
- M. Liu, M. Li, D. Golovnya, E. Rundensteiner, and K. Claypool, "Sequence pattern query processing over out-of-order event streams," in Proc. 25th Intl. Conf. Data Eng., (Shanghai, China), pp. 784--795, 2009. Google ScholarDigital Library
- R. S. Barga, J. Goldstein, M. Ali, and M. Hong, "Consistent streaming through time: a vision for event stream processing," in Proc. 3rd Biennial Conf. Innovative Data Systems Research, (Pacific Grove, CA), pp. 363--374, 2007.Google Scholar
- B. Chandramouli, J. Goldstein, and D. Maier, "High-performance dynamic pattern matching over disordered streams," in Proc. VLDB Endow., vol. 3, (Singapore), pp. 220--231, 2010. Google ScholarDigital Library
- Z. Jerzak and C. Fetzer, "BFSiena: a communication substrate for StreamMine," in Proc. 2nd Intl. Conf. Distributed Event-Based Systems, (Rome, Italy), pp. 321--324, 2008. Google ScholarDigital Library
- G. Mühl, L. Fiege, and P. Pietzuch, Distributed event-based systems. Springer, Berlin, 2006. Google ScholarDigital Library
- S. Babu, U. Srivastava, and J. Widom, "Exploiting k-constraints to reduce memory overhead in continuous queries over data streams," ACM Trans. Database Systems, vol. 29, no. 3, pp. 545--580, 2004. Google ScholarDigital Library
- C. Mutschler and M. Philippsen, "Distributed low-latency out-of-order event processing for high data rate sensor streams," in Proc. 27th Intl. Conf. Parallel and Distributed Processing Symposium, (Boston, MA), pp. 1133--1144, 2013.Google Scholar
- M. Allman, C. Hayes, and S. Ostermann, "An evaluation of TCP with larger initial windows," SIGCOMM Comput. Commun. Rev., vol. 28, no. 3, pp. 41--52, 1998. Google ScholarDigital Library
- T. v. d. Grün, N. Franke, D. Wolf, N. Witt, and A. Eidloth, "A real-time tracking system for football match and training analysis," in Microelectronic Systems, pp. 199--212, Springer Berlin, 2011.Google Scholar
Index Terms
Reliable speculative processing of out-of-order event streams in generic publish/subscribe middlewares
Recommendations
Adaptive Speculative Processing of Out-of-Order Event Streams
Special Issue on Event RecognitionDistributed event-based systems are used to detect meaningful events with low latency in high data-rate event streams that occur in surveillance, sports, finances, etc. However, both known approaches to dealing with the predominant out-of-order event ...
Distributed Low-Latency Out-of-Order Event Processing for High Data Rate Sensor Streams
IPDPS '13: Proceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed ProcessingEvent-based Systems (EBS) are used to detect and analyze meaningful events in surveillance, sports, finances and many other areas. With rising data and event rates and with correlations among these events, sequential event processing becomes infeasible ...
Demo: do event-based systems have a passion for sports?
DEBS '13: Proceedings of the 7th ACM international conference on Distributed event-based systemsThe ubiquity of sensor data calls for automatic processing to extract valuable information. Realtime Locating Systems (RTLS) provide many parallel position data streams for interacting objects, and event-based systems are the method of choice to analyze ...
Comments