skip to main content
10.1145/2488222.2488263acmconferencesArticle/Chapter ViewAbstractPublication PagesdebsConference Proceedingsconference-collections
research-article

Reliable speculative processing of out-of-order event streams in generic publish/subscribe middlewares

Authors Info & Claims
Published:29 June 2013Publication History

ABSTRACT

In surveillance, sports, finances, etc., distributed event-based systems are used to detect meaningful events with low latency in high data rate event streams. Both known approaches to deal with the predominant out-of-order event arrival at the distributed detectors have their shortcomings: buffering approaches introduce latencies for event ordering and stream revision approaches may result in system overloads due to unbounded retraction cascades. This paper presents a speculative processing technique for out-of-order event streams that enhances typical buffering approaches. In contrast to other stream revision approaches our novel technique encapsulates the event detector, uses the buffering technique to delay events but also speculatively processes a portion of it, and adapts the degree of speculation at runtime to fit the available system resources so that detection latency becomes minimal.

Our technique outperforms known approaches on both synthetical data and real sensor data from a Realtime Locating System (RTLS) with several thousands of out-of-order sensor events per second. Speculative buffering exploits system resources and reduces latency by 40% on average.

References

  1. M. Stonebraker, U. Çetintemel, and S. Zdonik, "The 8 requirements of real-time stream processing," SIGMOD Rec., vol. 34, no. 4, pp. 42--47, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. D. O'Keeffe and J. Bacon, "Reliable complex event detection for pervasive computing," in Proc. 4th Intl. Conf. Distributed Event-Based Systems, (Cambridge, UK), pp. 73--84, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. P. A. Tucker, D. Maier, T. Sheard, and L. Fegaras, "Exploiting punctuation semantics in continuous data streams," IEEE Trans. Knowledge and Data Engineering, vol. 15, no. 3, pp. 555--568, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. J. Li, K. Tufte, V. Shkapenyuk, V. Papadimos, T. Johnson, and D. Maier, "Out-of-order processing: a new architecture for high-performance stream systems," in Proc. VLDB Endow., vol. 1, (Auckland, NZ), pp. 274--288, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. U. Srivastava and J. Widom, "Flexible time management in data stream systems," in Proc. 23rd ACM Symp. Principles Database Systems, (Paris, France), pp. 263--274, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. E. Wu, Y. Diao, and S. Rizvi, "High-performance complex event processing over streams," in Proc. ACM Intl. Conf. Management of Data, (Chicago, IL), pp. 407--418, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. M. Li, M. Liu, L. Ding, E. Rundensteiner, and M. Mani, "Event stream processing with out-of-order data arrival," in Proc. 27th Intl. Conf. Distrib. Comp. Systems Workshops, (Toronto, CAN), pp. 67--74, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. J. Agrawal, Y. Diao, D. Gyllstrom, and N. Immerman, "Efficient pattern matching over event streams," in Proc. ACM Intl. Conf. Management of Data, (Vancouver, CAN), pp. 147--160, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. A. Demers, J. Gehrke, M. Hong, M. Riedewald, and W. White, "Towards expressive publish/subscribe systems," in Proc. 10th Intl. Conf. Extending Database Technology, (Munich, Germany), pp. 627--644, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. A. Demers, J. Gehrke, B. Panda, M. Riedewald, V. Sharma, and W. White, "Cayuga: a general purpose event monitoring system," in Proc. 3rd Biennial Conf. Innovative Data Systems Research, (Pacific Grove, CA), pp. 412--422, 2007.Google ScholarGoogle Scholar
  11. L. Brenna, A. Demers, J. Gehrke, M. Hong, J. Ossher, B. Panda, M. Riedewald, M. Thatte, and W. White, "Cayuga: a high-performance event processing engine," in Proc. ACM Intl. Conf. Management of Data, (Beijing, China), pp. 1100--1102, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. L. Brenna, J. Gehrke, M. Hong, and D. Johansen, "Distributed event stream processing with non-deterministic finite automata," in Proc. 3rd Intl. Conf. Distributed Event-Based Systems, (Nashville, TN), pp. 3:1--3:12, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. M. Balazinska, Y. Kwon, N. Kuchta, and D. Lee, "Moirae: history-enhanced monitoring," in Proc. 3rd Biennial Conf. Innovative Data Systems Research, (Pacific Grove, CA), pp. 375--386, 2007.Google ScholarGoogle Scholar
  14. A. S. Maskey and M. Cherniack, "Replay-based approaches to revision processing in stream query engines," in Proc. 2nd Intl. Workshop Scalable Stream Processing Systems, (Nantes, France), pp. 3--12, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. D. Anicic, S. Rudolph, P. Fodor, and N. Stojanovic, "Retractable complex event processing and stream reasoning," in Proc. 5th Intl. Conf. Rule-based Reasoning, Programming, and Applications, (Fort Lauderdale, FL), pp. 122--137, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. C.-W. Li, Y. Gu, G. Yu, and B. Hong, "Aggressive complex event processing with confidence over out-of-order streams," Comp. Science and Technol., vol. 26, no. 4, pp. 685--696, 2011.Google ScholarGoogle ScholarCross RefCross Ref
  17. E. Ryvkina, A. S. Maskey, M. Cherniack, and S. Zdonik, "Revision processing in a stream processing engine: a high-level design," in Proc. 22nd Intl. Conf. Data Engineering, (Atlanta, GA), pp. 141--143, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. A. Brito, C. Fetzer, H. Sturzrehm, and P. Felber, "Speculative out-of-order event processing with software transaction memory," in Proc. 2nd Intl. Conf. Distributed Event-Based Systems, (Rome, Italy), pp. 265--275, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. B. Wester, J. Cowling, E. B. Nightingale, P. M. Chen, J. Flinn, and B. Liskov, "Tolerating latency in replicated state machines through client speculation," in Proc. 6th USENIX Symp. Networked Systems Design and Implementation, (Boston, MA), pp. 245--260, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. M. Liu, M. Li, D. Golovnya, E. Rundensteiner, and K. Claypool, "Sequence pattern query processing over out-of-order event streams," in Proc. 25th Intl. Conf. Data Eng., (Shanghai, China), pp. 784--795, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. R. S. Barga, J. Goldstein, M. Ali, and M. Hong, "Consistent streaming through time: a vision for event stream processing," in Proc. 3rd Biennial Conf. Innovative Data Systems Research, (Pacific Grove, CA), pp. 363--374, 2007.Google ScholarGoogle Scholar
  22. B. Chandramouli, J. Goldstein, and D. Maier, "High-performance dynamic pattern matching over disordered streams," in Proc. VLDB Endow., vol. 3, (Singapore), pp. 220--231, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Z. Jerzak and C. Fetzer, "BFSiena: a communication substrate for StreamMine," in Proc. 2nd Intl. Conf. Distributed Event-Based Systems, (Rome, Italy), pp. 321--324, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. G. Mühl, L. Fiege, and P. Pietzuch, Distributed event-based systems. Springer, Berlin, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. S. Babu, U. Srivastava, and J. Widom, "Exploiting k-constraints to reduce memory overhead in continuous queries over data streams," ACM Trans. Database Systems, vol. 29, no. 3, pp. 545--580, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. C. Mutschler and M. Philippsen, "Distributed low-latency out-of-order event processing for high data rate sensor streams," in Proc. 27th Intl. Conf. Parallel and Distributed Processing Symposium, (Boston, MA), pp. 1133--1144, 2013.Google ScholarGoogle Scholar
  27. M. Allman, C. Hayes, and S. Ostermann, "An evaluation of TCP with larger initial windows," SIGCOMM Comput. Commun. Rev., vol. 28, no. 3, pp. 41--52, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. T. v. d. Grün, N. Franke, D. Wolf, N. Witt, and A. Eidloth, "A real-time tracking system for football match and training analysis," in Microelectronic Systems, pp. 199--212, Springer Berlin, 2011.Google ScholarGoogle Scholar

Index Terms

  1. Reliable speculative processing of out-of-order event streams in generic publish/subscribe middlewares

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in
        • Published in

          cover image ACM Conferences
          DEBS '13: Proceedings of the 7th ACM international conference on Distributed event-based systems
          June 2013
          360 pages
          ISBN:9781450317580
          DOI:10.1145/2488222

          Copyright © 2013 ACM

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 29 June 2013

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article

          Acceptance Rates

          DEBS '13 Paper Acceptance Rate16of58submissions,28%Overall Acceptance Rate130of553submissions,24%

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader