Skip to main content
SpringerLink
Log in

Deployment strategies for distributed complex event processing

  • Published:
Computing Aims and scope Submit manuscript

Abstract

Several complex event processing (CEP) middleware solutions have been proposed in the past. They act by processing primitive events generated by sources, extracting new knowledge in the form of composite events, and delivering them to interested sinks. Event-based applications often involve a large number of sources and sinks, possibly dispersed over a wide geographical area. To better support these scenarios, the CEP middleware can be internally built around several, distributed processors, which cooperate to provide the processing and routing service. This paper introduces and compares different deployment strategies for a CEP middleware, which define (i) how the processing load is distributed over different processors and (ii) how these processors interact to produce the required results and to deliver them to sinks. Our evaluation compares the presented solutions and shows their benefits with respect to a centralized deployment, both in terms of network traffic and in terms of forwarding delay.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Notes

  1. The use of advertisements has been first introduced (and widely adopted) by publish-subscribe systems [11].

  2. For simplicity, in the following discussion we assume sinks to be interested only in composite events. Primitive events can be easily handled using well known protocols for content-based routing (see for example [10, 19]).

  3. Forwarding of subscriptions is required only when a single tree strategy is adopted. Multiple trees strategies do not require subscriptions to be forwarded, since each sink is already at the root of its own tree, where composite events are detected.

  4. Notice that, while the example above only considers the types of events, our strategies also consider the constraints on the content of events expressed in rules during partitioning.

  5. Here the term is used to indicate the internal events used in Omnet++ to model a complex network, not a T-Rex event.

References

  1. Abadi DJ, Ahmad Y, Balazinska M, Cetintemel U, Cherniack M, Hwang JH, Lindner W, Maskey AS, Rasin A, Ryvkina E, Tatbul N, Xing Y, Zdonik SB (2005) The design of the borealis stream processing engine. In: CIDR 2005, ACM, Asilomar, CA, USA

  2. Agrawal J, Diao Y, Gyllstrom D, Immerman N (2008) Efficient pattern matching over event streams. In: SIGMOD ’08, ACM, New York, pp 147–160. doi:10.1145/1376616.1376634

  3. Ahmad Y, Berg B, Cetintemel U, Humphrey M, Hwang JH, Jhingran A, Maskey A, Papaemmanouil O, Rasin A, Tatbul N, Xing W, Xing Y, Zdonik S (2005) Distributed operation in the borealis stream processing engine. In: SIGMOD ’05, ACM, New York, pp 882-884. doi:10.1145/1066157.1066274

  4. Ahmad Y, Çetintemel U (2004) Network-aware query processing for stream-based applications. In: VLDB ’04, pp 456–467. (VLDB Endowment)

  5. Ali M (2010) An introduction to microsoft sql server streaminsight. In: Proceedings of the 1st international conference and exhibition on computing for geospatial research and application, COM.Geo ’10, ACM, New York, NY, USA, p 66:1. doi:10.1145/1823854.1823929

  6. Amini L, Jain N, Sehgal A, Silber J, Verscheure O (2006) Adaptive control of extreme-scale stream processing systems. In: ICDCS ’06, IEEE Computer Society, Washington, DC, p 71. doi:10.1109/ICDCS.2006.13

  7. Balazinska M, Balakrishnan H, Stonebraker M (2004) Contract-based load management in federated distributed systems.In: NSDI ’04, USENIX Association, Berkeley, p 15

  8. Brenna L, Demers A, Gehrke J, Hong M, Ossher J, Panda B, Riedewald M, Thatte M, White W (2007) Cayuga: a high-performance event processing engine. In: SIGMOD ’07, ACM, New York, pp 1100–1102. doi:10.1145/1247480.1247620

  9. Broda K, Clark K, Miller R, Russo A (2009) Sage: a logical agent-based environment monitoring and control system. In: AmI ’09, pp 112–117

  10. Carzaniga A, Rosenblum D, Wolf A (2001) Design and evaluation of a wide-area event notification service. ACM Trans Computer Syst 19(3):332–383. http://www.citeseer.nj.nec.com/482106.html

    Google Scholar 

  11. Carzaniga A, Rosenblum DS, Wolf AL (2000) Achieving scalability and expressiveness in an internet-scale event notification service. PODC ’00, Portland, pp 219–227

  12. Carzaniga A, Rutherford MJ, Wolf AL (2004) A routing scheme for content-based networking. In: INFOCOM ’04, Hong Kong

  13. Chakravarthy S, Krishnaprasad V, Anwar E, Kim SK (1994) Composite events for active databases: semantics, contexts and detection. In: Proceedings of the 20th international conference on very large data bases, VLDB ’94. Morgan Kaufmann Publishers Inc., San Francisco, pp 606–617

  14. Cugola G, Margara A (2009) Raced: an adaptive middleware for complex event detection. In: ARM ’09, ACM, New York, pp 1–6

  15. Cugola G, Margara A (2010) Tesla: a formally defined event specification language. In: DEBS ’10, ACM, New York, pp 50–61. doi:10.1145/1827418.1827427

  16. Cugola G, Margara A (2012) Complex event processing with t-rex. J Syst Softw 85(8):1709–1728. doi:10.1016/j.jss.2012.03.056. http://www.sciencedirect.com/science/article

  17. Cugola G, Margara A (2012) Low latency complex event processing on parallel hardware. J Parallel Distrib Comput 72(2):205–218. doi:10.1016/j.jpdc.2011.11.002

    Article  Google Scholar 

  18. Cugola G, Margara A (2012) ACM Comput Surv. Processing flows of information: from data stream to complex event processing 44(3):15:1–15:62. doi:10.1145/2187671.2187677

    Google Scholar 

  19. Cugola G, Margara A, Migliavacca M (2009) Context-aware publish-subscribe: model, implementation, and evaluation. In: ISCC ’09, IEEE Computer Society Press, Toronto

  20. Demers AJ, Gehrke J, Hong M, Riedewald M, White WM (2006) Towards expressive publish/subscribe systems. In: EDBT ’06, pp 627–644

  21. Eyers D, Freudenreich T, Margara A, Frischbier S, Pietzuch P, Eugster P (2012) Living in the present: on-the-fly information processing in scalable web architectures. In: Proceedings of the 2nd international workshop on cloud computing platforms, CloudCP ’12, ACM, New York, pp 6:1–6:6 USA. doi:10.1145/2168697.2168703

  22. Jain N, Kit D, Mahajan P, Yalagandula P, Dahlin M, Zhang Y (2007) STAR: self-tuning aggregation for scalable monitoring. In: VLDB’07, Viennna

  23. Jain N, Mahajan P, Kit D, Yalagandula P, Dahlin M, Zhang Y (2008) Network imprecision: a new consistency metric for scalable monitoring. In: OSDI’08. http://dl.acm.org/citation.cfm?id=1855741.1855748

  24. Khandekar R, Hildrum K, Parekh S, Rajan D, Wolf J, Wu KL, Andrade H, Gedik B (2009) Cola: optimizing stream processing applications via graph partitioning. In: Middleware ’09. Springer, New York, pp 1–20

  25. Kumar V, Cooper BF, Cai Z, Eisenhauer G, Schwan K (2005) Resource-aware distributed stream management using dynamic overlays. In: ICDCS ’05, IEEE Computer Society, Washington, pp 783–792. doi:10.1109/ICDCS.2005.69

  26. Lakshmanan GT, Li Y, Strom R (2008) Placement strategies for internet-scale data stream systems. In: IEEE Internet Comput 12(6):50–60. doi:10.1109/MIC.2008.129

  27. Li G, Jacobsen HA (2005) Composite subscriptions in content-based publish/subscribe systems. In: Middleware ’05, Springer, New York, Inc., New York

  28. Liebig C, Cilia M, Buchmann A (1999) Event composition in time-dependent distributed systems. In: Cooperative information systems, 1999, CoopIS ’99, Proceedings. 1999 IFCIS international conference on cooperative information system, pp 70–78. doi:10.1109/COOPIS.1999.792159

  29. Medina A, Lakhina A, Matta I, Byers J (2001) Brite: an approach to universal topology generation. In: MASCOTS ’01, IEEE Computer Society, Washington, DC, p 346

  30. Moody K, Bacon J, Evans D, Schwiderski-Grosche S (2010) Implementing a practical spatio-temporal composite event language. In: Sachs K, Petrov I, Guerrero P (eds) From active data management to event-based systems and more. Lecture notes in computer science, vol 6462, Springer Berlin, Heidelberg, pp 108–123. doi:10.1007/978-3-642-17226-7_7

  31. O’Keeffe D, Bacon J (2010) Reliable complex event detection for pervasive computing. In: Proceedings of the fourth ACM international conference on distributed event-based systems, DEBS ’10, ACM, New York, pp 73–84. doi:10.1145/1827418.1827429

  32. Pietzuch P, Ledlie J, Shneidman J, Roussopoulos M, Welsh M, Seltzer M (2006) Network-aware operator placement for stream-processing systems. In: ICDE ’06, IEEE Computer Society

  33. Pietzuch P, Shand B, Bacon J (2004) Composite event detection as a generic middleware extension. Network IEEE 18(1):44–55. doi:10.1109/MNET.2004.1265833

    Article  Google Scholar 

  34. Pietzuch PR, Shand B, Bacon J (2003) A framework for event composition in distributed systems. In: Proceedings of the ACM/IFIP/USENIX 2003 international conference on middleware, Middleware ’03. Springer-Verlag New York, Inc., New York, pp 62–82. http://dl.acm.org/citation.cfm?id=1515915.1515921

  35. Ré C, Letchner J, Balazinksa M, Suciu D (2008) Event queries on correlated probabilistic streams. In: Proceedings of the 2008 ACM SIGMOD international conference on management of data, SIGMOD ’08. ACM, New York, pp 715–728. doi:10.1145/1376616.1376688

  36. Repantis T, Gu X, Kalogeraki V (2006) Synergy: sharing-aware component composition for distributed stream processing systems. In: Middleware ’06. Springer-Verlag New York, Inc., New York

    Google Scholar 

  37. Schultz-Moeller NP, Migliavacca M, Pietzuch P (2009) Distributed complex event processing with query optimisation. In: DEBS ’09, ACM, ACM, Nashville

  38. Schwiderski-Grosche S, Moody K (2009) The spatec composite event language for spatio-temporal reasoning in mobile systems. In: Proceedings of the third ACM international conference on distributed event-based systems, DEBS ’09. ACM, New York, pp 11:1–11:12. doi:10.1145/1619258.1619273

  39. Srivastava U, Widom J (2004) Flexible time management in data stream systems. In: PODS ’04, ACM, New York, pp 263–274. doi:10.1145/1055558.1055596

  40. Varga A (2001) The omnet++ discrete event simulation system. In: ESM ’01

  41. Wasserkrug S, Gal A, Etzion O, Turchin Y (2012) Efficient processing of uncertain events in rule-based systems. In: IEEE Trans Knowl Data Eng 24(1):45–58. doi:10.1109/TKDE.2010.204

  42. Wolf J, Bansal N, Hildrum K, Parekh S, Rajan D, Wagle R, Wu KL, Fleischer L (2008) Soda: an optimizing scheduler for large-scale stream-based distributed computer systems. In: Middleware ’08, Springer-Verlag New York, Inc., New York, pp 306–325

  43. Yoneki E, Bacon J (2005) Unified semantics for event correlation over time and space in hybrid network environments. In: Proceedings of the 2005 confederated international conference on the move to meaningful internet systems, OTM’05. Springer, Berlin, Heidelberg, pp 366–384. doi:10.1007/11575771_24

  44. Zhou Y, Ooi BC, Tan KL, Wu J (2006) Efficient dynamic operator placement in a locally distributed continuous query system. In: OTM Conferences, vol 1, pp 54–71

Download references

Acknowledgments

This work was partially supported by the European Commission, Programme IDEAS-ERC, Project 227977-SMScom.

Author information

Authors and Affiliations

  1. Dipartimento di Elettronica e Informazione (DEI), Politecnico di Milano, Piazza Leonardo Da Vinci, 32, 20133 , Milan, Italy

    Gianpaolo Cugola & Alessandro Margara

Authors
  1. Gianpaolo Cugola
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessandro Margara
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alessandro Margara.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cugola, G., Margara, A. Deployment strategies for distributed complex event processing. Computing 95, 129–156 (2013). https://doi.org/10.1007/s00607-012-0217-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-012-0217-9

Keywords

Mathematics Subject Classification

Search

Navigation