Geetika T. Lakshmanan
Yuri G. Rabinovich
ABSTRACT
The quantity of events that a single application needs to process is constantly increasing. RFID related events have doubled within the past year and reached 4 trillion events per day, financial applications in large banks are processing 400 million events per day, and Massively Multiplayer Online (MMO) games are monitoring millions of events per second during peak periods. It is evident that scalability in event throughput is a major requirement for such applications. While the first generation of event processing systems is centralized, we see various solutions that attempt to use both scale-up and scale-out techniques. Alas, partitioning of the processing manually is difficult due to the semantic dependencies among event processing agents. It is also difficult to manually tune up the partition dynamically. This paper proposes a horizontal partition that is automatically created by analyzing the semantic dependencies among agents using a stratification principle. Each stratum contains a collection of independent agents, and events are routed to subsequent strata. We also implement a profiling-based technique for assigning agents to nodes in each stratum with the goal of maximizing throughput. A complementary step is to distribute load among different execution nodes dynamically based on their performance characteristics and the event traffic model. Experimental results show significant improvement in the ability to process high throughput of events relative to both centralized solutions as well as vertical partitions. We find this to be a promising approach to achieve high scalability particularly when the traffic model and network topology change frequently.
- Adaikkavalan R. and Chakravarthy S. 2006. SnoopIB: Interval-based Event Specification and Detection for Active Databases. DKE, 59(1), 2006, pp. 139--165. Google Scholar
Digital Library
- Adas A. 1997. Traffic models in broadband networks. IEEE Communications, 35(7):82--89, July 1997. Google ScholarDigital Library
- Adi A. and Etzion O. 2002. The situation manager rule language, Proc of RuleML, 2002.Google Scholar
- Adi A. and Etzion O. 2004. Amit - the situation manager. Proc. of VLDB J. 13(2): 177--203, 2004. Google ScholarDigital Library
- Baralis E., Ceri S. and Paraboschi S. 1996. Modularization Techniques for Active Rules Design. ACM Trans. Database Systems 21(1): 1--29, 1996 Google ScholarDigital Library
- Biger A. 2007. Complex Event Processing Scalability by Partition. M.Sc. Thesis, Technion. Israel Institute of Technology, 2007.Google Scholar
- Biger A., Rabinovich Y. and Etzion O. 2008. Stratified Implementation of Event Processing Network. Fast abstract in DEBS, 2008.Google Scholar
- Carzaniga A., Rosenblum D. S. and Wolf A. L. 2000. Achieving scalability and expressiveness in an internet-scale event notification service. Proc. of PODC, 2000. Google ScholarDigital Library
- Chakravarthy S. and Adaikkalavan R. 2008. Events and Streams: Harnessing and Unleashing Their Synergy. Proc. of DEBS, 2008. Google ScholarDigital Library
- Chakravarthy S. et al. 1994. Design of Sentinel: An Object-Oriented DBMS with Event-Based Rules. Information and Software Technology, 36(9):559--568, 1994.Google ScholarCross Ref
- Cohen N. H. and Kalleberg K. T. 2008. EventScript: an event-processing language based on regular expressions with actions. Proc. of LCTES, 2008. Google ScholarDigital Library
- Dinn A., Williams M. H. and Paton N. W. 1997. ROCK&ROLL: A Deductive Object-Oriented Database with Active hiand Spatial Extensions. Proc. of ICDE, 1997. Google ScholarDigital Library
- Gatziu S. and Dittrich K. R. 1993. Events in an Object-Oriented Database System. Proc. of Rules in Database Systems, 1993.Google Scholar
- Gatziu S. and Dittrich K. R. 1994. Detecting Composite Events in Active Databases using Petri Nets. Workshop on Research Issues in Data Engineering, 1994.Google Scholar
- Gu X., Yu P., and Wang H. 2007. Adaptive Load Diffusion for Multiway Windowed Stream Joins. Proc. of ICDE, 2007.Google Scholar
- Huebsch R., Hellerstein J. M., Lanham N., Loo B. T., Shenker S. and Stoica I. 2003. Querying the internet with PIER. Proc. of VLDB, 2003. Google ScholarDigital Library
- Jiang Q., Adaikkalavan R. and Chakravarthy S. 2007. MavEStream: Synergistic Integration of Stream and Event Processing. Proc. of ICDT, 2007. Google ScholarDigital Library
- Kulkarni D., Ravishankar C. V. and Cherniack M. 2008. Real-Time, Load-Adaptive Processing of Continuous Queries Over Data Streams. Proc. of DEBS, 2008. Google ScholarDigital Library
- Lieuwen D. L., Gehani N. H. and Arlein R. 1996. The Ode Active Database: Trigger Semantics and Implementation. Proc. of ICDE, 1996. Google ScholarDigital Library
- Liu B., Jbantova M. and Rundensteiner E. A. 2007. Optimizing State-Intensive Non-Blocking Queries Using Run-time Adaptation. Proc. of ICDE, 2007. Google ScholarDigital Library
- Luckham D. 2002. The Rapide pattern language. In The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems. Addison-Wesley, Boston, 2002, chapter 8.Google ScholarDigital Library
- Mehta M. and DeWitt D. 1995. Managing intra-operator parallelism in parallel database systems. Proc of VLDB, 1995. Google ScholarDigital Library
- Renesse R. V., Birman K. P. and Vogels W. 2003. Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining. ACM Transactions on Computer Systems, 21(2): 164--206, 2003. Google ScholarDigital Library
- Shah M., Hellerstein J., Chandrasekaran S. and Franklin M. 2003. Flux: An adaptive partitioning operator for continuous query systems. Proc. of ICDE, 2003.Google Scholar
- Sharon G. and Etzion O. 2008. Event Processing Networks: model and implementation. IBM System Journal, 47(2), 321--334, 2008 Google ScholarDigital Library
- Wu E., Diao Y. and Rizvi S. 2006 High Performance Complex Event Processing Over Streams. Proc. of SIGMOD, 2006. Google ScholarDigital Library
- Xing Y., Hwang J-H., Cetintemel U. and Zdonik S. 2006. Providing Resiliency to Load Variations in Distributed Stream Processing. Proc. of VLDB, 2006. Google ScholarDigital Library
- Xing Y., Zdonik S., and Hwang J-H. 2005. Dynamic Load Distribution in the Borealis Stream Processor. Proc. of ICDE, 2005. Google ScholarDigital Library
- Xu W., Hellerstein J. L., Kramer B. and Patterson D. 2005. Control Considerations for Scalable Event Processing. Proc. of DSOM, 2005. Google ScholarDigital Library
- Zhou Y., Ooi B. C., Tan K. L., Wu J. 2006. Efficient Dynamic Operator Placement in a Locally Distributed Continuous Query System. Proc. of COOPIS, 2006. Google ScholarDigital Library
Index Terms
- A stratified approach for supporting high throughput event processing applications
Recommendations
A scalable complex event processing system and evaluations of its performance
DEBS '12: Proceedings of the 6th ACM International Conference on Distributed Event-Based SystemsThis paper describes a scalable context delivery platform (SCTXPF) and our evaluations of it. The SCTXPF receives a large number of events from various event sources and a large number of complex event processing (CEP) rules from various services/...
A complex event processing for large-scale M2M services and its performance evaluations
DEBS '15: Proceedings of the 9th ACM International Conference on Distributed Event-Based SystemsThis paper describes a scalable complex event processing (CEP) system and rule allocation algorithm for large-scale M2M services and presents the results of its performance evaluations. Platforms providing M2M services deal with various data from ...
Data Dissemination supporting collaborative complex event processing: characteristics and open issues
DD4LCCI '10: Proceedings of the First International Workshop on Data Dissemination for Large Scale Complex Critical InfrastructuresMost distributed applications today receive events, process them and in turn create new events which are sent to other processes. Business intelligence, air traffic control, collaborative security, complex system software management are examples of such ...
Comments