Scalable complex event processing using adaptive load balancing

https://doi.org/10.1016/j.jss.2018.12.012Get rights and content

Highlights

  • We propose CCEP as a mechanism for clustering of heterogeneous CEP engines.

  • CCEP provides horizontal scaling of CEP via engine clustering.

  • CCEP supports CEP engine heterogeneity.

  • CCEP provides adaptive load balancing.

  • CCEP balances load of nodes with multiple types of resources based on a formal model.

Abstract

An essential requirement of large-scale event-driven systems is the real-time detection of complex patterns of events from a large number of basic events and derivation of higher-level events using complex event processing (CEP) mechanisms. Centralized CEP mechanisms are however not scalable and thus inappropriate for large-scale domains with many input events and complex patterns, rendering the horizontal scaling of CEP mechanisms a necessity. In this paper, we propose CCEP as a mechanism for clustering of heterogeneous CEP engines to provide horizontal scalability using adaptive load balancing. We experimentally compare the performance of CCEP with the performances of three CEP clustering mechanisms, namely VISIRI, SCTXPF, and RR. The results of experiments show that CCEP increases throughput by 40 percent and thus it is more scalable than the other three chosen mechanisms when the input event rate changes at runtime. Although CCEP increases the network utilization by about 40 percent, it keeps the load of the system two times more balanced and reduces the input event loss three times.

Introduction

Many large-scale computing systems are event-driven. Components of an event-driven computing system communicate with each other and their encompassing environment via events. Therefore, an event is produced for each state change in an event-driven system. A large-scale event-driven system produces a large number of events and the timely processing of such a high volume of data is an essential requirement. An important type of event processing is the real-time detection of complex patterns of events from a large number of input events, and derivation of higher-level events. Such processing is called Complex Event Processing (CEP) (Dayarathna and Perera, 2018). In a CEP system, event streams constitute the input data that are provided by many event producers, event patterns are defined by processing rules, and output is a stream of derived high-level events for some event consumers. CEP is an important requirement in many event-driven systems. CEP is employed in many domains such as in smart cities (Bonino and De Russis, 2018), road traffic control (Shinde and Dongre, 2017), internet of things (Sozert, 2017, Zu et al., 2017), social networks analysis (Shi et al., 2017), incident management (Mongiello et al., 2017), manufacturing intelligence (Sitaram and Subramaniam, 2016), interplanetary telecommunication (Choi et al., 2016), network security (Yin et al., 2016), and business process management (Meidan et al., 2017). A recent report (Research and Markets, 2016) has predicted that the market size of CEP applications will grow from 1.28 billion USD in 2015 to 4.95 billion USD by 2020 at a compound annual growth rate of 31.1%.

Big challenges have emerged with increasing numbers of deployments of CEP in various domains. On the one hand, the number of event types and the rates of input events have increased drastically. On the other hand, the complexity of rules and their numbers have increased. These attributes have triggered research on advancements to enhance the CEP performance.

One approach to enhance the performance of CEP is to optimize CEP queries (rules). This approach falls short of enhancing the performance of CEP too much because it does not use the potential of parallelization to enhance the performance and does not lead to a scalable CEP, which is critically required. This explains the motivations of many researches on presenting various techniques for parallelization and deployment of CEP on multi-core CPUs, GPUs, or even FPGAs. Parallelization and deployment on a single machine has proved to be inadequate for applications of CEP in large-scale domains that require more computing power than those available on a single machine with vertical scaling limitations.

Complementary research has focused on improving the CEP performance by presenting horizontally scaled CEP solutions. Some of the solutions that fall in this category, process rules over a network of operators, wherein each compute node processes a specific part of a given rule and a set of compute nodes processes that rule collaboratively. These solutions consider the sum of decomposability of all rules as the parallelization degree, disregarding the fact that neither all rules are decomposable nor a decomposed rule can be processed in a pipeline manner over the network. Furthermore, these solutions do not completely support the processing of stateful operators over the network. They also lack any mechanisms to balance the workloads of compute nodes in the network, resulting in the possible overloading of some nodes and the wastages of resources of other resourceful compute nodes. In addition, most of them ignore the network communication overhead that could be high and critical to real-time performance of CEP.

The deployment of CEP in a cloud infrastructure counts as another approach towards enhancing the CEP performance. In this approach, each container contains only one rule and a CEP engine, and the cloud infrastructure is responsible for managing the containers. In this approach, the overhead of an engine alongside the overhead of the management of a container incurs a very high cost on the processing of each rule. Also, rule migration is as expensive as the container migration.

A much more acceptable approach for enhancing the CEP performance is the clustering of CEP engines. A CEP engine is run on each compute node and CEP tasks are distributed on a cluster of compute nodes. Every node can process any rule of the system but only one of the rules. Horizontal scaling is provided without the need for rule decomposition. Instead, horizontal scaling is provided by taking advantage of popular CEP engines and their known rule syntaxes. Unlike the operator network approach, the CEP performance is enhanced without any changes to the rule definition language or the internal implementation of the engine. The parallelization degree is however limited to the number of rules, making this approach effective only when there are a large number of CEP rules. For example, in a large organization such as Iran Customs, there are hundreds of business processes and each process should be monitored using dozens of rules. Therefore, the rule set of the organization consists of thousands of rules and the rule set can be partitioned to many subsets to be processed by several CEP nodes.

In this paper, we propose a mechanism called CCEP for Clustering of CEP engines that provides both horizontal scalability and adaptive load balancing of a clustered set of compute nodes. In this mechanism, a user can employ her/his favorite CEP engine(s) to create a scalable cluster and use the rule language with which she/he is familiar. For load distribution among cluster nodes, we parallelize CEP using the partitioning of rules. The rule set of a system is partitioned into subsets in a way that the processing load of the subsets is balanced. Then, the processing of each partition (subset) is assigned to a CEP engine. Throughout the system operation, resources of each compute node are monitored and the load of the system is balanced by migration of rules between the engines upon changes to the rate of input events that causes load imbalance. The main contributions of CCEP can be summarized as presenting:

  • Horizontal scaling of CEP via engine clustering

  • Supporting heterogonous engines

  • Adaptive load balancing

  • Load balancing of nodes with multiple types of resources based on a formal model

We have organized the rest of the paper as follows. Section 2 presents background information about our work. Section 3 presents the required formalism of the problem domain and our solution. Section 4 explains CCEP and its architecture and implementation. Section 5 presents a use case and reports the results of our experiments with an implementation of CCEP and comparisons with three existing related mechanisms. Section 6 presents related works and Section 7 concludes the paper.

Section snippets

Background

In this section, we present the definitions of the concepts used in the research that is reported in this paper. At first, we define the event. An event is anything that occurs or is contemplated as having occurred in a particular system. But in a computing system, an event is an object that reports the actually occurred event (Etzion and Niblett, 2010). Definition 1, Definition 2 and Definition 3 define formally an event and its related concepts.

Definition 1

(Event Type) An event type τ=(typeid,ATR) refers

Problem formalism

To help presenting the CCEP more concisely, let us first define the problem that is addressed by CCEP, its basics and parameters formally. If we have n homogenous compute nodes and a CEP engine runs on each node, and there are m resource types on each node, then the current utilization of each resource at each node is denoted by Γ in Eq. (1).Γ=[γ11γ1mγn1γnm]

In Eq. (1), γij shows the current utilization of resource j at node i where i{1,2,,n}, j{1,2,,m} and 0 ≤ γij ≤ 1. Therefore, Γ

CCEP

In this section, we show how CCEP provides horizontal scalability and adaptive load balancing via clustering of CEP engines. CCEP uses rule partitioning for CEP parallelization. Therefore, as mentioned in Section 2.1, CCEP assumes that the system has a large number of rules compared to the number of compute nodes, and that each rule can be processed only by a single node. According to (Etzion and Niblett, 2010), for the derivation of each complex event, corresponding events of its rule are

Evaluation

Many scenarios are tested to evaluate CCEP and compare it to other related works. In this section, evaluation parameters and the results of experiments are reported, followed by an analytical discussion of the results.

Related work

As mentioned in Section 2.1, previous works have parallelized CEP using three methods. These methods are employed to scale CEP on three architectures. Table 3 presents a list of related works alongside their architectures and parallelization methods.

The first used architecture is the single machine architecture. CEP is scaled up on a single machine using multiple CPUs or multicore CPUs as is done and reported in (Balkesen et al., 2013, Cugola and Margara, 2012a, Cugola and Margara, 2012a,

Conclusion and future works

Given the rises in the rate and complexity of input events and the rises in the number of rules in modern CEP applications, horizontal scalability has proved essential for CEP systems. A scalable CEP system requires a good load balancing mechanism to adapt the system to dynamic changes to input events and rules. In this paper, we propose CCEP as a scalable CEP mechanism using adaptive load balancing. CCEP provides horizontal scalability for CEP using clustering of CEP engines. A CEP engine runs

Mohammad Ali Fardbastani is a Ph.D. candidate in the School of Computer Engineering of Iran University of Science and Technology (IUST). His research interests include Distributed Complex Event Processing (DCEP) systems, scalable distributed and parallel High Performance Computing (HPC), and real-time systems. He is experienced in system-level development of operating systems, virtualization, runtime systems, distributed adaptive resource sharing, and communication middleware for HPCs and

References (55)

  • G. Cugola et al.

    Deployment strategies for distributed complex event processing

    Computing

    (2013)
  • Cugola, G., Margara, A., Matteucci, M., Tamburrelli, G., 2014. Introducing uncertainty in complex event processing:...
  • M. Dayarathna et al.

    Scalable complex event processing on a notebook

  • M. Dayarathna et al.

    Recent advancements in event processing

    ACM Comput. Surv.

    (2018)
  • R. Dwarakanath et al.

    TrustCEP: adopting a trust-based approach for distributed complex event processing

  • R. Dwarakanath et al.

    Operator migration for distributed complex event processing in device-to-device based networks

  • O. Etzion et al.

    Event Processing in Action

    (2010)
  • S. Fathollahzadeh et al.

    Stateful complex event detection on event streams using parallelization of event stream aggregations and detection tasks

  • J. Fonseca et al.

    Doctoral symposium: a policy-based coordination architecture for distributed complex event processing in the internet of things, in: DEBS 2016

  • Y. Gong et al.

    Parallelized mobility-aware complex event processing

  • W.A. Higashino et al.

    CEPaaS: complex event processing as a service

  • M. Hirzel

    Partition and compose: parallel complex event processing

  • Isoyama, K., Kobayashi, Y., Sato, T., Kida, K., Yoshida, M., Tagato, H., 2012. A scalable complex event processing...
  • Y. Kobayashi et al.

    A complex event processing for large-scale M2M services and its performance evaluations

  • J. Krumeich et al.

    Event-driven business process management: where are we now?

    Bus. Process Manag. J.

    (2014)
  • Cited by (11)

    • SP-ant: An ant colony optimization based operator scheduler for high performance distributed stream processing on heterogeneous clusters

      2022, Expert Systems with Applications
      Citation Excerpt :

      As time passes, an exponential increase of data generated by IoT infrastructures such as wearable devices, traffic monitoring as well as real-time detection of complex patterns are witnessed (Fardbastani and Sharifi, 2019).

    • PARS: A parallel model for scaled processing of complex events

      2019, Journal of Systems and Software
      Citation Excerpt :

      The need for higher performance has triggered further research on distributed CEP systems that distribute detection logic over a network of operators, wherein individual operators have become a bottleneck and required operator parallelization. There have been different strategies (Fardbastani and Sharifi, 2019; Xiao et al., 2017; Mayer et al., 2017; Luthra et al., 2018) for parallelization of CEP. Generally, event query processing can take advantage of multiple compute nodes for parallelization of stateful operators by partitioning the input stream.

    • Self-adaptation on parallel stream processing: A systematic review

      2022, Concurrency and Computation: Practice and Experience
    • CLOSED: A Cloud-Edge Dynamic Collaborative Strategy for Complex Event Detection

      2022, Proceedings - IEEE International Conference on Web Services, ICWS 2022
    • STHITHIKA: Distributed complex event processing with query rewriting

      2021, MERCon 2021 - 7th International Multidisciplinary Moratuwa Engineering Research Conference, Proceedings
    View all citing articles on Scopus

    Mohammad Ali Fardbastani is a Ph.D. candidate in the School of Computer Engineering of Iran University of Science and Technology (IUST). His research interests include Distributed Complex Event Processing (DCEP) systems, scalable distributed and parallel High Performance Computing (HPC), and real-time systems. He is experienced in system-level development of operating systems, virtualization, runtime systems, distributed adaptive resource sharing, and communication middleware for HPCs and DCEPs. He received his B.Sc. and M.Sc. at University of Tehran.

    Mohsen Sharifi is a Professor of Software Engineering at the School of Computer Engineering of Iran University of Science and Technology. He directs a Distributed Systems research group and laboratory. His main interest is in the development of distributed systems, solutions, and applications, particularly for use in various fields of science. He has developed a high performance scalable cluster solution comprising any number of homogeneous or heterogeneous COTS computers for use in scientific applications requiring high performance, availability and scalability. The development of a true distributed operating system is on top of his wish list. He received his B.Sc., M.Sc. and Ph.D. in Computer Science from the Victoria University of Manchester in the United Kingdom in 1982, 1986, and 1990, respectively.

    View full text