Predictive complex event processing based on evolving Bayesian networks
Introduction
In the Big Data era users require new technology to process data stream with high speed and variety of data type. Besides the traditional stream processing technology, it is important to catch the relations inside online streaming data. Most of the data streams are composed of primitive events that produced by sensor networks, internet, social networks, etc. The semantic information inside primitive events is quite limited. In real application, people usually pay more attention to higher level information such as business logic and rules. For example, enormous events are generated in a trading system, but a fraud detecting system only care about the events that can cause frauds. Complex Event Processing (CEP) [1] is the technology that interprets and combines streams of primitive events to identify higher level composite events. CEP has been used in many areas, such as sensor networks for environmental monitoring, continuous analyzing of stocks to detect trends, etc.
Predictive complex event processing is the technology to identify events before they have happened, so that they can be eliminated or their effects mitigated [2]. For example, a financial institution wishes to detect frauds or a financial regulator wishes to catch illegal trading patterns in advance. Another example is to predict the traffic status in road networks and take some actions proactively to mitigate or eliminate undesired future states. Simple predictive CEP can be supported by rule-based method which means users define some patterns and the system then continuously monitor the event streams to predict future events. However, for complex cases it is not easy to define predictive CEP patterns exactly. In such circumstance, predictive analytics technology can be applied to support predictive CEP. Predictive analytics applies several statistical and data mining techniques such as clustering, classification, regression and so on. Recently, neural networks (especially deep neural networks) and Bayesian Networks (BN) are commonly used as prediction models. BN and it variations, such as Dynamic Bayesian Networks (DBN) [3], Adaptive Bayesian Networks (ABN) [4], etc., are widely used in predictive analytics because they have the following advantages: (i) they allow to express directly the fundamental qualitative relationship of direct causation, (ii) there exists mathematical methods to estimate the state of certain variables given the state of other variables, (iii) there are methods in order to explain to the user how the system came to its conclusions [5].
Currently predictive CEP with predictive analytics has some challenges. First, traditional predictive analytics methods are designed for database which means they assume all data is available at any time. However, in predictive CEP the system can only process data on single-pass and cannot control over the order of samples that arrive over time. Stream-based predictive analytics methods are needed which is more difficult. Second, the distribution of data can change over time. A model learned from historical data may not fit the new coming data well. This means real time modeling and learning algorithms are needed. In order to support predictive CEP with BN, an Evolving Bayesian Networks (EBN) model is needed that can adjust itself automatically when the distribution of data changes. There are some works on learning BN model incrementally but they simply assume all data is in memory and add new incoming data into existing data in each step. Finally, event streams often have high incoming rate, especially the event streams produced by wireless sensor networks. Predictive CEP is usually used to support online decision support system which need high performance even for large scale distributed event streams.
When used to handle the predictive CEP issue, the main weakness of the current predictive analytics methods is that they cannot get acceptable result due to the distribution drift of streaming event data. To address the challenges and overcome the weakness of current work, in this paper the authors propose a Predictive Complex Event Processing method based on Evolving Bayesian Networks (PCEP-EBN). The main contribution of this paper includes the following:
- -
The authors propose a predictive CEP framework based on BN model which uses Gaussian mixture model and EM algorithm for approximate inference.
- -
When learning and updating the BN structure, the authors propose incremental calculation methods for the score metric which support single-pass processing of event stream.
- -
The authors propose algorithms to support evolving BN structure based on hill-climbing method. The hill-climbing algorithm of this paper supports exploring multiple peaks on parallel and updating the score metric based on changed edges only. Evolving BN parameters is also supported using an incremental EM algorithm for Gaussian mixture model.
The authors evaluated the PCEP-EBN method in road traffic domain with both real application data and data produced by a simulated transportation system. The results show that PCEP-EBN is an effective method for predictive complex event processing and it outperforms other popular methods when processing traffic prediction in intelligent transportation systems.
Section snippets
Complex event processing and predictive analytics
CEP is a technique that recognizes complex events based on a set/sequence of occurrences of single events by continuously monitoring the event stream. Etzion et al. defined the basic concepts and architecture of complex event processing in their book [2]. The main component in CEP is Event Processing Agent (EPA) which can generate a set of complex events as output from a set of input events by applying some logic. Event Processing Network (EPN) is composed of a collection of EPAs, event
System architecture and basic CEP
The system architecture is shown in Fig. 1. Primitive event streams generated by various event sources are processed by CEP engine. The BEPA means basic event processing agent which recognize complex event based on patterns from event stream. Multiple EPAs are connected by event channels (ECs) to create an EPN. The skeleton in PEPA is a statistical summary of the dataset which contains all information necessary to calculate the scores of the Bayesian network. The predictive complex events
Bayesian network structure learning
Definition 5 evolving Bayesian network, EBN An evolving Bayesian network is a Bayesian network whose structure and parameters are changed according to the drift of data distribution. An EBN is represented by 〈G(t), Θ(t)〉 where G(t) and Θ(t) are the structure and parameters at time t respectively.
The EBN includes evolving of BN structure and parameters. For structure evolving, since search-and-score method is more suitable for evolving than constraint-based method, the authors use a search-and-score method with BDe metric [18] score
Evaluations of EBN
The authors evaluated the EBN method separately since it is the key part of this paper. The BN structure in Fig. 2 is a special structure and the authors can't find real data with corresponding result BN structures for accuracy evaluation. Therefore it is reasonable to draw samples from a known BN, apply structure learning on the synthetic data, and compare the learned structure with the original one. The original BN structure contains 783 event types (nodes) and 1752 edges. Since the authors
Conclusion and future work
In this paper the authors propose a predictive complex event processing method based on evolving Bayesian networks. The Bayesian model has two dimensions: event type and time. The Gaussian mixture model and EM algorithm are used as approximate method to infer the Bayesian model. This method supports calculating score metric incrementally. Bayesian network structure and parameters evolving algorithms are proposed. The evaluation results in road traffic domain with both real application data and
Acknowledgments
This project is supported by the “Study of Proactive Complex Event Processing for Large-scale Internet of Things” project of National Natural Science Foundation of China (61371116).
References (36)
- et al.
Predicting traffic flow using Bayesian networks
Transp. Res. Part B
(2008) - et al.
Causal network construction over event streams
Inf. Sci.
(2014) - et al.
Complex event processing over distributed probabilistic event streams
Comput. Math. Appl.
(2013) - et al.
Incremental partial least squares analysis of big streaming data
Pattern Recognit.
(2014) - et al.
Temporal variations of reference evapotranspiration and its sensitivity to meteorological factors in Heihe River Basin, China
Water Sci. Eng.
(2015) The Power of events: An Introduction to Complex Event Processing in Distributed Enterprise Systems
(2002)- et al.
Event Processing in Action
(2010) - et al.
Nonlinear networked control systems with random nature using neural approach and dynamic Bayesian networks
Int. J. Control, Autom. Syst.
(2008) - et al.
Adaptive Bayesian network for traffic flow prediction
Incremental hill-climbing search applied to Bayesian network structure learning
Event-processing network model and implementation
IBM Syst. J.
High-performance complex event processing over streams
Efficient pattern matching over event streams
Predictive complex event processing: a conceptual framework for combining complex event processing and predictive analytics
Complex event processing synergies with predictive analytics
Deep architecture for traffic flow prediction - deep belief networks with multitask learning
IEEE Trans. Intell. Transp. Syst.
Traffic flow prediction with big data: a deep learning approach
IEEE Trans. Intell. Transp. Syst.
A Bayesian network Model for traffic flow estimation using prior link flows
Cited by (26)
Few-shot node classification on attributed networks based on deep metric learning for Cyber–Physical–Social Services
2023, Pattern Recognition LettersAdaptive and personalized user behavior modeling in complex event processing platforms for remote health monitoring systems
2022, Artificial Intelligence in MedicineCitation Excerpt :Bayesian networks are one of the famous AI methods that can be used aligned with CEP. In [23], the authors proposed a complex predictive event processing method based on the evolution of the Bayesian networks. The Bayesian model is designed by an inference method based on the Gaussian mixed model and EM algorithm and based on event type and time.
Evolving Dynamic Bayesian Networks by an Analytical Threshold for Dealing with Data Imputation in Time Series Dataset
2022, Big Data ResearchCitation Excerpt :Nowadays in the Big Data Era, data are rapidly and continuously generated forming large datasets from many heterogeneous sources [1].
Towards a control-as-a-service architecture for smart environments
2021, Simulation Modelling Practice and TheoryCitation Excerpt :In a smart home scenario, this type of solution can detect the thermal discomfort of residents, either through temperature and humidity data or by the number of people occupying the place in a given time interval, and reacts by turning on an air-conditioning. A proactive CEP can identify, eliminate, or mitigate the effects of future events by applying forecasting and decision techniques using machine learning, data mining, or other approaches [22,26]. This type of application allows, for example, to detect that the consumption of electricity in a smart home will exceed the defined limits to ensure the comfort of residents.
Rules engine and complex event processor in the context of internet of things for precision agriculture
2018, Computers and Electronics in AgricultureCitation Excerpt :CEP is a technology that allows processing, analyze and correlate large amounts of heterogeneous data in the form of events to detect patterns with direct implications in making decisions relevant to a particular domain (Boubeta, 2014). According Wang et al. (2018), CEP interprets and combines flows of primitive events, captured in most cases in real time, to identify composite events of higher level, known as event patterns or complex events. In addition, a CEP allows the processing of data flows from multiple data sources (Akila et al., 2016) and interprets a set of rules written in an ad hoc language for defining rules (Cugola and Margara, 2012).
MARTIN: An End-To-end Microservice Architecture for Predictive Maintenance in Industry 4.0
2023, Proceedings - 2023 IEEE International Conference on Software Services Engineering, SSE 2023