Monitoring Anomalies in IT-Landscapes Using Clustering Techniques and Complex Event Processing

Abstract

Monitoring the behavior of IT-landscapes is the basis for the detection of breaches of non-functional requirements like security. Established methods, such as signature-based monitoring extract features from data instances and compare them to features of the signature database. However, signature-based monitoring techniques have an intrinsic limitation concerning unseen instances of aberrations (or attacks) because new instances have features which are not yet recognized in the signature database. Therefore, anomaly detection has been introduced to automatically detect non-conforming patterns in data. Unfortunately, it is often prohibitively hard to attain labeled training data to employ supervised-learning based approaches. Hence, the application of nonsupervised techniques such as clustering became popular. In this paper, we apply complex event processing rules and clustering techniques leveraging models of an IT-landscape considering workflows, services, and the network infrastructure to detect abnormal behavior. The service and infrastructure layer both have events on their own. Sequences of service events are well-defined, represent a workflow and are counter-checked via complex event processing rules. These service events however trigger infrastructure events, like database activity, and network traffic, which are not modeled. These infrastructure events are then related to the appropriate call traces and clustered among network profiles and database profiles. Outlying service events, nodes, and workflows are detected based on measured deviations to clusters. We present the main properties of our clustering-based anomaly detection approach and relate it to other techniques.

This work is supported by QE LaB - Living Models for Open Systems (FFG 822740), COSEMA - funded by the Tiroler Zukunftsstiftung, and SECTISSIMO (P-20388) FWF project.