Iterative Selection of Categorical Variables for Log Data Anomaly Detection

Abstract

Log data is a well-known source for anomaly detection in cyber security. Accordingly, a large number of approaches based on self-learning algorithms have been proposed in the past. Most of these approaches focus on numeric features extracted from logs, since these variables are convenient to use with commonly known machine learning techniques. However, system log data frequently involves multiple categorical features that provide further insights into the state of a computer system and thus have the potential to improve detection accuracy. Unfortunately, it is non-trivial to derive useful correlation rules from the vast number of possible values of all available categorical variables. Therefore, we propose the Variable Correlation Detector (VCD) that employs a sequence of selection constraints to efficiently disclose pairs of variables with correlating values. The approach also comprises of an online mode that continuously updates the identified variable correlations to account for system evolution and applies statistical tests on conditional occurrence probabilities for anomaly detection. Our evaluations show that the VCD is well adjustable to fit properties of the data at hand and discloses associated variables with high accuracy. Our experiments with real log data indicate that the VCD is capable of detecting attacks such as scans and brute-force intrusions with higher accuracy than existing detectors.

A Appendix

1.1 A.1 Threshold Parameter Selection

The filtering steps for correlations between variables and values presented in Sect. 4 make use of threshold parameters $\theta _{1}$-$\theta _{8}$ to narrow down the search space and select only those correlations that are likely to positively contribute to the detection of anomalies. This section investigates the influence of these threshold parameters on the resulting correlations and thereby supports the manual parameter selection process, in particular, by relating each parameter to specific properties of the data at hand. In the following, we first explain the generation of synthetic data for this evaluation and then describe our experiments.

Data. To measure the influence of thresholds on the correlation selection, it is necessary to control properties of the input data. Therefore, we generate synthetic data for our experiments. We use three variables $V_1$, $V_2$, and $V_3$, of which only $V_1$ and $V_2$ correlate with varying strength, and monitor the correlations found by the VCD for different threshold settings. We use values $\mathcal {V}_i = \left\{ 0, 1, ..., x \right\} , x \in \mathbb {N}$ for each variable and compute their occurrence probabilities as normalized geometric series. Equation 15 shows how the probabilities for values in $V_1$ and $V_3$ are computed, where $p_i = 1$ means that all values are equally likely to occur, and lower values mean that one or more values are dominating the probability distribution. Equation 16 shows how the conditional probabilities of values in $V_2$ given values from $V_1$ are computed. Thereby, $\rho$ specifies the correlation strength, i.e., larger values for $\rho$ indicate that the same values co-occur more frequently with each other, and $\zeta$ is a damping factor that reduces the correlation strength for larger $v_{i, j}$, i.e., higher values for $\zeta$ cause more co-occurrences between different values.

$$\begin{aligned} P\left( v_{i, j} \right) = \frac{p_i^j}{\sum _{j' = 0}^{\left| \mathcal {V}_i \right| } p_i^{j'}} \end{aligned}$$

(15)

$$\begin{aligned} P\left( v_{k, l} \mid v_{i, j} \right) = \frac{\left( 1 - \rho \right) ^{\left| j - l \right| } + \zeta ^{\left| \left| \mathcal {V}_i \right| - j \right| }}{\sum _{l' = 0}^{\left| \mathcal {V}_k \right| } \left( 1 - \rho \right) ^{\left| j - l' \right| } + \zeta ^{\left| \left| \mathcal {V}_i \right| - j \right| }} \end{aligned}$$

(16)

Figure 4 shows the co-occurrences of values from $V_1$ and $V_2$ for a sample configuration of $x = 9$, $p_1 = 0.7$, $\rho = 0.9$, and $\zeta = 0.4$. Due to the relatively strong correlation factor, most values in $V_1$ occur with the same value of $V_2$. The figure also shows that higher values of $V_1$ co-occur with more values of $V_2$ due to the damping factor, e.g., while $v_{1, 1}$ only occurs with four different values of $V_2$, $v_{1, 9}$ occurs with each value of $V_2$ at least once.

Fig. 4.

Value co-occurrences of damped correlation.

To evaluate the accuracy of the correlation selection procedure, we generate a ground truth of expected value correlations that contains all $v_{1, j} \leadsto v_{2, l}$ and $v_{2, l} \leadsto v_{1, j}$ that occur at least once in the data. We count correlations selected by the VCD and present in the ground truth as true positives (TP), correlations not present in the ground truth as false positives (FP), correlations missed by the VCD as false negatives (FN), and all other correlations as true negatives (TN). We use the F-score $F_1 = TP / \left( TP + 0.5 \cdot \left( FP + FN \right) \right)$ to measure the accuracy in the next section.

Fig. 5.

Influence of thresholds on accuracy of correlation selection.

Results. We first experiment with $\theta _{7}$, which is essential for selecting correlations that represent actual dependencies between the values and do not spuriously emerge from skewed value probability distributions. To analyze the relationship between $\theta _{7}$ and the correlation strength, we increase $\theta _{7}$ in steps of 0.05 and $\rho$ in steps of 0.1 in the range $\left[ 0, 1 \right]$ while leaving $p_1 = 0.7, p_3 = 0.7, \zeta =0.4$ constant, generate 10 data samples with 10000 events respectively as outlined in the previous section, and then compute the average F-score of these simulation runs. The results visualized in Fig. 5a show that weaker correlation strengths require $\theta _{7}$ to be sufficiently low to select all correct correlations and achieve the highest possible F-score of 1. However, setting $\theta _{7}$ to 0 causes a decrease of the F-score independent of the correlation strength. The reason for this is that correlations involving $V_3$ are not checked for dependency and are thus incorrectly selected, which increases the number of FP. We therefore conclude that $\theta _{7}$ should be set to a low, but non-zero value, e.g., 0.05. Note that the selection of $\theta _{7}$ is not affected by $\zeta$, since additional value co-occurrences only have little influence on the sum of variances as long as they are not dominating the distribution.

Threshold $\theta _{5}$ on the other hand relies on the total number of co-occurrences for a given value and is thus influenced by $\zeta$ in addition to $\rho$. Figure 5b shows the F-score for various combinations of $\theta _{5}$ and $\zeta$, while $\rho = 1$ is fixed. As expected, increasing values for $\zeta$ yield lower F-scores for a given $\theta _{5}$, because the number of distinct co-occurring values for any given value increases quickly (cf. Fig. 4). Accordingly, it is necessary to set $\theta _{5} \ge 1$ for $\zeta > 0.5$ to select any correlations. For $\zeta \le 0.5$, $\theta _{5}$ effectively steers the allowed number of distinct co-occurrences, e.g., for $\theta _{5} = 0.5$ at most 5 co-occurring values are allowed since $\left| \mathcal {V}_i \right| = 10, \forall i$.

We argue that the influence of other thresholds is trivial and therefore omit the plots for brevity. Table 4 shows a summary of all thresholds and the data properties with the highest influence on their selection. Note that $\theta _{8}$ is most influenced by $\theta _{5}$ and $\theta _{6}$ rather than a property of the input data, because these thresholds regulate the generation of value correlations that affect the selection criterion involving $\theta _{8}$. The table also provides default values that we identified as useful during our experiments and are used in the evaluations in Sect. 5.

These results indicate that the large number of parameters does not impede practical application of the VCD, since the thresholds are mostly independent from each other and allow to configure the correlation selection constraints specifically to counteract otherwise problematic properties of the data. For example, a high number of correlations involving many distinct values (i.e., $\left| \mathcal {V} \right|$ is large) or weakly correlated variables (i.e., $\rho$ is low) should be addressed by adjusting $\theta _{1}$ and $\theta _{7}$ accordingly to reduce the total number of correlations that are considered for anomaly detection as shown in Sect. 5.1.

Table 4. Dependencies and default values of thresholds.