One IDS Is Not Enough! Exploring Ensemble Learning for Industrial Intrusion Detection

Abstract

Industrial Intrusion Detection Systems (IIDSs) play a critical role in safeguarding Industrial Control Systems (ICSs) against targeted cyberattacks. Unsupervised anomaly detectors, capable of learning the expected behavior of physical processes, have proven effective in detecting even novel cyberattacks. While offering decent attack detection, these systems, however, still suffer from too many False-Positive Alarms (FPAs) that operators need to investigate, eventually leading to alarm fatigue. To address this issue, in this paper, we challenge the notion of relying on a single IIDS and explore the benefits of combining multiple IIDSs. To this end, we examine the concept of ensemble learning, where a collection of classifiers (IIDSs in our case) are combined to optimize attack detection and reduce FPAs. While training ensembles for supervised classifiers is relatively straightforward, retaining the unsupervised nature of IIDSs proves challenging. In that regard, novel time-aware ensemble methods that incorporate temporal correlations between alerts and transfer-learning to best utilize the scarce training data constitute viable solutions. By combining diverse IIDSs, the detection performance can be improved beyond the individual approaches with close to no FPAs, resulting in a promising path for strengthening ICS cybersecurity.

A WADI Results

In addition to the results of our experiments from Sect. 3, and Sect. 4, based on the SWaT dataset, we repeated the same analyses for the WADI dataset.

Table 5. No IIDS detects all attacks on WADI, and there is no single best detector that excels in all metrics. Also, the best IIDS for each metric differs from SWaT.

In the baseline results, we observe a similar insufficiency for WADI (cf. Table 5) as previously discussed for SWaT in Sect. 3.1. No single IIDS is capable of detecting all 14 cyberattacks, and there is no single best IIDS for all metrics. MinMax achieves the highest score in two metrics, but Seq2SeqNN detects two more scenarios. In total 13 of WADI’s 14 attacks would be detectable in combination.

Table 6. Weight-based ensembles yield similar results on WADI as on SWaT (cf. Table 2) and can outperform each base-IIDS in eTaF1. While they have the potential to improve upon the base-IIDS, finding suitable weights is again non-trivial.

Table 7. As for SWaT (cf. Table 3), temporal knowledge improves the individual IIDSs’ alerting behavior (upper part) and the ensembles’ performance (lower part).

We again assess the theoretical and practical potential for an ensemble on WADI as described in Sect. 4.1. Hereby, the eTaF1-optimized ensemble outperforms the best base-IIDS by $+11.9\%$ points in the eTaF1 score, detects more cyberattacks than any IIDS, and keeps the FPAs comparatively low (cf. upper part of Table 6). The manual voting strategies, however, fall short of this theoretical potential (cf. lower part of Table 6) with the best strategy lacking $-25\%$ points behind in the eTaF1 score. Nonetheless, the $\ge$2-Alerts ensemble indicates all 13 cyberattacks detected by any base-IIDS while maintaining only four FPAs.

Incorporating temporal knowledge (cf. Sect. 5) enhances the optimum by $+2.5\%$ points in eTaF1 and improves the manual strategies, especially in FPAs (cf. Table 7). We see a substantial improvement in the eTaF1 score of the best manual vote ($+5.8\%$ points for $\ge$5-Alerts), and the $\ge$2-Alerts strategy still detects 13 cyberattacks, now with just 2 FPAs, matching the number of FPAs achieved by Opt. Weights. Unfortunately, this great result is not expressed by eTaF1.

These results support the previous conclusion that weight-based ensembles are useful given suitable weights and thresholds, yet finding them remains non-trivial. Lastly, adding time-awareness yielded a significant performance boost.