Inter-dataset generalization strength of supervised machine learning methods for intrusion detection
Introduction
Intrusion detection is a cornerstone of cybersecurity and an active field of research since the 1980s. Although the early research focused more on host intrusion detection systems (HIDS), the principal aims of an intrusion detection system (IDS) have not changed. A well-functioning IDS should be able to detect a wide range of intrusions, possibly in real-time, with high discriminating power, improving itself through self-learning, while being modifiable in its design and execution [10]. The advent of computer networking and its ever greater adoption, shifted part of the research away from HIDS to network intrusion detection systems (NIDS).
This article details two experiments, starting with the analysis of two modern intrusion detection datasets CIC-IDS2017 and CSE-CIC-IDS2018 by means of supervised machine learning. The second part is an investigation into the generalizing capabilities of these supervised learners by exposing pre-trained models to unseen attack data from the other dataset. The first part is self-evidently useful, because it is a necessary prerequisite to start the second part. That second part is more innovative and focuses on a research problem that has largely been ignored by intrusion detection researchers. Much effort and resources are spent on improving classification results within data sets (mostly through algorithmic tweaks, ensembles or data preprocessing techniques). Although this is an essential part, those articles conclude with the implicit assumption that an increase in classification performance within the data set, will transfer into a better IDS in real-world scenarios. Omitting to investigate how well the new algorithms perform on unseen attack data, is very likely due to the lack of same-feature, labeled, publicly available data sets. The sparse landscape of good data sets has fairly recently been criticized in [19] and [7]. Those publications predate the CICIDS collection which for the first time offers researchers the possibility to test trained models on new, albeit similar, data. Hopefully, as the IDS data generation matures further, from one-time efforts to dynamic generation, generalization testing should become standard practice.
The article is structured as follows. First an overview of the related work in intrusion detection and network security dataset generation is given, then the implementation of the analysis is described (Section 3). The fourth part contains the discussion of the results obtained on both datasets with recommendations for algorithms based on raw classification performance and run time characteristics (Section 4). The fifth Section 5 details the results of the second experiment that looks into generalization strength of the tested algorithms.
Key findings in this work are the outstanding performance both in terms of classification and time metrics of tree-based classifiers, especially ensemble learners and the surprising effectiveness of simple distance-based methods on both datasets. These classification results are however not likely to be good proxies for real-world performance of supervised learners of the tested families in intrusion detection. This claim can be made because the methods fail to generalize correctly even under the most favorable conditions.
Section snippets
Intrusion detection
The field of network intrusion detection developed two main approaches to solve the problem of determining whether observed traffic is legitimate. The chronologically first approach is the use of signature-based systems (also called misuse detection systems). Within this category different strategies have been studied [2], including state modelling, string matching, simple rule-based systems and expert systems (emulating human expert knowledge, by applying an inference system on a knowledge
Architecture and implementation
The evaluation of this dataset is a project in Python, supported by the Pandas [22], Sklearn [26] and XGboost [5] modules. The following subsections detail the engineering effort and choices to produce a robust, portable solution to evaluate any dataset. Google’s guide [38], Rules of Machine Learning: Best Practices for ML Engineering, by Martin Zinkevich, has been influential on the implementation (mainly rules 2, 4, 24, 25, 32 and 40), as well as the detailed guides offered by Scikit-Learn.
Evaluation results
This section describes the results from the retesting with optimized parameters. In total 12 algorithms were tested. It should be noted that for two of these no cross-validation was done. The N-centroid classifier does not use optimized parameters, due to a limitation of Scikit-learn. For the RBF-SVC classifier, parameter optimization was skipped due to the excessive run times of forced single-core execution. The results are described in their respective algorithmic classes in subsections 4.1
Model generalization strength
As a practical defensive tool, intrusion detection should ideally be tested in the real world. Experiments that push the boundaries in this regard are very rare [27], [34]. Equally uncommon is the adoption of machine learning based solutions into commercial IDSs. This split between the research community that studies intrusion detection through machine learning methods and the real-world adoption of systems built on the research findings has been described most succinctly in [31]. The authors
Conclusions and future work
This article contains detailed experiment results on CIC-IDS2017 and CSE-CIC-IDS2018, two modern data sets geared towards the application of machine learning to network intrusion detection systems. The design and implementation have been laid out in Section 3, focusing on the principles and application of solid machine learning engineering.
The baseline result section of this article (4) conveys the results of applying twelve supervised learning algorithms with optimized parameters to the data.
CRediT authorship contribution statement
Laurens D’hooge: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing - original draft, Writing - review & editing, Visualization. Tim Wauters: Conceptualization, Writing - review & editing, Supervision, Project administration. Bruno Volckaert: Conceptualization, Writing - review & editing, Supervision, Funding acquisition. Filip De Turck: Resources, Supervision, Funding acquisition.
Declaration of Competing Interest
The authors declare that they do not have any financial or nonfinancial conflict of interests
References (38)
- et al.
Ensemble based collaborative and distributed intrusion detection systems: A survey
J Netw Comput Appl
(2016) - et al.
Deep learning approach for IDS
Fourth international congress on information and communication technology
(2020) - et al.
Outside the closed world: on using machine learning for network intrusion detection
2010 IEEE Symposium on Security and Privacy
(2010) - et al.
A novel intrusion detector based on deep learning hybrid methods
2019 IEEE 5th intl conference on big data security on cloud (BigDataSecurity), IEEE Intl Conference on high performance and smart computing,(HPSC) and IEEE Intl conference on intelligent data and security (IDS)
(2019) - et al.
A deep learning approach for intrusion detection using recurrent neural networks
IEEE Access
(2017) - Attak H., Combalia M., Gardikis G., Gastón B., Jacquin L., Litke A., et al. Application of distributed computing and...
Intrusion detection systems: a survey and taxonomy
Tech. Rep.
(2000)- et al.
Analysis of the 1999 darpa/lincoln laboratory ids evaluation data with netadhict
Computational intelligence for security and defense applications, 2009. CISDA 2009. IEEE symposium on
(2009) - et al.
A survey of data mining and machine learning methods for cyber security intrusion detection
IEEE Commun Surv Tutor
(2016) - et al.
Xgboost: a scalable tree boosting system
CoRR
(2016)
Generation of a new ids test dataset: Time to retire the kdd collection
2013 IEEE wireless communications and networking conference (WCNC)
Ddos intrusion detection through machine learning ensemble
2019 IEEE 19th international conference on software Quality, Reliability and Security Companion (QRS-C)
A holistic approach for detecting ddos attacks by using ensemble unsupervised machine learning
Requirements and model for IDES-a real-time intrusion-detection expert system
An adaptive ensemble machine learning model for intrusion detection
IEEE Access
Challenging the anomaly detection paradigm: A provocative discussion
Proceedings of the 2006 Workshop on New Security Paradigms
Bl-ids: Detecting web attacks using bi-lstm model based on deep learning
International conference on security and privacy in new computing environments
Sshcure: a flow-based ssh intrusion detection system
IFIP international conference on autonomous infrastructure, management and security
Cited by (61)
Paying attention to cyber-attacks: A multi-layer perceptron with self-attention mechanism
2023, Computers and SecurityGeneralizing intrusion detection for heterogeneous networks: A stacked-unsupervised federated learning approach
2023, Computers and SecurityHDFEF: A hierarchical and dynamic feature extraction framework for intrusion detection systems
2022, Computers and SecurityCitation Excerpt :Focusing on feature selection In Das et al. (2021), the most competitive feature selection (FS) method was discovered from a large number of well-known FS methods using the ensemble feature selection technique and then integrated with an extended version of the ensemble learning framework. D’hooge et al. (2020) verified that XGBoost performs best in tree-based classifiers by using the feature scaling method after normalizing the feature space of all attributes. In PBCNN (Yu et al., 2021), the hierarchy of network flow is exploited.
Robust Botnet Detection Approach for Known and Unknown Attacks in IoT Networks Using Stacked Multi-classifier and Adaptive Thresholding
2024, Arabian Journal for Science and EngineeringA hybrid ensemble machine learning model for detecting APT attacks based on network behavior anomaly detection
2023, Concurrency and Computation: Practice and Experience