Abstract
The evolving behavior of the attacks may affect the decision boundaries of the trained machine learning models. The issue has not been well investigated, especially with hypervisor-based security solutions where virtual machine (VM)’s network artifacts are introspected and analyzed. In this paper, we proposed a sustainable and explainable flow-filtering-based concept drift-driven network intrusion detection approach, called ‘SFC-NIDS’ which introspects network activities by analyzing VM traffic profile. The VM traffic is captured and pre-processed at the hypervisor to extract important network artifacts. The redundant and trivial network flows have been filtered using the proposed gradient descent-based flow filtering mechanism and validated using explainability. SFC-NIDS employs auto-encoders to reconstruct the traffic features to capture additional patterns. Afterward, the 1D-convolution neural network has been employed to learn and detect malicious attack flows. The model’s sustainability is ensured by integrating the drift detection mechanism with the decision model to retrain it with evolving attack patterns. The approach has been validated with virtual network traffic artifacts collected at the hypervisor and provides 98.9% accuracy, 99.03%, and F1-Score. In addition, the approach has also been validated using the KDD99 dataset, showcasing an accuracy of 99.97% and an F1-Score of 99.98%.
Similar content being viewed by others
Data availability
The data will be made available on request.
Notes
References
Ahmad, Z., Shahid Khan, A., Wai Shiang, C., Abdullah, J., Ahmad, F.: Network intrusion detection system: a systematic study of machine learning and deep learning approaches. Trans. Emerg. Telecommun. Technol. 32(1), 1–12 (2021)
Du, J., Yang, K., Hu, Y., Jiang, L.: NIDS-CNNLSTM: network intrusion detection classification model based on deep learning. IEEE Access 11, 24808–24821 (2023)
Mishra, P., Varadharajan, V., Tupakula, U., Pilli, E.S.: A detailed investigation and analysis of using machine learning techniques for intrusion detection. IEEE Commun. Surv. Tutor. 21(1), 686–728 (2018)
Srinivas, B., Mandal, I., Keshavarao, S.: Virtual machine migration-based intrusion detection system in cloud environment using deep recurrent neural network. Cybern. Syst. (2022). https://doi.org/10.1080/01969722.2022.2122008
Sakr, M.M., Tawfeeq, M.A., El-Sisi, A.B.: Network intrusion detection system based PSO-SVM for cloud computing. Int. J. Comput. Netw. Inf. Secur. 11(3), 22–29 (2019)
Leon, R.S., Kiperberg, M., Leon Zabag, A.A., Zaidenberg, N.J.: Hypervisor-assisted dynamic malware analysis. Cybersecurity 4(1), 1–14 (2021)
Mishra, P., Pilli, E.S., Varadharajan, V., Tupakula, U.: Out-VM monitoring for malicious network packet detection in cloud. In: 2017 ISEA Asia Security and Privacy Conference, ISEASP 2017, Surat, India, pp. 1–10 (2017)
Mittal, A., Dua, M.: Static-dynamic features and hybrid deep learning models based spoof detection system for ASV. Complex Intell. Syst. 8(2), 1153–1166 (2022)
Muallem, A., Shetty, S., Pan, J., Zhao, J., Biswal, B.: Hoeffding tree algorithms for anomaly detection in streaming HTBPS: a survey. J. Inf. Secur. 08, 339–361 (2017)
Seraphim, I., Eswaran, P.: Analysis on intrusion detection system using machine learning techniques. In: Computer Networks, Big Data and IoT, pp. 423–441. Springer, Singapore (2021)
Pradheep, D., Gokul, R., Naveen, V., Vijayarani, J.: Anomaly intrusion detection based on concept drift. Glob. J. Comput. Sci. Technol. 20(E2), 1–12 (2020)
Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., Zhang, G.: Learning under concept drift: a review. IEEE Trans. Knowl. Data Eng. 31(12), 2346–2363 (2018)
Ashiku, L., Dagli, C.: Network intrusion detection system using deep learning. Procedia Comput. Sci. 185, 239–247 (2021)
Kumar, D., Pateriya, R.K., Gupta, R.K., Dehalwar, V., Sharma, A.: Ddos detection using deep learning. Procedia Comput. Sci. 218, 2420–2429 (2023)
Kiranyaz, S., Avci, O., Abdeljaber, O., Ince, T., Gabbouj, M., Inman, D.J.: 1d convolutional neural networks and applications: a survey. Mech. Syst. Signal Process. 151, 1–21 (2021)
Bifet, A., Gavaldà, R.: Learning from time-changing data with adaptive windowing. In: 7th SIAM International Conference on Data Mining, vol. 7, pp. 1–17 (2007)
Seth, S., Singh, G., Chahal, K.: Drift-based approach for evolving data stream classification in intrusion detection system. In: WCNC-2021: Workshop on Computer Networks & Communications, Chennai, India pp. 23–30 (2021)
Shaji, N.S., Muthalagu, R., Pawar, P.M.: SD-IIDS: intelligent intrusion detection system for software-defined networks. Multimedia Tools Appl 83(4), 11077–11109 (2023)
Priya, S., Uthra, R.A.: Deep learning framework for handling concept drift and class imbalanced complex decision-making on streaming data. Complex Intell. Syst. 9, 3499–3515 (2021)
Yuan, X., Wang, R., Zhuang, Y., Zhu, K., Hao, J.: A concept drift based ensemble incremental learning approach for intrusion detection. In: 2018 IEEE International Conference on Internet of Things (IThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), pp. 350–357. IEEE (2018)
Andresini, G., Appice, A., Loglisci, C., Belvedere, V., Redavid, D., Malerba, D.: A Network Intrusion Detection System for Concept Drifting Network Traffic Data, pp. 111–121. Springer, Cham (2021)
Kuppa, A., Le-Khac, N.-A.: Learn to adapt: Robust drift detection in security domain. Comput. Electr. Eng. 102, 1–13 (2022)
Andresini, G., Pendlebury, F., Pierazzi, F., Loglisci, C., Appice, A., Cavallaro, L.: SOMNIA: towards concept-drift robustness in network intrusion detection. In: Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security (AISec), pp. 111–122. ACM, New York (2021)
Nautiyal, A., Saklani, S., Pant, A., Agarwal, Y., Gaur, A., Mishra, P.: VNSecure: an explainable virtual network attack detection. In: IC3-2023: Proceedings of the 2023 Fifteenth International Conference on Contemporary Computing, pp. 153–160 (2023)
Horchulhack, P., Viegas, E.K., Lopez, M.A.: A stream learning intrusion detection system for concept drifting network traffic. In: 2022 6th Cyber Security in Networking Conference (CSNet), pp. 1–7. IEEE (2022)
Jain, M., Kaur, G.: Distributed anomaly detection using concept drift detection based hybrid ensemble techniques in streamed network data. Clust. Comput. 24, 2099–2114 (2021)
Patil, R., Dudeja, H., Modi, C.: Designing an efficient security framework for detecting intrusions in virtual network of cloud computing. Comput. Secur. 85, 402–422 (2019)
Rajeswari, P.V.N., Shashi, M., Rao, T.K., Rajya Lakshmi, M., Kiran, L.V.: Effective intrusion detection system using concept drifting data stream and support vector machine. Concurr. Comput. Pract. Exp. 34(21), 7118 (2022)
Hinder, F., Vaquet, V., Brinkrolf, J., Hammer, B.: Model-based explanations of concept drift. Neurocomputing 555, 126640 (2023). https://doi.org/10.1016/j.neucom.2023.126640
Liu, W., Zhu, C., Ding, Z., Zhang, H., Liu, Q.: Multiclass imbalanced and concept drift network traffic classification framework based on online active learning. Eng. Appl. Artif. Intell. 117, 105607 (2023). https://doi.org/10.1016/j.engappai.2022.105607
Coppolino, L., D’Antonio, S., Nardone, R., Romano, L.: A self-adaptation-based approach to resilience improvement of complex internets of utility systems. Environ. Syst. Decis. 3, 708–720 (2023)
Wankhade, K.K., Jondhale, K.C., Dongre, S.S.: A clustering and ensemble based classifier for data stream classification. Appl. Soft Comput. 102, 107076 (2021)
Jain, M., Kaur, G., Saxena, V.: A k-means clustering and svm based hybrid concept drift detection technique for network anomaly detection. Expert Syst. Appl. 193, 1–18 (2022)
Chisnall, D.: The Definitive Guide to the Xen Hypervisor, 1st edn. Prentice Hall Press, Upper Saddle (2007)
Zhang, C., Yao, M., Chen, W., Zhang, S., Chen, D., Wu, Y.: Gradient descent optimization in deep learning model training based on multistage and method combination strategy. Secur. Commun. Netw. 2021, 1–15 (2021)
Alzubaidi, L., Zhang, J., Humaidi, A.J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M.A., Al-Amidie, M., Farhan, L.: Review of deep learning: concepts, cnn architectures, challenges, applications, future directions. J. Big Data 8, 1–74 (2021)
Bottou, L.: Stochastic gradient descent tricks. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade, pp. 421–436. Springer, Berlin (2012)
Ruder, S.: An overview of gradient descent optimization algorithms. arXiv preprint (2016). arXiv:1609.04747
Bank, D., Koenigstein, N., Giryes, R.: Autoencoders. Machine Learning for Data Science Handbook: Data Mining and Knowledge Discovery Handbook, pp. 353–374. Springer, Berlin (2023)
Sarhan, M., Layeghy, S., Portmann, M.: Evaluating standard feature sets towards increased generalisability and explainability of ml-based network intrusion detection. Big Data Res. 30(C), 1–9 (2022)
Santos, C.F.G.D., Papa, J.P.: Avoiding overfitting: a survey on regularization methods for convolutional neural networks. ACM Comput. Surv. (CSUR) 54(10s), 1–25 (2022)
Ghosh, S., Das, N., Nasipuri, M.: Reshaping inputs for convolutional neural network: Some common and uncommon methods. Pattern Recogn. 93, 79–94 (2019)
Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint, pp. 1–9 (2014). arXiv:1412.6806
Agrahari, S., Singh, A.K.: Concept drift detection in data stream mining: a literature review. J. King Saud Univ. Comput. Inf. Sci. 34(10, Part B), 9523–9540 (2022)
Huang, D.T.J., Koh, Y.S., Dobbie, G., Bifet, A.: Drift detection using stream volatility. In: Machine Learning and Knowledge Discovery in Databases, pp. 417–432. Springer, Cham (2015)
Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS’17, pp. 4768–4777. Curran Associates, Red Hook (2017)
Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you? explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)
Joshi, L.M., Kumar, M., Bharti, R.: Understanding threats in hypervisor, its forensics mechanism and its research challenges. Int. J. Comput. Appl. 119(1), 1–5 (2015)
Deylami, H., Gutierrez, J., Sinha, R.: Kororā: a secure live virtual machine job migration framework for cloud systems integrity. Array 19, 100312 (2023)
Ortega-Fernandez, I., Sestelo, M., Burguillo, J.C., Pinon-Blanco, C.: Network intrusion detection system for DDoS attacks in ICS using deep autoencoders. Wirel. Netw. (2023). https://doi.org/10.1007/s11276-022-03214-3
Elmasry, W., Akbulut, A., Zaim, A.H.: A design of an integrated cloud-based intrusion detection system with third party cloud service. Open Comput. Sci. 11(1), 365–379 (2021)
Arun kumar, M., Ashok kumar, K.: Malicious attack detection approach in cloud computing using machine learning techniques. Soft. Comput. 26(23), 13097–13107 (2022)
Shlens, J.: Notes on kullback-leibler divergence and likelihood. arXiv preprint (2014). arXiv:1404.2000
Acknowledgements
The authors would like to express their gratitude to the Science and Engineering Research Board, Department of Science and Technology (SERB-DST) for their intellectual generosity and research assistance.
Funding
This work is supported by SERB-POWER Grant [File No. SPG/2021/002003] and SERB-POWER Mobility Grant [File No. SPM/2022/000004] under Science and Engineering Research Board, Department of Science and Technology (SERB-DST), Govt. of India.
Author information
Authors and Affiliations
Contributions
Arjun Singh: Software, validation, investigation, data curation, writing—-original draft, writing—review and editing. Preeti Mishra: Conceptualization, investigation, supervision, writing—original draft, writing—review and editing. Vinod P.: Conceptualization, investigation, supervision, writing—original draft, writing—review and editing. Avantika Gaur: Software, validation, investigation, writing—original draft, writing—review and editing. Mauro Conti: Supervision, review and editing.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Ethical approval
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Singh, A., Mishra, P., Vinod, P. et al. SFC-NIDS: a sustainable and explainable flow filtering based concept drift-driven security approach for network introspection. Cluster Comput 27, 9757–9782 (2024). https://doi.org/10.1007/s10586-024-04444-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10586-024-04444-0