Skip to main content

Advertisement

Springer Nature Link
Log in

Deep SARSA-based reinforcement learning approach for anomaly network intrusion detection system

  • Regular contribution
  • Published:
International Journal of Information Security Aims and scope Submit manuscript

Abstract

The growing evolution of cyber-attacks imposes a risk in network services. The search of new techniques is essential to detect and classify dangerous attacks. In that regard, deep reinforcement learning (DRL) is emerging both as a promising solution in various fields and an autonomous agent capable to interact with the environment and make decisions without the knowledge of human experts. In this work, we propose a deep reinforcement learning model that highlights the advantages of combining a SARSA-based reinforcement learning algorithm with a deep neural network for intrusion detection system. The main objective of our proposed deep SARSA model is to enhance the detection accuracy of modern and complex attacks in the network environment. We validated the performance of our method using two prominent benchmark including NSL-KDD and UNSW-NB15. By comparing it with various classic machine learning and deep learning approaches and other related published results, our experimental results show that the proposed approach outperforms the other models taking into consideration various metrics such as accuracy, recall, precision and F1-score.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

My manuscript has no associated data.

References

  1. Javaid, A., Niyaz, Q., Sun, W., Alam, M.: A deep learning approach for network intrusion detection system. In Proceedings of the 9th EAI International Conference on Bio-inspired Information and Communications Technologies formerly BIONETICS), (ICST (Institute for Computer (Sciences, Social-Informatics and Telecommunications Engineering), pp. 21–26 (2016)

  2. Qu, F., Zhang, J., Shao, Z., Qi, S.: An intrusion detection model based on deep belief network. In: Proceedings of the 2017 VI International Conference on Network Communication and Computing, pp. 97–101, December (2017)

  3. Lin, W.H., Lin, H.C., Wang, P., Wu, B.H., Tsai, J.Y.: Using convolutional neural networks to network intrusion detection for cyber threats. In: International Conference on Applied System Invention (ICASI), pp. 1107–111 (2018)

  4. Alavizadeh, H., Jang-Jaccard, J., Alavizadeh, H.: Deep Q-Learning based reinforcement learning approach for network intrusion detection. arXiv:2111.13978v1 [cs.CR] 27 Nov (2021)

  5. Frikha, M.S., Gammar, S.M., Lahmadi, A., et al.: Reinforcement and deep reinforcement learning for wireless internet of things: a survey. Comput. Commun. 178, 98–113 (2021)

    Article  Google Scholar 

  6. Spano, S., Cardarilli, G.C., Di Nunzio, L., et al.: An efficient hardware implementation of reinforcement learning: The q-learning algorithm. Ieee Access 7, 186340–186351 (2019)

    Article  Google Scholar 

  7. Li, H., Zhang, Q., Zhao, D.: Deep reinforcement learning-based automatic exploration for navigation in unknown environment. IEEE Trans. Neural Netw. Learn. Syst. 31(6), 2064–2076 (2019)

    Article  Google Scholar 

  8. Mnih, V., Kavukcuoglu, K., Silver, D. et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)

  9. Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  10. Zhao, D., Wang, H., Shao, K. et al.: Deep reinforcement learning with experience replay based on SARSA. In: IEEE Computational Intelligence (2017)

  11. Faisal, J., Dohyeun, K., et al.: An ensemble of a prediction and learning mechanism for improving accuracy of anomaly detection in network intrusion environments. Sustainability 13(18), 10057 (2021)

    Article  Google Scholar 

  12. Caminero, G., Lopez-Martin, M., Carro, B.: Adversarial environment reinforcement learning algorithm for intrusion detection. Comput. Netw. 159, 96–109 (2019). https://doi.org/10.1016/j.comnet.2019.05.013

    Article  Google Scholar 

  13. Naseer, S., Saleem, Y., Khalid, S., Bashir, M.K., Han, J., Iqbal, M.M., Han, K.: Enhanced network anomaly detection based on deep neural networks. IEEE Access 6, 48231–48246 (2018)

    Article  Google Scholar 

  14. Thomas, R., Pavithran, D.: A survey of intrusion detection models based on NSL-KDD data set. In: 2018 Fifth HCT Information Technology Trends (ITT), pp. 286–291 (2018)

  15. Dhanabal, L., Shantharajah, S.P.: A study on NSL-KDD dataset for intrusion detection system based on classification algorithms. Int. J. Adv. Res. in Comput. Commun. Eng. 4(6), 446–452 (2015)

    Google Scholar 

  16. da Costa, K.A.P., Papa, J.P., de Oliveira-Lisboa, C., Munoz, R., de Albuquerque, V.H.C.: Internet of things: a survey on machine learning-based intrusion detection approaches. Comput. Netw. 151, 147–157 (2019). https://doi.org/10.1016/j.comnet.2019.01.023

    Article  Google Scholar 

  17. Ikram, S.T., Cherukuri, A.K.: Improving accuracy of intrusion detection model using PCA and optimized SVM. J. Comput. Inf. Technol. 24(2), 133–148 (2016)

    Article  Google Scholar 

  18. Çavuşoğlu, Ü.: A new hybrid approach for intrusion detection using machine learning methods. Appl. Intell. 49, 2735–276 (2019)

    Article  Google Scholar 

  19. Berman, D.S., Buczak, A.L., Chavis, J.S., Corbett, C.L.: A survey of deep learning methods for cyber security. Information 10(4), 122 (2019). https://doi.org/10.3390/info10040122

    Article  Google Scholar 

  20. Ding, Y., Zhai, Y.: Intrusion detection system for NSL-KDD dataset using convolutional neural networks. In: Proceedings of the 2nd International Conference on Computer Science and Artificial Intelligence , pp. 81–85 (2018)

  21. Gurung, S., Ghose, M.K., Subedi, A.: Deep learning approach on network intrusion detection system using NSL-KDD dataset. Int. J. Comput. Netw. Inf. Secur. (IJCNIS) 3, 8–14 (2019)

    Google Scholar 

  22. Al-Turaiki, I., et al.: Anomaly-based network intrusion detection using bidirectional long short term memory and convolutional neural network. ISC Int. J. Inf. Secur. 12(3), 37–44 (2020)

    Google Scholar 

  23. Zhang, C., Ruan, F., Yin, L., Chen, X., Zhai, L., Liu, F.: A deep learning approach for network intrusion detection based on NSL-KDD dataset. In: 2019 IEEE 13th International Conference on Anti-counterfeiting, Security, and Identification (ASID), pp. 41–45, IEEE (2019)

  24. Servin, A.: Towards traffic anomaly detection via reinforcement learning and data flow. Department of Computer Science, University of York, United Kingdom (2007)

  25. Servin, A.: Multi-agent reinforcement learning for intrusion detection. Ph.D. thesis, University of York (2009)

  26. Blanco, R., Cilla, J.J., Briongos, S., Malagon, P., Moya, J.M.: Applying costsensitive classifiers with reinforcement learning to IDS. In: International Conference on Intelligent Data Engineering and Automated Learning, Springer, pp. 531–538 (2018)

  27. Nguyen, T.T., &Reddi, V.J.: Deep reinforcement learning for cyber security, arXiv:1906.05799 [cs.CR] (2019)

  28. Ma, X., Shi, W.: AESMOTE: Adversarial reinforcement learning with SMOTE for anomaly detection. In: IEEE Transactions on Network Science and Engineering. doi: https://doi.org/10.1109/TNSE.2020.3004312 (2020)

  29. Lopez-Martin, M., Carro, B., Sanchez-Esguevillas, A.: Application of deep reinforcement learning to intrusion detection for supervised problems. Expert Syst. Appl. 141, 112963 (2020)

    Article  Google Scholar 

  30. Moustafa, N., Slay, J.: UNSW-NB15: a comprehensive data set for network intrusion detection systems. MilCIS 2015

  31. Moustafa, N., Slay, J.: The evaluation of network anomaly detection systems: statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set. Inf. Secur. J.: Global Perspect. 25(1–3), 18–31 (2016)

    Google Scholar 

  32. Potluri, S., Ahmed, S., Diedrich, C.: Convolutional neural networks for multi‐class intrusion detection system. In: International Conference on Mining Intelligence and Knowledge Exploration, pp. 225–238. Springer (2018)

  33. Vinayakumar, R., et al.: Deep learning approach for intelligent intrusion detection system. IEEE Access 7, 41525–41550 (2019)

    Article  Google Scholar 

  34. Elmrabit, N. et al.: Evaluation of machine learning algorithms for anomaly detection. In 2020 International Conference on Cyber Security and Protection of Digital Services (Cyber Security). IEEE (2020)

  35. Kasongo, S.M., Sun, Y.: Performance analysis of intrusion detection systems using a feature selection method on the UNSW-NB15 dataset. J. Big Data 7(1), 1–20 (2020)

    Article  Google Scholar 

  36. Wang, Z.: Deep learning-based intrusion detection with adversaries. IEEE Access 6, 38367–38384 (2018)

    Article  Google Scholar 

  37. Yin, Y. et al.: IGRF-RFE: A hybrid feature selection method for MLP-based network intrusion detection on UNSW-NB15 dataset. arXiv preprint arXiv:2203.16365 (2022)

  38. T. Schaul et al.: Prioritized experience replay, arXiv:1511.05952v4 [cs.LG], (2016)

Download references

Acknowledgements

The authors would like to acknowledge the financial support of this work by grants from General Direction of Scientific Research (DGRST), Tunisia, under the ARUB program.

Author information

Authors and Affiliations

  1. Research Team in Intelligent Machines (RTIM), National Engineering School of Gabes, University of Gabes, 6072, Omar Ibn El Khattab, Avenue Zrig, Tunisia

    Safa Mohamed & Ridha Ejbali

Authors
  1. Safa Mohamed
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ridha Ejbali
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Safa Mohamed.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mohamed, S., Ejbali, R. Deep SARSA-based reinforcement learning approach for anomaly network intrusion detection system. Int. J. Inf. Secur. 22, 235–247 (2023). https://doi.org/10.1007/s10207-022-00634-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10207-022-00634-2

Keywords