Skip to main content
Springer Nature Link
Log in

Data-Driven Evaluation of Intrusion Detectors: A Methodological Framework

  • Conference paper
  • First Online:
Foundations and Practice of Security (FPS 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13877))

Included in the following conference series:

  • 694 Accesses

Abstract

Intrusion detection systems are an important domain in cybersecurity research. Countless solutions have been proposed, continuously improving upon one another. Yet, and despite the introduction of distinct approaches, including machine-learning methods, the evaluation methodology has barely evolved.

In this paper, we design a comprehensive evaluation framework for Machine Learning (ML)-based intrusion detection systems (IDS) and take into account the unique aspects of ML algorithms, their strengths and weaknesses. The framework design is inspired by both i) traditional IDS evaluation methods and ii) recommendations for evaluating ML algorithms in diverse application areas. Data quality being the key to machine learning, we focus on data-driven evaluation by exploring data-related issues. Our approach goes beyond evaluating intrusion detection performance (also known as effectiveness) and aims at proposing standard data manipulation methods to tackle robustness and stability. Finally, we evaluate our framework through a qualitative comparison with other IDS evaluation approaches from the state of the art.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    We believe however that the term “anomaly-based IDS” should solely apply to IDS trained on normal traffic only.

References

  1. Abbas, A., Khan, M.A., Latif, S., Ajaz, M., Shah, A.A., Ahmad, J.: A new ensemble-based intrusion detection system for internet of things. Arab. J. Sci. Eng. 47(2), 1805–1819 (2022). https://doi.org/10.1007/s13369-021-06086-5

    Article  Google Scholar 

  2. Abdelmoumin, G., Whitaker, J., Rawat, D.B., Rahman, A.: A survey on data-driven learning for intelligent network intrusion detection systems. Electronics 11(2), 213 (2022)

    Article  Google Scholar 

  3. Al-Qatf, M., Lasheng, Y., Al-Habib, M., Al-Sabahi, K.: Deep learning approach combining sparse autoencoder with SVM for network intrusion detection. IEEE Access 6, 52843–52856 (2018)

    Article  Google Scholar 

  4. Alrawashdeh, K., Purdy, C.: Toward an online anomaly intrusion detection system based on deep learning. In: 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 195–200 (2016)

    Google Scholar 

  5. Axelsson, S.: The base-rate fallacy and the difficulty of intrusion detection. ACM Trans. Inf. Syst. Secur. (TISSEC) 3(3), 186–205 (2000)

    Article  Google Scholar 

  6. Aygun, R.C., Yavuz, A.G.: Network anomaly detection with stochastically improved autoencoder based models. In: 2017 IEEE 4th International Conference on Cyber Security and Cloud Computing (CSCloud), pp. 193–198 (2017)

    Google Scholar 

  7. Bekkar, M., Djemaa, H.K., Alitouche, T.A.: Evaluation measures for models assessment over imbalanced data sets. J. Inf. Eng. Appl. 3(10), 27–38 (2013)

    Google Scholar 

  8. Bermúdez-Edo, M., Salazar-Hernández, R., Díaz-Verdejo, J., García-Teodoro, P.: Proposals on assessment environments for anomaly-based network intrusion detection systems. In: Lopez, J. (ed.) CRITIS 2006. LNCS, vol. 4347, pp. 210–221. Springer, Heidelberg (2006). https://doi.org/10.1007/11962977_17

    Chapter  Google Scholar 

  9. Bronzino, F., Schmitt, P., Ayoubi, S., Kim, H., Teixeira, R.C., Feamster, N.: Traffic refinery. Proc. ACM Meas. Anal. Comput. Syst. 5, 1–24 (2021)

    Article  Google Scholar 

  10. Cárdenas, A., Baras, J., Seamon, K.: A framework for the evaluation of intrusion detection systems. In: 2006 IEEE Symposium on Security and Privacy (S &P’06), pp. 15–77 (2006)

    Google Scholar 

  11. Chalapathy, R., Chawla, S.: Deep learning for anomaly detection: a survey (2019)

    Google Scholar 

  12. Charmet, F., et al.: Explainable artificial intelligence for cybersecurity: a literature survey. Ann. Telecommun. 77, 789–812 (2022). https://doi.org/10.1007/s12243-022-00926-7

    Article  Google Scholar 

  13. Gao, N., Gao, L., Gao, Q., Wang, H.: An intrusion detection model based on deep belief networks. In: 2014 Second International Conference on Advanced Cloud and Big Data, pp. 247–252 (2014)

    Google Scholar 

  14. García Cordero, C., Hauke, S., Mühlhäuser, M., Fischer, M.: Analyzing flow-based anomaly intrusion detection using replicator neural networks. In: 2016 14th Annual Conference on Privacy, Security and Trust (PST), pp. 317–324 (2016)

    Google Scholar 

  15. Gharib, A., Sharafaldin, I., Lashkari, A.H., Ghorbani, A.A.: An evaluation framework for intrusion detection dataset. In: 2016 International Conference on Information Science and Security (ICISS), pp. 1–6. IEEE (2016)

    Google Scholar 

  16. Goncalves, A., Ray, P., Soper, B., Stevens, J., Coyle, L., Sales, A.P.: Generation and evaluation of synthetic patient data. BMC Med. Res. Methodol. 20(1), 108 (2020)

    Article  Google Scholar 

  17. Gu, G., Fogla, P., Dagon, D., Lee, W., Skorić, B.: Measuring intrusion detection capability: an information-theoretic approach. In: Proceedings of the 2006 ACM Symposium on Information, Computer and Communications Security, pp. 90–101 (2006)

    Google Scholar 

  18. Imoize, A.L., Oyedare, T., Otuokere, M.E., Shetty, S.: Software intrusion detection evaluation system: a cost-based evaluation of intrusion detection capability. Commun. Netw. 10(4), 211–229 (2018)

    Article  Google Scholar 

  19. Imrana, Y., et al.: \(\chi ^2\)-BidlSTM: a feature driven intrusion detection system based on \(\chi ^2\) statistical model and bidirectional LSTM. Sensors 22(5), 2018 (2022)

    Article  Google Scholar 

  20. Intrator, Y., Katz, G., Shabtai, A.: MDGAN: boosting anomaly detection using multi-discriminator generative adversarial networks. ArXiv abs/1810.05221 (2018)

    Google Scholar 

  21. Khan, M.A.: HCRNNIDS: hybrid convolutional recurrent neural network-based network intrusion detection system. Processes 9(5), 834 (2021)

    Article  Google Scholar 

  22. Kim, J., Kim, J., Thu, H.L.T., Kim, H.: Long short term memory recurrent neural network classifier for intrusion detection. In: 2016 International Conference on Platform Technology and Service (PlatCon), pp. 1–5 (2016)

    Google Scholar 

  23. Kwon, D., Natarajan, K., Suh, S.C., Kim, H., Kim, J.: An empirical study on network anomaly detection using convolutional neural networks. In: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), pp. 1595–1598 (2018)

    Google Scholar 

  24. Lin, Z., Shi, Y., Xue, Z.: IDSGAN: generative adversarial networks for attack generation against intrusion detection. ArXiv abs/1809.02077 (2018)

    Google Scholar 

  25. Lopez-Martin, M., Carro, B., Sanchez-Esguevillas, A., Lloret, J.: Conditional variational autoencoder for prediction and feature recovery applied to intrusion detection in IoT. Sensors 17(9), 1967 (2017)

    Article  Google Scholar 

  26. Magán-Carrión, R., Urda, D., Díaz-Cano, I., Dorronsoro, B.: Towards a reliable comparison and evaluation of network intrusion detection systems based on machine learning approaches. Appl. Sci. 10(5), 1775 (2020)

    Article  Google Scholar 

  27. Malaiya, R.K., Kwon, D., Kim, J., Suh, S.C., Kim, H., Kim, I.: An empirical evaluation of deep learning for network anomaly detection. In: 2018 International Conference on Computing, Networking and Communications (ICNC), pp. 893–898 (2018)

    Google Scholar 

  28. Mehedi, S.T., Anwar, A., Rahman, Z., Ahmed, K., Rafiqul, I.: Dependable intrusion detection system for IoT: a deep transfer learning-based approach. IEEE Trans. Ind. Inform. 19(1), 1006–1017 (2022)

    Article  Google Scholar 

  29. Mell, P., Lippmann, R., Chung, Haines, J., Zissman, M.: An overview of issues in testing intrusion detection systems (2003)

    Google Scholar 

  30. Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A., Payne, B.D.: Evaluating computer intrusion detection systems: a survey of common practices. ACM Comput. Surv. (CSUR) 48(1), 1–41 (2015)

    Article  Google Scholar 

  31. Mirsky, Y.: Autoencoders for online network intrusion detection. ArXiv abs/1802.09089 (2018)

    Google Scholar 

  32. Ring, M., Wunderlich, S., Scheuring, D., Landes, D., Hotho, A.: A survey of network-based intrusion detection data sets. Comput. Secur. 86, 147–167 (2019)

    Article  Google Scholar 

  33. Sarhan, M., Layeghy, S., Portmann, M.: Evaluating standard feature sets towards increased generalisability and explainability of ML-based network intrusion detection (2021)

    Google Scholar 

  34. Shahriar, M.H., Haque, N.I., Rahman, M.A., Alonso, M.: G-IDS: generative adversarial networks assisted intrusion detection system. In: 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC), pp. 376–385 (2020)

    Google Scholar 

  35. Staudemeyer, R.C.: Applying long short-term memory recurrent neural networks to intrusion detection. S. Afr. Comput. J. 56, 136–154 (2015)

    Google Scholar 

  36. Tang, T.A., Mhamdi, L., McLernon, D., Zaidi, S.A.R., Ghogho, M.: Deep learning approach for network intrusion detection in software defined networking. In: 2016 International Conference on Wireless Networks and Mobile Communications (WINCOM), pp. 258–263 (2016)

    Google Scholar 

  37. Tavallaee, M., Stakhanova, N., Ghorbani, A.A.: Toward credible evaluation of anomaly-based intrusion-detection methods. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 40(5), 516–524 (2010)

    Google Scholar 

  38. Thing, V.L.L.: IEEE 802.11 network anomaly detection and attack classification: a deep learning approach. In: 2017 IEEE Wireless Communications and Networking Conference (WCNC), pp. 1–6 (2017)

    Google Scholar 

  39. Ulvila, J.W., Gaffney, J.E., Jr.: Evaluation of intrusion detection systems. J. Res. Nat. Inst. Stand. Technol. 108(6), 453 (2003)

    Article  Google Scholar 

  40. Viegas, E.K., Santin, A.O., Oliveira, L.S.: Toward a reliable anomaly-based intrusion detection in real-world environments. Comput. Netw. 127, 200–216 (2017)

    Article  Google Scholar 

  41. Wasielewska, K., Soukup, D., Čejka, T., Camacho, J.: Evaluation of detection limit in network dataset quality assessment with permutation testing. In: 4th Workshop on Machine Learning for Cybersecurity (MLCS) (2022)

    Google Scholar 

  42. Yin, C., Zhu, Y., Liu, S., Fei, J., Zhang, H.: An enhancing framework for botnet detection using generative adversarial networks. In: 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD), pp. 228–234 (2018)

    Google Scholar 

  43. Yu, Y., Long, J., Cai, Z.: Network intrusion detection through stacking dilated convolutional autoencoders. Secur. Commun. Netw. 2017, 4184196 (2017)

    Article  Google Scholar 

  44. Zhang, X., Ran, J., Mi, J.: An intrusion detection system based on convolutional neural network for imbalanced network traffic. In: 2019 IEEE 7th International Conference on Computer Science and Network Technology (ICCSNT), pp. 456–460 (2019)

    Google Scholar 

  45. Zixu, T., Liyanage, K.S.K., Gurusamy, M.: Generative adversarial network and auto encoder based anomaly detection in distributed IoT networks. In: GLOBECOM 2020–2020 IEEE Global Communications Conference, pp. 1–7 (2020)

    Google Scholar 

  46. Zolotukhin, M., Hämäläinen, T., Kokkonen, T., Siltanen, J.: Increasing web service availability by detecting application-layer DDoS attacks in encrypted traffic. In: 2016 23rd International Conference on Telecommunications (ICT), pp. 1–6 (2016)

    Google Scholar 

Download references

Acknowledgements

This work is funded by the GRIFIN project (ANR-20-CE39-0011).

Author information

Authors and Affiliations

  1. LORIA, Universite de Lorraine, Vandœuvre-lès-Nancy, France

    Solayman Ayoubi & Thomas Silverston

  2. SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, Palaiseau, France

    Gregory Blanc & Houda Jmila

  3. Sorbonne Université, CNRS, LIP6, Institut Universitaire de France, Paris, France

    Sébastien Tixeuil

Authors
  1. Solayman Ayoubi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gregory Blanc
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Houda Jmila
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thomas Silverston
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sébastien Tixeuil
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Solayman Ayoubi .

Editor information

Editors and Affiliations

  1. University of Ottawa, Ottawa, ON, Canada

    Guy-Vincent Jourdan

  2. Université Grenoble Alpes, Grenoble, France

    Laurent Mounier

  3. University of Ottawa, Ottawa, ON, Canada

    Carlisle Adams

  4. University Toulouse III Paul Sabatier, Toulouse, France

    Florence Sèdes

  5. Telecom SudParis, Palaiseau, France

    Joaquin Garcia-Alfaro

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ayoubi, S., Blanc, G., Jmila, H., Silverston, T., Tixeuil, S. (2023). Data-Driven Evaluation of Intrusion Detectors: A Methodological Framework. In: Jourdan, GV., Mounier, L., Adams, C., Sèdes, F., Garcia-Alfaro, J. (eds) Foundations and Practice of Security. FPS 2022. Lecture Notes in Computer Science, vol 13877. Springer, Cham. https://doi.org/10.1007/978-3-031-30122-3_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-30122-3_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-30121-6

  • Online ISBN: 978-3-031-30122-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics