Abstract
High-quality datasets are an essential requirement for leveraging machine learning (ML) in data processing and recently in network security as well. However, the quality of datasets is overlooked or underestimated very often. Having reliable metrics to measure and describe the input dataset enables the feasibility assessment of a dataset. Imperfect datasets may require optimization or updating, e.g., by including more data and merging class labels. Applying ML algorithms will not bring practical value if a dataset does not contain enough information. This work addresses the neglected topics of dataset evaluation and missing metrics. We propose three novel metrics to estimate the quality of an input dataset and help with its improvement or building a new dataset. This paper describes experiments performed on public datasets to show the benefits of the proposed metrics and theoretical definitions for more straightforward interpretation. Additionally, we have implemented and published Python code so that the metrics can be adopted by the worldwide scientific community.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anderson, B., McGrew, D.: Machine learning for encrypted malware traffic classification: accounting for noisy labels and non-stationarity. In: 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2017)
Brabec, J., et al.: On model evaluation under non-constant class imbalance. In: Computational Science (ICCS) (2020)
Celdrán, A.H., et al.: RITUAL: a platform quantifying the trustworthiness of supervised machine learning. In: 18th International Conference on Network and Service Management (CNSM) (2022)
Chen, H., et al.: Data curation and quality assurance for machine learning-based cyber intrusion detection (2021)
Zelaya, C.V.G.: Towards explaining the effects of data preprocessing on machine learning. In: 35th International Conference on Data Engineering (2019)
Hwang, I., et al.: SimEX: express prediction of inter-dataset similarity by a fleet of autoencoders. arXiv preprint arXiv:2001.04893 (2020)
Jeřábek, K., Hynek, K., Čejka, T., Ryšavý, O.: Collection of datasets with DNS over https traffic. Data Brief 42, 108310 (2022)
Koh, P.W., et al.: WILDS: a benchmark of in-the-wild distribution shifts. In: Proceedings of the 38th International Conference on Machine Learning (2021)
Komorniczak, J., Ksieniewicz, P.: Problexity - an open-source python library for binary classification problem complexity assessment. arXiv preprint arXiv:2207.06709 (2022)
Lanvin, M., et al.: Errors in the CICIDS2017 dataset and the significant differences in detection performances it makes (2023). https://hal.science/hal-03775466
Lee, Y.W., et al.: AIMQ: a methodology for information quality assessment. Inf. Manag. (2002)
Lorena, A.C., Garcia, L.P.F., Lehmann, J., Souto, M.C.P., Ho, T.K.: How complex is your classification problem? A survey on measuring classification complexity. ACM Comput. Surv. 52(5) (2019)
Luxemburk, J., Čejka, T.: Fine-grained TLS services classification with reject option. Comput. Netw. 220, 109467 (2023)
Maillo, J., Triguero, I., Herrera, F.: Redundancy and complexity metrics for big data classification: towards smart data. IEEE Access 8, 87918–87928 (2020)
Obaid, H.S., et al.: The impact of data pre-processing techniques and dimensionality reduction on the accuracy of machine learning. In: 9th Annual Information Technology, Electromechanical Engineering and Microelectronics Conference (IEMECON) (2019)
Papadogiannaki, E., Ioannidis, S.: A survey on encrypted network traffic analysis applications, techniques, and countermeasures. ACM Comput. Surv. 54(6) (2021)
Pendlebury, F., et al.: TESSERACT: eliminating experimental bias in malware classification across space and time. In: Proceedings of the 28th USENIX Conference on Security Symposium, USA (2019)
Pesarin, F., Salmaso, L.: A review and some new results on permutation testing for multivariate problems. Stat. Comput. 22(2), 639–646 (2012)
Sharafaldin, I., Lashkari, A.H., Ghorbani, A.A.: Toward generating a new intrusion detection dataset and intrusion traffic characterization. In: International Conference on Information Systems Security and Privacy (2018)
Soukup, D., et al.: Towards evaluating quality of datasets for network traffic domain. In: 17th International Conference on Network and Service Management (CNSM) (2021)
Wasielewska, K., et al.: Dataset quality assessment with permutation testing showcased on network traffic datasets (2022). http://dx.doi.org/10.36227/techrxiv.20145539.v1
Yoon, J., Arik, S., Pfister, T.: Data valuation using reinforcement learning. In: Daumé, H., III., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 10842–10851. PMLR (2020)
Zhang, Y., Zhao, S., Sang, Y.: Towards unknown traffic identification using deep auto-encoder and constrained clustering. In: Computational Science – ICCS (2019)
Acknowledgement
This research was funded by the Ministry of the Interior of the Czech Republic, grant No. VJ02010024: Flow-Based Encrypted Traffic Analysis, and by the MEYS of the Czech Republic, grant No. LM2023054: e-Infrastructure, and by internal grant of CTU in Prague, grant No. SGS23/207/OHK3/3T/18.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 IFIP International Federation for Information Processing
About this paper
Cite this paper
Soukup, D., Uhříček, D., Vašata, D., Čejka, T. (2024). Machine Learning Metrics for Network Datasets Evaluation. In: Meyer, N., Grocholewska-Czuryło, A. (eds) ICT Systems Security and Privacy Protection. SEC 2023. IFIP Advances in Information and Communication Technology, vol 679. Springer, Cham. https://doi.org/10.1007/978-3-031-56326-3_22
Download citation
DOI: https://doi.org/10.1007/978-3-031-56326-3_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-56325-6
Online ISBN: 978-3-031-56326-3
eBook Packages: Computer ScienceComputer Science (R0)
