Abstract
Encryption technology has become ubiquitous in network communication and encrypted malicious traffic detection becomes an important part of malware detection and cyber attack detection. Existing machine learning models and deep learning models are mainly trained based on packet length sequence information and time series information. Recent studies have shown that these models perform poorly in real network environments. In response to this challenge, this paper proposes a novel malicious traffic detection method based on certificate information extracted during the TLS (Transport Layer Security) encrypted handshake protocol. Our approach demonstrates that certificate information exhibits a strong correlation with the maliciousness of traffic, while remaining unaffected by the complexities of the real network environment. The experimental results illustrate that our method has high accuracy and low time overheading.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
DataCon 2020 dataset (2020). https://datacon.qianxin.com/opendata
Brim software (2022). https://github.com/brimdata/brimcap
MTA dataset (2023). https://malware-traffic-analysis.net
Althouse, J.: ja4 (2024). https://github.com/FoxIO-LLC/ja4
Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)
Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794 (2016)
Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 13(1), 21–27 (1967)
Fu, C., Li, Q., Xu, K., Wu, J.: Point cloud analysis for ML-based malicious traffic detection: reducing majorities of false positive alarms. In: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 1005–1019 (2023)
Garcia, S., Grill, M., Stiborek, J., Zunino, A.: An empirical comparison of botnet detection methods. Comput. Secur. 45, 100–123 (2014)
Google: Google transparency report (2024). https://transparencyreport.google.com/
Althouse, J., Atkinson, J., Atkins, J.: ja3 (2017). https://github.com/salesforce/ja3
Ke, G., et al.: LightGBM: a highly efficient gradient boosting decision tree. In: Advances in Neural Information Processing Systems 30 (2017)
Quinlan, J.R.: C4. 5: Programs for Machine Learning. Elsevier (2014)
Sharafaldin, I., Lashkari, A.H., Ghorbani, A.A., et al.: Toward generating a new intrusion detection dataset and intrusion traffic characterization. In: ICISSp, vol. 1, pp. 108–116 (2018)
Shen, M., Liu, Y., Zhu, L., Xu, K., Du, X., Guizani, N.: Optimizing feature selection for efficient encrypted traffic classification: a systematic approach. IEEE Netw. 34(4), 20–27 (2020)
Shen, M., et al.: Machine learning-powered encrypted network traffic analysis: a comprehensive survey. IEEE Commun. Surv. Tutor. 25(1), 791–824 (2022)
Shen, M., Zhang, J., Zhu, L., Xu, K., Du, X.: Accurate decentralized application identification via encrypted traffic analysis using graph neural networks. IEEE Trans. Inf. Forensics Secur. 16, 2367–2380 (2021)
Torroledo, I., Camacho, L.D., Bahnsen, A.C.: Hunting malicious TLS certificates with deep neural networks. In: Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security, pp. 64–73 (2018)
Xie, R., et al.: Rosetta: enabling robust TLS encrypted traffic classification in diverse network environments with TCP-aware traffic augmentation. In: Proceedings of the ACM Turing Award Celebration Conference-China 2023, pp. 131–132 (2023)
Zscaler: 2022 encrypted attacks report (2024). https://www.zscaler.com/blogs/security-research/2022-encrypted-attacks-report/
Acknowledgement
This work was supported by Major Scientific and Technological Innovation Projects of shandong Province (2020CXGC010116) and the National Natural Science Foundation of China (No. 62172042).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Suo, Y., Xue, J., Guo, W., Du, W., Han, W., Xu, C. (2025). Certificate-Based Transport Layer Security Encrypted Malicious Traffic Detection in Real-Time Network Environments. In: Zhu, T., Li, J., Castiglione, A. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2024. Lecture Notes in Computer Science, vol 15251. Springer, Singapore. https://doi.org/10.1007/978-981-96-1525-4_20
Download citation
DOI: https://doi.org/10.1007/978-981-96-1525-4_20
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-96-1524-7
Online ISBN: 978-981-96-1525-4
eBook Packages: Computer ScienceComputer Science (R0)