skip to main content
research-article

TLS-MHSA: An Efficient Detection Model for Encrypted Malicious Traffic based on Multi-Head Self-Attention Mechanism

Published:14 October 2023Publication History
Skip Abstract Section

Abstract

In recent years, the use of TLS (Transport Layer Security) protocol to protect communication information has become increasingly popular as users are more aware of network security. However, hackers have also exploited the salient features of the TLS protocol to carry out covert malicious attacks, which threaten the security of network space. Currently, the commonly used traffic detection methods are not always reliable when applied to the problem of encrypted malicious traffic detection due to their limitations. The most significant problem is that these methods do not focus on the key features of encrypted traffic. To address this problem, this study proposes an efficient detection model for encrypted malicious traffic based on transport layer security protocol and a multi-head self-attention mechanism called TLS-MHSA. Firstly, we extract the features of TLS traffic during pre-processing and perform traffic statistics to filter redundant features. Then, we use a multi-head self-attention mechanism to focus on learning key features as well as generate the most important combined features to construct the detection model, thereby detecting the encrypted malicious traffic. Finally, we use a public dataset to verify the effectiveness and efficiency of the TLS-MHSA model, and the experimental results show that the proposed TLS-MHSA model has high precision, recall, F1-measure, AUC-ROC as well as higher stability than seven state-of-the-art detection models.

REFERENCES

  1. [1] Amoli Payam Vahdani, Hamalainen Timo, David Gil, Zolotukhin Mikhail, and Mirzamohammad Mahsa. 2016. Unsupervised network intrusion detection systems for zero-day fast-spreading attacks and botnets. JDCTA (International Journal of Digital Content Technology and its Applications 10, 2 (2016), 113.Google ScholarGoogle Scholar
  2. [2] Anderson Blake and McGrew David. 2016. Identifying encrypted malware traffic with contextual flow data. In Proceedings of the 2016 ACM Workshop on Artificial Intelligence and Security. Association for Computing Machinery, New York, NY, USA, 3546. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Anderson Blake, Paul Subharthi, and McGrew David. 2018. Deciphering malware’s use of TLS (without decryption). Journal of Computer Virology and Hacking Techniques 14, 3 (2018), 195211.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Bernaille Laurent, Teixeira Renata, and Salamatian Kave. 2006. Early application identification. In Proceedings of the 2006 ACM CoNEXT Conference. Association for Computing Machinery, New York, NY, USA, 112. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Cao Zigang, Xiong Gang, Zhao Yong, Li Zhenzhen, and Guo Li. 2014. A survey on encrypted traffic classification. In Applications and Techniques in Information Security, Batten Lynn, Li Gang, Niu Wenjia, and Warren Matthew (Eds.). Springer Berlin, Berlin, 7381. Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Chen Jinfu, Lv Tianxiang, Cai Saihua, Song Luo, and Yin Shang. 2023. A novel detection model for abnormal network traffic based on bidirectional temporal convolutional network. Information and Software Technology 157 (2023), 107166. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Cisco. 2019. Joy. https://github.com/cisco/joy, Retrieved on 2019-11-8.Google ScholarGoogle Scholar
  8. [8] Garcia Sebastian, Grill Martin, Stiborek Jan, and Zunino Alejandro. 2014. An empirical comparison of botnet detection methods. Computers & Security 45 (2014), 100123.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Guo Han, Cao Juan, Zhang Yazi, Guo Junbo, and Li Jintao. 2018. Rumor detection with hierarchical social attention network. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM’18). Association for Computing Machinery, New York, NY, USA, 943951. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Lan Jin, Lu Jia Z., Wan Guo G., Wang Yuan Y., Huang Chen Y., Zhang Shi B., Huang Yu Y., Ma Jin N., and Damaševičius Robertas. 2022. E-MinBatch GraphSAGE: An industrial internet attack detection model. Sec. and Commun. Netw. 2022 (Jan.2022), 12 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Lin Kunda, Xu Xiaolong, and Gao Honghao. 2021. TSCRNN: A novel classification scheme of encrypted traffic based on flow spatiotemporal features for efficient management of IIoT. Computer Networks 190 (2021), 107974.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Lin Lan, Luo Huan, Huang Renjie, and Ye Mao. 2019. Recurrent models of visual co-attention for person re-identification. IEEE Access 7 (2019), 88658875.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Lin Peng, Ye Kejiang, Hu Yishen, Lin Yanying, and Xu Cheng-Zhong. 2023. A novel multimodal deep learning framework for encrypted traffic classification. IEEE/ACM Transactions on Networking 31, 3 (2023), 13691384. Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] MontazeriShatoori Mohammadreza, Davidson Logan, Kaur Gurdip, and Lashkari Arash Habibi. 2020. Detection of DoH tunnels using time-series classification of encrypted traffic. In 2020 IEEE Intl. Conf. on Dependable, Autonomic and Secure Computing, Intl. Conf. on Pervasive Intelligence and Computing, Intl. Conf. on Cloud and Big Data Computing, Intl. Conf. on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech). IEEE, Calgary, AB, Canada, 6370. Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Niu Zequn, Xue Jingfeng, Qu Dacheng, Wang Yong, Zheng Jun, and Zhu Hongfei. 2022. A novel approach based on adaptive online analysis of encrypted traffic for identifying malware in IIoT. Information Sciences 601 (2022), 162174.Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Roques Olivier, Maffeis S., and Cova M.. 2019. Detecting Malware in TLS Traffic. Master’s thesis. Imperial College London.Google ScholarGoogle Scholar
  17. [17] Sharafaldin Iman, Lashkari Arash Habibi, and Ghorbani Ali A.. 2018. Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSP 1 (2018), 108116.Google ScholarGoogle Scholar
  18. [18] Shekhawat Anish Singh, Troia Fabio Di, and Stamp Mark. 2019. Feature analysis of encrypted malicious traffic. Expert Systems with Applications 125 (2019), 130141.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Sivanathan Arunan, Gharakheili Hassan Habibi, Loi Franco, Radford Adam, Wijenayake Chamith, Vishwanath Arun, and Sivaraman Vijay. 2018. Classifying IoT devices in smart environments using network traffic characteristics. IEEE Transactions on Mobile Computing 18, 8 (2018), 17451759.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Stergiopoulos George, Talavari Alexander, Bitsikas Evangelos, and Gritzalis Dimitris. 2018. Automatic detection of various malicious traffic using side channel features on TCP packets. In Computer Security, Lopez Javier, Zhou Jianying, and Soriano Miguel (Eds.). Springer International Publishing, Cham, 346362. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Stratosphere. 2015. Stratosphere Laboratory Datasets. https://www.stratosphereips.org/datasets-overview, Retrieved March 13, 2020.Google ScholarGoogle Scholar
  22. [22] Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N., Kaiser Łukasz, and Polosukhin Illia. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdfGoogle ScholarGoogle Scholar
  23. [23] Velan Petr, Čermák Milan, Čeleda Pavel, and Drašar Martin. 2015. A survey of methods for encrypted traffic classification and analysis. International Journal of Network Management 25, 5 (2015), 355374.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Vormayr Gernot, Fabini Joachim, and Zseby Tanja. 2020. Why are my flows different? A tutorial on flow exporters. IEEE Communications Surveys & Tutorials 22, 3 (2020), 20642103.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Wang Tianwei, Zhu Yuanzhi, Jin Lianwen, Luo Canjie, Chen Xiaoxue, Wu Yaqiang, Wang Qianying, and Cai Mingxiang. 2020. Decoupled attention network for text recognition. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA, 1221612224. Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Yang Luming, Fu Shaojing, Wang Yongjun, Liang Kaitai, Mo Fan, and Liu Bo. 2022. DEV-ETA: An interpretable detection framework for encrypted malicious traffic. Comput. J. 66, 5 (032022), 12131227. . arXiv:https://academic.oup.com/comjnl/article-pdf/66/5/1213/50397336/bxac008.pdfGoogle ScholarGoogle ScholarCross RefCross Ref
  27. [27] Ying Haochao, Zhuang Fuzhen, Zhang Fuzheng, Liu Yanchi, Xu Guandong, Xie Xing, Xiong Hui, and Wu Jian. 2018. Sequential recommender system based on hierarchical attention network. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI’18). AAAI Press, Palo Alto, CA, 39263932. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Yu Tangda, Zou FuTai, Li Linsen, and Yi Ping. 2019. An encrypted malicious traffic detection system based on neural network. In 2019 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC). IEEE, Guilin, China, 6270. Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Zheng Juan, Zeng Zhiyong, Feng Tao, and Maglaras Leandros. 2022. GCN-ETA: High-efficiency encrypted malicious traffic detection. Sec. and Commun. Netw. 2022 (Jan.2022), 11 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. TLS-MHSA: An Efficient Detection Model for Encrypted Malicious Traffic based on Multi-Head Self-Attention Mechanism

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Privacy and Security
        ACM Transactions on Privacy and Security  Volume 26, Issue 4
        November 2023
        260 pages
        ISSN:2471-2566
        EISSN:2471-2574
        DOI:10.1145/3614236
        • Editor:
        • Ninghui Li
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 14 October 2023
        • Online AM: 7 August 2023
        • Accepted: 27 July 2023
        • Revised: 31 May 2023
        • Received: 22 November 2022
        Published in tops Volume 26, Issue 4

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
      • Article Metrics

        • Downloads (Last 12 months)508
        • Downloads (Last 6 weeks)126

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text