A Machine Learning-Based Framework for Detecting Malicious HTTPS Traffic

Malicious traffic detection plays an essential role for Network Operators to prevent attackers from manipulating the network systems. In the past, many Network Intrusion Detection Systems (e.g., Snort, etc.) were designed to inspect the packets using pre-defined rules in order to identify the malicious traffic. Despite achieving good performance in the context of non-encrypted traffic, these systems are ineffective nowadays due to encrypted traffic (e.g., HTTPS, QUIC, etc.) and complex network behaviors of compromised computers. Therefore, many studies focus on malicious traffic detection mechanisms using Machine Learning (ML), which analyzes flow-based features using ML algorithms to detect the traffic generated by malware. There are two main kinds of features for malicious traffic detection containing protocol-agnostic features and TLS/SSL features. Using all these features can result in high time complexity and performance degradation, so it cannot meet the real-time requirement of the Intrusion Detection Systems. Therefore, in this paper, we take into account different kinds of flow-based features and implement a feature selection to select an appropriate feature set to improve the detection accuracy and execution time for malicious traffic detection. Besides, the proposed framework is evaluated using various datasets: CTU-13, MCFP, and CIC-AndMal201. The experimental results show that the framework can achieve an accuracy of 99 percent in considered scenarios.