Malicious Domain Detection on Imbalanced Data with Deep Reinforcement Learning

Abstract

Domain name system (DNS) is the key infrastructure of the Internet, yet has been deliberately abused by cyber attackers. Previous works detect malicious domain mainly based on the statistical features or the association features, which ignore the serialization impact and pay little attention to the imbalanced data problem. To address these problems, we propose a deep reinforcement learning based malicious domain detection model. We consider the malicious domain detection as a sequential decision process and employ Double Deep Q Network (DDQN) to address it. Furthermore, we devise a specific reward function to adapt to the imbalanced classification task. The specific reward function will guide the agent to learn the optimal classification policy. Extensive experiments are carried out on the real-world dataset, and experimental results demonstrate the effectiveness of our proposed method in detecting malicious domain in imbalanced DNS traffic.