Abstract
In this paper, we propose a scalable framework, called Daedalus, to analyze streams of NIDS (network-based intrusion detection system) logs in near real-time and to extract useful threat security intelligence. The proposed system pre-processes huge amounts of BRO NIDS logs received from different participating organizations and applies an elaborated anomaly detection technique in order to distinguish between normal and abnormal or anomalous network behaviors. As such, Daedalus detects network traffic anomalies by extracting a set of features of interest from the connection logs and then applying a time series-based technique in order to detect abnormal behavior in near real-time. Moreover, we correlate IP blocks extracted from the logs with some external security signature-based feeds that detect factual malicious activities (e.g., malware families and hashes, ransomware distribution, and command and control centers) in order to validate the proposed approach. Performed experiments demonstrate that Daedalus accurately identifies the malicious activities with an average \(F_{1}\) score of \(92.88\%\). We further compare our proposed approach with existing K-Means approaches and demonstrate the accuracy and efficiency of our system.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Antonakakis, M., et al.: Understanding the mirai botnet. In: Proceedings of the 26th USENIX Security Symposium (2017)
Eberhart, R., Kennedy, J.: A new optimizer using particle swarm theory. In: Proceedings of the Sixth International Symposium on Micro Machine and Human Science. MHS 1995, pp. 39–43. IEEE (1995)
Goldberg, D., Shan, Y.: The importance of features for statistical anomaly detection. In: HotCloud (2015)
Hamamoto, A.H., Carvalho, L.F., Sampaio, L.D.H., Abrão, T., Proença Jr., M.L.: Network anomaly detection system using genetic algorithm and fuzzy logic. Expert Syst. Appl. 92, 390–402 (2018)
Hu, W., Liao, Y., Vemuri, V.R.: Robust anomaly detection using support vector machines. In: Proceedings of the International Conference on Machine Learning, pp. 282–289 (2003)
Lakhina, A., Crovella, M., Diot, C.: Mining anomalies using traffic feature distributions. In: ACM SIGCOMM Computer Communication Review, vol. 35, pp. 217–228. ACM (2005)
Machaka, P., Bagula, A., Nelwamondo, F.: Using exponentially weighted moving average algorithm to defend against ddos attacks. In: 2016 Pattern Recognition Association of South Africa and Robotics and Mechatronics International Conference (PRASA-RobMech), pp. 1–6. IEEE (2016)
Maimo, L.F., Gomez, A.L.P., Clemente, F.J.G., Pérez, M.G., Pérez, G.M.: A self-adaptive deep learning-based system for anomaly detection in 5g networks. IEEE Access 6, 7700–7712 (2018)
Marini, F., Walczak, B.: Particle swarm optimization (PSO). A tutorial. Chemom. Intell. Lab. Syst. 149, 153–165 (2015)
Mendel, J.M.: Fuzzy logic systems for engineering: a tutorial. Proc. IEEE 83(3), 345–377 (1995)
Mirsky, Y., Doitshman, T., Elovici, Y., Shabtai, A.: Kitsune: an ensemble of autoencoders for online network intrusion detection. In: 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18–21, 2018 (2018)
Mushtaq, R.: Augmented dickey fuller test
Sbert, M., Shen, H.-W., Viola, I., Chen, M., Bardera, A., Feixas, M.: Tutorial on information theory in visualization. In: SIGGRAPH Asia 2017 Courses, p. 17. ACM (2017)
Sharafaldin, I., Lashkari, A.H., Ghorbani, A.A.: Toward generating a new intrusion detection dataset and intrusion traffic characterization. In: ICISSP, pp. 108–116 (2018)
Shinde, R., et al.: Survey on ransomware: a new era of cyber attack
Shirani, P., Azgomi, M.A., Alrabaee, S.: A method for intrusion detection in web services based on time series. In: 2015 IEEE 28th Canadian Conference on Electrical and Computer Engineering (CCECE), pp. 836–841. IEEE (2015)
Anomaly detection with k-means clustering (2015). http://amid.fish/anomaly-detection-with-k-means-clustering
An exponentially weighted moving average implementation that decays based on the elapsed time since the last update, approximating a time windowed moving average (2017). https://gist.github.com/jhalterman/f7b18b30160ae7817bb93894056eb380
htop(1) - linux man page https://linux.die.net/man/1/htop
How to check if time series data is stationary with python (2016). https://machinelearningmastery.com/time-series-data-stationary-python/
Ransomware tracker website (2018). https://ransomwaretracker.abuse.ch/tracker/
The bro network security monitor. https://www.bro.org/
Bro log files. https://www.bro.org/sphinx/script-reference/log-files.html
Exploring the exponentially weighted moving average (2018). https://www.investopedia.com/articles/07/ewma.asp
UNBCIC 2017 IDS Dataset (2017). http://www.unb.ca/cic/datasets/ids-2017.html
Wang, X., Zhang, H., Zhang, C., Cai, X., Wang, J., Ye, M.: Time series prediction using LS-SVM with particle swarm optimization. In: Wang, J., Yi, Z., Zurada, J.M., Lu, B.-L., Yin, H. (eds.) ISNN 2006. LNCS, vol. 3972, pp. 747–752. Springer, Heidelberg (2006). https://doi.org/10.1007/11760023_110
Zhang, X., Gu, C., Lin, J.: Support vector machines for anomaly detection. In: The Sixth World Congress on Intelligent Control and Automation. WCICA 2006, vol. 1, pp. 2594–2598. IEEE (2006)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Chohra, A., Debbabi, M., Shirani, P. (2019). Daedalus: Network Anomaly Detection on IDS Stream Logs. In: Zincir-Heywood, N., Bonfante, G., Debbabi, M., Garcia-Alfaro, J. (eds) Foundations and Practice of Security. FPS 2018. Lecture Notes in Computer Science(), vol 11358. Springer, Cham. https://doi.org/10.1007/978-3-030-18419-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-18419-3_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-18418-6
Online ISBN: 978-3-030-18419-3
eBook Packages: Computer ScienceComputer Science (R0)