Abstract
Information systems record the current states and the access records in logs, so logs become the data basis for detecting anomalies of system security. To realize log anomaly detection, frameworks based on text, sequence, and graph are applied. However, the existing frameworks could not extract the complex associations in logs, which leads to low accuracy. To meet the requirements of the hyperautomation framework for log analysis, this paper proposes GenGLAD, a generated graph based log anomaly detection framework. The generated graph is used to express the log associations, and the node embedding of the generated graph is obtained based on random walk and word2vec. Finally, we use clustering to realize unsupervised anomaly detection. Experiments verify the detection effect of GenGLAD. Compared with the existing detection frameworks, GenGLAD achieves the highest accuracy and improves the comprehensive detection effect.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. CoRR abs/1803.01271 (2018)
Bornet, P., Barkin, I., Wirtz, J.: Intelligent Automation: Welcome to the World of Hyperautomation - Learn How to Harness Artificial Intelligence to Boost Business & Make Our World More Human. WorldScientific (2021)
Du, M., Li, F., Zheng, G., Srikumar, V.: DeepLog: anomaly detection and diagnosis from system logs through deep learning. In: ACM SIGSAC Conference on Computer and Communication Security, pp. 1285–1298 (2017)
Duan, X., Ying, S., Yuan, W., Cheng, H., Yin, X.: A generative adversarial networks for log anomaly detection. Comput. Syst. Sci. Eng. 37(1), 135–148 (2021)
Gai, K., Du, Z., et al.: Efficiency-aware workload optimizations of heterogeneous cloud computing for capacity planning in financial industry. In: IEEE 2nd CSCloud (2015)
Gai, K., Qiu, M., Elnagdy, S.: A novel secure big data cyber incident analytics framework for cloud-based cybersecurity insurance. In: IEEE BigData Security Conference (2016)
Gai, K., Zhang, Y., et al.: Blockchain-enabled service optimizations in supply chain digital twin. In: IEEE TSC (2022)
Gai, K., et al.: Electronic health record error prevention approach using ontology in big data. In: IEEE 17th HPCC (2015)
Gao, X., Qiu, M.: Energy-based learning for preventing backdoor attack. In: KSEM (3), pp. 706–721 (2022)
Han, X., Pasquier, T.F.J., Bates, A., Mickens, J., Seltzer, M.I.: Unicorn: runtime provenance-based detector for advanced persistent threats. In: 27th Network and Distributed System Security Symposium, NDSS 2020 (2020)
Hu, F., Lakdawala, S., et al.: Low-power, intelligent sensor hardware interface for medical data preprocessing. IEEE Trans. Inform. Tech. Biomed. 13(4), 656–663 (2009)
Kent, A.: Cyber security data sources for dynamic network research. In: Dynamic Networks and Cyber-Security, pp. 37–65 (05 2016)
Kulyadi, S.P., Mohandas, P., et al.: Anomaly detection using generative adversarial networks on firewall log message data. In: 13th IEEE Conference on Electronics, Computers and Artificial Intelligence ECAI, pp. 1–6 (2021)
Li, J., Ming, Z., et al.: Resource allocation robustness in multi-core embedded systems with inaccurate information. J. Sys. Arch. 57(9), 840–849 (2011)
Li, Y., Gai, K., et al.: Intercrossed access controls for secure financial services on multimedia big data in cloud systems. In: ACM TMCCA (2016)
Lindauer, B.: Insider threat test dataset (2020). https://kilthub.cmu.edu/articles/dataset/Insider_Threat_Test_Dataset/12841247
Lindemann, B., Maschler, B., Sahlab, N., Weyrich, M.: A survey on anomaly detection for technical systems using LSTM networks. Comput. Ind. 131, 103498 (2021)
Liu, F., Wen, Y., et al.: Log2vec: a heterogeneous graph embedding based approach for detecting cyber threats within enterprise. In: ACM SIGSAC Conference on Computer and Communications Security, pp. 1777–1794 (2019)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: Bengio, Y., LeCun, Y. (eds.) 1st International Conference on Learning Representations, ICLR 2013, Workshop Track Proceedings (2013)
Mikolov, T., Sutskever, I., et al.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, vol. 26. Curran Associates, Inc. (2013)
Moon, G.E., Newman-Griffis, D., et al.: Parallel data-local training for optimizing word2vec embeddings for word and graph embeddings. In: IEEE/ACM Workshop on Machine Learning in High Performance Computing Environment, MLHPC@SC, 2019, pp. 44–55 (2019)
Nehinbe, D.J.: A review of technical issues on ids and alerts. Global J. Comput. Sci. Technol. 17, 55–62 (2018)
Nguyen, T.-B.-T., Liao, T.-L., Vu, T.-A.: Anomaly detection using one-class SVM for logs of juniper router devices. In: Duong, T.Q., Vo, N.-S., Nguyen, L.K., Vien, Q.-T., Nguyen, V.-D. (eds.) INISCOM 2019. LNICST, vol. 293, pp. 302–312. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30149-1_24
Niu, J., Gao, Y., et al.: Selecting proper wireless network interfaces for user experience enhancement with guaranteed probability. JPDC 72(12), 1565–1575 (2012)
Pawlicki, M., Kozik, R., Choras, M.: A survey on neural networks for (cyber-) security and (cyber-) security of neural networks. Neurocomputing 500, 1075–1087 (2022)
Qiu, H., Dong, T., et al.: Adversarial attacks against network intrusion detection in IoT systems. IEEE IoT J. 8(13), 10327–10335 (2020)
Qiu, H., Zheng, Q., et al.: Topological graph convolutional network-based urban traffic flow and density prediction. IEEE Trans. ITS (2020)
Qiu, M., Chen, Z., et al.: Energy-aware data allocation with hybrid memory for mobile cloud systems. IEEE Sys. J. 11(2), 813–822 (2014)
Qiu, M., Gai, K., Xiong, Z.: Privacy-preserving wireless communications using bipartite matching in social big data. FGCS 87, 772–781 (2018)
Qiu, M., Jia, Z., et al.: Voltage assignment with guaranteed probability satisfying timing constraint for real-time multiproceesor DSP. J. Signal Proc. Sys. 46, 55–73 (2007)
Qiu, M., Li, H., Sha, E.: Heterogeneous real-time embedded software optimization considering hardware platform. In: ACM Symposium on Applied Computing, pp. 1637–1641 (2009)
Qiu, M., Qiu, H.: Review on image processing based adversarial example defenses in computer vision. In: IEEE 6th International Conference on BigData Security, pp. 94–99 (2020)
Qiu, M., Qiu, H., et al.: Secure data sharing through untrusted clouds with blockchain-enabled key management. In: 3rd SmartBlock Conference, pp. 11–16 (2020)
Qiu, M., Sha, E., et al.: Energy minimization with loop fusion and multi-functional-unit scheduling for multidimensional DSP. JPDC 68(4), 443–455 (2008)
Qiu, M., Xue, C., Shao, Z., et al.: Efficient algorithm of energy minimization for heterogeneous wireless sensor network. In: IEEE EUC Conference, pp. 25–34 (2006)
Qiu, M., Xue, C., et al.: Energy minimization with soft real-time and DVS for uniprocessor and multiprocessor embedded systems. In: IEEE DATE Conference, pp. 1–6 (2007)
Qiu, M., Yang, L., Shao, Z., Sha, E.: Dynamic and leakage energy minimization with soft real-time loop scheduling and voltage assignment. IEEE TVLSI 18(3), 501–504 (2009)
Qiu, M., Zhang, L., Ming, Z., Chen, Z., Qin, X., Yang, L.T.: Security-aware optimization for ubiquitous computing systems with SEAT graph approach. J. Comput. Syst. Sci. 79(5), 518–529 (2013)
Shao, Z., Wang, M., et al.: Real-time dynamic voltage loop scheduling for multi-core embedded systems. IEEE Trans. Circuits Syst. II 54(5), 445–449 (2007)
Wang, S., Balarezo, J.F., Kandeepan, S., Al-Hourani, A., Chavez, K.G., Rubinstein, B.: Machine learning in network anomaly detection: a survey. IEEE Access 9, 152379–152396 (2021)
Wang, Z., Tian, J., Fang, H., Chen, L., Qin, J.: Lightlog: a lightweight temporal convolutional network for log anomaly detection on the edge. Comput. Netw. 203, 108616 (2022)
Xie, Y., Ji, L., Cheng, X.: An attention-based GRU network for anomaly detection from system logs. IEICE Trans. Inf. Syst. 103D(8), 1916–1919 (2020)
Zeng, L., Xiao, Y., Chen, H., Sun, B., Han, W.: Computer operating system logging and security issues: a survey. Secur. Commun. Netw. 9(17), 4804–4821 (2016)
Zhang, C., Wang, X., Zhang, H., Zhang, H., Han, P.: Log sequence anomaly detection based on local information extraction and globally sparse transformer model. IEEE Trans. Netw. Serv. Manag. 18(4), 4119–4133 (2021)
Zhang, H., Duan, D., Zhang, Q.: RWREL: a fast training framework for random walk-based knowledge graph embedding. In: ACAI 2021: 4th International Conference on Algorithms, Computing and Artificial Intelligence, pp. 67:1–67:6. ACM (2021)
Zhang, L., Qiu, M., Tseng, W., Sha, E.: Variable partitioning and scheduling for MPSOC with virtually shared scratch pad memory. J. Signal Proc. Sys. 58(2), 247–265 (2018)
Zhou, F., Qu, H.: A GMM-based anomaly IP detection model from security logs. In: Qiu, M. (ed.) SmartCom 2020. LNCS, vol. 12608, pp. 97–105. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-74717-6_11
Acknowledgements
This work was supported by State Grid Zhoushan Electric Power Supply Company of Zhejiang Power Corporation under grant No. B311ZS220002 (Research on hyperautomation for information comprehensive inspection).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, H. et al. (2023). GenGLAD: A Generated Graph Based Log Anomaly Detection Framework. In: Qiu, M., Lu, Z., Zhang, C. (eds) Smart Computing and Communication. SmartCom 2022. Lecture Notes in Computer Science, vol 13828. Springer, Cham. https://doi.org/10.1007/978-3-031-28124-2_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-28124-2_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-28123-5
Online ISBN: 978-3-031-28124-2
eBook Packages: Computer ScienceComputer Science (R0)