Abstract
Log analysis can diagnose software system issues. Log anomaly detection always faces the challenge of class distribution imbalance and data noise. In addition, existing methods often overlook log event structural relationships, causing instability. In this work, we propose AdvGraLog, a Generative Adversarial Network (GAN) model based on log graph representation, to detect anomalies when the reconstruction error of discriminator is terrible. We construct log graphs and employ Graph Neural Network (GNN) to obtain a comprehensive graph representation. We use a GAN generator to transform original negative logs into adversarial samples. Discriminator adopts an AutoEncoder (AE) to detect anomalies by comparing reconstruction error to a threshold. Adversarial training enhances adversarial sample quality and boosts the discriminator’s anomaly recognition. Experimental results demonstrate the superiority of our proposed method over baseline approaches in real-world datasets. Supplementary experiments further validate the effectiveness of our model in handling imbalanced log data and augmenting model robustness.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Akcay, S., Atapour-Abarghouei, A., Breckon, T.P.: GANomaly: semi-supervised anomaly detection via adversarial training. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11363, pp. 622–637. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20893-6_39
Avola, D., et al.: A novel GAN-based anomaly detection and localization method for aerial video surveillance at low altitude. Remote Sens. 14(16), 4110 (2022)
Capra, L.: Graph transformation systems: a semantics based on (stochastic) symmetric nets. In: Pang, J., Zhang, L. (eds.) SETTA 2020. LNCS, vol. 12153, pp. 35–51. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-62822-2_3
Du, M., Li, F.: Spell: streaming parsing of system event logs. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 859–864. IEEE (2016)
Du, M., Li, F., Zheng, G., Srikumar, V.: DeepLog: anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1285–1298 (2017)
Fenton, N.E., Ohlsson, N.: Quantitative analysis of faults and failures in a complex software system. IEEE Trans. Softw. Eng. 26(8), 797–814 (2000)
Han, X., Yuan, S.: Unsupervised cross-system log anomaly detection via domain adaptation. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 3068–3072 (2021)
He, P., Zhu, J., Zheng, Z., Lyu, M.R.: Drain: an online log parsing approach with fixed depth tree. In: 2017 IEEE International Conference on Web Services (ICWS), pp. 33–40. IEEE (2017)
Jiang, W., Hong, Y., Zhou, B., He, X., Cheng, C.: A GAN-based anomaly detection approach for imbalanced industrial time series. IEEE Access 7, 143608–143619 (2019)
Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759 (2016)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Le, V.H., Zhang, H.: Log-based anomaly detection without log parsing. In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 492–504. IEEE (2021)
Le, V.H., Zhang, H.: Log-based anomaly detection with deep learning: How far are we? In: Proceedings of the 44th International Conference on Software Engineering, pp. 1356–1367 (2022)
Lin, Q., Zhang, H., Lou, J.G., Zhang, Y., Chen, X.: Log clustering based problem identification for online service systems. In: Proceedings of the 38th International Conference on Software Engineering Companion, pp. 102–111 (2016)
Liu, Z., Xia, X., Lo, D., Xing, Z., Hassan, A.E., Li, S.: Which variables should i log? IEEE Trans. Softw. Eng. 47(9), 2012–2031 (2019)
Makanju, A.A., Zincir-Heywood, A.N., Milios, E.E.: Clustering event logs using iterative partitioning. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1255–1264 (2009)
Meng, W., et al.: LogAnomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs. In: IJCAI, vol. 19, pp. 4739–4745 (2019)
Mi, H., Wang, H., Zhou, Y., Lyu, M.R.T., Cai, H.: Toward fine-grained, unsupervised, scalable performance diagnosis for production cloud computing systems. IEEE Trans. Parallel Distrib. Syst. 24(6), 1245–1255 (2013)
Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018)
Oliner, A.J., Aiken, A., Stearley, J.: Alert detection in system logs. In: 2008 Eighth IEEE International Conference on Data Mining, pp. 959–964. IEEE (2008)
Park, S., Lee, K.H., Ko, B., Kim, N.: Unsupervised anomaly detection with generative adversarial networks in mammography. Sci. Rep. 13(1), 2925 (2023)
Pimentel, M.A., Clifton, D.A., Clifton, L., Tarassenko, L.: A review of novelty detection. Signal Process. 99, 215–249 (2014)
Rouillard, J.P.: Real-time log file analysis using the simple event correlator (SEC). In: LISA, vol. 4, pp. 133–150 (2004)
Sagar, B., Manjul, M., et al.: Anomaly detection in wireless sensor network using generative adversarial network (GAN). In: Automation and Computation, pp. 45–49 (2023)
Vaarandi, R.: Mining event logs with SLCT and LogHound. In: NOMS 2008–2008 IEEE Network Operations and Management Symposium, pp. 1071–1074. IEEE (2008)
Wan, Y., Liu, Y., Wang, D., Wen, Y.: GLAD-PAW: graph-based log anomaly detection by position aware weighted graph attention network. In: Karlapalem, K., et al. (eds.) PAKDD 2021. LNCS (LNAI), vol. 12712, pp. 66–77. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-75762-5_6
Xia, B., Yin, J., Xu, J., Li, Y.: LogGAN: a sequence-based generative adversarial network for anomaly detection based on system logs. In: Liu, F., Xu, J., Xu, S., Yung, M. (eds.) SciSec 2019. LNCS, vol. 11933, pp. 61–76. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-34637-9_5
Xie, Y., Zhang, H., Babar, M.A.: LogGD: detecting anomalies from system logs with graph neural networks. In: 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), pp. 299–310. IEEE (2022)
Xu, W., Huang, L., Fox, A., Patterson, D., Jordan, M.I.: Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, pp. 117–132 (2009)
Yan, Y., Jiang, S., Zhang, S., Huang, Y.: CSFL: fault localization on real software bugs based on the combination of context and spectrum. In: Qin, S., Woodcock, J., Zhang, W. (eds.) SETTA 2021. LNCS, vol. 13071, pp. 219–238. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-91265-9_12
Yang, R., Qu, D., Gao, Y., Qian, Y., Tang, Y.: nLSALog: an anomaly detection framework for log sequence in security management. IEEE Access 7, 181152–181164 (2019)
Yen, T.F., et al.: Beehive: large-scale log analysis for detecting suspicious activity in enterprise networks. In: Proceedings of the 29th Annual Computer Security Applications Conference, pp. 199–208 (2013)
Zenati, H., Foo, C.S., Lecouat, B., Manek, G., Chandrasekhar, V.R.: Efficient GAN-based anomaly detection. arXiv preprint arXiv:1802.06222 (2018)
Zenati, H., Romain, M., Foo, C.S., Lecouat, B., Chandrasekhar, V.: Adversarially learned anomaly detection. In: 2018 IEEE International Conference on Data Mining (ICDM), pp. 727–736. IEEE (2018)
Zhang, C., et al.: DeepTraLog: trace-log combined microservice anomaly detection through graph-based deep learning. In: Proceedings of the 44th International Conference on Software Engineering, pp. 623–634 (2022)
Zhang, H.: On the distribution of software faults. IEEE Trans. Softw. Eng. 34(2), 301–302 (2008)
Zhang, H.: An investigation of the relationships between lines of code and defects. In: 2009 IEEE International Conference on Software Maintenance, pp. 274–283. IEEE (2009)
Zhang, H., Zhang, X.: Comments on “data mining static code attributes to learn defect predictors’’. IEEE Trans. Softw. Eng. 33(9), 635–637 (2007)
Zhang, X., et al.: Robust log-based anomaly detection on unstable log data. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 807–817 (2019)
Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., Lyu, M.R.: Tools and benchmarks for automated log parsing. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 121–130 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
He, Z., Tang, Y., Zhao, K., Liu, J., Chen, W. (2024). Graph-Based Log Anomaly Detection via Adversarial Training. In: Hermanns, H., Sun, J., Bu, L. (eds) Dependable Software Engineering. Theories, Tools, and Applications. SETTA 2023. Lecture Notes in Computer Science, vol 14464. Springer, Singapore. https://doi.org/10.1007/978-981-99-8664-4_4
Download citation
DOI: https://doi.org/10.1007/978-981-99-8664-4_4
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8663-7
Online ISBN: 978-981-99-8664-4
eBook Packages: Computer ScienceComputer Science (R0)