Abstract
System logs produced by modern computer systems are valuable resources for detecting anomalies, debugging performance issues, and recovering application failures. With the increasing scale and complexity of the log data, manual log inspection is infeasible and man-power expensive. In this paper, we proposed LogAttn, an autoencoder model that combines an encoder-decoder structure with an attention mechanism for unsupervised log anomaly detection. The unstructured normal log data is proceeded by a log parser that uses a semantic analyse and clustering algorithm to parse log data into a sequence of event count vectors and semantic vectors. The encoder combines deep neural networks with an attention mechanism that learns the weights of different features to form a latent feature representation, which is further used by a decoder to reconstruct the log event sequence. If the reconstruction error is above a predefined threshold, it detects an anomaly in the log sequence and reports the result to the administrator. We conduct extensive experiments based on three real-world log datasets, which show that LogAttn achieves the best comprehensive performance compared to the state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Arora, S., Liang, Y., Ma, T.: A simple but tough-to-beat baseline for sentence embeddings. In: International Conference on Learning Representations (ICLR 2017) (2017)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018)
Breier, J., Branišová, J.: Anomaly detection from log files using data mining techniques. In: Kim, Kuinam J. (ed.) Information Science and Applications. LNEE, vol. 339, pp. 449–457. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-46578-3_53
Chen, M., Zheng, A.X., Lloyd, J., Jordan, M.I., Brewer, E.: Failure diagnosis using decision trees. In: International Conference on Autonomic Computing (2004)
Du, M., Li, F., Zheng, G., Srikumar, V.: Deeplog: anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1285–1298 (2017)
Erven, T., Harremoës, P.: Rényi divergence and kullback-leibler divergence. IEEE Trans. Inf. Theory 60, 3797–3820 (2014)
Ester, M., Kriegel, H., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD 1996) (1996)
Fu, Q., Lou, J.G., Wang, Y., Li, J.: Execution anomaly detection in distributed systems through unstructured log analysis. In: IEEE international conference on data mining (ICDM 2009), pp. 149–158. IEEE (2009)
Gai, K., Qiu, M., Zhao, H., Sun, X.: Resource management in sustainable cyber-physical systems using heterogeneous cloud computing. IEEE Trans. Sustain. Comput. 3, 60–72 (2018)
Han, J., Kamber, M.: Data Mining: Concepts and Techniques.Morgan Kaufmann, Massachusetts (2011)
He, P., Zhu, J., He, S., Li, J., Lyu, M.R.: Towards automated log parsing for large-scale log data analysis. IEEE Trans. Dependable Secure Comput. 15(6), 931–944 (2017)
He, P., Zhu, J., Zheng, Z., Lyu, M.R.: Drain: an online log parsing approach with fixed depth tree. In: IEEE International Conference on Web Services, pp. 33–40. IEEE (2017)
He, S., Lin, Q., Lou, J.G., Zhang, H., Lyu, M.R., Zhang, D.: Identifying impactful service system problems via log analysis. In: 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 60–70 (2018)
He, S., Zhu, J., He, P., Lyu, M.R.: Experience report: system log analysis for anomaly detection. In: IEEE 27th International Symposium on Software Reliability Engineering (ISSRE 2016), pp. 207–218. IEEE (2016)
Kabinna, S., Bezemer, C.-P., Shang, W., Syer, M.D., Hassan, A.E.: Examining the stability of logging statements. Empir. Softw. Eng. 23(1), 290–333 (2017). https://doi.org/10.1007/s10664-017-9518-0
Khatuya, S., Ganguly, N., Basak, J., Bharde, M., Mitra, B.: Adele: anomaly detection from event log empiricism. In: IEEE Conference on Computer Communications (INFOCOM 2018), pp. 2114–2122. IEEE (2018)
Liang, Y., Zhang, Y., Xiong, H., Sahoo, R.: Failure prediction in ibm bluegene/l event logs. In: IEEE International Conference on Data Mining (ICDM 2007), pp. 583–588. IEEE (2007)
Lin, Q., Zhang, H., Lou, J.G., Zhang, Y., Chen, X.: Log clustering based problem identification for online service systems. In: IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C 2016), pp. 102–111. IEEE (2016)
Lou, J.G., Fu, Q., Yang, S., Xu, Y., Li, J.: Mining invariants from console logs for system problem detection. In: USENIX Annual Technical Conference (ATC 2010), pp. 1–14 (2010)
Makanju, A.A., Zincir-Heywood, A.N., Milios, E.E.: Clustering event logs using iterative partitioning. In: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD 2009), pp. 1255–1264 (2009)
Meng, W., Liu, Y., Zhu, Y., Zhang, S., Zhou, R.: Loganomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs. In: Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI 2019) (2019)
Oliner, A., Stearley, J.: What supercomputers say: a study of five system logs. In: IEEE/IFIP 37th International Conference on Dependable Systems and Networks (DSN 2007), pp. 575–584. IEEE (2007)
Pecchia, A., Cotroneo, D., Kalbarczyk, Z., Iyer, R.K.: Improving log-based field failure data analysis of multi-node computing systems. In: IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN 2011), pp. 97–108. IEEE (2011)
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP 2014), pp. 1532–1543 (2014)
Siffer, A., Fouque, P.A., Termier, A., Largouët, C.: Anomaly detection in streams with extreme value theory. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2017) (2017)
Tang, L., Li, T., Perng, C.S.: LogSig: generating system events from raw textual logs. In: Proceedings of the 20th ACM international conference on Information and knowledge management (CIKM 2011), pp. 785–794 (2011)
Xu, W., Huang, L., Fox, A., Patterson, D., Jordan, M.I.: Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, pp. 117–132 (2009)
Zhang, X., Li, Z., Chen, J., He, X., Cheng, Q.: Robust log-based anomaly detection on unstable log data. In: Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (2019)
Zhou, C., Paffenroth, R.C.: Anomaly detection with robust deep autoencoders. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2017), pp. 665–674 (2017)
Acknowledgment
This work was partially supported by the National Key R&D Program of China (Grant No. 2018YFB1004704), the National Natural Science Foundation of China (Grant Nos. 61972196, 61832008, 61832005), the Key R&D Program of Jiangsu Province, China (Grant No. BE2018116), the Collaborative Innovation Center of Novel Software Technology and Industrialization, and the Sino-German Institutes of Social Computing.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, L. et al. (2021). LogAttn: Unsupervised Log Anomaly Detection with an AutoEncoder Based Attention Mechanism. In: Qiu, H., Zhang, C., Fei, Z., Qiu, M., Kung, SY. (eds) Knowledge Science, Engineering and Management. KSEM 2021. Lecture Notes in Computer Science(), vol 12817. Springer, Cham. https://doi.org/10.1007/978-3-030-82153-1_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-82153-1_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-82152-4
Online ISBN: 978-3-030-82153-1
eBook Packages: Computer ScienceComputer Science (R0)