Abstract
Log sequences generated by heterogeneous systems are critical for understanding computer system behaviour and ensuring operational and security integrity. However, the diverse formats, structures, and content of logs pose challenges for traditional log anomaly detection approaches that rely on log parsing, which can be imperfect and incomplete in information extraction. To address these challenges, we propose HEART (HEterogeneous Log Anomaly Detection using Robust Transformers), an end-to-end framework for log-based anomaly detection. HEART eliminates the need for log parsing and leverages Transfer Learning (TL) and Transformer models to operate directly on raw log events from multiple systems. We enhance existing tokenizers with domain-specific tokens, applied to BERT and RoBERTa, and introduce two novel Transformer models, LogAnBERT and LogBERTa, trained from scratch on log events. We comprehensively evaluate HEART in intra-system and cross-system scenarios, demonstrating its competitive performance with enhanced anomaly detection using fewer training parameters. Our findings highlight the importance of adapting Transformers and tokenizers for log anomaly detection, enabling improved system monitoring and security across domains. HEART is a significant contribution, being the first end-to-end TL framework for log-based anomaly detection in heterogeneous systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The code, datasets, and Transformer models developed and evaluated in this work are available upon reasonable request to the corresponding author.
- 2.
- 3.
- 4.
- 5.
References
Almodovar, C., Sabrina, F., Karimi, S., Azad, S.: Can language models help in system security? Investigating log anomaly detection using BERT. In: Proceedings of the The 20th Annual Workshop of the Australasian Language Technology Association, pp. 139–147. Australasian Language Technology Association, Adelaide, Australia (2022). https://aclanthology.org/2022.alta-1.19
Arp, D., et al.: Dos and don’ts of machine learning in computer security. In: Proceedings of the USENIX Security Symposium (2022). https://doi.org/10.48550/arXiv.2010.09470
Chen, R., et al.: LogTransfer: Cross-system log anomaly detection for software systems with transfer learning. In: 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE) pp. 37–47 (2020). https://doi.org/10.1109/ISSRE5003.2020.00013. ISSN: 2332-6549
Chen, S., Liao, H.: Bert-log: anomaly detection for system logs based on pre-trained language model. Appl. Artif. Intell. 36(1), 2145642 (2022). https://doi.org/10.1080/08839514.2022.2145642
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding (2019). https://doi.org/10.48550/arXiv.1810.04805
Du, M., Li, F., Zheng, G., Srikumar, V.: DeepLog: anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1285–1298. ACM (2017). https://doi.org/10.1145/3133956.3134015
Friedman, M.: The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J. Am. Stat. Assoc. 32(200), 675–701 (1937). https://doi.org/10.1080/01621459.1937.10503522
Guo, H., Yuan, S., Wu, X.: LogBERT: log anomaly detection via BERT. In: 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2021). https://doi.org/10.1109/IJCNN52387.2021.9534113. ISSN: 2161-4407
He, P., Zhu, J., Zheng, Z., Lyu, M.R.: Drain: an online log parsing approach with fixed depth tree. In: 2017 IEEE International Conference on Web Services (ICWS), pp. 33–40. IEEE (2017). https://doi.org/10.1109/ICWS.2017.13
Le, V.H., Zhang, H.: Log-based anomaly detection without log parsing. In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 492–504 (2021). https://doi.org/10.1109/ASE51524.2021.9678773. ISSN: 2643-1572
Lin, Q., Zhang, H., Lou, J.G., Zhang, Y., Chen, X.: Log clustering based problem identification for online service systems. In: 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 102–111 (2016). https://doi.org/10.1145/2889160.2889232
Liu, Y., et al.: Roberta: a robustly optimized BERT pretraining approach. ArXiv abs/1907.11692 (2019). https://doi.org/10.48550/arXiv.1907.11692
Mvula, P.K., Branco, P., Jourdan, G.V., Viktor, H.L.: A systematic literature review of cyber-security data repositories and performance assessment metrics for semi-supervised learning. Discov. Data 1(1), 4 (2023). https://doi.org/10.1007/s44248-023-00003-x
Oliner, A., Stearley, J.: What supercomputers say: a study of five system logs. In: 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2007), pp. 575–584. IEEE (2007). https://doi.org/10.1109/DSN.2007.103
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019)
Shao, Y., et al.: Log anomaly detection method based on BERT model optimization. In: 2022 7th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), pp. 161–166 (2022). https://doi.org/10.1109/ICCCBDA55098.2022.9778900
Weiss, K., Khoshgoftaar, T.M., Wang, D.D.: A survey of transfer learning. J. Big Data 3(1), 1–40 (2016). https://doi.org/10.1186/s40537-016-0043-6
Xu, W., Huang, L., Fox, A., Patterson, D., Jordan, M.I.: Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles - SOSP 2009, p. 117. ACM Press (2009). https://doi.org/10.1145/1629575.1629587
Zhang, J., Li, Z., Zhang, X., Lin, F., Wang, C., Cai, X.: PoSBert: log classification via modified BERT based on part-of-speech weight. In: 2022 5th International Conference on Pattern Recognition and Artificial Intelligence (PRAI), pp. 979–983 (2022). https://doi.org/10.1109/PRAI55851.2022.9904207
Zhang, S., et al.: Syslog processing for switch failure diagnosis and prediction in datacenter networks. In: 2017 IEEE/ACM 25th International Symposium on Quality of Service (IWQoS), pp. 1–10 (2017). https://doi.org/10.1109/IWQoS.2017.7969130
Zhang, S., Liu, Y., Zhang, X., Cheng, W., Chen, H., Xiong, H.: CAT: beyond efficient transformer for content-aware anomaly detection in event sequences. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4541–4550 (2022). https://doi.org/10.1145/3534678.3539155
Zhao, L., Pan, S., Xiang, E., Zhong, E., Lu, Z., Yang, Q.: Active transfer learning for cross-system recommendation. Proc. AAAI Conf. Artif. Intell. 27(1), 1205–1211 (2013). https://doi.org/10.1609/aaai.v27i1.8458
Acknowledgements
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Vector Institute, and The IBM Center for Advanced Studies (CAS) Canada within Project 1059. We are also grateful to the Digital Research Alliance of Canada (the Alliance) for their continuous support and access to their High-Performance Computing clusters.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Mvula, P.K., Branco, P., Jourdan, GV., Viktor, H.L. (2023). HEART: Heterogeneous Log Anomaly Detection Using Robust Transformers. In: Bifet, A., Lorena, A.C., Ribeiro, R.P., Gama, J., Abreu, P.H. (eds) Discovery Science. DS 2023. Lecture Notes in Computer Science(), vol 14276. Springer, Cham. https://doi.org/10.1007/978-3-031-45275-8_45
Download citation
DOI: https://doi.org/10.1007/978-3-031-45275-8_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45274-1
Online ISBN: 978-3-031-45275-8
eBook Packages: Computer ScienceComputer Science (R0)