Abstract
Log-based anomaly detection plays an essential role in the fast-emerging Artificial Intelligence for IT Operations (AIOps) of software systems. Many log-based anomaly detection methods have been proposed. Due to the variety and unstructured characteristics of logs, log parsing is the first necessary step for parsing logs into structured ones in log-based anomaly detection methods. Prior studies have found that the effectiveness of log parsing will impact the performance of log-based anomaly detection. However, few studies comprehensively investigate whether better log parsing implies better anomaly detection. In this paper, we conduct a comprehensively empirical study to investigate the impact of six state-of-the-art log parsers belonging to four categories (including heuristic-based, frequency-based, clustering-based, and subsequence-based) on six state-of-the-art log-based anomaly detection methods (including machine-learning-based and deep-learning-based methods). Experimental results on three public datasets show that (1) High parsing accuracy does not definitely imply high anomaly detection performance. Both parsing accuracy and the number of parsed event templates should be considered when choosing log parsers for anomaly detection. (2) The log parsers have an impact on the efficiency of anomaly detection methods. With the increase in the number of parsed event templates, the efficiency of anomaly detection decreases. In detail, the heuristic-based parsers have less impact on the efficiency of anomaly detection methods, followed by frequency-based parsers. (3) All the anomaly detection methods perform more effectively and efficiently with the heuristic-based log parsers. Thus, the heuristic-based log parsers are recommended for a new practitioner on anomaly detection. We believe that our work, with the evaluation results and the corresponding findings, can help researchers and practitioners better understand the impact of log parsers on anomaly detection and provide guidelines for choosing a suitable log parser for their anomaly detection method.
Similar content being viewed by others
References
Abdi H et al (2007) Bonferroni and šidák corrections for multiple comparisons. Encyclopedia of Measurement and Statistics 3:103–107
Babenko A, Mariani L, Pastore F (2009) Ava: automated interpretation of dynamically detected anomalies. In: Proceedings of the eighteenth international symposium on software testing and analysis, pp 237–248
Berrocal E, Yu L, Wallace S, Papka M E, Lan Z (2014) Exploring void search for fault detection on extreme scale systems. In: 2014 IEEE International conference on cluster computing (CLUSTER). IEEE, pp 1–9
Bodik P, Goldszmidt M, Fox A, Woodard D B, Andersen H (2010) Fingerprinting the datacenter: automated classification of performance crises. In: Proceedings of the 5th European conference on computer systems, pp 111–124
Breier J, Branišová J (2015) Anomaly detection from log files using data mining techniques. In: Information science and applications. Springer, pp 449–457
Chen A R (2019) An empirical study on leveraging logs for debugging production failures. In: 2019 IEEE/ACM 41st international conference on software engineering: companion proceedings (ICSE-C). IEEE, pp 126–128
Chen M, Zheng A X, Lloyd J, Jordan M I, Brewer E (2004) Failure diagnosis using decision trees. In: International conference on autonomic computing, 2004. Proceedings. IEEE, pp 36–43
Chen Y, Yang X, Lin Q, Zhang H, Gao F, Xu Z, Dang Y, Zhang D, Dong H, Xu Y et al (2019) Outage prediction and diagnosis for cloud service systems. In: The world wide web conference, pp 2659–2665
Dai H, Li H, Chen C S, Shang W, Chen T H (2020) Logram: efficient log parsing using n-gram dictionaries. IEEE Trans Softw Eng
Damasio C V, Fröhlich P, Nejdl W, Pereira L M, Schroeder M (2002) Using extended logic programming for alarm-correlation in cellular phone networks. Appl Intell 17(2):187–202
Dang Y, Lin Q, Huang P (2019) Aiops: real-world challenges and research innovations. In: 2019 IEEE/ACM 41st international conference on software engineering: companion proceedings (ICSE-C). IEEE, pp 4–5
Du M, Li F (2018) Spell: online streaming parsing of large unstructured system logs. IEEE Trans Knowl Data Eng 31(11):2213–2227
Du M, Li F, Zheng G, Srikumar V (2017) Deeplog: anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp 1285–1298
El-Sayed N, Zhu H, Schroeder B (2017) Learning from failure across multiple clusters: a trace-driven approach to understanding, predicting, and mitigating job terminations. In: 2017 IEEE 37th international conference on distributed computing systems (ICDCS). IEEE, pp 1333–1344
Fu Q, Lou J G, Wang Y, Li J (2009) Execution anomaly detection in distributed systems through unstructured log analysis. In: 2009 Ninth IEEE international conference abstracting log lines to log event types for mining software system logsce on data mining. IEEE, pp 149–158
Hamooni H, Debnath B, Xu J, Zhang H, Jiang G, Mueen A (2016) Logmine: fast pattern recognition for log analytics. In: Proceedings of the 25th ACM international on conference on information and knowledge management, pp 1573–1582
Hansen S E, Atkins E T (1993) Automated system monitoring and notification with swatch. In: LISA, vol 93, pp 145–152
He P, Zhu J, He S, Li J, Lyu M R (2016a) An evaluation study on log parsing and its use in log mining. In: 2016 46th Annual IEEE/IFIP international conference on dependable systems and networks (DSN). IEEE, pp 654–661
He S, Zhu J, He P, Lyu M R (2016b) Experience report: system log analysis for anomaly detection. In: 2016 IEEE 27th international symposium on software reliability engineering (ISSRE), IEEE, pp 207–218
He P, Zhu J, Zheng Z, Lyu M R (2017) Drain: an online log parsing approach with fixed depth tree. In: 2017 IEEE International conference on web services (ICWS). IEEE, pp 33–40
He S, Lin Q, Lou J G, Zhang H, Lyu M R, Zhang D (2018a) Identifying impactful service system problems via log analysis. In: Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 60–70
He S, Lin Q, Lou J G, Zhang H, Lyu M R, Zhang D (2018b) Identifying impactful service system problems via log analysis. In: Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 60–70
Huang P, Guo C, Lorch J R, Zhou L, Dang Y (2018) Capturing and enhancing in situ system observability for failure detection. In: 13th Symposium on operating systems design and implementation (OSDI), pp 1–16
Jia T, Chen P, Yang L, Li Y, Meng F, Xu J (2017) An approach for anomaly diagnosis based on hybrid graph model with logs for distributed services. In: 2017 IEEE International conference on web services (ICWS). IEEE, pp 25–32
Jiang Z M, Hassan A E, Flora P, Hamann G (2008) Abstracting execution logs to execution events for enterprise applications (short paper). In: 2008 The eighth international conference on quality software. IEEE, pp 181–186
Jiang W, Hu C, Pasupathy S, Kanevsky A, Li Z, Zhou Y (2009) Understanding customer problem troubleshooting from storage system logs. In: Proccedings of the 7th conference on file and storage technologies, pp 43–56
Liang Y, Zhang Y, Xiong H, Sahoo R (2007) Failure prediction in ibm bluegene/l event logs. In: Seventh IEEE international conference on data mining (ICDM 2007). IEEE, pp 583–588
Lin Q, Zhang H, Lou J G, Zhang Y, Chen X (2016) Log clustering based problem identification for online service systems. In: 2016 IEEE/ACM 38th international conference on software engineering companion (ICSE-c). IEEE, pp 102–111
Lin Q, Hsieh K, Dang Y, Zhang H, Sui K, Xu Y, Lou J G, Li C, Wu Y, Yao R et al (2018) Predicting node failure in cloud service systems. In: Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 480–490
Liu F, Wen Y, Zhang D, Jiang X, Xing X, Meng D (2019) Log2vec: a heterogeneous graph embedding based approach for detecting cyber threats within enterprise. In: Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pp 1777–1794
loggly (2021) [EB/OL]. https://www.loggly.com/blog/
Logstash (2021) [EB/OL]. https://logz.io
Lou J G, Fu Q, Yang S, Xu Y, Li J (2010) Mining invariants from console logs for system problem detection. In: USENIX Annual technical conference, pp 1–14
Lu J, Li F, Li L, Feng X (2018a) Cloudraid: hunting concurrency bugs in the cloud via log-mining. In: Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 3–14
Lu S, Wei X, Li Y, Wang L (2018b) Detecting anomaly in big data system logs using convolutional neural network. In: 2018 IEEE 16th international conference on dependable, autonomic and secure computing, 16th international conference on pervasive intelligence and computing, 4th international conference on big data intelligence and computing and cyber science and technology congress (DASC/picom/datacom/cyberscitech). IEEE, pp 151–158
Makanju A, Zincir-Heywood A N, Milios E E (2011) A lightweight algorithm for message type extraction in system application logs. IEEE Trans Knowl Data Eng 24(11):1921–1936
Meng W, Liu Y, Zhu Y, Zhang S, Pei D, Liu Y, Chen Y, Zhang R, Tao S, Sun P et al (2019) Loganomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs. In: IJCAI, vol 7, pp 4739–4745
Messaoudi S, Panichella A, Bianculli D, Briand L, Sasnauskas R (2018) A search-based approach for accurate identification of log message formats. In: 2018 IEEE/ACM 26th international conference on program comprehension (ICPC). IEEE, pp 167–16,710
Mizutani M (2013) Incremental mining of system log format. In: 2013 IEEE International conference on services computing. IEEE, pp 595–602
Nagappan M, Vouk M A (2010) Abstracting log lines to log event types for mining software system logs. In: 2010 7th IEEE working conference on mining software repositories (MSR 2010). IEEE, pp 114– 117
Nagappan M, Wu K, Vouk M A (2009) Efficiently extracting operational profiles from execution logs using suffix arrays. In: 2009 20th International symposium on software reliability engineering (ISSRE). IEEE, pp 41–50
Nandi A, Mandal A, Atreja S, Dasgupta G B, Bhattacharya S (2016) Anomaly detection using program control flow graph mining from execution logs. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 215–224
Oliner A, Stearley J (2007) What supercomputers say: a study of five system logs. In: 37th Annual IEEE/IFIP international conference on dependable systems and networks (DSN). IEEE, pp 575–584
Shang W, Jiang Z M, Hemmati H, Adams B, Hassan A E, Martin P (2013) Assisting developers of big data analytics applications when deploying on hadoop clouds. In: 2013 35th International conference on software engineering (ICSE). IEEE, pp 402–411
Shima K (2016) Length matters: clustering system log messages using length of words. arXiv:161103213
Splunk (2021) [EB/OL]. https://docs.splunk.com/Documentation/Splunk/7.3.1/_Knowledge/AboutSplunkregularexpressions/
Tang L, Li T, Perng C S (2011) Logsig: generating system events from raw textual logs. In: Proceedings of the 20th ACM international conference on Information and knowledge management, pp 785– 794
Vaarandi R (2003) A data clustering algorithm for mining patterns from event logs. In: Proceedings of the 3rd IEEE workshop on IP operations & management (IPOM). IEEE, pp 119–126
Vaarandi R (2006) Simple event correlator for real-time security log monitoring. Hakin9 Magazine 1(6):28–39
Vaarandi R, Pihelgas M (2015) Logcluster-a data clustering and pattern mining algorithm for event logs. In: 2015 11th International conference on network and service management (CNSM). IEEE, pp 1–7
Wilcoxon F (1992) Individual comparisons by ranking methods. In: Breakthroughs in statistics. Springer, pp 196–202
Xia B, Bai Y, Yin J, Li Y, Xu J (2020) Loggan: a log-level generative adversarial network for anomaly detection using permutation event modeling. Inf Syst Front 1–14
Xu W, Huang L, Fox A, Patterson D, Jordan M I (2009) Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd symposium on operating systems principles, pp 117–132
Yin K, Yan M, Xu L, Xu Z, Li Z, Yang D, Zhang X (2020) Improving log-based anomaly detection with component-aware analysis. In: 2020 IEEE International conference on software maintenance and evolution (ICSME). IEEE, pp 667–671
Yuan D, Mai H, Xiong W, Tan L, Zhou Y, Pasupathy S (2010) Sherlog: error diagnosis by connecting clues from run-time logs. In: Proceedings of the fifteenth international conference on architectural support for programming languages and operating systems, pp 143–154
Zar J H (2005) Spearman rank correlation. Encyclopedia of biostatistics 7
Zhang X, Xu Y, Lin Q, Qiao B, Zhang H, Dang Y, Xie C, Yang X, Cheng Q, Li Z et al (2019) Robust log-based anomaly detection on unstable log data. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 807–817
Zhou X, Peng X, Xie T, Sun J, Ji C, Liu D, Xiang Q, He C (2019) Latent error prediction and fault localization for microservice applications by learning from system trace logs. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 683–694
Zhu J, He S, Liu J, He P, Xie Q, Zheng Z, Lyu M R (2019) Tools and benchmarks for automated log parsing. In: 2019 IEEE/ACM 41st international conference on software engineering: software engineering in practice (ICSE-SEIP). IEEE, pp 121–130
Acknowledgments
This work was supported in part by the National Key Research and Development Project (No. 2018YFB2101200), the Fundamental Research Funds for the Central Universities (No. 2022CDJKYJH001), the Natural Science Foundation of Chongqing (No. cstc2021jcyj-msxmX0538), the National Natural Science Foundation of China (No. 62002034) and the Postdoctoral Foundation of Chongqing (No. 2020LY13).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of Interest
The authors have no conflict of interest.
Additional information
Communicated by: Philipp Leitner
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix A: The Impact of the Number of Parsed Event Templates on Anomaly Detection Methods’ Effectiveness
Appendix A: The Impact of the Number of Parsed Event Templates on Anomaly Detection Methods’ Effectiveness
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Fu, Y., Yan, M., Xu, Z. et al. An empirical study of the impact of log parsers on the performance of log-based anomaly detection. Empir Software Eng 28, 6 (2023). https://doi.org/10.1007/s10664-022-10214-6
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-022-10214-6