Abaci-finder: Linux kernel crash classification through stack trace similarity learning
Introduction
The Linux operating system (OS) dump a crash reports when a kernel error occurs. A crash report includes crash title description, machine environment information, and a stack trace with multiple stack frames and their timestamps. To conduct bug triaging, bug localization, and bug fixing, Linux kernel developers usually search for similar crash reports that have been released in the mainline kernel, stable kernel, or other enterprise kernels, based on the stack trace in the crash report. However, it is challenging for developers to classify crash reports manually because kernel crash reports are numerous and diverse in continuous kernel testing. This raises the need for automatic stack-trace-based crash classification for kernel developers.
Existing crash classification models rely on string matching algorithms, statistical-feature-based classification, or neural networks. Classic string matching algorithms, such as edit distance, prefix match, and Longest Common Subsequence (LCS) approaches are used in early studies. Statistical-feature-based classification first extracts features from the text representations of stack traces through approaches such as bag-of-words (BoW), N-grams, or term frequency-inverse document frequency (TF-IDF). Then, classification is performed by an explicit similarity metric or a traditional machine learning process. Though these approaches can be easily applied to kernel crash classification, they make it hard to explore high-level correlations between traces because they ignore the semantics and ordering of stack traces. As a result, these methods' classification effectiveness is limited.
Recently, neural-network-based approaches have shown their superior performance in stack-trace-based crash deduplication and crash similarity calculation. They represent stack traces as sequences that contain tokens from a predefined dictionary and use a recurrent neural network as a classifier to output an end-to-end feature for classification. These models show superior performance because they can explore the semantic context of stack traces and do not rely on handcraft feature engineering.
However, existing methods are intended for crash classification in user-space applications, especially Java applications, which is not suitable for the classification of Linux kernel crashes. The stack traces of kernel crashes and application crashes are different. In particular, the format of kernel stack traces may be various due to different kernel configurations. The useful structured information in a stack frame is limited, whereas an application crash stack contains ordered information including package names, class names, function names, and line numbers. In addition, the top and bottom functions of the kernel crash stack are mostly common functions that are not useful for classification, while specific frames pointing to the root cause of bugs are essential to classification and should be paid more attention. In summary, the performance of existing methods of crash classification is limited due to the characteristics of Linux kernel stack traces.
In this paper, we propose abaci-finder, a deep learning (DL)-based classification framework specific to Linux kernel crashes. Abaci-finder first models a kernel stack trace as a sequence of function names with offsets in stack frames. Then, we propose a vectorization method called kstack2vec, which is used to extract trace features, with consideration for both semantic information hidden in function names and kernel-specific information of offsets. Finally, to classify kernel crashes, exploit an attention-based BiLSTM neural network, which considers both the semantic context of frames and the bias of specific frames by an attention mechanism. We conduct experiments on a real Linux crash dataset, and the experimental results indicate that abaci-finder outperforms existing methods. Moreover, we conduct an industry practice of enterprise Linux kernel crash classification and find 48 kernel bugs in the enterprise kernel that were fixed in the upstream version of kernel. Through our work, enterprise developers could fix kernel crashes more efficiently with the guidance of upstream patches. The contributions of this paper can be summarized as follows:
- •
We propose a DL-based classification framework specific to Linux kernel crashes. To the best of our knowledge, it is the first DL approach to classify kernel crashes by their stack trace features.
- •
We propose a kernel stack trace vectorization method called kstack2vec, which considers both the semantic information hidden in traces and kernel-specific offsets information.
- •
The experimental results show the superior performance of abaci-finder. The proposed framework finds 48 enterprise Linux kernel bugs that were fixed in upstream.
The rest of this paper is organized as follows. Section 2 introduces the background of Linux crash reports and applications for stack-trace-based classification currently in enterprise use. Section 3 reviews the related works on stack-trace-based crash classification. Section 4 presents the abaci-finder framework. Section 5 reports the experimental dataset, settings, compared approaches, experimental results, discussion, and industry practice. Section 6 draws conclusion and future work.
Section snippets
Background
In this section, we give a brief introduction to Linux crashes and crash report data. Then, we introduce applications for Linux crash classification used in enterprise to show the motivation for our work.
Related work
Crash reports, which are generated by users or software itself, are intended to help developers diagnose and fix software bugs [26]. In this paper, we focus on the classification of automatically generated reports rather than natural-language-based reports. In this scope, many studies have been proposed for different purposes, such as patch recommending [1], report bucketing [7], or crash deduplication [32]. These approaches rely mainly on explicit string matching algorithms,
Framework
In this section, we provide an overview and details of the proposed abaci-finder framework.
Experiments
In this section, we first introduce the collected dataset and the experimental settings include compared methods and metrics. Then, the experimental results and discussions are reported. Finally, we conduct an industry practice of crash classification for enterprise Linux kernel crashes by abaci-finder.
Conclusion
In this paper, we propose a stack-trace-based crash classification framework called abaci-finder, which is specific to Linux kernel crash classification. Abaci-finder first extracts the stack trace, which is a sequence of stack frames that includes function name with offsets, from the Linux crash report by preprocessing. The stack trace feature is derived by the kernel stack frame vectorization methods called kstack2vec, which is based on the function name tokenization, offsets integration, and
CRediT authorship contribution statement
Heyuan Shi: Conceptualization, Methodology, Supervision, Writing – original draft, Writing – review & editing. Guyu Wang: Software, Validation, Visualization. Ying Fu: Conceptualization, Data curation, Investigation, Methodology. Chao Hu: Investigation, Resources. Houbing Song: Supervision. Jian Dong: Resources, Supervision. Kun Tang: Investigation, Resources. Kai Liang: Writing – review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgement
We would like to thank Renze Zhang and Xinwei Liu for their invaluable help on Linux crash report analysis. Furthermore, we would like to thank Yang Lee and Jiapeng Chong for their extensive help in the enterprise practice preparation of the present article. This work was partially supported by the National Natural Science Foundation of China (Grant numbers: 62177046, 61977062), Hunan Provincial Natural Science Foundation (Grant numbers: 2021JJ30866).
Heyuan Shi received the B.S. degree in school of information science and engineering, Central South University, Changsha, China, in 2015, and the Ph.D. degree in School of Software, Tsinghua University, Beijing, China. His current research interests include software safety, machine learning and operating systems.
References (43)
- et al.
Class-incremental learning for wireless device identification in iot
IEEE Int. Things J.
(2021) - et al.
Machine learning for the detection and identification of internet of things (iot) devices: a survey
IEEE Int. Things J.
(2021) - et al.
Automatic prediction of the severity of bugs using stack traces and categorical features
Inf. Softw. Technol.
(2020) - et al.
Finding similar failures using callstack similarity
- et al.
Quickly finding known software problems via automated symptom matching
- et al.
The unreasonable effectiveness of traditional information retrieval in crash report deduplication
The Linux kernel documentation: bug hunting
The Linux kernel documentation: dynamic debug
- et al.
Automated bug detection and replay for COTS Linux kernel modules with concolic execution
- et al.
Rebucket: a method for clustering duplicate crash reports based on call stack similarity
BERT: pre-training of deep bidirectional transformers for language understanding
Classifying field crash reports for fixing bugs: a case study of mozilla firefox
The Linux kernel documentation: kernel stacks
The Linux kernel documentation: how to get printk format specifiers right
The kernel concurrency sanitizer (kcsan)
Bayesian-network-based reliability analysis of PLC systems
IEEE Trans. Ind. Electron.
Design and optimization of multiclocked embedded systems using formal techniques
IEEE Trans. Ind. Electron.
Design of mixed synchronous/asynchronous systems with multiple clocks
IEEE Trans. Parallel Distrib. Syst.
Data-centered runtime verification of wireless medical cyber-physical system
IEEE Trans. Ind. Inform.
Dependable model-driven development of CPS: from stateflow simulation to verified implementation
ACM Trans. Cyber Phys. Syst.
S3M: Siamese stack (trace) similarity measure
Cited by (2)
A Survey on Bug Deduplication and Triage Methods from Multiple Points of View
2023, Applied Sciences (Switzerland)DeepCrash: Deep metric learning for crash bucketing based on stack trace
2022, MaLTeSQuE 2022 - Proceedings of the 6th International Workshop on Machine Learning Techniques for Software Quality Evaluation, co-located with ESEC/FSE 2022
Heyuan Shi received the B.S. degree in school of information science and engineering, Central South University, Changsha, China, in 2015, and the Ph.D. degree in School of Software, Tsinghua University, Beijing, China. His current research interests include software safety, machine learning and operating systems.
Guyu Wang received the MS degree in School of Computer Science and Engineering from Beihang University, China. He is currently working in Alibaba Group. His current research interests include cyber threat intelligence, machine learning and data mining.
Houbing Song (Senior Member, IEEE) received the Ph.D. degree in electrical engineering from the University of Virginia, Charlottesville, VA, USA, in August 2012, and the M.S. degree in civil engineering from the University of Texas, TX, USA, in December 2006. In August 2017, he joined the Department of Electrical Engineering and Computer Science, Embry-Riddle Aeronautical University, FL, where he is currently an Assistant Professor and the Director of the Security and Optimization for Networked Globe Laboratory (SONG Lab). His current research interests include AI/Machine Learning, Cyber-Physical Systems, Cybersecurity and Privacy, Unmanned Aircraft Systems, Communications and Networking.
Jian Dong received the B.S. degree in electrical engineering from Hunan University, Changsha, China, in 2004, and the Ph.D. degree in electrical engineering from the Huazhong University of Science and Technology (HUST), Wuhan, China, in 2010. From 2006 to 2010, he was a Research Assistant with the National Key Laboratory of Science and Technology on Multispectral Information Processing, HUST. From 2012 to 2019, he was an Associate Professor with the School of Information Science and Engineering, Central South University, Changsha. From 2016 to 2017, he was a Visiting Scholar with the Eledia Research Center, University of Trento, Italy. He is currently a Full Professor with the School of Computer Science and Engineering, Central South University. He has published six books and over 100 peer reviewed articles on international journals and conferences. He holds over 16 innovation patents. His current research interests include antennas, wireless communications, and numerical optimization techniques.
Kun Tang received the B.S. degree in telecommunications from Wuhan University of Technology, Wuhan, China, in 2006, and M.S. degree in The University of New South Wales, Sydney, Australia, in 2011, and Ph.D. degree in Telecommunications from the Central South University, Changsha, China, in 2018. He is now a post-doctor with the school of Electronic and Information at South China University of Technology. His research interests are in the areas of cognitive radio networks, millimeter-wave communications, and network security.
Chao Hu received the Ph.D. in management from the Central South University. His research field includes information and network management, regional medical treatment, social computing and edge computing.
Kai Liang received the B.S. degree in computer science and technology from the Hunan Agricultural University, Changsha, China, in 2017. He is currently pursuing the master degree with the School of Computer Science and Engineering, Central South University, Changsha, China.
Ying Fu received the BS degree in school of software engineering, Beijing University of Posts and Telecommunications, Beijing, China, in 2017, and the master degree in School of Software, Tsinghua University, Beijing, China. Her current research interests include software safety and security of blockchain systems and smart contract.