Abaci-finder: Linux kernel crash classification through stack trace similarity learning

https://doi.org/10.1016/j.jpdc.2022.06.003Get rights and content

Highlights

  • Attention-based BiLSTM neural network to classify kernel crashes by stack traces.

  • Kernel stack trace vectorization using function name semantics and offsets information.

  • Automatic find kernel bugs in enterprise Linux that have been fixed in upstream.

Abstract

Developers often classify crashes by stack traces to analyze, locate and fix kernel bugs. Existing stack-trace-based crash classification approaches rely on string matching and statistical features, which ignore crash semantic contexts and cannot explore high-order correlations. Deep-learning-based approaches use crash embeddings and output end-to-end features for classification. However, they ignore kernel-specific information, which limits classification performance. Regarding these issues, we propose abaci-finder, a deep-learning-based classification framework specific to Linux kernel crashes. We first model the kernel stack trace as a stack frames sequence and then perform stack trace preprocessing. Then, we propose a vectorization method specific to kernel stack traces, called kstack2vec, to extract features with consideration for function semantics and kernel-specific offsets information. Finally, we exploit an attention-based BiLSTM neural network for classification, with consideration for both frame context and key frames in traces. The experiments on the real Linux kernel crash dataset indicate that abaci-finder outperforms existing methods of crash classification.

Introduction

The Linux operating system (OS) dump a crash reports when a kernel error occurs. A crash report includes crash title description, machine environment information, and a stack trace with multiple stack frames and their timestamps. To conduct bug triaging, bug localization, and bug fixing, Linux kernel developers usually search for similar crash reports that have been released in the mainline kernel, stable kernel, or other enterprise kernels, based on the stack trace in the crash report. However, it is challenging for developers to classify crash reports manually because kernel crash reports are numerous and diverse in continuous kernel testing. This raises the need for automatic stack-trace-based crash classification for kernel developers.

Existing crash classification models rely on string matching algorithms, statistical-feature-based classification, or neural networks. Classic string matching algorithms, such as edit distance, prefix match, and Longest Common Subsequence (LCS) approaches are used in early studies. Statistical-feature-based classification first extracts features from the text representations of stack traces through approaches such as bag-of-words (BoW), N-grams, or term frequency-inverse document frequency (TF-IDF). Then, classification is performed by an explicit similarity metric or a traditional machine learning process. Though these approaches can be easily applied to kernel crash classification, they make it hard to explore high-level correlations between traces because they ignore the semantics and ordering of stack traces. As a result, these methods' classification effectiveness is limited.

Recently, neural-network-based approaches have shown their superior performance in stack-trace-based crash deduplication and crash similarity calculation. They represent stack traces as sequences that contain tokens from a predefined dictionary and use a recurrent neural network as a classifier to output an end-to-end feature for classification. These models show superior performance because they can explore the semantic context of stack traces and do not rely on handcraft feature engineering.

However, existing methods are intended for crash classification in user-space applications, especially Java applications, which is not suitable for the classification of Linux kernel crashes. The stack traces of kernel crashes and application crashes are different. In particular, the format of kernel stack traces may be various due to different kernel configurations. The useful structured information in a stack frame is limited, whereas an application crash stack contains ordered information including package names, class names, function names, and line numbers. In addition, the top and bottom functions of the kernel crash stack are mostly common functions that are not useful for classification, while specific frames pointing to the root cause of bugs are essential to classification and should be paid more attention. In summary, the performance of existing methods of crash classification is limited due to the characteristics of Linux kernel stack traces.

In this paper, we propose abaci-finder, a deep learning (DL)-based classification framework specific to Linux kernel crashes. Abaci-finder first models a kernel stack trace as a sequence of function names with offsets in stack frames. Then, we propose a vectorization method called kstack2vec, which is used to extract trace features, with consideration for both semantic information hidden in function names and kernel-specific information of offsets. Finally, to classify kernel crashes, exploit an attention-based BiLSTM neural network, which considers both the semantic context of frames and the bias of specific frames by an attention mechanism. We conduct experiments on a real Linux crash dataset, and the experimental results indicate that abaci-finder outperforms existing methods. Moreover, we conduct an industry practice of enterprise Linux kernel crash classification and find 48 kernel bugs in the enterprise kernel that were fixed in the upstream version of kernel. Through our work, enterprise developers could fix kernel crashes more efficiently with the guidance of upstream patches. The contributions of this paper can be summarized as follows:

  • We propose a DL-based classification framework specific to Linux kernel crashes. To the best of our knowledge, it is the first DL approach to classify kernel crashes by their stack trace features.

  • We propose a kernel stack trace vectorization method called kstack2vec, which considers both the semantic information hidden in traces and kernel-specific offsets information.

  • The experimental results show the superior performance of abaci-finder. The proposed framework finds 48 enterprise Linux kernel bugs that were fixed in upstream.

The rest of this paper is organized as follows. Section 2 introduces the background of Linux crash reports and applications for stack-trace-based classification currently in enterprise use. Section 3 reviews the related works on stack-trace-based crash classification. Section 4 presents the abaci-finder framework. Section 5 reports the experimental dataset, settings, compared approaches, experimental results, discussion, and industry practice. Section 6 draws conclusion and future work.

Section snippets

Background

In this section, we give a brief introduction to Linux crashes and crash report data. Then, we introduce applications for Linux crash classification used in enterprise to show the motivation for our work.

Related work

Crash reports, which are generated by users or software itself, are intended to help developers diagnose and fix software bugs [26]. In this paper, we focus on the classification of automatically generated reports rather than natural-language-based reports. In this scope, many studies have been proposed for different purposes, such as patch recommending [1], report bucketing [7], or crash deduplication [32]. These approaches rely mainly on explicit string matching algorithms,

Framework

In this section, we provide an overview and details of the proposed abaci-finder framework.

Experiments

In this section, we first introduce the collected dataset and the experimental settings include compared methods and metrics. Then, the experimental results and discussions are reported. Finally, we conduct an industry practice of crash classification for enterprise Linux kernel crashes by abaci-finder.

Conclusion

In this paper, we propose a stack-trace-based crash classification framework called abaci-finder, which is specific to Linux kernel crash classification. Abaci-finder first extracts the stack trace, which is a sequence of stack frames that includes function name with offsets, from the Linux crash report by preprocessing. The stack trace feature is derived by the kernel stack frame vectorization methods called kstack2vec, which is based on the function name tokenization, offsets integration, and

CRediT authorship contribution statement

Heyuan Shi: Conceptualization, Methodology, Supervision, Writing – original draft, Writing – review & editing. Guyu Wang: Software, Validation, Visualization. Ying Fu: Conceptualization, Data curation, Investigation, Methodology. Chao Hu: Investigation, Resources. Houbing Song: Supervision. Jian Dong: Resources, Supervision. Kun Tang: Investigation, Resources. Kai Liang: Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

We would like to thank Renze Zhang and Xinwei Liu for their invaluable help on Linux crash report analysis. Furthermore, we would like to thank Yang Lee and Jiapeng Chong for their extensive help in the enterprise practice preparation of the present article. This work was partially supported by the National Natural Science Foundation of China (Grant numbers: 62177046, 61977062), Hunan Provincial Natural Science Foundation (Grant numbers: 2021JJ30866).

Heyuan Shi received the B.S. degree in school of information science and engineering, Central South University, Changsha, China, in 2015, and the Ph.D. degree in School of Software, Tsinghua University, Beijing, China. His current research interests include software safety, machine learning and operating systems.

References (43)

  • J. Devlin et al.

    BERT: pre-training of deep bidirectional transformers for language understanding

  • T. Dhaliwal et al.

    Classifying field crash reports for fixing bugs: a case study of mozilla firefox

  • C. Du

    The Linux kernel documentation: kernel stacks

  • R. Dunlap et al.

    The Linux kernel documentation: how to get printk format specifiers right

  • M. Elver

    The kernel concurrency sanitizer (kcsan)

  • Y. Jiang et al.

    Bayesian-network-based reliability analysis of PLC systems

    IEEE Trans. Ind. Electron.

    (2013)
  • Y. Jiang et al.

    Design and optimization of multiclocked embedded systems using formal techniques

    IEEE Trans. Ind. Electron.

    (2015)
  • Y. Jiang et al.

    Design of mixed synchronous/asynchronous systems with multiple clocks

    IEEE Trans. Parallel Distrib. Syst.

    (2015)
  • Y. Jiang et al.

    Data-centered runtime verification of wireless medical cyber-physical system

    IEEE Trans. Ind. Inform.

    (2017)
  • Y. Jiang et al.

    Dependable model-driven development of CPS: from stateflow simulation to verified implementation

    ACM Trans. Cyber Phys. Syst.

    (2019)
  • A. Khvorov et al.

    S3M: Siamese stack (trace) similarity measure

  • Cited by (2)

    Heyuan Shi received the B.S. degree in school of information science and engineering, Central South University, Changsha, China, in 2015, and the Ph.D. degree in School of Software, Tsinghua University, Beijing, China. His current research interests include software safety, machine learning and operating systems.

    Guyu Wang received the MS degree in School of Computer Science and Engineering from Beihang University, China. He is currently working in Alibaba Group. His current research interests include cyber threat intelligence, machine learning and data mining.

    Houbing Song (Senior Member, IEEE) received the Ph.D. degree in electrical engineering from the University of Virginia, Charlottesville, VA, USA, in August 2012, and the M.S. degree in civil engineering from the University of Texas, TX, USA, in December 2006. In August 2017, he joined the Department of Electrical Engineering and Computer Science, Embry-Riddle Aeronautical University, FL, where he is currently an Assistant Professor and the Director of the Security and Optimization for Networked Globe Laboratory (SONG Lab). His current research interests include AI/Machine Learning, Cyber-Physical Systems, Cybersecurity and Privacy, Unmanned Aircraft Systems, Communications and Networking.

    Jian Dong received the B.S. degree in electrical engineering from Hunan University, Changsha, China, in 2004, and the Ph.D. degree in electrical engineering from the Huazhong University of Science and Technology (HUST), Wuhan, China, in 2010. From 2006 to 2010, he was a Research Assistant with the National Key Laboratory of Science and Technology on Multispectral Information Processing, HUST. From 2012 to 2019, he was an Associate Professor with the School of Information Science and Engineering, Central South University, Changsha. From 2016 to 2017, he was a Visiting Scholar with the Eledia Research Center, University of Trento, Italy. He is currently a Full Professor with the School of Computer Science and Engineering, Central South University. He has published six books and over 100 peer reviewed articles on international journals and conferences. He holds over 16 innovation patents. His current research interests include antennas, wireless communications, and numerical optimization techniques.

    Kun Tang received the B.S. degree in telecommunications from Wuhan University of Technology, Wuhan, China, in 2006, and M.S. degree in The University of New South Wales, Sydney, Australia, in 2011, and Ph.D. degree in Telecommunications from the Central South University, Changsha, China, in 2018. He is now a post-doctor with the school of Electronic and Information at South China University of Technology. His research interests are in the areas of cognitive radio networks, millimeter-wave communications, and network security.

    Chao Hu received the Ph.D. in management from the Central South University. His research field includes information and network management, regional medical treatment, social computing and edge computing.

    Kai Liang received the B.S. degree in computer science and technology from the Hunan Agricultural University, Changsha, China, in 2017. He is currently pursuing the master degree with the School of Computer Science and Engineering, Central South University, Changsha, China.

    Ying Fu received the BS degree in school of software engineering, Beijing University of Posts and Telecommunications, Beijing, China, in 2017, and the master degree in School of Software, Tsinghua University, Beijing, China. Her current research interests include software safety and security of blockchain systems and smart contract.

    View full text