Leveraging textual properties of bug reports to localize relevant source files

https://doi.org/10.1016/j.ipm.2018.07.004Get rights and content

Highlights

  • A multi-component bug localization approach that leverages different textual properties of bug reports and source files.

  • The approach utilizes information retrieval, textual matching, stack trace analysis, and multi-label classification.

  • It is applied to real-world software projects and achieved good performance.

  • It Improves the MRR and MAP metrics compared to several existing state-of-the-art approaches.

Abstract

Bug reports are an essential part of a software project's life cycle since resolving them improves the project's quality. When a new bug report is received, developers usually need to reproduce the bug and perform code review to locate the bug and assign it to be fixed. However, the huge number of bug reports and the increasing size of software projects make this process tedious and time-consuming. To solve this issue, bug localization techniques try to rank all the source files of a project with respect to how likely they are to contain a bug. This process reduces the search space of source files and helps developers to find relevant source files quicker. In this paper, we propose a multi-component bug localization approach that leverages different textual properties of bug reports and source files as well as the relations between previously fixed bug reports and a newly received one. Our approach uses information retrieval, textual matching, stack trace analysis, and multi-label classification to improve the performance of bug localization. We evaluate the performance of the proposed approach on three open source software projects (i.e., AspectJ, SWT, and ZXing) and the results show that it can rank appropriate source files for more than 52% of bugs by recommending only one source file and 78% by recommending ten files. It also improves the MRR and MAP values compared to several existing state-of-the-art bug localization approaches.

Introduction

A software bug is an error, flaw, failure or fault in a computer program that produces an incorrect or unexpected result. The quality of a software project is an important aspect for its success. Although different software quality control processes like software testing and software inspection are used, the increasing size of software and the limited development resources make developers release their software while still containing bugs. To deal with this problem, software projects set up bug tracking systems such as Bugzilla1 and MantisBT2 to keep track of bugs in their product effectively and then ask users all over the world to report bugs that they have confronted. However, this process can also create a new challenge for developers.

During the development and post-release life cycle of a software product, especially in a popular and well-known system, the number of bug reports can grow large and easily become tedious to handle. For example, there were 3,389 bug reports created for the Eclipse platform product in 2013 alone (Ye, Bunescu, & Liu, 2014) and Mozilla project receives on average 300 new bug reports every day (Shokripour, Anvik, Kasirun, & Zamani, 2013). These numbers show that it's essential for the development teams to efficiently locate buggy files and resolve the problem.

After a bug is reported and confirmed, the development team should find which source files contain the problem and who will be assigned to make the code changes necessary to fix the fault (Jeong, Kim, & Zimmermann, 2009). As the software projects also become bigger and more complicated, one of the most time-consuming and difficult tasks is finding the buggy files that are responsible for a reported bug among hundreds or thousands of source files (Anvik, Hiew, & Murphy, 2005). Furthermore, developers who have to maintain and work on unfamiliar projects need to put a lot of effort in comprehending and understanding different parts of the program to find higher level concepts that are described by the bug report. Therefore, it would be beneficial to have an automatic system help us in this process that recommend buggy source files to developers based on the given bug report.

Researchers have proposed approaches to automate the process of finding the faulty source files. These approaches rank source files according to their relevance to the given bug report and make the developers’ search space smaller. Developers then examine the files from the beginning of the ranked list to find the relevant faulty files.

One category of these approaches is spectrum-based that need the spectra information between different executions of the program to help locate faults (Abreu, Zoeteweij, Golsteijn and van Gemund, 2009, Jones and Harrold, 2005, Le, Lo, Le Goues and Grunske, 2016, Liu et al., 2006). Spectrum-based approaches usually need a test suite to execute the program and collect its execution information; while the lack of test cases in software projects makes it hard to collect many correct and failing execution traces. Another category of bug localization approaches is information retrieval (IR)-based techniques that are static and don't require runtime information of the program (Rahman, Ganguly and Sakib, 2015, Saha, Lease, Khurshid and Perry, 2013, Wang and Lo, 2014, Wong et al., 2014, Zhou, Tong, Chen and Han, 2017, Zhou, Zhang and Lo, 2012). These approaches locate the bug-relevant files based on a given bug report. In fact, the text in the summary and description of a given bug report is considered as a query and source files are documents that are ranked based on their relevance to the bug report's content.

As shown in Fig. 1, there are similar word tokens in different parts of a bug report3 and its associated source file. The intuition here is that software elements that share many common word tokens with the given bug report are likely to be relevant to the bug and can help locate the appropriate source file for each bug report quicker (Zhou et al., 2012). Apart from word tokens, full identifiers are often present in bug reports and can further help in going toward the right source file (Saha et al., 2013). Some matching positions in the source file are more important than others. We can give more weight to matching terms or word tokens in these positions in order to avoid their effect loss in finding the relevant source file. Words in bug reports also have different importance. For example, this can be seen in Fig. 2 as words with specific part-of-speech or POS tags are more important, and mostly nouns and verbs are the common words between the bug report and source code.

Some bug reports contain a stack trace and the relevant file name for a bug report can be found there. It has been shown that the relevant source file of about 90% of bugs is located within the top 10 stack frames (Schröter, Bettenburg, & Premraj, 2010), and as expected, most of the times it is in the first frame. Fig. 3 shows an example of a bug report4 containing a stack trace of a component that has crashed. The relevant source file for this bug report is “ProgramElement.java”, which is located in the first frame of the stack trace.

Sometimes the terms used to describe the bug in the bug report are not the same as the ones used in the source code, but if we consider them semantically, they are close in meaning. For example in Fig. 4, the word “camera” is used, and the related source file to this bug report has the term “capture” in its name. We know that “camera” and “capture” are related to each other; therefore we can use semantic similarity to overcome this lexical mismatch between the bug report and source file. There are many different mismatches in what bug reports contain and what developers consider useful in them to better localize the bugs (Bettenburg et al., 2008), but lexical mismatch makes it harder for a bug localization system. In this case, some domain knowledge about the software project is needed to locate relevant files (Ye et al., 2014).

Previously fixed source files may be responsible for newer similar bugs (Murphy-Hill, Zimmermann, Bird, & Nagappan, 2013), and sometimes people submit bug reports similar to these already fixed bug reports that may affect the same source file. Since usually there are already fixed bug reports available in bug tracking systems of software projects, common tokens shared between new and fixed reports can be used to further improve the bug localization performance.

Although many bug localization approaches are proposed, there are still some gaps that can be covered to improve their accuracy. Motivated by these observations, in this paper, we propose a five-component bug localization approach that combines various textual aspects of bug reports and source files to locate relevant source files for each bug report. The components consist of a token matching component that tries to find exact matches of some specific textual tokens of bug reports in specific positions of source files; a Vector Space Model (VSM) similarity component that utilizes a variation of the revised Vector Space Model (rVSM) (Zhou et al., 2012) with specifically extracted word tokens to better score source files for a bug report; a stack trace component that uses stack traces in bug reports if available; a semantic similarity component, and finally a fixed bug reports component that uses previously fixed bug reports and a multi-label classification algorithm to score suspicious faulty source files according to existing solved bug reports.

As the main contribution of this paper, we first develop these five components to get separate ranking scores for each source file. Then we combine these scores to get a final ranking score and use that to rank relevant source files with respect to each bug report. After that, we try to show how good our approach works on the bug reports of real-world software projects. Therefore, we evaluate our approach on three open source software projects, namely, AspectJ, SWT, and ZXing. We also compare our approach with several state-of-the-art bug localization approaches including BugLocator proposed by Zhou et al. (2012), BLUiR proposed by Saha et al. (2013), BRTracer proposed by Wong et al. (2014), AmaLgam proposed by Wang and Lo (2014), and Rahman et al. (2015) approach. Results show that on average for the benchmark projects, our approach can rank appropriate source files for 52.5% of bugs in top 1 and 78.2% in top 10 and also improves the mean reciprocal rank (MRR) and mean average precision (MAP) values. Considering the MAP value, the improvement is 37.8%, 25.4%, 16.8%, and 12.5% over BugLocator, BLUiR, BRTracer, and AmaLgam, respectively. In the same order, the MRR value has the improvement of 38.2%, 25.9%, 17.8%, and 13.1%.

This paper significantly extends our preliminary work (Gharibi, Rasekh, & Sadreddini, 2017). Two new components (i.e., stack trace and semantic similarity components) are added, and previous components are further improved. More extensive evaluation is also done. We added one more software project to our benchmark dataset and compared our results to four more previous studies with and without the use of previously fixed bug reports. In addition, the rest of the paper is also substantially expanded with new content.

The remainder of the paper is structured as follows. In Section 2, we review how this work relates to other research. The details of our proposed approach are described in Section 3. The experimental setup of the model is explained in Section 4. Section 5, presents the results and discusses our main findings. Finally, this is followed by Section 6, where we draw conclusions and describe future work.

Section snippets

Related work

In this section, we briefly review the related work. We divide related studies into four categories. First, textual and IR-based bug localization approaches are discussed. Then we describe some feature location works and spectrum-based methods. Finally, other related studies are also shown.

Proposed approach

In this section, we describe our proposed approach for building an automatic bug localization tool that locates buggy files based on the information it gathers from the given bug reports. Our approach consists of a parsing and preprocessing phase, and five components that are the token matching, VSM similarity, stack trace, semantic similarity, and fixed bug reports. Each of the mentioned components gives a score to source files with respect to each bug report. Then these scores are combined to

Experimental setup

In this section, first, we describe the dataset and evaluation metrics used to evaluate the performance of the proposed approach. Then, we discuss our research questions and present the implementation details.

Overall results

Table 3 shows the experimental results on AspectJ, SWT and ZXing projects and also the comparison with the reported results of the five state-of-the-art bug localization approaches including BugLocator (Zhou et al., 2012), BLUiR (Saha et al., 2013), BRTracer (Rahman, Ganguly and Sakib, 2015, Wong et al., 2014), Rahman et al. (2015), and AmaLgam (Wang & Lo, 2014). Since these approaches all have used the same dataset as us, we compare our results with their performance as it's reported in their

Conclusion and future work

The developer teams of a software project need to locate appropriate files for each received bug report to solve the system's problem. For this matter, searching through a considerable amount of source files is inefficient and assigning bug reports to the wrong developers wastes their time. To overcome this manual effort, bug localization techniques try to automate the process and rank relevant source files for each bug report. This way buggy files will rank higher, and the developer can find

References (71)

  • M. Borg et al.

    Recovering from a decade: A systematic mapping of information retrieval approaches to software traceability

    Empirical Software Engineering

    (2014)
  • M. Borg et al.

    Supporting change impact analysis using a recommendation system: An industrial case study in a safety-critical context

    IEEE Transactions on Software Engineering

    (2017)
  • J. Cleland-Huang et al.

    Software traceability: Trends and future directions

    Future of software engineering

    (2014)
  • B. Dit et al.

    Feature location in source code: A taxonomy and survey

    Journal of Software: Evolution and Process

    (2013)
  • B.P. Eddy et al.

    Impact of structural weighting on a latent Dirichlet allocation–based feature location technique

    Journal of Software: Evolution and Process

    (2018)
  • G. Gay et al.

    On the use of relevance feedback in IR-based concept location

  • R. Gharibi et al.

    Locating relevant source files for bug reports using textual analysis

  • G. Jeong et al.

    Improving bug triage with bug tossing graphs

  • H. Jiang et al.

    PRST: A pagerank-based summarization technique for summarizing bug reports with duplicates

    International Journal of Software Engineering and Knowledge Engineering

    (2017)
  • J.A. Jones et al.

    Empirical evaluation of the tarantula automatic fault-localization technique

  • L. Jonsson et al.

    Automated bug assignment: Ensemble-based machine learning in large scale industrial contexts

    Empirical Software Engineering

    (2016)
  • D. Kim et al.

    Where should we fix this bug? A two-phase recommendation model

    IEEE Transactions on Software Engineering

    (2013)
  • S. Kim et al.

    Predicting faults from cached history

  • P.S. Kochhar et al.

    Practitioners’ expectations on automated fault localization

  • A.N. Lam et al.

    Combining deep learning with information retrieval to localize buggy files for bug reports

  • T.-D.B. Le et al.

    A learning-to-rank based fault localization approach using likely invariants

  • T.-D.B. Le et al.

    Information retrieval and spectrum based bug localization: Better together

  • T.-D.B. Le et al.

    Predicting effectiveness of IR-based bug localization techniques

  • C. Liu et al.

    Statistical debugging: A hypothesis testing-based approach

    IEEE Transactions on Software Engineering

    (2006)
  • X.-Y. Liu et al.

    Measuring semantic similarity in WordNet

  • S.K. Lukins et al.

    Source code retrieval for bug localization using latent Dirichlet allocation

  • S. Mani et al.

    Ausum: Approach for unsupervised bug report summarization

  • C.D. Manning et al.

    Introduction to information retrieval

    (2008)
  • A. Marcus et al.

    An information retrieval approach to concept location in source code

  • A. McCallum et al.

    A comparison of event models for naive bayes text classification

  • Cited by (0)

    View full text