Robustness of spectrum-based fault localisation in environments with labelling perturbations

Highlights

• It is the first work to explore the robustness of SBFL under labelling perturbations.

• Influence of labelling perturbations on three relations is theoretically analysed.

• Effect of multiple mislabelled cases on the outputting ranks is theoretically analysed.

• The robustness of 23 classes of risk evaluation formulas are empirically studied.

• A new evaluation metric for SBFL is proposed.

Abstract

Most fault localisation techniques take as inputs a faulty program and a test suite, and produce as output a ranked list of suspicious code locations at which the program may be defective. If only a small portion of the executions are labelled erroneously, we expect a fault localisation technique to be robust to these errors. However, it is not known which fault localisation techniques with high accuracy are robust and which techniques are best at finding faults under the trade-off between accuracy and robustness.

In this paper, a theoretical analysis of the impacts of labelling perturbations on spectrum-based fault localisation techniques (SBFL) is presented from different aspects first. We theoretically analyse the influence of labelling perturbations on three relations among risk evaluation formulas and the effect of mislabelling cases on the ranking of faulty statements. Then, we conduct controlled experiments on 18 programs with 3079 faulty versions from different domains to compare the robustness of 23 classes of risk evaluation formulas. Besides, experiments are conducted for evaluating the robustness of two neural network-based techniques. The impacts of perturbation degrees, number of faults and types of labelling perturbation on the robustness of formulas are empirically studied, and several interesting findings are obtained.

Introduction

Software and software systems can be found everywhere in daily life and software failures are frequently encountered. Many failures are due to faults (or bugs) that are embedded in programs during development, and debugging is a very effective way of identifying their presence. It is commonly recognized that debugging is important but resource-consuming in software engineering, in which fault localisation is one of the essential activities. Due to a substantial amount of manual involvement, fault localisation is a very resource-consuming task in the software development lifecycle. Therefore, many researchers have proposed various automatic and effective techniques for fault localisation to decrease its cost and to increase the software quality.

One promising approach towards fault localisation is spectrum-based fault localisation (referred to as SBFL in this article). SBFL refers to the automatic mechanism for predicting fault positions in a faulty program by analysing the dynamic program spectra that are captured in program runs. Typically, a program element that is always exercised in failed runs and never exercised in passed runs has a high chance of explaining the observed failures and is deemed very suspicious, in terms of relations to one or more faults. Many heuristics (Jones and Harrold, 2005, Abreu et al., 2007, Gong et al., 2012, Steimann et al., 2013, Neelofar et al., 2017) and mathematical models (Liblit et al., 2005, Liu et al., 2006, Zhang et al., 2011, Gore and Reynolds, 2012, Tang et al., 2017) have been proposed.

SBFL has received substantial attention due to its simplicity and effectiveness. It takes as inputs a faulty program and a test suite and produces as output a ranked list of suspicious code locations at which the program may be defective. Ideally, there should be no perturbations to the input of SBFL. In this way, output values of an SBFL technique will not be changed by the perturbation parameters of input values. Unfortunately, perturbations could occur in real testing process. They may produce errors to the obtained test information, which are propagated by SBFL techniques and cause unexpected results. Consequently, we have a problem: will a small perturbation of the input cause a large variance?

There could be various types of perturbations in the process of testing and debugging. As an attempt to study the above problem, we focus on labelling perturbations in this work. This type of perturbations is caused by the incorrect labelling on a small number of test cases in a test suite, for example mislabelling a test case as passed although it is actually failed or vice versa. It could be very common due to the facts such as human errors, imperfect development of test systems, differences between the test environment and the actual execution environment, and etc. We expect a fault localisation technique to be robust to the perturbations. However, it has been observed that, even under a small labelling perturbation, there may be a substantial impact on the results of fault localisation. In the example shown in Table 1,¹ although there is only one test case mislabelled as failed, the faulty statement is given the lowest suspiciousness degree by Naish1, which has been evaluated as one of the “maximal” risk evaluation formula under the single-fault scenario. The ranking of the faulty statement drops from the first to almost the last. For other formulas, for example Jaccard, we can find a similar situation as shown in Table 1.

In this paper, we first provide a theoretical investigation of the impacts of labelling perturbations on the accuracy of risk evaluation formulas. The preservation of three relations among the formulas, namely, strict equivalence relations (Naish et al., 2011), Xie and Chen's equivalence relations (Xie et al., 2013a), and Xie and Chen's order relations (Xie et al., 2013a), is proved under the scenario of labelling perturbations. In addition, the problem of how the labelling perturbation influences the outputting rank list of formulas is theoretically studied both in the scenario of all mislabelling activities are in the same direction and the mislabelling activities are in different directions.

To further explore the impacts of labelling perturbations on different risk evaluation formulas, we conducted controlled experiments using the Siemens suite, UNIX utility software, space and Defects4J. The robustness of 23 classes of risk evaluation formulas and their impact factors, including perturbation degrees, number of faults and types of labelling perturbation are empirically studied. We observe that (1) Different risk evaluation formulas usually have different robustness values and the robustness values of risk evaluation formulas are not positive or negative correlation to their Expense; (2) The robustness of most risk evaluation formulas decreases with the increase of perturbation degrees; (3) Most formulas show an increasing trend of robustness with the increase of the number of faults; (4) On average, the impacts of mislabelling passed cases as failed cases are greater than the ones of mislabelling failed cases as passed cases. Based on the findings, a new metric is proposed for evaluating risk evaluation formulas by synthetically determining their robustness and accuracy. Experiments show the rationality of the metric. Besides, we also perform the experiments to evaluate the robustness of two neural network-based fault localization techniques.

The rest of this article is organized as follows: Section 2 provides the problem description and motivation of this work. Section 3 introduces the theoretical analysis from two aspects. Section 4 presents an empirical study on 18 programs with 3079 faulty versions from different domains. Section 5 discusses the threats to the validity of this work. In Section 6, a review of previous theoretical and empirical studies is presented. Finally, the conclusions for this work is discussed in Section 7.