Simultaneous debugging of software faults

Abstract

(Semi-)automated diagnosis of software faults can drastically increase debugging efficiency, improving reliability and time-to-market. Current automatic diagnosis techniques are predominantly of a statistical nature and, despite typical defect densities, do not explicitly consider multiple faults, as also demonstrated by the popularity of the single-fault benchmark set of programs. We present a reasoning approach, called Zoltar-M(ultiple fault), that yields multiple-fault diagnoses, ranked in order of their probability. Although application of Zoltar-M to programs with many faults requires heuristics (trading-off completeness) to reduce the inherent computational complexity, theory as well as experiments on synthetic program models and multiple-fault program versions available from the software infrastructure repository (SIR) show that for multiple-fault programs this approach can outperform statistical techniques, notably spectrum-based fault localization (SFL). As a side-effect of this research, we present a new SFL variant, called Zoltar-S(ingle fault), that is optimal for single-fault programs, outperforming all other variants known to date.

Introduction

Automatic software fault localization (also known as fault diagnosis) techniques aid developers to pinpoint the root cause of detected failures, thereby reducing the debugging effort. Two approaches can be distinguished:

• the spectrum-based fault localization (SFL) approach, which correlates software component activity with program failures (a statistical approach) (Abreu et al., 2007, Gupta et al., 2005, Jones et al., 2002, Liu et al., 2005, Renieris and Reiss, 2003, Zeller, 2002), and

• the model-based diagnosis or debugging (MBD) approach, which deduces component failure through logic reasoning (de Kleer and Williams, 1987, Feldman et al., 2008, Mayer and Stumptner, 2008, Wotawa et al., 2002).

Because of its low computational complexity, SFL has gained large popularity. Although inherently not restricted to single faults, in most cases these statistical techniques are applied and evaluated in a single-fault context, as demonstrated by the benchmark set of programs widely used by researchers,¹ which is seeded with only one fault per program (version). In practice, however, the defect density of even small programs typically amounts to multiple faults. Although the root cause of a particular program failure need not constitute multiple faults that are acting simultaneously, many failures will be caused by different faults. Hence, the problem of multiple-fault localization (diagnosis) deserves detailed study.

Unlike SFL, MBD traditionally deals with multiple faults. However, apart from much higher computational complexity, the logic models that are used in the diagnostic inference are typically based on static program analysis. Consequently, they do not exploit execution behavior, which, in contrast, is the essence of the SFL approach. Combining the dynamic approach of SFL with the multiple-fault logic reasoning approach of MBD, in this paper, we present a multiple-fault reasoning approach that is based on the dynamic, spectrum-based observations of SFL. Additional reasons to study the merits of this approach are the following.

• Diagnoses are returned in terms of multiple faults, whereas statistical techniques return a one-dimensional list of single fault locations only. The information on fault multiplicity is attractive from parallel debugging point of view (Jones et al., 2007).

• Unlike statistical approaches, multiple-fault diagnoses only include valid candidates, and are asymptotically optimal with increasing test information (Abreu et al., 2008).

• The ranking of the diagnoses is based on probability instead of similarity. This implies that the quality of a diagnosis can be expressed in terms of information entropy or any other metric that is based on probability theory (Pietersma and van Gemund, 2006).

• The reasoning approach naturally accommodates additional (model) information about component behavior, increasing diagnostic performance when more information about component behavior is available.

To illustrate the difference between multiple-fault and the statistical approach, consider a triple-fault (sub)program with faulty components c₁, c₂, and c₃. Whereas under ideal testing circumstances a traditional SFL approach would produce multiple single-fault diagnoses (in terms of the component indices) like {{1}, {2}, {3}, {4}, {5}, …} (ordered in terms of statistical similarity), a multiple-fault approach would simply produce one single multiple-fault diagnosis {{1, 2, 3}}. Although the statistical similarity of the first three items in the former diagnosis would be highest, the latter, single diagnosis unambiguously reveals the actual triple fault.

Despite the above advantages, a reasoning approach is more costly than statistical approaches because an exponential number of multiple-fault candidates need to be processed instead of just the (M, being M the number of components in the system under analysis) single-fault candidates. In this paper, we compare our reasoning approach to several statistical approaches. Our study is based on random synthetic spectra, as well as on several benchmark programs, extended by us to accommodate multiple faults. More specifically, this paper makes the following five contributions.

• We introduce a multiple-fault diagnosis approach that originates from the model-based diagnosis area, but which is specifically adapted to the interaction dynamics of software. The approach is coined Zoltar-M (Zoltar for the name of our debugging tool set (Janssen et al., 2009),² M for multiple-fault).

• We show how our reasoning approach applies to single-fault programs, yielding a provably optimal SFL variant, called Zoltar-S (S for single-fault), as of yet unknown in literature.

• We introduce a general, multiple-fault, probabilistic program (spectrum) model, parameterized in terms of size, testing code coverage, and testing fault coverage, to theoretically study Zoltar-M, compared to statistical techniques such as Tarantula and Zoltar-S.

• We extend the traditional, single-fault benchmark set of programs (referred to as SIR-S) with a multiple-fault version (SIR-M), by combining the existing single-fault versions, to empirically evaluate debugging performance under realistic, multiple-fault conditions.

• We investigate the ability of all techniques to deduce program fault multiplicity, which is aimed at providing a good estimate to guide parallel debugging, using an approach that substantially differs from Jones et al. (2007).

To the best of our knowledge, this is the first paper to specifically address software multiple-fault localization using a spectrum-based, logic reasoning approach, yielding two new localization techniques Zoltar-S and Zoltar-M, implemented within our Zoltar SFL framework. Our experiments confirm that Zoltar-S is superior to all known similarity coefficients for the Siemens-S benchmark. More importantly however, our experiments for multiple-fault programs show that although for synthetic spectra Zoltar-M is outperformed by Zoltar-S, for our SIR-M experiments Zoltar-M outperforms all similarity coefficients known to date.

The paper is organized as follows. In the next section, we present the concepts and terminology used throughout the paper. In Section 3, our multiple-fault localization approach is described, as well as a derivation of the optimal similarity coefficient for single-fault programs. In Section 4, the approaches are theoretically evaluated, and in Section 5, real programs are used to assess the capabilities of the studied techniques for fault localization. Related work is discussed in Section 6. Preliminary results of Sections 4 Theoretical evaluation, 5 Empirical evaluation appeared in Abreu et al. (2009a). We conclude and discuss future work in Section 7.