Friedrich Steimann
ABSTRACT
Resuming past work on coverage-based fault localization, we find that empirical assessments of its accuracy are subject to so many imponderables that they are of limited value. To improve on this situation, we have compiled a comprehensive list of threats to be considered when attempting such assessments in the future. In addition, we propose the establishment of theoretical lower and upper bounds of fault localization accuracy that depend on properties of the subject programs (including their test suites) only. We make a suggestion for a lower bound and show that well-known fault locators do not uniformly perform better.
- R Abreu, P Zoeteweij, AJC van Gemund “An evaluation of similarity coefficients for software fault localization” in: Proc. of PRDC (2006) 39–46. Google Scholar
Digital Library
- R Abreu, P Zoeteweij, R Golsteijn, AJC van Gemund “A practical evaluation of spectrum-based fault localization” Journal of Systems and Software 82:11 (2009) 1780–1792. Google ScholarDigital Library
- R Abreu, P Zoeteweij, AJC van Gemund “Spectrum-based multiple fault localization” in: Proc. of ASE (2009) 88–99. Google ScholarDigital Library
- S Ali, JH Andrews, T Dhandapani, W Wang “Evaluating the accuracy of fault localization techniques” in: Proc. of ASE (2009) 76–87. Google ScholarDigital Library
- JH Andrews, LC Briand, Y Labiche “Is mutation an appropriate tool for testing experiments?” in: Proc. of ICSE (2005) 402–411. Google ScholarDigital Library
- GK Baah, A Podgurski, MJ Harrold “Mitigating the confounding effects of program dependences for effective fault localization” in: Proc. of SIGSOFT FSE (2011) 146–156. Google ScholarDigital Library
- J Bentley More Programming Pearls: Confessions of a Coder (Addison Wesley 1988). Google Scholar
- MY Chen, E Kiciman, E Fratkin, A Fox, A., EA Brewer “Pinpoint: Problem determination in large, dynamic internet services” in: Proc. of DSN (2002) 595–604. Google ScholarDigital Library
- H Cleve, A Zeller “Locating causes of program failures” in: Proc. of ICSE (2005) 342–351. Google ScholarDigital Library
- V Dallmeier, C Lindig, A Zeller “Lightweight defect localization for Java” in: Proc. of ECOOP (2005) 528–550. Google ScholarDigital Library
- N DiGiuseppe, JA Jones “On the influence of multiple faults on coverage-based fault localization” in: Proc. of ISSTA (2011) 210–220. Google ScholarDigital Library
- G Fraser, A Arcuri “Sound empirical evidence in software testing” in: Proc. of ICSE (2012) 178–188. Google ScholarDigital Library
- A Gonzalez-Sanchez, R Abreu, HG Gross, AJC von Gemund “Prioritizing tests for fault localization through ambiguity group reduction” in: Proc. of ASE (2011) 83-92. Google ScholarDigital Library
- R Gore, PF Reynolds Jr. “Reducing confounding bias in predicate-level statistical debugging metrics” in: Proc. of ICSE (2012) 463–473. Google ScholarDigital Library
- JA Jones, MJ Harrold, JT Stasko “Visualization of test information to assist fault localization” in: Proc. of ICSE (2002) 467–477. Google ScholarDigital Library
- JA Jones, MJ Harrold “Empirical evaluation of the tarantula automatic fault-localization technique” in: Proc. of ASE (2005) 273–282. Google ScholarDigital Library
- JA Jones, MJ Harrold, JF Bowring “Debugging in parallel” in: Proc. of ISSTA (2007) 16–26. Google ScholarDigital Library
- JA Jones Semi-Automatic Fault Localization PhD Thesis (Georgia Institute of Technology, 2008). Google ScholarDigital Library
- L Naish, H Jie L, K Ramamohanarao “A model for spectrabased software diagnosis” ACM Trans. Softw. Eng. Methodol. 20(3):11 (2011). Google ScholarDigital Library
- AS Namin, S Kakarla “The use of mutation in testing experiments and its sensitivity to external threats” in: Proc. of ISSTA (2011) 342–352. Google ScholarDigital Library
- C Parnin, A Orso “Are automated debugging techniques actually helping programmers?” in: Proc of ISSTA (2011) 199– 209. Google ScholarDigital Library
- M Renieris, SP Reiss “Fault localization with nearest neighbor queries” in: Proc. of ASE (2003) 30–39.Google Scholar
- D Saff “Theory-infected: or how I learned to stop worrying and love universal quantification” in: OOPSLA Companion (2007) 846–847. Google ScholarDigital Library
- RA Santelices, JA Jones, Y Yu, MJ Harrold “Lightweight fault-localization using multiple coverage types” in: Proc. of ICSE (2009) 56–66. Google ScholarDigital Library
- F Steimann, T Eichstädt-Engelen, M Schaaf “Towards raising the failure of unit tests to the level of compiler-reported errors” in: Proc. of TOOLS Europe (2008) 60–79.Google Scholar
- F Steimann, M Bertschler “A simple coverage-based locator for mul-tiple faults” in: Proc of ICST (2009) 366–375. Google ScholarDigital Library
- F Steimann, M Frenkel “Improving coverage-based localization of multiple faults using algorithms from Integer Linear Programming” in: Proc. of ISSRE (2012).Google ScholarDigital Library
- X Wang, SC Cheung, WK Chan, Z Zhang “Taming coincidental correctness: Coverage refinement with context patterns to improve fault localization” in: Proc. of ICSE (2009) 45–55. Google ScholarDigital Library
- X Xie, T Chen, FC Kuo, B Xu “A theoretical analysis of the risk evaluation formulas for spectrum-based fault Localization” ACM TOSEM (in print).Google Scholar
- X Xu, V Debroy, WE Wong, D Guo “Ties within fault localization rankings: Exposing and addressing the problem” International Journal of Software Engineering and Knowledge Engineering 21:6 (2011) 803–827.Google ScholarCross Ref
- S Yoo, M Harman, D Clark “Fault localization prioritization: Comparing information theoretic and coverage based approaches” ACM TOSEM (in press).Google Scholar
- Z Zhang, WK Chan, TH Tse, YT Yu, P Hu “Non-parametric statistical fault localization” Journal of Systems and Software 84:6 (2011) 885–905. Google ScholarDigital Library
Index Terms
- Threats to the validity and value of empirical assessments of the accuracy of coverage-based fault locators
Recommendations
Fault density, fault types, and spectra-based fault localization
This paper presents multiple empirical experiments that investigate the impact of fault quantity and fault type on statistical, coverage-based fault localization techniques and fault-localization interference. Fault-localization interference is a ...
On the influence of multiple faults on coverage-based fault localization
ISSTA '11: Proceedings of the 2011 International Symposium on Software Testing and AnalysisThis paper presents an empirical study on the effects of the quantity of faults on statistical, coverage-based fault localization techniques. The former belief was that the effectiveness of fault-localization techniques was inversely proportional to the ...
Iterative Feedback-Based Fault Localization Approach
SEAA '11: Proceedings of the 2011 37th EUROMICRO Conference on Software Engineering and Advanced ApplicationsMost current fault-localization approaches merely use the information from testing to locate faults, assuming that test cases satisfying a certain test adequacy criterion can provide adequate information for fault localization. Unfortunately however, ...
Comments