skip to main content
10.1145/2483760.2483767acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article

Threats to the validity and value of empirical assessments of the accuracy of coverage-based fault locators

Published: 15 July 2013 Publication History

Abstract

Resuming past work on coverage-based fault localization, we find that empirical assessments of its accuracy are subject to so many imponderables that they are of limited value. To improve on this situation, we have compiled a comprehensive list of threats to be considered when attempting such assessments in the future. In addition, we propose the establishment of theoretical lower and upper bounds of fault localization accuracy that depend on properties of the subject programs (including their test suites) only. We make a suggestion for a lower bound and show that well-known fault locators do not uniformly perform better.

References

[1]
R Abreu, P Zoeteweij, AJC van Gemund “An evaluation of similarity coefficients for software fault localization” in: Proc. of PRDC (2006) 39–46.
[2]
R Abreu, P Zoeteweij, R Golsteijn, AJC van Gemund “A practical evaluation of spectrum-based fault localization” Journal of Systems and Software 82:11 (2009) 1780–1792.
[3]
R Abreu, P Zoeteweij, AJC van Gemund “Spectrum-based multiple fault localization” in: Proc. of ASE (2009) 88–99.
[4]
S Ali, JH Andrews, T Dhandapani, W Wang “Evaluating the accuracy of fault localization techniques” in: Proc. of ASE (2009) 76–87.
[5]
JH Andrews, LC Briand, Y Labiche “Is mutation an appropriate tool for testing experiments?” in: Proc. of ICSE (2005) 402–411.
[6]
GK Baah, A Podgurski, MJ Harrold “Mitigating the confounding effects of program dependences for effective fault localization” in: Proc. of SIGSOFT FSE (2011) 146–156.
[7]
J Bentley More Programming Pearls: Confessions of a Coder (Addison Wesley 1988).
[8]
MY Chen, E Kiciman, E Fratkin, A Fox, A., EA Brewer “Pinpoint: Problem determination in large, dynamic internet services” in: Proc. of DSN (2002) 595–604.
[9]
H Cleve, A Zeller “Locating causes of program failures” in: Proc. of ICSE (2005) 342–351.
[10]
V Dallmeier, C Lindig, A Zeller “Lightweight defect localization for Java” in: Proc. of ECOOP (2005) 528–550.
[11]
N DiGiuseppe, JA Jones “On the influence of multiple faults on coverage-based fault localization” in: Proc. of ISSTA (2011) 210–220.
[12]
G Fraser, A Arcuri “Sound empirical evidence in software testing” in: Proc. of ICSE (2012) 178–188.
[13]
A Gonzalez-Sanchez, R Abreu, HG Gross, AJC von Gemund “Prioritizing tests for fault localization through ambiguity group reduction” in: Proc. of ASE (2011) 83-92.
[14]
R Gore, PF Reynolds Jr. “Reducing confounding bias in predicate-level statistical debugging metrics” in: Proc. of ICSE (2012) 463–473.
[15]
JA Jones, MJ Harrold, JT Stasko “Visualization of test information to assist fault localization” in: Proc. of ICSE (2002) 467–477.
[16]
JA Jones, MJ Harrold “Empirical evaluation of the tarantula automatic fault-localization technique” in: Proc. of ASE (2005) 273–282.
[17]
JA Jones, MJ Harrold, JF Bowring “Debugging in parallel” in: Proc. of ISSTA (2007) 16–26.
[18]
JA Jones Semi-Automatic Fault Localization PhD Thesis (Georgia Institute of Technology, 2008).
[19]
L Naish, H Jie L, K Ramamohanarao “A model for spectrabased software diagnosis” ACM Trans. Softw. Eng. Methodol. 20(3):11 (2011).
[20]
AS Namin, S Kakarla “The use of mutation in testing experiments and its sensitivity to external threats” in: Proc. of ISSTA (2011) 342–352.
[21]
C Parnin, A Orso “Are automated debugging techniques actually helping programmers?” in: Proc of ISSTA (2011) 199– 209.
[22]
M Renieris, SP Reiss “Fault localization with nearest neighbor queries” in: Proc. of ASE (2003) 30–39.
[23]
D Saff “Theory-infected: or how I learned to stop worrying and love universal quantification” in: OOPSLA Companion (2007) 846–847.
[24]
RA Santelices, JA Jones, Y Yu, MJ Harrold “Lightweight fault-localization using multiple coverage types” in: Proc. of ICSE (2009) 56–66.
[25]
F Steimann, T Eichstädt-Engelen, M Schaaf “Towards raising the failure of unit tests to the level of compiler-reported errors” in: Proc. of TOOLS Europe (2008) 60–79.
[26]
F Steimann, M Bertschler “A simple coverage-based locator for mul-tiple faults” in: Proc of ICST (2009) 366–375.
[27]
F Steimann, M Frenkel “Improving coverage-based localization of multiple faults using algorithms from Integer Linear Programming” in: Proc. of ISSRE (2012).
[28]
X Wang, SC Cheung, WK Chan, Z Zhang “Taming coincidental correctness: Coverage refinement with context patterns to improve fault localization” in: Proc. of ICSE (2009) 45–55.
[29]
X Xie, T Chen, FC Kuo, B Xu “A theoretical analysis of the risk evaluation formulas for spectrum-based fault Localization” ACM TOSEM (in print).
[30]
X Xu, V Debroy, WE Wong, D Guo “Ties within fault localization rankings: Exposing and addressing the problem” International Journal of Software Engineering and Knowledge Engineering 21:6 (2011) 803–827.
[31]
S Yoo, M Harman, D Clark “Fault localization prioritization: Comparing information theoretic and coverage based approaches” ACM TOSEM (in press).
[32]
Z Zhang, WK Chan, TH Tse, YT Yu, P Hu “Non-parametric statistical fault localization” Journal of Systems and Software 84:6 (2011) 885–905.

Cited By

View all
  • (2024)FILO: Automated FIx-LOcus Identification for Android Framework Compatibility IssuesInformation10.3390/info1508042315:8(423)Online publication date: 23-Jul-2024
  • (2024)An Empirical Study of Fault Localization on Novice Programs and Addressing the Tie ProblemInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402450042635:01(19-44)Online publication date: 21-Nov-2024
  • (2024)Test Scheduling Across Heterogeneous Machines While Balancing Running Time, Price, and Flakiness2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00048(449-460)Online publication date: 6-Oct-2024
  • Show More Cited By

Index Terms

  1. Threats to the validity and value of empirical assessments of the accuracy of coverage-based fault locators

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and Analysis
    July 2013
    381 pages
    ISBN:9781450321594
    DOI:10.1145/2483760
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 15 July 2013

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Fault localization
    2. Threats to validity

    Qualifiers

    • Research-article

    Conference

    ISSTA '13
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 58 of 213 submissions, 27%

    Upcoming Conference

    ISSTA '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)20
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 10 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)FILO: Automated FIx-LOcus Identification for Android Framework Compatibility IssuesInformation10.3390/info1508042315:8(423)Online publication date: 23-Jul-2024
    • (2024)An Empirical Study of Fault Localization on Novice Programs and Addressing the Tie ProblemInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402450042635:01(19-44)Online publication date: 21-Nov-2024
    • (2024)Test Scheduling Across Heterogeneous Machines While Balancing Running Time, Price, and Flakiness2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00048(449-460)Online publication date: 6-Oct-2024
    • (2024)Causal Program Dependence AnalysisScience of Computer Programming10.1016/j.scico.2024.103208(103208)Online publication date: Sep-2024
    • (2024)Boosting fault localization of statements by combining topic modeling and OchiaiInformation and Software Technology10.1016/j.infsof.2024.107499173(107499)Online publication date: Sep-2024
    • (2024)An empirical study of fault localization in Python programsEmpirical Software Engineering10.1007/s10664-024-10475-329:4Online publication date: 13-Jun-2024
    • (2024)A Systematic Exploration of Mutation‐Based Fault Localization FormulaeSoftware Testing, Verification and Reliability10.1002/stvr.1905Online publication date: 11-Nov-2024
    • (2023)Triggering Modes in Spectrum-Based Multi-location Fault LocalizationProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613884(1774-1785)Online publication date: 30-Nov-2023
    • (2023)Improving Spectrum-Based Localization of Multiple Faults by Iterative Test Suite ReductionProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598148(1445-1457)Online publication date: 12-Jul-2023
    • (2023)Reliable Fix Patterns Inferred from Static Checkers for Automated Program RepairACM Transactions on Software Engineering and Methodology10.1145/357963732:4(1-38)Online publication date: 26-May-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media