Skip to main content
SpringerLink
Log in

A program slicing-based method for effective detection of coincidentally correct test cases

  • Published:
Computing Aims and scope Submit manuscript

Abstract

Despite the proven applicability of the spectrum-based fault localization (SBFL) methods, their effectiveness may be degraded due to the presence of coincidental correctness, which occurs when faults fail to propagate, i.e., their execution does not result in failures. This article aims at improving SBFL effectiveness by mitigating the effect of coincidentally correct test cases. In this regard, given a test suite in which each test has been classified as failing or passing and each faulty program has a single-bug, we present a program slicing-based technique to identify a set of program entities that directly affect the program output when executed with failing test cases, called failure candidate causes (FCC). We then use FCC set to identify test cases that can be marked as being coincidentally correct. These tests are identified based on two heuristics: the average suspiciousness score of the statements that directly affect the program output and the coverage ratio of those statements. To evaluate our approach, we used several evaluation metrics and conducted extensive experiments on programs containing single and multiple bugs, including both real and seeded faults. The empirical results demonstrate that the proposed heuristics can alleviate the coincidental correctness problem and improve the accuracy of SBFL techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Ju X, Jiang S, Chen X, Wang X, Zhang Y, Cao H (2014) HSFal: Effective fault localization using hybrid spectrum of full slices and execution slices. J Syst Softw 90:3–17

    Article  Google Scholar 

  2. Feyzi F, Parsa S (2017) FPA-FL: Incorporating static fault-proneness analysis into statistical fault localization. J Syst Softw. https://doi.org/10.1016/j.jss.2017.11.002

    Google Scholar 

  3. Wong WE, Debroy V, Gao R, Li Y (2014) The Dstar method for effective software fault localization. IEEE Trans Reliab 63(1):290–308

    Article  Google Scholar 

  4. Wong WE, Debroy V, Xu D (2012) Towards better fault localization: a crosstab-based statistical approach. IEEE Trans Syst Man Cybern C Appl Rev 42(3):378–396

    Article  Google Scholar 

  5. Wong WE, Debroy V, Choi B (2010) A family of code coverage-based heuristics for effective fault localization. J Syst Softw 83(2):188–208

    Article  Google Scholar 

  6. Feyzi F, Parsa S (2017) Inforence: effective fault localization based on information-theoretic analysis and statistical causal inference. Front Comput Sci. https://doi.org/10.1007/s11704-017-6512-z

    Google Scholar 

  7. Feyzi F, Nikravan E, Parsa S (2016) FPA-Debug: Effective statistical fault localization considering fault-proneness analysis. ArXiv preprint arXiv:1612.05780

  8. Abreu R, Zoeteweij P, Golsteijn R, Van Gemund AJ (2009) A practical evaluation of spectrum-based fault localization. J Syst Softw 82(11):1780–1792

    Article  Google Scholar 

  9. Naish L, Lee HJ, Ramamohanarao K (2011) A model for spectra-based software diagnosis. ACM Trans Softw Eng Methodol 20(3):11

    Article  Google Scholar 

  10. Jones JA, Harrold, MJ (2005, November) Empirical evaluation of the tarantula automatic fault-localization technique. In: Proceedings of the 20th IEEE/ACM international conference on automated software engineering. ACM, pp 273–282

  11. Masri W, Podgurski A (2009) Measuring the strength of information flows in programs. ACM Trans Softw Eng Methodol 19(2):5

    Article  Google Scholar 

  12. Hierons RM (2006) Avoiding coincidental correctness in boundary value analysis. ACM Trans Softw Eng Methodol 15(3):227–241

    Article  Google Scholar 

  13. Wang X, Cheung SC, Chan WK, Zhang Z (2009, May) Taming coincidental correctness: coverage refinement with context patterns to improve fault localization. In: Proceedings of the 31st international conference on software engineering. IEEE Computer Society, pp 45–55

  14. Masri W, Assi RA (2014) Prevalence of coincidental correctness and mitigation of its impact on fault localization. ACM Trans Softw Eng Methodol 23(1):8

    Article  Google Scholar 

  15. Masri W, Abou-Assi R, El-Ghali M, Al-Fatairi N (2009, June) An empirical study of the factors that reduce the effectiveness of coverage-based fault localization. In: Proceedings of the 2nd international workshop on defects in large software systems: held in conjunction with the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2009). ACM, pp 1–5

  16. Voas JM (1992) PIE: A dynamic failure-based technique. IEEE Trans Softw Eng 18(8):717–727

    Article  Google Scholar 

  17. Androutsopoulos K, Clark D, Dan H, Hierons RM, Harman M (2014, May) An analysis of the relationship between conditional entropy and failed error propagation in software testing. In: Proceedings of the 36th international conference on software engineering. ACM, pp 573–583

  18. Laski JW, Szermer W, Luczycki P (1995) Error masking in computer programs. Softw Test Verif Reliab 5(2):81–105

    Article  Google Scholar 

  19. Clark D, Hierons R (2012) Squeeziness: an information theoretic measure for avoiding fault masking. Inf Process Lett 112(8–9):335–340

    Article  MathSciNet  MATH  Google Scholar 

  20. Miao Y, Chen Z, Li S, Zhao Z, Zhou Y (2012, June) Identifying coincidental correctness for fault localization by clustering test cases. In: SEKE. pp 267–272

  21. Masri W, Assi RA (2010, April) Cleansing test suites from coincidental correctness to enhance fault-localization. In: 2010 Third international conference on software testing, verification and validation. IEEE, pp 165–174

  22. Xue X, Pang Y, Namin AS (2014, July) Trimming test suites with coincidentally correct test cases for enhancing fault localizations. In: Computer software and applications conference (COMPSAC), 2014 IEEE 38th annual. IEEE, pp 239–244

  23. Zhang X, Gupta N, Gupta R (2006, June) Pruning dynamic slices with confidence. In: ACM SIGPLAN notices, vol 41, No. 6. ACM, pp 169–180

  24. Pearson S, Campos J, Just R, Fraser G, Abreu R, Ernst MD, ... Keller B (2017) Evaluating and improving fault localization. In: Proceedings of the 39th international conference on software engineering. IEEE, pp 609–620

  25. Shu G, Sun B, Henderson TA, Podgurski A (2013, March). JavaPDG: A new platform for program dependence analysis. In: Software testing, verification and validation (ICST), 2013 IEEE sixth international conference on. IEEE, pp 408–415

  26. Ferrante J, Ottenstein KJ, Warren JD (1987) The program dependence graph and its use in optimization. ACM Trans Program Lang Syst 9(3):319–349

    Article  MATH  Google Scholar 

  27. Chekam TT, Papadakis M, Traon YL (2016) Assessing and comparing mutation-based fault localization techniques. ArXiv preprint arXiv:1607.05512

  28. Just R, Jalali D, Inozemtseva L, Ernst MD, Holmes R, Fraser G (2014) Are mutants a valid substitute for real faults in software testing?. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering. ACM, pp 654–665

  29. Just R, Jalali D, Ernst MD (2014) Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In: Proceedings of the 2014 international symposium on software testing and analysis. ACM, pp 437–440

  30. http://valgrind.org/

  31. http://www.elis.ugent.be/diablo/

  32. Zhang X, Gupta R (2004) Whole execution traces. In: IEEE/ACM 37th international symposium on microarchitecture

  33. Wang T, Roychoudhury A (2004) Using compressed bytecode traces for slicing Java programs. In: Proceedings of the 26th international conference on software engineering, ICSE 2004. IEEE, pp 512–521

  34. Ott RL (1993) An introduction to statistical methods and data analysis, 4th edn. Duxbury Press, North Scituate

    Google Scholar 

  35. Steimann F, Frenkel M, Abreu R (July 2013) Threats to the validity and value of empirical assessments of the accuracy of coverage-based fault locators. In: ISSTA. pp 314–324

  36. Wong WE, Gao R, Li Y, Abreu R, Wotawa F (2016) A survey of software fault localization. IEEE Trans Softw Eng 42(8):707–740

    Article  Google Scholar 

  37. Yan S, Chen Z, Zhao Z, Zhang C, Zhou Y (2010) A dynamic test cluster sampling strategy by leveraging execution spectra information. In: ICST. pp 147–154

  38. Dickinson W, Leon D, Podgurski A (2001) Finding failures by cluster analysis of execution profiles. In: ICSE. pp 339–348

  39. Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques. Morgan Kaufmann, Los Altos

    MATH  Google Scholar 

  40. Podgurski A, Leon D, Francis P, Masri W, Minch M, Sun J, Wang B (2003). Automated support for classifying software failure reports. In: Proceedings of the international conference on software engineering. pp 465–474

  41. Yu Y, Jones JA, Harrold MJ (2008) An empirical study of the effects of test-suite reduction on fault localization. In: Proceedings of the 30th international conference on Software engineering. pp 201–210

  42. Woodward MR, Al-Khanjari ZA. Testability, fault size and the domain-to-range ratio: an eternal triangle. In: The international symposium on software testing and analysis, ISSTA ’00. pp 168–172

  43. Chen J, Li Q, Zhao J, Li X (2010) Test adequacy criterion based on coincidental correctness probability. In: The second Asia-Pacific symposium on Internetware, Internetware’10. ACM, Suzhou, pp 20:1–20:4

  44. Masri W, Assi RA, Zaraket F, Fatairi N (2012, April) Enhancing fault localization via multivariate visualization. In: 2012 IEEE fifth international conference on software testing, verification and validation. IEEE, pp 737–741

  45. Valizadegan H, Tan PN (2007, April) Kernel based detection of mislabeled training examples. In: SDM. pp 309–319

  46. Papadakis M, Le Traon Y (2015) Metallaxis-FL: mutation-based fault localization. Softw Test Verif Reliab 25(5–7):605–628

    Article  Google Scholar 

  47. Wen W (2012, June) Software fault localization based on program slicing spectrum. In: Proceedings of the 34th international conference on software engineering. IEEE Press, pp 1511–1514

  48. Mao X, Lei Y, Dai Z, Qi Y, Wang C (2014) Slice-based statistical fault localization. J Syst Softw 89:51–62

    Article  Google Scholar 

  49. Agrawal H, Horgan JR, London S, Wong WE (1995, October) Fault localization using execution slices and dataflow tests. In: Software reliability engineering, 1995. Proceedings, sixth international symposium on. IEEE, pp 143–151

  50. Gupta N, He H, Zhang X, Gupta R (2005, November) Locating faulty code using failure-inducing chops. In: Proceedings of the 20th IEEE/ACM international conference on automated software engineering. ACM, pp 263–272

  51. Zeller A, Hildebrandt R (2002) Simplifying and isolating failure-inducing input. IEEE Trans Softw Eng 28(2):183–200

    Article  Google Scholar 

  52. Zhang X, Gupta N, Gupta R (2006, May) Locating faults through automated predicate switching. In: Proceedings of the 28th international conference on Software engineering. ACM, pp 272–281

  53. Jeffrey D, Gupta N, Gupta R (2008, July). Fault localization using value replacement. In: Proceedings of the 2008 international symposium on Software testing and analysis. ACM, pp 167–178

  54. Zhang X, Tallam S, Gupta N, Gupta R (2007, June) Towards locating execution omission errors. In: ACM sigplan notices, vol 42, No. 6. ACM, pp 415–424

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Iran University of Science and Technology, Tehran, Iran

    Farid Feyzi & Saeed Parsa

Authors
  1. Farid Feyzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Saeed Parsa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saeed Parsa.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feyzi, F., Parsa, S. A program slicing-based method for effective detection of coincidentally correct test cases. Computing 100, 927–969 (2018). https://doi.org/10.1007/s00607-018-0591-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-018-0591-z

Keywords

Mathematics Subject Classification

Search

Navigation