Skip to main content
Springer Nature Link
Log in

On similarity-awareness in testing-based fault localization

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

In the process of software development and maintenance, software debugging is an inevitable and time-consuming task. To accelerate software debugging, various approaches have been proposed to automate fault localization. Among them, testing-based fault-localization approaches are most promising, which use the execution information of many test cases to localize the faults. However, these existing testing-based fault-localization approaches ignore the similarity between test cases, which may harm the effectiveness of these approaches according to our previous research. Therefore, in this paper we propose a similarity-aware fault-localization approach, which takes each test case as a fuzzy set to deal with the similarity between test cases and calculates statements’ suspicions based on the probability theory. To investigate whether SAFL can address the similarity issue effectively, we manually injected redundant test cases in a test suite and performed an experimental study on the original test suite and the test suite with redundancy, respectively. The experimental results demonstrate that in our experiments SAFL is an effective fault-localization approach, whether there is manually injected redundancy in the test suite. To compare SAFL with most existing testing-based fault-localization approaches, we performed another experimental study on Siemens program suite, which is extensively used in the evaluation of many other testing-based fault-localization approaches. This experimental study confirms the effectiveness of SAFL. Based on the two experimental studies, it seems that in our experiments SAFL cannot only deal with test suites containing much redundancy effectively but also perform effectively for test suites without much redundancy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Agrawal, H., DeMillo, R.A., Hathaway, R., Hsu, W., Krauser, E.W., Martin, R.J., Mathur, A.P., Spafford, E.H.: Design of mutant operators for the C programming language. Technical Report SERC-TR-41-P, Purdue University, West Lafayette, IN (1989)

  • Agrawal, H., Horgan, J., London, S., Wong, W.: Fault location using execution slices and dataflow tests. In: IEEE Software Reliability Engineering, pp. 143–151 (1995)

  • Ball, T., Eick, S.G.: Software visualization in the large. Computer 29(4), 33–43 (1996)

    Article  Google Scholar 

  • Barbosa, E., Maldonado, J., Vincenzi, A.: Toward the determination of sufficient mutant operators for C. software testing. Verif. Reliab. 11(2), 113–136 (2001)

    Article  Google Scholar 

  • Baudry, B., Fleurey, F., Traon, Y.L.: Improving test suites for efficient fault localization. In: Proceedings of the 28th International Conference on Software Engineering, pp. 82–91 (2006)

  • Black, J., Melachrinoudis, E., Kaeli, D.: Bi-criteria models for all-uses test suite reduction. In: Proceedings of the 26th International Conference on Software Engineering, pp. 106–115 (2004)

  • Bush, W.R., Pincus, J.D., Sielaff, D.J.: A static analyzer for finding dynamic programming errors. Softw. Pract. Exp. 30(7), 775–802 (2000)

    Article  MATH  Google Scholar 

  • Chan, F.T., Chen, T.Y.: AIDA—a dynamic data flow anomaly detection system for Pascal programs. Softw. Pract. Exp. 17(3), 227–239 (1987)

    Article  MATH  Google Scholar 

  • Chaim, M.L., Maldonado, J.C., Jino, M.: A debugging strategy based on the requirements of testing. Softw. Maint. Evol. Res. Pract. 16, 277–308 (2004)

    Article  Google Scholar 

  • Cleve, H., Zeller, A.: Locating causes of program failures. In: Proceedings of the 27th International Conference on Software Engineering, pp. 342–351 (2005)

  • Collofello, J.S., Woodfield, S.N.: Evaluating the effectiveness of reliable-assurance techniques. Syst. Softw. 9(3), 191–195 (1989)

    Article  Google Scholar 

  • Demillo, R., Pan, H., Spafford, E.: Failure and fault analysis for software debugging. In: Proceedings of the 21st International Computer Software and Applications Conference, pp. 515–521 (1997)

  • Dickinson, W., Leon, D., Podgurski, A.: Finding failures by cluster analysis of execution profiles. In: Proceedings of the 23rd International Conference on Software Engineering, pp. 339–348 (2001)

  • Dor, N., Rodeh, M., Sagiv, M.: CSSV: towards a realistic tool for statically detecting all buffer overflows in C. In: Proceedings of the ACM SIGPLAN 2003 Conference on Programming Language Design and Implementation, pp. 155–167 (2003)

  • Dubois, D., Prade, H.: Fuzzy Sets and Systems, pp. 9–20. Academic Press, New York (1980)

    MATH  Google Scholar 

  • Engler, D., Chelf, B., Chou, A., Hallem, S.: Checking system rules using system-specific, programmer-written compiler extensions. In: Proceedings of Operating Systems Design and Implementation, pp. 1–16 (2000)

  • Engler, D., Chen, D.Y., Hallem, S., Chou, A., Chelf, B.: Bugs as deviant behavior: a general approach to inferring errors in systems code. In: Proceedings of 18th ACM Symposium on Operating Systems Principles, pp. 57–72 (2001)

  • Freund, J.E., Simon, G.A.: Statistics: A First Course, 6th edn., pp. 479–483. Prentice Hall, New York (1995)

    Google Scholar 

  • Hangal, S., Lam, M.S.: Tracking down software bugs using automatic anomaly detection. In: Proceedings of the 24th International Conference on Software Engineering, pp. 291–301 (2002)

  • Hao, D., Pan, Y., Zhang, L., Mei, H., Sun, J.: A Similarity-aware approach to testing based fault localization. In: Proceedings of 20th IEEE International Conference on Automated Software Engineering, pp. 291–294 (2005a)

  • Hao, D., Zhang, L., Zhong, H., Mei, H., Sun, J.: Eliminating harmful redundancy for testing-based fault localization using test suite reduction: an experimental study. In: Proceedings of the 21st IEEE International Conference on Software Maintenance, pp. 683–686 (2005b)

  • Harrold, M.J., Gupta, R., Soffa, M.L.: A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Methodology 2(3), 270–285 (1993)

    Article  Google Scholar 

  • Howden, W.E.: Weak mutation testing and completeness of test sets. IEEE Trans. Softw. Eng. 8(4), 371–379 (1982)

    Article  Google Scholar 

  • Huang, J.C.: Detection of data flow anomaly through program instrumentation. IEEE Trans. Softw. Eng. 5(3), 226–236 (1979)

    Article  Google Scholar 

  • Hutchins, M., Foster, H., Goradia, T., Ostrand, T.: Experiments on the effectiveness of dataflow- and controlflow-based test adequacy criteria. In: Proceedings of the 16th International Conference on Software Engineering, pp. 191–200 (1994)

  • Jones, J.A., Harrold, M.J., Stasko, J.: Visualization of test information to assist fault localization. In: Proceedings of the 24th International Conference on Software Engineering, pp. 467–477 (2002)

  • Jones, J.A., Harrold, M.J.: Empirical evaluation of the tarantula automatic fault-localization technique. In: Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering, pp. 273–282 (2005)

  • Larochelle, D., Evans, D.: Statically detecting likely buffer overflow vulnerabilities. In: Proceedings of the 10th USENIX Security Symposium, pp. 13–17 (2001)

  • Liblit, B., Aiken, A., Zheng, A.X., Jordan, M.I.: Bug isolation via remote program sampling. In: Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pp. 141–154 (2003)

  • Liblit, B., Naik, M., Zheng, A.X., Aiken, A.: Jordan M.I., Scalable statistical bug isolation. In: Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pp. 15–26 (2005)

  • Liu, C., Yan, X., Fei, L., Han, J., Midkiff, S.P.: SOBER: statistical model-based bug localization. In: Proceedings of the 13th ACM SIGSOFT Symposium on Foundations of Software Engineering, pp. 286–295 (2005)

  • Liu, C., Fei, L., Yan, X., Han, J., Midkiff, S.P.: Statistical debugging: a hypothesis testing-based approach. IEEE Trans. Softw. Eng. 32(10), 831–848 (2006)

    Article  Google Scholar 

  • Maldonado, J.C., Delamaro, M.E., Fabbri, S.C.P.F., Simö, A.S., Sugeta, T., Vincenzi, A.M.R., Masiero, P.C.: Proteum: a Family of Tools to Support Specification and Program Testing Based on Mutation. Kluwer International Series on Advances in Database Systems, pp. 113–116 (2001)

  • Mayer, W., Stumptner, M., Wieland, D., Wotawa, F.: Towards an integrated debugging environment. In: Proceedings of the 15th European Conference on Artificial Intelligence (ECAI), pp. 422–426 (2002)

  • Mayer, W., Stumptner, M.: Extending diagnosis to debug programs with exceptions. In: Proceedings of the 18th IEEE International Conference on Automated Software Engineering, pp. 240–244 (2003)

  • Newmark, J.: Statistics and Probability in Modern Life, 5th edn., pp. 679–700. Sauders College, Philadelphia (1992)

    Google Scholar 

  • Offutt, A., Pan, J., Tewary, K., Zhang, T.: An experimental evaluation of data flow and mutation testing. Softw. Pract. Exp. 26(2), 165–176 (1996)

    Article  Google Scholar 

  • Renieris, M., Reiss, S.P.: Fault localization with nearest neighbor queries. In: Proceedings of the 18th International Conference on Automated Software Engineering, pp. 30–39 (2003)

  • Ruthruff, J.R., Burnett, M., Rothermel, G.: An experimental study of fault localization for end-user programmers. In: Proceedings of the 27th IEEE International Conference on Software Engineering, pp. 352–361 (2005)

  • Weiser, M.: Program slicing. In: IEEE Transactions on Software Engineering, pp. 352–357 (1984)

  • Wong, W.E., Qi, Y.: An execution slice and inter-block data dependency-based approach for fault localization. In: Proceedings of 11th Asia-Pacific Software Engineering Conference, pp. 366–373 (2004)

  • Xie, Y., Chou, A., Engler, D.: ARCHER: using symbolic, path-sensitive analysis to detect memory access errors. In: Proceedings of the 11th ACM SIGSOFT Symposium on Foundations of Software Engineering, pp. 327–336 (2003)

  • Xie, Y., Engler, D.: Using redundancies to find errors. IEEE Trans. Softw. Eng. 29(10), 915–928 (2003)

    Article  Google Scholar 

  • Zadeh, L.A.: Fuzzy sets. Inf. Control 8, 338–353 (1965)

    Article  MATH  MathSciNet  Google Scholar 

  • Zeller, A., Hildebrandt, R.: Simplifying and isolating failure-inducing input. IEEE Trans. Softw. Eng. 28(2), 183–200 (2002)

    Article  Google Scholar 

  • Zeller, A.: Isolating cause-effect chains from computer programs. In: Proceedings of the 10th ACM SIGSOFT Symposium on Foundations of Software Engineering, pp. 1–10 (2002)

Download references

Author information

Authors and Affiliations

  1. Key Laboratory of High Confidence Software Technologies, Ministry of Education, Institute of Software, School of Electronics Engineering and Computer Science, Peking University, Beijing, 100871, China

    Dan Hao, Lu Zhang, Ying Pan, Hong Mei & Jiasu Sun

Authors
  1. Dan Hao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Lu Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Ying Pan
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Hong Mei
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Jiasu Sun
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Lu Zhang.

Additional information

A preliminary version of this paper appears in (Hao et al. 2005a).

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hao, D., Zhang, L., Pan, Y. et al. On similarity-awareness in testing-based fault localization. Autom Softw Eng 15, 207–249 (2008). https://doi.org/10.1007/s10515-008-0025-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-008-0025-9

Keywords