Skip to main content
Springer Nature Link
Log in

Test input reduction for result inspection to facilitate fault localization

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

Testing-based fault-localization (TBFL) approaches often require the availability of high-statement-coverage test suites that sufficiently exercise the areas around the faults. However, in practice, fault localization often starts with a test suite whose quality may not be sufficient to apply TBFL approaches. Recent capture/replay or traditional test-generation tools can be used to acquire a high-statement-coverage test collection (i.e., test inputs only) without expected outputs. But it is expensive or even infeasible for developers to manually inspect the results of so many test inputs. To enable practical application of TBFL approaches, we propose three strategies to reduce the test inputs in an existing test collection for result inspection. These three strategies are based on the execution traces of test runs using the test inputs. With the three strategies, developers can select only a representative subset of the test inputs for result inspection and fault localization. We implemented and applied the three test-input-reduction strategies to a series of benchmarks: the Siemens programs, DC, and TCC. The experimental results show that our approach can help developers inspect the results of a smaller subset (less than 10%) of test inputs, whose fault-localization effectiveness is close to that of the whole test collection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Agrawal, H., Horgan, J., London, S., Wong, W.: Fault localization using execution slices and dataflow. In: Proc. 6th International Symposium on Software Reliability Engineering, pp. 143–151, October 1995

  • Barbosa, E., Maldonado, J., Vincenzi, A.: Toward the determination of sufficient mutant operators for C. Softw. Test. Verif. Reliab. 11(2), 113–136 (2001)

    Article  Google Scholar 

  • Baresi, L., Young, M.: Test oracles. Technical Report CIS-TR-01-02, University of Oregon, Dept. of Computer and Information Science, Eugene, Oregon, USA, August 2001

  • Baudry, B., Fleurey, F., Traon, Y.L.: Improving test suites for efficient fault localization. In: Proc. 28th International Conference on Software Engineering, pp. 82–91, May 2006

  • Cleve, H., Zeller, A.: Locating causes of program failure. In: Proc. 27th International Conference on Software Engineering, pp. 342–351, May 2005

  • Do, H., Elbaum, S., Rothermel, G.: Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Emp. Softw. Eng. 10(4), 405–435 (2005)

    Article  Google Scholar 

  • Elbaum, S., Malishevsky, A.G., Rothermel, G.: Test case prioritization: A family of empirical studies. IEEE Trans. Softw. Eng. 28(2), 159–182 (2002)

    Article  Google Scholar 

  • Enderton, H.B.: Elements of Set Theory. Academic Press, San Diego (1977)

    MATH  Google Scholar 

  • Graves, T.L., Harrold, M.J., Kim, J.-M., Porter, A., Rothermel, G.: An empirical study of regression test selection techniques. ACM Trans. Softw. Eng. Methodol. 10(2), 184–208 (2001)

    Article  MATH  Google Scholar 

  • Guo, L., Roychoudhury, A., Wang, T.: Accurately choosing execution runs for software fault localization. In: Proc. International Conference on Compiler Construction, pp. 80–95, Mar. 2006

  • Hao, D.: Testing-based interactive fault localization. In: Proc. 28th International Conference on Software Engineering, Doctoral Symposium Track, pp. 957–960, May 2006

  • Hao, D., Pan, Y., Zhang, L., Zhao, W., Mei, H., Sun, J.: A similarity-aware approach to testing based fault localization. In: Proc. 20th IEEE International Conference on Automated Software Engineering, pp. 291–294 (2005a)

  • Hao, D., Zhang, L., Zhong, H., Mei, H., Sun, J.: Eliminating harmful redundancy for testing-based fault localization using test suite reduction: An experimental study. In: Proc. 21st IEEE International Conference on Software Maintenance, pp. 683–686 (2005b)

  • Hao, D., Zhang, L., Mei, H., Sun, J.: Towards interactive fault localization using test information. In: Proc. 13th Asia Pacific Software Engineering Conference, pp. 277–284 (2006)

  • Hao, D., Zhang, L., Pan, Y., Mei, H., Sun, J.: On similarity-awareness in testing-based fault localization. Autom. Softw. Eng. J. 15(2), 207–249 (2008)

    Article  Google Scholar 

  • Hao, D., Zhang, L., Zhang, L., Sun, J., Mei, H.: VIDA: Visual interactive debugging. In: Proc. 31st International Conference on Software Engineering, Formal Research Demonstrations, pp. 583–586, May 2009

  • Harrold, M.J., Gupta, R., Soffa, M.L.: A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Methodol. 2(3), 270–285 (1993)

    Article  Google Scholar 

  • Hutchins, M., Foster, H., Goradia, T., Ostrand, T.: Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria. In: Proc. 16th International Conference on Software Engineering, pp. 191–200, May 1994

  • Jeffrey, D., Gupta, N.: Test suite reduction with selective redundancy. In: Proc. 21st IEEE International Conference on Software Maintenance, pp. 549–558, Sept. 2005

  • Jeffrey, D., Gupta, N.: Improving fault detection capability by selectively retaining test cases during test suite reduction. IEEE Trans. Softw. Eng. 33(2), 108–123 (2007)

    Article  Google Scholar 

  • Jones, J.A., Harrold, M.J.: Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Trans. Softw. Eng. 29(3), 195–209 (2003)

    Article  Google Scholar 

  • Jones, J.A., Harrold, M.J.: Empirical evaluation of tarantula automatic fault-localization technique. In: Proc. 20th International Conference on Automated Software Engineering, pp. 273–282 (2005)

  • Jones, J.A., Harrold, M.J., Stasko, J.: Visualization of test information to assist fault localization. In: Proc. 24th International Conference on Software Engineering, pp. 467–477, May 2002

  • Joseph, N.: Statistics and Probability in Modern Life. Saunders, Philadelphia (1992)

    Google Scholar 

  • Liblit, B., Aiken, A., Zheng, A.X., Jordan, M.I.: Bug isolation via remote program sampling. In: Proc. ACM SIGPLAN 2003 Conference on Programming Languages Design and Implementation, pp. 141–154 (2003)

  • Liu, C., Yuan, X., Fei, L., Han, J., Midkiff, S.P.: SOBER: Statistical model-based bug localization. In: Proc. 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 286–295, September 2005

  • Ma, X.y., He, Z.f., Sheng, B.k., Ye, C.q.: A genetic algorithm for test-suite reduction. In: Proc. the International Conference on Systems, Man and Cybernetics, pp. 133–139, October 2005

  • Pacheco, C., Ernst, M.D.: Eclat: Automatic generation and classification of test inputs. In: Proc. 19th European Conference on Object-Oriented Programming, pp. 504–527 (2005)

  • Renieris, M., Reiss, S.P.: Fault localization with nearest neighbor queries. In: Proc. 18th International Conference on Automated Software Engineering, pp. 30–39 (2003)

  • Rothermel, G., Harrold, M.J.: A safe, efficient regression test selection technique. ACM Trans. Softw. Eng. Methodol. 6(2), 173–210 (1997)

    Article  Google Scholar 

  • Rothermel, G., Harrold, M.J.: Experimental studies of a safe regression test selection technique. IEEE Trans. Softw. Eng. 24(6), 401–419 (1998)

    Article  Google Scholar 

  • Rothermel, G., Untch, R.H., Chu, C., Harrold, M.J.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001)

    Article  Google Scholar 

  • Rothermel, G., Harrold, M.J., von Ronne, J., Hong, C.: Empirical studies of test-suite reduction. Softw. Test. Verif. Reliab. 12(4), 219–249 (2002)

    Article  Google Scholar 

  • Saff, D., Artzi, S., Perkins, J.H., Ernst, M.D.: Automatic test factoring for Java. In: Proc. 20th International Conference on Automated Software Engineering, pp. 114–123, November 2005

  • Sen, K., Marinov, D., Agha, G.: CUTE: A concolic unit testing engine for C. In: Proc. 13th ACM SIGSOFT Symposium on the Foundations of Software Engineering, pp. 263–272 (2005)

  • Sprenkle, S., Sampath, S., Gibson, E., Pollock, L., Souter, A.: An empirical comparison of test suite reduction techniques for user-session-based testing of web applications. In: Proc. 21st IEEE International Conference on Software Maintenance, pp. 587–596, Sept. 2005

  • Srivastava, A., Thiagarajan, J.: Effectively prioritizing tests in development environment. In: Proc. 2002 International Symposium on Software Testing and Analysis, pp. 97–106, July 2002

  • Trochim, W.M.K.: Research Methods Knowledge Base. Thomson Custom Pub., New York (2007)

    Google Scholar 

  • Walcott, K.R., Kapfhammer, G.M., Soffa, M.L., Roos, R.S.: Time-aware test suite prioritization. In: Proc. International Symposium on Software Testing and Analysis, pp. 1–12 (2006)

  • Wang, T., Roychoudhury, A.: Automated path generation for software fault localization. In: Proc. 20th International Conference on Automated Software Engineering, pp. 347–351 (2005)

  • Wohlin, C., Runeson, P., Host, M.: Experimentation in Software Engineering: An Introduction. Springer, Berlin (1999)

    Google Scholar 

  • Xie, T., Notkin, D.: Tool-assisted unit-test generation and selection based on operational abstractions. Autom. Softw. Eng. J. 13(3), 345–371 (2006)

    Article  Google Scholar 

  • Xie, Y., Engler, D.: Using redundancies to find errors. IEEE Trans. Softw. Eng. 29(10), 915–928 (2003)

    Article  Google Scholar 

  • Yu, Y., Jones, J.A., Harrold, M.J.: An empirical study of the effects of test-suite reduction on fault localization. In: Proc. 30th International Conference on Software Engineering, pp. 201–210, May 2008

Download references

Author information

Authors and Affiliations

  1. Key Laboratory of High Confidence Software Technologies, Ministry of Education, Institute of Software, School of Electronics Engineering and Computer Science, Peking University, Beijing, 100871, People’s Republic of China

    Dan Hao, Lu Zhang, Xiaoyin Wang, Jiasu Sun & Hong Mei

  2. Department of Computer Science, North Carolina State University, Raleigh, NC, 27695, USA

    Tao Xie

Authors
  1. Dan Hao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Tao Xie
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Lu Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Xiaoyin Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Jiasu Sun
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Hong Mei
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Hong Mei.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hao, D., Xie, T., Zhang, L. et al. Test input reduction for result inspection to facilitate fault localization. Autom Softw Eng 17, 5 (2010). https://doi.org/10.1007/s10515-009-0056-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10515-009-0056-x

Keywords