Skip to main content
SpringerLink
Log in

Challenges in Using Search-Based Test Generation to Identify Real Faults in Mockito

  • Conference paper
  • First Online:
Search Based Software Engineering (SSBSE 2016)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 9962))

Included in the following conference series:

Abstract

The cost of test creation can potentially be reduced through automated generation. However, to impact testing practice, automated tools must be at least as effective as manual generation. The Mockito project—a framework for mocking portions of a system—offers an opportunity to assess the capabilities of test generation tools on a complex real-world system. We have identified 17 faults in the Mockito project, added those to the Defects4J database, and assessed the ability of the EvoSuite tool to detect these faults. In our study, EvoSuite was only able to detect one of the 17 faults. Analysis of the 16 undetected faults yields lessons in how to improve generation tools. We offer these faults to the community to assist in benchmarking future test generation advances.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://mockito.org/.

  2. 2.

    https://github.com/mockito/mockito/commit/7a647a702c8af81ccf5d37b09c11529c6c0cb1b7.

  3. 3.

    https://code.google.com/archive/p/mockito/issues/138.

  4. 4.

    https://github.com/mockito/mockito/commit/3eec7451d6c83c280743c39b39c77a179abb30f9.

  5. 5.

    https://github.com/mockito/mockito/issues/152.

  6. 6.

    https://github.com/mockito/mockito/commit/dc205824dbc289acbcde919e430176ad72da847f.

  7. 7.

    https://github.com/mockito/mockito/commit/77cb2037314dd024eb53ffe2e9e06304088a2d53.

References

  1. Ali, S., Briand, L.C., Hemmati, H., Panesar-Walawege, R.K.: A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Softw. Eng. 36(6), 742–762 (2010)

    Article  Google Scholar 

  2. Feldt, R., Poulding, S.: Finding test data with specific properties via metaheuristic search. In: 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), pp. 350–359, November 2013

    Google Scholar 

  3. Fraser, G., Arcuri, A.: Whole test suite generation. IEEE Trans. Softw. Eng. 39(2), 276–291 (2013)

    Article  Google Scholar 

  4. Fraser, G., Staats, M., McMinn, P., Arcuri, A., Padberg, F.: Does automated white-box test generation really help software testers? In: Proceedings of the 2013 International Symposium on Software Testing and Analysis, ISSTA 2013, pp. 291–301. ACM, New York (2013). http://doi.acm.org/10.1145/2483760.2483774

  5. Gay, G., Staats, M., Whalen, M., Heimdahl, M.: The risks of coverage-directed test case generation. IEEE Trans. Softw. Eng. PP(99) (2015)

    Google Scholar 

  6. Just, R., Jalali, D., Ernst, M.D.: Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pp. 437–440. ACM, New York (2014). http://doi.acm.org/10.1145/2610384.2628055

  7. Shamshiri, S., Just, R., Rojas, J.M., Fraser, G., McMinn, P., Arcuri, A.: Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges. In: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), ASE 2015. ACM, New York (2015)

    Google Scholar 

  8. Weiss, T.: We analyzed 30,000 GitHub projects - here are the top 100 libraries in Java, JS and Ruby (2013). http://blog.takipi.com/we-analyzed-30000-github-projects-here-are-the-top-100-libraries-in-java-js-and-ruby/

Download references

Author information

Authors and Affiliations

  1. University of South Carolina, Columbia, SC, USA

    Gregory Gay

Authors
  1. Gregory Gay
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gregory Gay .

Editor information

Editors and Affiliations

  1. University College London, London, United Kingdom

    Federica Sarro

  2. Michigan State University , East Lansing, Michigan, USA

    Kalyanmoy Deb

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Gay, G. (2016). Challenges in Using Search-Based Test Generation to Identify Real Faults in Mockito. In: Sarro, F., Deb, K. (eds) Search Based Software Engineering. SSBSE 2016. Lecture Notes in Computer Science(), vol 9962. Springer, Cham. https://doi.org/10.1007/978-3-319-47106-8_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-47106-8_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-47105-1

  • Online ISBN: 978-3-319-47106-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation