skip to main content
10.1145/1529282.1529382acmconferencesArticle/Chapter ViewAbstractPublication PagessacConference Proceedingsconference-collections
research-article

An empirical study of incorporating cost into test suite reduction and prioritization

Published: 08 March 2009 Publication History

Abstract

Software developers use testing to gain and maintain confidence in the correctness of a software system. Automated reduction and prioritization techniques attempt to decrease the time required to detect faults during test suite execution. This paper uses the Harrold Gupta Soffa, delayed greedy, traditional greedy, and 2-optimal greedy algorithms for both test suite reduction and prioritization. Even though reducing and reordering a test suite is primarily done to ensure that testing is cost-effective, these algorithms are normally configured to make greedy choices with coverage information alone. This paper extends these algorithms to greedily reduce and prioritize the tests by using both test cost (e.g., execution time) and the ratio of code coverage to test cost. An empirical study with eight real world case study applications shows that the ratio greedy choice metric aids a test suite reduction method in identifying a smaller and faster test suite. The results also suggest that incorporating test cost during prioritization allows for an average increase of 17% and a maximum improvement of 141% for a time sensitive evaluation metric called coverage effectiveness.

References

[1]
M. J. Crawley. The R Book. John Wiley & Sons, Inc., 2007.
[2]
H. Do, G. Rothermel, and A. Kinneer. Empirical studies of test case prioritization in a JUnit testing environment. In Proc. of ISSRE, 2004.
[3]
S. Elbaum, A. Malishevsky, and G. Rothermel. Test case prioritization: A family of empirical studies. IEEE TSE, 28(2), 2002.
[4]
M. J. Harrold, R. Gupta, and M. L. Soffa. A methodology for controlling the size of a test suite. ACM TOSEM, 2(3), 1993.
[5]
G. M. Kapfhammer. A Comprehensive Framework for Testing Database-Centric Applications. PhD thesis, University of Pittsburgh, Pittsburgh, Pennsylvania, 2007.
[6]
G. M. Kapfhammer and M. L. Soffa. Using coverage effectiveness to evaluate test suite prioritizations. In Proc. of WEASELTech, 2007.
[7]
G. M. Kapfhammer and M. L. Soffa. Database-aware test coverage monitoring. In Proc. of ISEC, 2008.
[8]
G. M. Kapfhammer, M. L. Soffa, and D. Mosse. Testing in resource constrained execution environments. In Proc. of ASE, 2005.
[9]
Z. Li, M. Harman, and R. M. Hierons. Search algorithms for regression test case prioritization. IEEE TSE, 33(4), 2007.
[10]
A. G. Malishevsky, J. Ruthruff, G. Rothermel, and S. Elbaum. Cost-cognizant test case prioritization. Technical Report TR-UNL-CSE-2006-0004, University of Nebraska - Lincoln, 2006.
[11]
T. J. McCabe and C. W. Butler. Design complexity measurement and testing. CACM, 32(12), 1989.
[12]
S. McMaster and A. Memon. Call stack coverage for test suite reduction. In Proc. of ICSM, 2005.
[13]
S. McMaster and A. Memon. Call stack coverage for GUI test-suite reduction. In Proc. of ISSRE, 2006.
[14]
S. McMaster and A. M. Memon. Fault detection probability analysis for coverage-based test suite reduction. In Proc. of ICSM, 2007.
[15]
J. Misurda, J. A. Clause, J. L. Reed, B. R. Childers, and M. L. Soffa. Demand-driven structural testing with dynamic instrumentation. In Proc. of ICSE, 2005.
[16]
G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. IEEE TSE, 27(10), 2001.
[17]
A. Smith, J. Geiger, G. M. Kapfhammer, and M. L. Soffa. Test suite reduction and prioritization with call trees. In Proc. of ASE, 2007.
[18]
S. Tallam and N. Gupta. A concept analysis inspired greedy algorithm for test suite minimization. In Proc. of PASTE, 2005.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SAC '09: Proceedings of the 2009 ACM symposium on Applied Computing
March 2009
2347 pages
ISBN:9781605581668
DOI:10.1145/1529282
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 March 2009

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. prioritization
  2. reduction
  3. regression testing

Qualifiers

  • Research-article

Conference

SAC09
Sponsor:
SAC09: The 2009 ACM Symposium on Applied Computing
March 8, 2009 - March 12, 2008
Hawaii, Honolulu

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25
The 40th ACM/SIGAPP Symposium on Applied Computing
March 31 - April 4, 2025
Catania , Italy

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)6
  • Downloads (Last 6 weeks)2
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)An Empirical Study of Greedy Test Suite Minimization Techniques Using Mutation CoverageIEEE Access10.1109/ACCESS.2023.328907311(65427-65442)Online publication date: 2023
  • (2023)A systematic review on search‐based test suite reductionIET Software10.1049/sfw2.1210417:2(93-136)Online publication date: 20-Feb-2023
  • (2023)Generic and industrial scale many-criteria regression test selectionJournal of Systems and Software10.1016/j.jss.2023.111802205(111802)Online publication date: Nov-2023
  • (2023)A Systematic Literature Review on Test Case Prioritization TechniquesAgile Software Development10.1002/9781119896838.ch7(101-159)Online publication date: 8-Feb-2023
  • (2021)Systematic Literature Review on Test Case Selection and Prioritization: A Tertiary StudyApplied Sciences10.3390/app11241212111:24(12121)Online publication date: 20-Dec-2021
  • (2021)Empirically Comparing the Test Suite Reduction Techniques in Continuous Integration ProcessThe 2nd European Symposium on Computer and Communications10.1145/3478301.3478304(14-19)Online publication date: 16-Apr-2021
  • (2020)An Effective Regression Test Case Selection Using Hybrid Whale Optimization AlgorithmInternational Journal of Distributed Systems and Technologies10.4018/IJDST.202001010511:1(53-67)Online publication date: 1-Jan-2020
  • (2019)How Do Static and Dynamic Test Case Prioritization Techniques Perform on Modern Software Systems? An Extensive Study on GitHub ProjectsIEEE Transactions on Software Engineering10.1109/TSE.2018.282227045:11(1054-1080)Online publication date: 1-Nov-2019
  • (2019)Combining Code and Requirements Coverage with Execution Cost for Test Suite ReductionIEEE Transactions on Software Engineering10.1109/TSE.2017.277783145:4(363-390)Online publication date: 1-Apr-2019
  • (2019)Introducing a Fuzzy Model for Cost Cognizant Software Test Case PrioritizationTENCON 2019 - 2019 IEEE Region 10 Conference (TENCON)10.1109/TENCON.2019.8929716(504-509)Online publication date: Oct-2019
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media