research-article

A Unified Test Case Prioritization Approach

Authors:

Lingming Zhang,

Gregg Rothermel,

Hong MeiAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 24, Issue 2

Article No.: 10, Pages 1 - 31

https://doi.org/10.1145/2685614

Published: 23 December 2014 Publication History

Abstract

Test case prioritization techniques attempt to reorder test cases in a manner that increases the rate at which faults are detected during regression testing. Coverage-based test case prioritization techniques typically use one of two overall strategies: a total strategy or an additional strategy. These strategies prioritize test cases based on the total number of code (or code-related) elements covered per test case and the number of additional (not yet covered) code (or code-related) elements covered per test case, respectively. In this article, we present a unified test case prioritization approach that encompasses both the total and additional strategies. Our unified test case prioritization approach includes two models (basic and extended) by which a spectrum of test case prioritization techniques ranging from a purely total to a purely additional technique can be defined by specifying the value of a parameter referred to as the f_p value. To evaluate our approach, we performed an empirical study on 28 Java objects and 40 C objects, considering the impact of three internal factors (model type, choice off_p value, and coverage type) and three external factors (coverage granularity, test case granularity, and programming/testing paradigm), all of which can be manipulated by our approach. Our results demonstrate that a wide range of techniques derived from our basic and extended models with uniform f_p values can outperform purely total techniques and are competitive with purely additional techniques. Considering the influence of each internal and external factor studied, the results demonstrate that various values of each factor have nontrivial influence on test case prioritization techniques.

References

[1]

J. H. Andrews, L. C. Briand, and Y. Labiche. 2005. Is mutation an appropriate tool for testing experiments&quest; In Proceedings of the International Conference on Software Engineering (ICSE'05). 402--411.

Digital Library

[2]

J. H. Andrews, L. C. Briand, Y. Labiche, and A. Siami Namin 2006. Using mutation analysis for assessing and comparing testing Coverage criteria. IEEE Trans. Softw. Engin. 32, 8, 608--624.

Digital Library

[3]

C. Cadar, D. Dunbar, and D. Engler. 2008. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8^th USENIX Symposium on Operating Systems Design and Implementation (OSDI'08). 209--224.

Digital Library

[4]

H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel. 2008. An empirical study of the effect of time constraints on the cost-benefits of regression testing. In Proceedings of the ACM Symposium on Foundations of Software Engineering (FSE'08). 71--82.

Digital Library

[5]

H. Do and G. Rothermel. 2006a. An empirical study of regression testing techniques incorporating context and lifecycle factors and improved cost-benefit models. In Proceedings of the Symposium on the Foundations of Software Engineering (FSE'06). 141--151.

Digital Library

[6]

H. Do and G. Rothermel. 2006b. On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Trans. Softw. Engin. 32, 9, 733--752.

Digital Library

[7]

H. Do and G. Rothermel. 2008. Using sensitivity analysis to create simplified economic models for regression testing. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'08). 51--62.

Digital Library

[8]

H. Do, G. Rothermel, and A. Kinneer. 2004. Empirical studies of test case prioritization in a JUnit testing environment. In Proceedings of the International Symposium on Software Reliability Engineering (ISSRE'04). 113--124.

Digital Library

[9]

S. Elbaum, A. Malishevsky, and G. Rothermel. 2000. Prioritizing test cases for regression testing. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'00). 102--112.

Digital Library

[10]

S. Elbaum, A. Malishevsky, and G. Rothermel. 2001. Incorporating varying test costs and fault severities into test case prioritization. In Proceedings of the International Conference on Software Engineering (ICSE'01). 329--338.

Digital Library

[11]

S. Elbaum, A. Malishevsky, and G. Rothermel. 2002. Test case prioritization: A family of empirical studies. IEEE Trans. Softw. Engin. 28, 2, 159--182.

Digital Library

[12]

C. Fang, Z. Chen, and B. Xu. 2012. Comparing logic coverage criteria on test case prioritization. Sci. China Inf. Sci. 55, 12, 2826--2840.

[13]

M. Gligoric, A. Groce, C. Zhang, R. Sharma, A. Alipour, and D. Marinov. 2013. Comparing non-adequate test suites using coverage criteria. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'13). 302--313.

Digital Library

[14]

D. Hao, T. Lan, H. Zhang, C. Guo, and L. Zhang. 2013a. Is this a bug or an obsolete test&quest; In Proceedings of the 27^th European Conference on Object-Oriented Programming (ECOOP'13). 602--628.

Digital Library

[15]

D. Hao, X. Zhao, and L. Zhang. 2013b. Adaptive test-case prioritization guided by output inspection. In Proceedings of the 37^th Annual IEEE Computer Software and Applications Conference (COMPSAC'13). 169--179.

Digital Library

[16]

D. Hao, L. Zhang, X. Wu, H. Mei, and G. Rothermel. 2012. On-demand test suite reduction. In Proceedings of the International Conference on Software Engineering (ICSE'12). 738--748.

Digital Library

[17]

A. J. Hayter. 1986. The maximum familywise error rate of Fisher's least significant difference test. J. Amer. Statist. Assoc. 81, 396, 1000--1004.

[18]

S.-S. Hou, L. Zhang, T. Xie, and J. Sun. 2008. Quota-constrained test-case prioritization for regression testing of service-centric systems. In Proceedings of the International Conference on Software Maintenance (ICSM'08). 257--266.

[19]

H. Hsu and A. Orso. 2009. MINTS: A general framework and tool for supporting test-suite minimization. In Proceedings of the International Conference on Software Engineering (ICSE'09). 419--429.

Digital Library

[20]

B. Jiang, Z. Zhang, W. K. Chan, and T. H. Tse. 2009. Adaptive random test case prioritization. In Proceedings of the Conference on Automated Software Engineering (ASE'09). 257--266.

Digital Library

[21]

J. A. Jones and M. J. Harrold. 2001. Test-suite reduction and prioritization for modified condition/decision coverage. In Proceedings of the International Conference on Software Maintenance (ICSM'01). 92--101.

Digital Library

[22]

R. Just. 2014. The major mutation framework: Efficient and scalable mutation analysis for Java. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'14). 433--436.

Digital Library

[23]

R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. 2014. Are mutants a valid substitute for real faults in software testing&quest; In Proceedings of the Symposium on the Foundations of Software Engineering (FSE'14).

Digital Library

[24]

R. Just, F. Schweiggert, and M. Kapfhammer. 2011. MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler. In Proceedings of the International Conference on Automated Software Engineering (ASE'11). 612--615.

Digital Library

[25]

J. M. Kim and A. Porter. 2002. A history-based test prioritization technique for regression testing in resource constrained environments. In Proceedings of the International Conference on Software Engineering (ICSE'02). 119--129.

Digital Library

[26]

B. Korel, L. Tahat, and M. Harman. 2005. Test prioritization using system models. In Proceedings of the International Conference on Software Maintenance (ICSM'05). 559--568.

Digital Library

[27]

Z. Li, M. Harman, and R. Hierons. 2007. Search algorithms for regression test case prioritisation. IEEE Trans. Softw. Engin. 33, 4, 225--237.

Digital Library

[28]

S. Lin. 1965. Computer solutions of the travelling salesman problem. Bell Syst. Tech. J. 44, 5, 2245--2269.

[29]

Y.-S. Ma, J. Offutt, and Y. R. Kwon. 2005. MuJava: An automated class mutation system. J. Softw. Test. Verif. Reliab. 15, 2, 97--133.

Digital Library

[30]

A. Malishevsky, J. R. Ruthruff, G. Rothermel, and S. Elbaum. 2006. Cost-cognizant test case prioritization. Tech. rep., Department of Computer Science and Engineering, University of Nebraska.

[31]

T. J. McCabe. 1976. A complexity measure. IEEE Trans. Softw. Engin. SE-2, 4, 308--320.

Digital Library

[32]

H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, and G. Rothermel. 2012. A static approach to prioritizing JUnit test cases. IEEE Trans. Softw. Engin. 38, 6, 1258--1275.

Digital Library

[33]

L. Mei, W. K. Chan, and T. H. Tse. 2008. Data flow testing of service-oriented workflow applications. In Proceedings of the International Conference on Software Engineering (ICSE'08). 371--380.

Digital Library

[34]

L. Mei, Z. Zhang, W. K. Chan, and T. H. Tse. 2009. Test case prioritization for regression testing of service-oriented business applications. In Proceedings of the International Conference on World Wide Web (WWW'09). 901--910.

Digital Library

[35]

N. Nagappan, T. Ball, and Z. Zeller. 2006. Mining metrics to predict component failures. In Proceedings of the International Conference on Software Engineering (ICSE'06). 452--461.

Digital Library

[36]

T. J. Ostrand, E. J. Weyuker, and R. M. Bell. 2005. Predicting the location and number of faults in large software systems. IEEE Trans. Softw. Engin. 31, 4, 340--355.

Digital Library

[37]

H. Park, H. Ryu, and J. Baik. 2008. Historical value-based approach for cost-cognizant test case prioritization to improve the effectiveness of regression testing. In Proceedings of the International Conference on Secure Software Integration and Reliability Improvement (SSIRI'08). 39--46.

Digital Library

[38]

M. Qu, M. B. Cohen, and K. M. Woolf. 2007. Combinatorial interaction regression testing: A study of test case generation and prioritization. In Proceedings of the International Conference on Software Maintenance (ICSM'07). 255--264.

[39]

X. Qu, M. B. Cohen, and G. Rothermel. 2008. Configuration-aware regression testing: An empirical study of sampling and prioritization. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'08). 75--86.

Digital Library

[40]

G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. 1999. Test case prioritization: An empirical study. In Proceedings of the International Conference on Software Maintenance (ICSM'99). 179--188.

Digital Library

[41]

D. Schuler and A. Zeller. 2009. Javalanche: Efficient mutation testing for Java. In Proceedings of the ACM Symposium on Foundations of Software Engineering (FSE'09). 297--298.

Digital Library

[42]

A. Srivastava and J. Thiagarajan. 2002. Effectively prioritizing tests in development environment. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'02). 97--106.

Digital Library

[43]

K. R. Walcott, M. L. Soffa, G. M. Kapfhammer, and R. S. Roos. 2006. Time aware test suite prioritization. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'06). 1--11.

Digital Library

[44]

W. E. Wong, J. R. Horgan, S. London, and H. Agrawal. 1997. A study of effective regression testing in practice. In Proceedings of the International Symposium on Software Reliability Engineering (ISSRE'97). 230--238.

Digital Library

[45]

T. Xie, L. Zhang, X. Xiao, Y.-F. Xiong, and D. Hao. 2014. Cooperative software testing and analysis advances and challenges. J. Comput. Sci. Technol. 29, 4, 713--723.

[46]

L. Zhang, D. Hao, L. Zhang, G. Rothermel, and H. Mei. 2013a. Bridging the gap between the total and additional test-case prioritization strategies. In Proceedings of the International Conference on Software Engineering (ICSE'13). 192--201.

Digital Library

[47]

L. Zhang, L. Zhang, and S. Khurshid. 2013b. Injecting mechanical faults to localize developer faults for evolving software. In Proceedings of the International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA'13). 765--784.

Digital Library

[48]

L. Zhang, S. Hou, C. Guo, T. Xie, and H. Mei. 2009a. Time-aware test-case prioritization using integer linear programming. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'09). 213--224.

Digital Library

[49]

L. Zhang, J. Zhou, D. Hao, L. Zhang, and H. Mei. 2009b. Jtop: Managing JUnit test cases in absence of coverage information. In Proceedings of the Automated Software Engineering (ASE'09). 673--675.

Digital Library

[50]

L. Zhang, J. Zhou, D. Hao, L. Zhang, and H. Mei. 2009c. Prioritizing JUnit test cases in absence of coverage information. In Proceedings of the International Conference on Software Maintenance (ICSM'09). 19--28.

[51]

L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. 2011. An empirical study of JUnit test-suite reduction. In Proceedings of the International Symposium on Software Reliability Engineering (ISSRE'11). 170--179.

Digital Library

[52]

S. Zhang, D. Jalali, J. Wuttke, K. Muslu, W. Lam, M. Ernst, and D. Notkin. 2014. Empirically revisiting the test independence assumption. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'14). 385--396.

Digital Library

Cited By

Su QLi XRen YQiu RHu CYin Y(2025)Attention Transfer Reinforcement Learning for Test Case Prioritization in Continuous IntegrationApplied Sciences10.3390/app1504224315:4(2243)Online publication date: 19-Feb-2025
https://doi.org/10.3390/app15042243
Zhao RZhang SZhu ZShang YWang W(2025)E2E test execution optimization for web application based on state reuseJournal of Software: Evolution and Process10.1002/smr.271437:1Online publication date: 22-Jan-2025
https://dl.acm.org/doi/10.1002/smr.2714
Xie HLei YLi MYan MZhang SFilkov VRay BZhou M(2024)Combining Coverage and Expert Features with Semantic Representation for Coincidental Correctness DetectionProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695542(1770-1782)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695542
Show More Cited By

Index Terms

A Unified Test Case Prioritization Approach
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Combined Source Code Approach for Test Case Prioritization
ICISS '18: Proceedings of the 1st International Conference on Information Science and Systems

Regression testing is an activity in the software testing process to ensure the software is validated and verified after modification occurred on software. It is costly process procedure which has been expected to reach half cost of the software ...
A Static Approach to Prioritizing JUnit Test Cases

Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most ...
Test Case Prioritization for Continuous Regression Testing: An Industrial Case Study
ICSM '13: Proceedings of the 2013 IEEE International Conference on Software Maintenance

Regression testing in continuous integration environment is bounded by tight time constraints. To satisfy time constraints and achieve testing goals, test cases must be efficiently ordered in execution. Prioritization techniques are commonly used to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 24, Issue 2

December 2014

224 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/2702120

Editor:
David S. Rosenblum
National University of Singapore, Singapore

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 December 2014

Accepted: 01 September 2014

Revised: 01 July 2014

Received: 01 January 2014

Published in TOSEM Volume 24, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

80
Total Citations
View Citations
1,139
Total Downloads

Downloads (Last 12 months)48
Downloads (Last 6 weeks)11

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Su QLi XRen YQiu RHu CYin Y(2025)Attention Transfer Reinforcement Learning for Test Case Prioritization in Continuous IntegrationApplied Sciences10.3390/app1504224315:4(2243)Online publication date: 19-Feb-2025
https://doi.org/10.3390/app15042243
Zhao RZhang SZhu ZShang YWang W(2025)E2E test execution optimization for web application based on state reuseJournal of Software: Evolution and Process10.1002/smr.271437:1Online publication date: 22-Jan-2025
https://dl.acm.org/doi/10.1002/smr.2714
Xie HLei YLi MYan MZhang SFilkov VRay BZhou M(2024)Combining Coverage and Expert Features with Semantic Representation for Coincidental Correctness DetectionProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695542(1770-1782)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695542
Laaber CYue TAli S(2024)Evaluating Search-Based Software Microbenchmark PrioritizationIEEE Transactions on Software Engineering10.1109/TSE.2024.338083650:7(1687-1703)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3380836
Levasseur MBadri M(2024)Prioritizing unit tests using object-oriented metrics, centrality measures, and machine learning algorithmsInnovations in Systems and Software Engineering10.1007/s11334-024-00550-9Online publication date: 2-Feb-2024
https://doi.org/10.1007/s11334-024-00550-9
Altiero FCorazza ADi Martino SPeron ALibero Lucio Starace L(2024)Regression test prioritization leveraging source code similarity with tree kernelsJournal of Software: Evolution and Process10.1002/smr.265336:8Online publication date: 5-Aug-2024
https://dl.acm.org/doi/10.1002/smr.2653
Zhang YXie XLi YLin YChen SLiu YLi X(2023)Demystifying Performance Regressions in String SolversIEEE Transactions on Software Engineering10.1109/TSE.2022.316837349:3(947-961)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3168373
Yang LChen JYou HHan JJiang JSun ZLin XLiang FKang Y(2023)Can Code Representation Boost IR-Based Test Case Prioritization?2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00077(240-251)Online publication date: 9-Oct-2023
https://doi.org/10.1109/ISSRE59848.2023.00077
Zhao YHao DZhang L(2023)Revisiting Machine Learning based Test Case Prioritization for Continuous Integration2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00032(232-244)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00032
Yu JWu YXie XLe WMa LChen YHu JZhang FGrundy JPollock LPenta M(2023)GameRTS: A Regression Testing Framework for Video GamesProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00122(1393-1404)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00122
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents