research-article

Time-aware test-case prioritization using integer linear programming

Authors:

Hong MeiAuthors Info & Claims

ISSTA '09: Proceedings of the eighteenth international symposium on Software testing and analysis

Pages 213 - 224

https://doi.org/10.1145/1572272.1572297

Published: 19 July 2009 Publication History

Abstract

Techniques for test-case prioritization re-order test cases to increase their rate of fault detection. When there is a fixed time budget that does not allow the execution of all the test cases, time-aware techniques for test-case prioritization may achieve a better rate of fault detection than traditional techniques for test-case prioritization. In this paper, we propose a novel approach to time-aware test-case prioritization using integer linear programming. To evaluate our approach, we performed experiments on two subject programs involving four techniques for our approach, two techniques for an approach to time-aware test-case prioritization based on genetic algorithms, and four traditional techniques for test-case prioritization. The empirical results indicate that two of our techniques outperform all the other techniques for the two subjects under the scenarios of both general and version-specific prioritization. The empirical results also indicate that some traditional techniques with lower analysis time cost for test-case prioritization may still perform competitively when the time budget is not quite tight.

References

[1]

S. Alspaugh, K. R. Walcott, M. Belanich, G. M. Kapfhammer, and M. L. Soffa. Efficient time-aware prioritization with knapsack solvers. In Proc. WEASELTech, pages 17--31, 2007.

Digital Library

[2]

J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In Proc. ICSE, pages 402--411, 2005.

Digital Library

[3]

T. Ball. On the limit of control flow analysis for regression test selection. In Proc. ISSTA, pages 134--142, 1998.

Digital Library

[4]

D. Binkley. Semantics guided regression test cost reduction. IEEE TSE, 23(8):498--516, 1998.

Digital Library

[5]

J. Black, E. Melachrinoudis, and D. Kaeli. Bi-criteria models for all-uses test suite reduction. In Proc. ICSE, pages 106--115, 2004.

Digital Library

[6]

T. Y. Chen and M. F. Lau. A new heuristic for test suite reduction. IST, 40(5-6):347--354, 1998.

[7]

Y. Chen, D. Rosenblum, and K. Vo. TestTube: A system for selective regression testing. In Proc. ICSE, pages 113--124, 1994.

Digital Library

[8]

H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel. An empirical study of the effect of time constraints on the cost-benefits of regression testing. In Proc. FSE, pages 71--82, 2008.

Digital Library

[9]

H. Do and G. Rothermel. On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE TSE, 32(9):733--752, 2006.

Digital Library

[10]

H. Do, G. Rothermel, and A. Kinneer. Empirical studies of test case prioritization in a JUnit testing environment. In Proc. ISSRE, pages 113--124, 2004.

Digital Library

[11]

S. Elbaum, A. Malishevsky, and G. Rothermel. Prioritizing test cases for regression testing. In Proc. ISSTA, pages 102--112, 2000.

Digital Library

[12]

S. Elbaum, A. Malishevsky, and G. Rothermel. Incorporating varying test costs and fault severities into test case prioritization. In Proc. ICSE, pages 329--338, 2001.

Digital Library

[13]

S. Elbaum, A. Malishevsky, and G. Rothermel. Test case prioritization: A family of empirical studies. IEEE TSE, 28(2):159--182, 2002.

Digital Library

[14]

M. J. Harrold, R. Gupta, and M. L. Soffa. A methodology for controlling the size of a test suite. ACM TOSEM, 2(3):270--285, 1993.

Digital Library

[15]

S. Hou, L. Zhang, T. Xie, and J. Sun. Quota-constrained test-case prioritization for regression testing of service-centric systems. In Proc. ICSM, pages 257--266, 2008.

[16]

J. A. Jones and M. J. Harrold. Test-suite reduction and prioritization for modified condition/decision coverage. IEEE TSE, 29(3):195--209, 2003.

Digital Library

[17]

J. M. Kim and A. Porter. A history-based test prioritization technique for regression testing in resource constrained environments. In Proc. ICSE, pages 119--129, 2002.

Digital Library

[18]

B. Korel, G. Koutsogiannakis, and L. Tahat. Application of system models in regression test suite prioritization. In Proc. ICSM, pages 247--256, 2008.

[19]

B. Korel, L. Tahat, and M. Harman. Test prioritization using system models. In Proc. ICSM, pages 559--568, 2005.

Digital Library

[20]

H. Leung and L. White. A study of integration testing and software regression at the integration level. In Proc. ICSM, pages 290--300, 1990.

[21]

Z. Li, M. Harman, and R. Hierons. Search algorithms for regression test case prioritisation. IEEE TSE, 33(4):225--237, 2007.

Digital Library

[22]

S. Lin. Computer solutions of the travelling salesman problem. Bell System Technical Journal, 44(5):2245--2269, 1965.

[23]

A. Malishevsky, J. R. Ruthru, G. Rothermel, and S. Elbaum. Cost-cognizant test case prioritization. Technical report, Department Computer Science and Engineering of University of Nebraska, 2006.

[24]

N. Mansour and K. El-Fakin. Simulated annealing and genetic algorithms for optimal regression testing. Journal of Software Maintenance: Research and Practice, 11(1):19--34, 1999.

Digital Library

[25]

I. Moore. Jester-a JUnit test tester. In Proc. International Conference on Extreme Programming and Flexible Processes, pages 84--87, 2001.

[26]

A. J. Offutt, G. Rothermel, and C. Zapf. An experimental evaluation of selective mutation. In Proc. ICSE, pages 100--107, 1993.

Digital Library

[27]

H. Park, H. Ryu, and J. Baik. Historical value-based approach for cost-cognizant test case prioritization to improve the effectiveness of regression testing. In Proc. SSIRI, pages 39--46, 2008.

Digital Library

[28]

X. Qu, M. B. Cohen, and G. Rothermel. Configuration-aware regression testing: an empirical study of sampling and prioritization. In Proc. ISSTA, pages 75--86, 2008.

Digital Library

[29]

T. Ralphs and M. Guzelsoy. The SYMPHONY callable library for mixed integer programming. In Proc. INFORMS Computing Society Conference, pages 61--73, 2005.

[30]

C. R. Reeves. Modern Heuristic technqiues for combinatorial problems. John Wiley&Sons, 1993.

Digital Library

[31]

G. Rothermel and M. Harrold. A safe, efficient regression test selection technique. ACM TOSEM, 6(2):173--210, 1997.

Digital Library

[32]

G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Test case prioritization: an empirical study. In Proc. ICSM, pages 179--188, 1999.

Digital Library

[33]

G. Rothermel, R. J. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. IEEE TSE, 27(10):929--948, 2001.

Digital Library

[34]

A. Srivastava and J. Thiagarajan. Effectively prioritizing tests in development environment. In Proc. ISSTA, pages 97--106, 2002.

Digital Library

[35]

K. R. Walcott, M. L. Soffa, G. M. Kapfhammer, and R. S. Roos. Time aware test suite prioritization. In Proc. ISSTA, pages 1--11, 2006.

Digital Library

[36]

H. Williams. Model Building in Mathematical Programming. John Wiley, New York, 1993.

[37]

W. Wong, J. Horgan, S. London, and H. Agrawal. A study of effective regression testing in practice. In Proc. ISSRE, pages 230--238, 1997.

Digital Library

[38]

S. Yoo and M. Harman. Pareto efficient multi objective test case selection. In Proc. ISSTA, pages 140--150, 2007.

Digital Library

[39]

H. Zhong, L. Zhang, and H. Mei. An experimental study of four typical test suite reduction techniques. IST, 50(6):534--546, 2008.

Digital Library

Cited By

Felding EStrandberg PQuttineh NAfzal WHong JPark J(2024)Resource Constrained Test Case Prioritization with Simulated Annealing in an Industrial ContextProceedings of the 39th ACM/SIGAPP Symposium on Applied Computing10.1145/3605098.3635971(1694-1701)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1145/3605098.3635971
Ba-Quttayyan BMohd HYusof Y(2024)A Regression Test Case Prioritization Framework for Software SustainabilityComputing and Informatics10.1007/978-981-99-9589-9_24(315-329)Online publication date: 26-Jan-2024
https://doi.org/10.1007/978-981-99-9589-9_24
Pan WKessentini MMing HYang Z(2023)EASE: An Effort-aware Extension of Unsupervised Key Class Identification ApproachesACM Transactions on Software Engineering and Methodology10.1145/363571433:4(1-43)Online publication date: 2-Dec-2023
https://dl.acm.org/doi/10.1145/3635714
Show More Cited By

Index Terms

Time-aware test-case prioritization using integer linear programming
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Test-case prioritization with model-checkers
SE'07: Proceedings of the 25th conference on IASTED International Multi-Conference: Software Engineering

The order in which test-cases are executed has an influence on the rate at which faults can be detected. In this paper we demonstrate how test-case prioritization can be performed with the use of model-checkers. For this, different well known ...
Adaptive Test-Case Prioritization Guided by Output Inspection
COMPSAC '13: Proceedings of the 2013 IEEE 37th Annual Computer Software and Applications Conference

Test-case prioritization is to schedule the execution order of test cases so as to maximize some objective (e.g., revealing faults early). The existing test-case prioritization approaches separate the process of test-case prioritization and the process ...
Test-case prioritization: achievements and challenges

Test-case prioritization, proposed at the end of last century, aims to schedule the execution order of test cases so as to improve test effectiveness. In the past years, test-case prioritization has gained much attention, and has significant ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA '09: Proceedings of the eighteenth international symposium on Software testing and analysis

July 2009

306 pages

ISBN:9781605583389

DOI:10.1145/1572272

General Chair:
Gregg Rothermel
University of Nebraska-Lincoln, USA
,
Program Chair:
Laura Dillon
Michigan State University, USA

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '09

Sponsor:

ISSTA '09: International Symposium on Software Testing and Analysis

July 19 - 23, 2009

IL, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

126
Total Citations
View Citations
822
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Felding EStrandberg PQuttineh NAfzal WHong JPark J(2024)Resource Constrained Test Case Prioritization with Simulated Annealing in an Industrial ContextProceedings of the 39th ACM/SIGAPP Symposium on Applied Computing10.1145/3605098.3635971(1694-1701)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1145/3605098.3635971
Ba-Quttayyan BMohd HYusof Y(2024)A Regression Test Case Prioritization Framework for Software SustainabilityComputing and Informatics10.1007/978-981-99-9589-9_24(315-329)Online publication date: 26-Jan-2024
https://doi.org/10.1007/978-981-99-9589-9_24
Pan WKessentini MMing HYang Z(2023)EASE: An Effort-aware Extension of Unsupervised Key Class Identification ApproachesACM Transactions on Software Engineering and Methodology10.1145/363571433:4(1-43)Online publication date: 2-Dec-2023
https://dl.acm.org/doi/10.1145/3635714
Zhao YHao DZhang L(2023)Revisiting Machine Learning based Test Case Prioritization for Continuous Integration2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00032(232-244)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00032
Dobslaw FWan RHao Y(2023)Generic and industrial scale many-criteria regression test selectionJournal of Systems and Software10.1016/j.jss.2023.111802205(111802)Online publication date: Nov-2023
https://doi.org/10.1016/j.jss.2023.111802
Singh ASinghrova ABhatia RRattan D(2023)A Systematic Literature Review on Test Case Prioritization TechniquesAgile Software Development10.1002/9781119896838.ch7(101-159)Online publication date: 8-Feb-2023
https://doi.org/10.1002/9781119896838.ch7
Ling XAgrawal RMenzies T(2022)How Different is Test Case Prioritization for Open and Closed Source Projects?IEEE Transactions on Software Engineering10.1109/TSE.2021.306322048:7(2526-2540)Online publication date: 1-Jul-2022
https://doi.org/10.1109/TSE.2021.3063220
Zhu PLi YLi TRen HSun X(2022)Advanced Crowdsourced Test Report Prioritization Based on Adaptive StrategyIEEE Access10.1109/ACCESS.2022.317608610(53522-53532)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3176086
Huang RTowey DXu YZhou YYang N(2022)Dissimilarity‐based test case prioritization through data fusionSoftware: Practice and Experience10.1002/spe.306852:6(1352-1377)Online publication date: 18-Jan-2022
https://doi.org/10.1002/spe.3068
Samad AMahdin HKazmi RIbrahim RBaharum Z(2021)Multiobjective Test Case Prioritization Using Test Case EffectivenessScientific Programming10.1155/2021/99889872021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/9988987
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten