Skip to main content
SpringerLink
Log in

Test case prioritization: a systematic mapping study

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Test case prioritization techniques, which are used to improve the cost-effectiveness of regression testing, order test cases in such a way that those cases that are expected to outperform others in detecting software faults are run earlier in the testing phase. The objective of this study is to examine what kind of techniques have been widely used in papers on this subject, determine which aspects of test case prioritization have been studied, provide a basis for the improvement of test case prioritization research, and evaluate the current trends of this research area. We searched for papers in the following five electronic databases: IEEE Explorer, ACM Digital Library, Science Direct, Springer, and Wiley. Initially, the search string retrieved 202 studies, but upon further examination of titles and abstracts, 120 papers were identified as related to test case prioritization. There exists a large variety of prioritization techniques in the literature, with coverage-based prioritization techniques (i.e., prioritization in terms of the number of statements, basic blocks, or methods test cases cover) dominating the field. The proportion of papers on model-based techniques is on the rise, yet the growth rate is still slow. The proportion of papers that use datasets from industrial projects is found to be 64 %, while those that utilize public datasets for validation are only 38 %. On the basis of this study, the following recommendations are provided for researchers: (1) Give preference to public datasets rather than proprietary datasets; (2) develop more model-based prioritization methods; (3) conduct more studies on the comparison of prioritization methods; (4) always evaluate the effectiveness of the proposed technique with well-known evaluation metrics and compare the performance with the existing methods; (5) publish surveys and systematic review papers on test case prioritization; and (6) use datasets from industrial projects that represent real industrial problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. [[ref#]] refers to papers in Appendix A.

References

  • Catal, C., & Diri, B. (2009). A systematic review of software fault prediction studies. Expert System with Applications, 36(4), 7346–7354.

    Article  Google Scholar 

  • Do, H., Elbaum, S., & Rothermel, G. (2005). Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering, 10(4), 405–435.

    Article  Google Scholar 

  • Do, H., Mirarab, S., Tahvildari, L., & Rothermel, G. (2010). The effects of time constraints on test case prioritization: A series of controlled experiments. IEEE Transactions on Software Engineering SE-2, 36(5), 593–617.

    Article  Google Scholar 

  • Do, H., Rothermel, G., & Kinneer, A. (2006). Prioritizing JUnit test cases: An empirical assessment and cost-benefits analysis. Empirical Software Engineering, 11(1), 33–70.

    Article  Google Scholar 

  • Elbaum, S., Kallakuri, P., Malishevsky, A. G., Rothermel, G., & Kanduri, S. (2003). Understanding the effects of changes on the cost-effectiveness of regression testing techniques. Journal of Software Testing, Verification, and Reliability, 13(2), 65–83.

    Article  Google Scholar 

  • Elbaum, S. G., Malishevsky, A. G., & Rothermel, G. (2000). Prioritizing test cases for regression testing. In Proceedings of the 2000 ACM SIGSOFT international symposium on software testing and analysis (ISSTA 2000), ACM, New York, NY, pp. 102–112, 2000.

  • Engström, E., Runeson, P., & Skoglund, M. (2010). A systematic review on regression test selection techniques. Information and Software Technology, 52(1), 14–30.

    Article  Google Scholar 

  • Engström, E., Skoglund, M., & Runeson, P. (2008). Empirical evaluations of regression test selection techniques: A systematic review. In Proceedings of the second ACM-IEEE international symposium on empirical software engineering and measurement (ESEM ‘08). ACM, New York, NY, USA, pp. 22–31.

  • Jorgensen, M., & Shepperd, M. (2007). A systematic review of software development cost estimation studies. IEEE Transactions on Software Engineering, 33(1), 33–53.

    Article  Google Scholar 

  • Kapfhammer, G. M. (2011). Empirically evaluating regression testing techniques: Challenges, solutions, and a potential way forward. In Proceedings of the 2011 IEEE fourth international conference on software testing, verification and validation workshops (ICSTW ‘11). IEEE Computer Society, Washington, DC, USA, pp. 99–102.

  • Kapfhammer, G. M., & Soffa, M. L. (2007). Using coverage effectiveness to evaluate test suite prioritizations. Proceedings of 1st ACM international workshop on empirical assessment of software engineering languages and technologies: held in conjunction with the 22nd IEEE/ACM international conference on automated software engineering (ASE) 2007 (WEASELTech ‘07), pp. 19–20.

  • King, G. (2007). An Introduction to the dataverse network as an infrastructure for data sharing. Sociological Methods and Research, 36(2), 173–199.

    Article  MathSciNet  Google Scholar 

  • Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009). Systematic literature reviews in software engineering: A systematic literature review. Information and Software Technology, 51(1), 7–15.

    Article  Google Scholar 

  • Kitchenham, B. A., Budgen, D., & Brereton, P. (2010). The value of mapping studies—A participant-observer case study. Proceedings of evaluation and assessment in software engineering (EASE 10), pp. 1–9.

  • Kitchenham, B. A., Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Technical report EBSE-2007-01.

  • Kitchenham, B. A., Dyba, T., & Jorgensen, M. (2004). Evidence-based software engineering. Proceedings of the 26th international conference on software engineering (ICSE ‘04), pp. 273–281.

  • Korel, B., & Koutsogiannakis, G. (2009). Experimental comparison of code-based and model-based test prioritization. Proceedings of the IEEE international conference on software testing, verification, and validation workshops (ICSTW ‘09), pp. 77–84.

  • Korel, B., & Koutsogiannakis, G. (2009). Experimental comparison of code-based and model-based test prioritization. In Proceedings of the IEEE international conference on software testing, verification, and validation workshops (ICSTW ‘09). IEEE Computer Society, Washington, DC, USA, pp. 77–84.

  • Krishnamoorthi, R., & Sahaaya Arul Mary, S. A. (2009). Factor oriented requirement coverage based system test case prioritization of new and regression test cases. Information and Software Technology, 51(4), 799–808.

    Article  Google Scholar 

  • Ma, Z., & Zhao, J. (2008). Test case prioritization based on analysis of program structure. Proceedings of the 2008 15th Asia-Pacific software engineering conference, pp. 471–478.

  • Petersen, K., Feldt, R., Shahid, M., & Mattsson, M. (2008). systematic mapping studies in software engineering. Proceedings of 12th international conference on evaluation and assessment in software engineering (EASE), Department of Informatics, University of Bari, Italy.

  • Qu, X., Cohen, M. B., & Woolf, K. M. (2007). Combinatorial interaction regression testing: A study of test case generation and prioritization. Proceedings of ICSM 2007, pp. 255–264.

  • Srikanth, H., & Williams, L. (2005). On the economics of requirements-based test case prioritization. Proceedings of the seventh international workshop on economics-driven software engineering research (EDSER ‘05), pp. 1–3.

  • Yoo, S., & Harman, M. (2012). Regression testing minimization, selection and prioritization: A survey. Software Testing, Verification and Reliability, 22(2), 67–120.

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the Associate Editor and the four reviewers for their constructive comments on the paper.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Istanbul Kultur University, 34156, Istanbul, Turkey

    Cagatay Catal

  2. Department of Computer Engineering, Atilim University, 06836, Ankara, Turkey

    Deepti Mishra

Authors
  1. Cagatay Catal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deepti Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cagatay Catal.

Appendices

Appendix 1: References for the 120 Included Papers [[1–120]]

The actual mapping of individual papers in each category is shown at the end of the reference. Classification properties are given with C1 (Research Approach), C2 (Study Context), C3 (Dataset Type), C4 (Prioritization Approach), C5 (Research Topic), and C6 (Evaluation Metrics) codes, and the values indicate their subcategories. “Appendix 3” details the subcategories of each property. For example (C1:1, C2:2, C3:1, C4:2, C5:1, C6:2), code at the end of a reference indicates that the research approach is theory, the dataset is from the industrial project, the dataset type is public, the prioritization approach is change-based, the research topic is prioritization method, and the evaluation metric is ASFD.

References from this Appendix must be cited using the [[ref#]] reference style. Please report any problems regarding this classification to: c.catal@iku.edu.tr.

  1. [1]

    David Leon and Andy Podgurski. 2003. A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases. In Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE ‘03). IEEE Computer Society, Washington, DC, USA, 442–. (C1:3, C2:2, C3:1, C4:1&2, C5:2, C6:1)

  2. [2]

    Hyunsook Do and Gregg Rothermel. 2005. A Controlled Experiment Assessing Test Case Prioritization Techniques via Mutation Faults. In Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM ‘05). IEEE Computer Society, Washington, DC, USA, 411–420. (C1:3, C2:2, C3:1, C4:1, C5:2, C6:1)

  3. [3]

    Jung-Min Kim and Adam Porter. 2002. A history-based test prioritization technique for regression testing in resource constrained environments. In Proceedings of the 24th International Conference on Software Engineering (ICSE ‘02). ACM, New York, NY, USA, 119–129. (C1:4, C2:1, C3:2, C4:5, C5:1, C6:9)

  4. [4]

    Xiang Chen, Qing Gu, Xinping Wang, Ang Li, Daoxu Chen. 2009. A Hybrid Approach to Build Prioritized Pairwise Interaction Test Suites. In Computational Intelligence and Software Engineering, 2009. CiSE 2009. International Conference on pp. 1–4. (C1:4, C2:3, C3:4, C4:1, C5:1, C6:8)

  5. [5]

    Siavash Mirarab and Ladan Tahvildari. 2007. A Prioritization Approach for Software Test Cases based on Bayesian Networks. In Proceedings of the 10th International Conference on Fundamental Approaches to Software Engineering (FASE’07), Springer-Verlag, Berlin, Heidelberg, 276–290. (C1:4, C2:2, C3:1, C4:4, C5:1, C6:1)

  6. [6]

    Sihan Li, Naiwen Bian, Zhenyu Chen, Dongjiang You, Yuchen He. 2010. A Simulation Study on Some Search Algorithms for Regression Test Case Prioritization. In International Conference on Quality Software (QSIC) 2010, pp. 72–81. (C1:3, C2:3, C3:4, C4:1, C5:2, C6:9)

  7. [7]

    Bo Jiang, Zhenyu Zhang, W. K. Chan, and T. H. Tse. 2009. Adaptive Random Test Case Prioritization. In Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering (ASE ‘09). IEEE Computer Society, Washington, DC, USA, 233–244. (C1:4, C2:2, C3:1, C4:1, C5:1, C6:1)

  8. [8]

    Xiaoqing Bai, C. Peng Lam, Huaizhong Li. 2004. An Approach to Generate the Thin-Threads from the UML Diagrams. In 28th Annual International Computer Software and Applications Conference (COMPSAC’04), vol. 1, pp. 546–552. (C1:4, C2:3, C3:4, C4:9, C5:4, C6:8)

  9. [9]

    Heiko Stallbaum, Andreas Metzger, and Klaus Pohl. 2008. An Automated Technique for Risk-based Test Case Generation and Prioritization. In Proceedings of the 3rd International Workshop on Automation of Software Test (AST ‘08). ACM, New York, NY, USA, 67–70. (C1:4, C2:1, C3:1, C4:9, C5:1&3, C6:9)

  10. [10]

    Sejun Kim and Jongmoon Baik. 2010. An Effective Fault Aware Test Case Prioritization by Incorporating a Fault Localization Technique. In Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ‘10). ACM, New York, NY, USA, Article 5, 10 pages. (C1:4, C2:2, C3:1, C4:1, C5:1, C6:1)

  11. [11]

    Adam M. Smith and Gregory M. Kapfhammer. 2009. An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization. In Proceedings of the 2009 ACM symposium on Applied Computing (SAC ‘09). ACM, New York, NY, USA, 461–467. (C1:4, C2:1, C3:2, C4:1, C5:1, C6:7)

  12. [12]

    Hyunsook Do, Siavash Mirarab, Ladan Tahvildari, and Gregg Rothermel. 2008. An Empirical Study of the Effect of Time Constraints on the Cost-benefits of Regression Testing. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT ‘08/FSE-16). ACM, New York, NY, USA, 71–82. (C1:3, C2:2, C3:1, C4:1&4, C5:2, C6:9)

  13. [13]

    Siavash Mirarab and Ladan Tahvildari. 2008. An Empirical Study on Bayesian Network-based Approach for Test Case Prioritization. In Proceedings of the 2008 International Conference on Software Testing, Verification, and Validation (ICST ‘08). IEEE Computer Society, Washington, DC, USA, 278–287. (C1:4, C2:2, C3:1, C4:4, C5:1, C6:1)

  14. [14]

    Sheng Huang, Yang Chen, Jun Zhu, Zhong Jie Li, and Hua Fang Tan. 2009. An Optimized Change-driven Regression Testing Selection Strategy for Binary Java Applications. In Proceedings of the 2009 ACM Symposium on Applied Computing (SAC ‘09). ACM, New York, NY, USA, 558–565. (C1:4, C2:2, C3:2, C4:1, C5:1, C6:1)

  15. [15]

    Arshad Mansoor. 2010. Analytical Survey on Automated Software Test Data Evaluation. In Proceedings of the 4th International Conference on New Trends in Information Science and Service Science (NISS), pp. 580–585. (C1:6, C2:3, C3:4, C4:10, C5:4, C6:8)

  16. [16]

    Bogdan Korel, George Koutsogiannakis, Luay H. Tahat. 2008. Application of System Models in Regression Test Suite Prioritization. In Proceedings of the IEEE International Conference on Software Maintenance (ICSM), 247–256. (C1:4, C2:2, C3:1, C4:7, C5:1, C6:6)

  17. [17]

    Shan–Shan Hou, Lu Zhang, Tao Xie, Hong Mei, Jia-Su Sun. 2007. Applying Interface-Contract Mutation in Regression Testing of Component-Based Software. In Proceedings of the IEEE International Conference on Software Maintenance, ICSM 2007, pp. 174–183. (C1:3, C2:2, C3:3, C4:9, C5:4, C6:9)

  18. [18]

    Khin Haymar Saw Hla, YoungSik Choi, and Jong Sou Park. 2008. Applying Particle Swarm Optimization to Prioritizing Test Cases for Embedded Real Time Software Retesting. In Proceedings of the 2008 IEEE 8th International Conference on Computer and Information Technology Workshops (CITWORKSHOPS ‘08). IEEE Computer Society, Washington, DC, USA, 527–532. (C1:4, C2:2, C3:1, C4:1, C5:1, C6:9)

  19. [19]

    Gary Chen, Jamie Rogers. 2010. Arranging Software Test Cases through an Optimization Method. In Proceedings of the Technology Management for Global Economic Growth PICMET 2010, 1596–1600. (C1:4, C2:3, C3:4, C4:9&3, C5:1, C6:8)

  20. [20]

    Xiang Chen, Qing Gu, Xin Zhang, and Daoxu Chen. 2009. Building Prioritized Pairwise Interaction Test Suites with Ant Colony Optimization. In Proceedings of the 2009 Ninth International Conference on Quality Software (QSIC ‘09). IEEE Computer Society, Washington, DC, USA, 347–352. (C1:4, C2:3, C3:4, C4:9, C5:1, C6:9)

  21. [21]

    X. Qu, M. B. Cohen, and K. M. Woolf. 2007. Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization. In Proceedings of the Intl. Conference on Software Maintenance, pages 255–264. (C1:3, C2:2, C3:1, C4:9, C5:2, C6:1&5)

  22. [22]

    Leonardo Mariani, Sofia Papagiannakis, and Mauro Pezze. 2007. Compatibility and Regression Testing of COTS-Component-Based Software. In Proceedings of the 29th International Conference on Software Engineering (ICSE ‘07). IEEE Computer Society, Washington, DC, USA, 85–95. (C1:4, C2:2, C3:1, C4:7, C5:1, C6:9)

  23. [23]

    Xiao Qu. 2009. Configuration Aware Prioritization Techniques in Regression Testing. In Proceedings of the ICSE Companion 2009, 375–378. (C1:4, C2:2, C3:1, C4:1, C5:1, C6:5)

  24. [24]

    Xiao Qu, Myra B. Cohen, and Gregg Rothermel. 2008. Configuration-aware Regression Testing: An Empirical Study of Sampling and Prioritization. In Proceedings of the 2008 International Symposium on Software Testing and Analysis (ISSTA ‘08). ACM, New York, NY, USA, 75–86. (C1:4, C2:2, C3:1, C4:1, C5:1, C6:5)

  25. [25]

    Renee C. Bryce. Constructing Interaction Test Suites with Greedy Algorithms, In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE05), Extended Abstract—Doctoral Symposium. Long Beach, California (November 2005), pp. 440–443. (C1:5, C2:3, C3:4, C4:1, C5:4, C6:8)

  26. [26]

    Jianjun Yuan, Zuowen Jiang. 2010. Constructing Prioritized Interaction Test Suite with Interaction Relationship. In Proceedings of the Second International Workshop on Education Technology and Computer Science (etcs 2010), vol. 3, pp. 181–184. (C1:4, C2:3, C3:4, C4:1, C5:4, C6:8)

  27. [27]

    Fevzi Belli, Mubariz Eminov, Nida Gökçe. Coverage-Oriented, Prioritized Testing—A Fuzzy Clustering Approach and Case Study. In Proceedings of 3rd Latin-American Symposium (LADC’2007), pp. 95–110. (C1:4, C2:2, C3:2, C4:7&1, C5:1, C6:8)

  28. [28]

    Yu-Chi Huang, Chin-Yu Huang, Jun-Ru Chang, Tsan-Yuan Chen. 2010. Design and Analysis of Cost-Cognizant Test Case Prioritization Using Genetic Algorithm with Test History. In Proceedings of the IEEE 34th Annual Computer Software and Applications Conference (compsac 2010), pp. 413–418. (C1:4, C2:2, C3:1, C4:5&1, C5:1, C6:4)

  29. [29]

    Chin-Yu Huang, Jun-Ru Chang, and Yung-Hsin Chang. 2010. Design and Analysis of GUI Test-case Prioritization using Weight-based Methods. J. Syst. Softw. 83, 4 (April 2010), 646–659. (C1:4, C2:2, C3:1, C4:7, C5:1, C6:1)

  30. [30]

    Renee C. Bryce, Sreedevi Sampath, and Atif M. Memon. 2011. Developing a Single Model and Test Prioritization Strategies for Event-Driven Software. IEEE Trans. Softw. Eng. 37, 1 (January 2011), 48–64. (C1:4, C2:2, C3:2, C4:1, C5:1, C6:1)

  31. [31]

    Mark Last, Shay Eyal, Abraham Kandel. 2005. Effective Black-Box Testing with Genetic Algorithms. In Proceedings of Haifa Verification Conference ‘2005. pp. 134 ~ 148. (C1:4, C2:1, C3:2, C4:1, C5:1, C6:9)

  32. [32]

    Sara Alspaugh, Kristen R. Walcott, Michael Belanich, Gregory M. Kapfhammer, and Mary Lou Soffa. 2007. Efficient Time-aware Prioritization with Knapsack Solvers. In Proceedings of the 1st ACM International Workshop on Empirical Assessment of Software Engineering Languages and Technologies: held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 2007 (WEASELTech ‘07). ACM, New York, NY, USA, 13–18. (C1:3, C2:2, C3:1, C4:1&9, C5:2, C6:9)

  33. [33]

    Hyunsook Do, Gregg Rothermel, and Alex Kinneer. 2004. Empirical Studies of Test Case Prioritization in a JUnit Testing Environment. In Proceedings of the 15th International Symposium on Software Reliability Engineering (ISSRE ‘04). IEEE Computer Society, Washington, DC, USA, 113–124. (C1:3, C2:2, C3:1, C4:1, C5:2, C6:1)

  34. [34]

    Alexander P. Conrad, Robert S. Roos, and Gregory M. Kapfhammer. 2010. Empirically Studying the Role of Selection Operators during Search-Based Test Suite Prioritization. In the Proceedings of the ACM SIGEVO Genetic and Evolutionary Computation Conference, Portland, Oregon, July 2010. (C1:4, C2:2, C3:1, C4:1, C5:1, C6:7)

  35. [35]

    T. M. S. Ummu Salima, A. Askarunisha, and N. Ramaraj. 2007. Enhancing the Efficiency of Regression Testing through Intelligent Agents. In Proceedings of the International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007)—Volume 01 (ICCIMA ‘07), Vol. 1. IEEE Computer Society, Washington, DC, USA, 103–108. (C1:1, C2:3, C3:4, C4:1, C5:4, C6:1)

  36. [36]

    Bogdan Korel and George Koutsogiannakis. 2009. Experimental Comparison of Code-Based and Model-Based Test Prioritization. In Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation Workshops (ICSTW ‘09). IEEE Computer Society, Washington, DC, USA, 77–84. (C1:3, C2:2, C3:2, C4:1, C5:2, C6:6)

  37. [37]

    Dennis Jeffrey and Neelam Gupta. 2008. Experiments with Test Case Prioritization using Relevant Slices. J. Syst. Softw. 81, 2 (February 2008), 196–221. (C1:4, C2:2, C3:1, C4:1&9, C5:1, C6:1)

  38. [38]

    R. Krishnamoorthi and S. A. Sahaaya Arul Mary. 2009. Factor Oriented Requirement Coverage based System Test Case Prioritization of New and Regression Test Cases. Inf. Softw. Technol. 51, 4 (April 2009), 799–808. (C1:4, C2:1, C3:2, C4:3&6, C5:1, C6:2)

  39. [39]

    Y. T. Yu, S. P. Ng, and Eric Y. K. Chan. 2003. Generating, Selecting and Prioritizing Test Cases from Specifications with Tool Support. In Proceedings of the Third International Conference on Quality Software (QSIC ‘03). IEEE Computer Society, Washington, DC, USA, 83–. (C1:1, C2:3, C3:4, C4:9, C5:4, C6:8)

  40. [40]

    Hyuncheol Park, Hoyeon Ryu, and Jongmoon Baik. 2008. Historical Value-Based Approach for Cost-Cognizant Test Case Prioritization to Improve the Effectiveness of Regression Testing. In Proceedings of the 2008 Second International Conference on Secure System Integration and Reliability Improvement (SSIRI ‘08). IEEE Computer Society, Washington, DC, USA, 39–46. (C1:4, C2:2, C3:1, C4:8, C5:1, C6:4)

  41. [41]

    Bo Jiang, Zhenyu Zhang, T. H. Tse, and T. Y. Chen. 2009. How Well Do Test Case Prioritization Techniques Support Statistical Fault Localization. In Proceedings of the 2009 33rd Annual IEEE International Computer Software and Applications Conference—Volume 01 (COMPSAC ‘09), Vol. 1. IEEE Computer Society, Washington, DC, USA, 99–106. (C1:3, C2:2, C3:1, C4:1&2, C5:4, C6:9)

  42. [42]

    Yalda Fazlalizadeh, Alireza Khalilian, Mohammad Abdollahi Azgomi, and Saeed Parsa. 2009. Incorporating Historical Test Case Performance Data and Resource Constraints into Test Case Prioritization. In Proceedings of the 3rd International Conference on Tests and Proofs (TAP ‘09), Catherine Dubois (Ed.). Springer-Verlag, Berlin, Heidelberg, 43–57. (C1:4, C2:2, C3:1, C4:5, C5:1, C6:1)

  43. [43]

    Ramasamy Krishnamoorthi, S.A. Sahaaya Arul Mary. 2008. Incorporating Varying Requirement Priorities and Costs in Test Case Prioritization for New and Regression Testing. In Proceedings of the Computing, Communication and Networking, 2008. ICCCn 2008. pp. 1–9. (C1:4, C2:1, C3:2, C4:3&6, C5:1, C6:2)

  44. [44]

    Sebastian Elbaum, Alexey Malishevsky, and Gregg Rothermel. 2001. Incorporating Varying Test Costs and Fault Severities into Test Case Prioritization. In Proceedings of the 23rd International Conference on Software Engineering (ICSE ‘01). IEEE Computer Society, Washington, DC, USA, 329–338. (C1:3, C2:2, C3:1, C4:1, C5:3, C6:4)

  45. [45]

    Lingming Zhang, Ji Zhou, Dan Hao, Lu Zhang, and Hong Mei. 2009. Jtop: Managing JUnit Test Cases in Absence of Coverage Information. In Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering (ASE ‘09). IEEE Computer Society, Washington, DC, USA, 677–679. (C1:5, C2:3, C3:4, C4:7, C5:4, C6:8)

  46. [46]

    Praveen Ranjan Srivastava. 2008. Model for Optimizing Software Testing Period using Non homogenous Poisson Process based on Cumulative Test Case Prioritization. In Proceedings of the TENCON 2008–2008 IEEE Region 10 Conference, pp. 1–6, 19–21 Nov. 2008. (C1:4, C2:3, C3:4, C4:4, C5:1, C6:8)

  47. [47]

    Bogdan Korel, George Koutsogiannakis, and Luay H. Tahat. 2007. Model-based Test Prioritization Heuristic Methods and Their Evaluation. In Proceedings of the 3rd International Workshop on Advances in Model-based Testing (A-MOST ‘07). ACM, New York, NY, USA, 34–43. (C1:4, C2:2, C3:2, C4:7, C5:1, C6:6)

  48. [48]

    Fevzi Belli, Mubariz Eminov, and Nida Gokce. 2009. Model-based Test Prioritizing: A Comparative Soft-computing Approach and Case Studies. In Proceedings of the 32nd Annual German Conference on Advances in Artificial Intelligence (KI’09), Springer-Verlag, Berlin, Heidelberg, 427–434. (C1:4, C2:2, C3:2, C4:7&1, C5:1, C6:8)

  49. [49]

    Alexey G. Malishevsky, Gregg Rothermel, Sebastian Elbaum. 2002. Modeling the Cost-Benefits Tradeoffs for Regression Testing Techniques. In Proceedings of the International Conference on Software Maintenance (ICSM’02) (ICSM ‘02). IEEE Computer Society, Washington, DC, USA, 204-. (C1:3, C2:2, C3:2, C4:1, C5:4, C6:9)

  50. [50]

    Gregg Rothermel, Sebastian Elbaum, Alexey G. Malishevsky, Praveen Kallakuri, and Xuemei Qiu. 2004. On Test Suite Composition and Cost-effective Regression Testing. ACM Trans. Softw. Eng. Methodol. 13, 3 (July 2004), 277–331. (C1:3, C2:2, C3:2, C4:1, C5:4, C6:1)

  51. [51]

    Hema Srikanth and Laurie Williams. 2005. On the Economics of Requirements-based Test Case Prioritization. In Proceedings of the Seventh International Workshop on Economics-driven Software Engineering Research (EDSER ‘05). ACM, New York, NY, USA, 1–3. (C1:4, C2:1, C3:2, C4:6&3, C5:1, C6:2)

  52. [52]

    Bo Jiang and W. K. Chan. 2010. On the Integration of Test Adequacy, Test Case Prioritization, and Statistical Fault Localization. In Proceedings of the 2010 10th International Conference on Quality Software (QSIC ‘10). IEEE Computer Society, Washington, DC, USA, 377–384. (C1:3, C2:2, C3:1, C4:1&9, C5:4, C6:9)

  53. [53]

    Hyunsook Do and Gregg Rothermel. 2006. On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques. IEEE Trans. Softw. Eng. 32, 9 (September 2006), 733–752. (C1:3, C2:2, C3:1, C4:1, C5:2, C6:1)

  54. [54]

    Murali Krishna Ramanathan, Mehmet Koyuturk, Ananth Grama, and Suresh Jagannathan. 2008. PHALANX: A Graph-theoretic Framework for Test Case Prioritization. In Proceedings of the 2008 ACM Symposium on Applied Computing (SAC ‘08). ACM, New York, NY, USA, 667–673. (C1:3, C2:2, C3:1, C4:1&9, C5:2, C6:1)

  55. [55]

    Ke Zhai and W. K. Chan. 2010. Point-of-Interest Aware Test Case Prioritization: Methods and Experiments. In Proceedings of the 2010 10th International Conference on Quality Software (QSIC ‘10). IEEE Computer Society, Washington, DC, USA, 449–456. (C1:3, C2:1, C3:2, C4:9, C5:2, C6:1)

  56. [56]

    Mark Sherriff, Mike Lake, and Laurie Williams. 2007. Prioritization of Regression Tests using Singular Value Decomposition with Empirical Change Records. In Proceedings of the The 18th IEEE International Symposium on Software Reliability (ISSRE ‘07). IEEE Computer Society, Washington, DC, USA, 81–90. (C1:4, C2:2, C3:2, C4:5&9, C5:1, C6:8)

  57. [57]

    Sapna P.G., Hrushikesha Mohanty. 2009. Prioritization of Scenarios Based on UML Activity Diagrams. In Proceedings of the First International Conference on Computational Intelligence, Communication Systems and Networks (cicsyn 2009), pp. 271–276. (C1:4, C2:1, C3:4, C4:7, C5:1, C6:8)

  58. [58]

    Sangeeta Sabharwal, Ritu Sibal, Chayanika Sharma. 2010. Prioritization of Test Case Scenarios Derived from Activity Diagram using Genetic Algorithm. In Proceedings of the Computer and Communication Technology (ICCCT), 2010 International Conference on, pp. 481–485. (C1:4, C2:1, C3:4, C4:1, C5:1, C6:8)

  59. [59]

    Yuen-Tak Yu and Man Fai Lau. 2002. Prioritization of Test Cases in MUMCUT Test Sets: An Empirical Study. In Proceedings of the 7th Ada-Europe International Conference on Reliable Software Technologies (Ada-Europe ‘02), Springer-Verlag, London, UK, UK, 245–256. (C1:3, C2:2, C3:1, C4:9, C5:2, C6:1)

  60. [60]

    Renee C. Bryce, Charles J. Colbourn. 2006. Prioritized Interaction Testing for Pair-wise Coverage with Seeding and Constraints, Information and Software Technology, Volume 48, Issue 10, Advances in Model-based Testing, October 2006, Pages 960–970. (C1:4, C2:3, C3:4, C4:1, C5:1, C6:8)

  61. [61]

    Jianhua Gao and Jie Zhu. 2009. Prioritized Test Generation Strategy for Pair-Wise Testing. In Proceedings of the 2009 15th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC ‘09). IEEE Computer Society, Washington, DC, USA, 99–102. (C1:4, C2:3, C3:4, C4:1, C5:1, C6:8)

  62. [62]

    Fevzi Belli, Mubariz Eminov, and Nida Gokce. 2007. Prioritizing Coverage-Oriented Testing Process—An Adaptive-Learning-Based Approach and Case Study. In Proceedings of the 31st Annual International Computer Software and Applications Conference–Volume 02 (COMPSAC ‘07), Vol. 2. IEEE Computer Society, Washington, DC, USA, 197–203. (C1:4, C2:2, C3:2, C4:1, C5:1, C6:8)

  63. [63]

    Lingming Zhang, Ji Zhou, Dan Hao, Lu Zhang and Hong Mei. 2009. Prioritizing JUnit Test Cases in Absence of Coverage Information. In Proceedings of 25th International Conference on Software Maintenance (ICSM 2009), 19–28. (C1:4, C2:2, C3:1, C4:7, C5:1, C6:1)

  64. [64]

    Hyunsook Do, Gregg Rothermel, Alex Kinneer. 2006. Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis. Empirical Software Engineering. 11, 1, 33–70. (C1:3, C2:2, C3:1, C4:1, C5:1, C6:1)

  65. [65]

    Dianxiang Xu and Junhua Ding. 2010. Prioritizing State-Based Aspect Tests. In Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation (ICST ‘10). IEEE Computer Society, Washington, DC, USA, 265–274. (C1:4, C2:2, C3:2, C4:7, C5:1, C6:9)

  66. [66]

    Gregg Rothermel, Roland J. Untch, and Chengyun Chu. 2001. Prioritizing Test Cases For Regression Testing. IEEE Trans. Softw. Eng. 27, 10 (October 2001), 929–948. (C1:3, C2:2, C3:1, C4:1, C5:1, C6:1)

  67. [67]

    Y. Fazlalizadeh, A. Khalilian, M. Azgomi, and S. Parsa. 2009. Prioritizing Test Cases for Resource Constraint Environments using Historical Test Case Performance Data. 2nd IEEE International Conference on Computer Science and Information Technology, 190–195. (C1:4, C2:2, C3:1, C4:5, C5:1, C6:1)

  68. [68]

    Alberto Gonzalez-Sanchez, Eric Piel, Hans-Gerhard Gross, and Arjan J. C. van Gemund. 2010. Prioritizing Tests for Software Fault Localization. In Proceedings of the 2010 10th International Conference on Quality Software (QSIC ‘10). IEEE Computer Society, Washington, DC, USA, 42–51. (C1:4, C2:2, C3:1, C4:9, C5:1, C6:1)

  69. [69]

    Emad Shihab, Zhen Ming Jiang, Bram Adams, Ahmed E. Hassan, and Robert Bowerman. 2010. Prioritizing Unit Test Creation for Test-Driven Maintenance of Legacy Systems. In Proceedings of the 2010 10th International Conference on Quality Software (QSIC ‘10). IEEE Computer Society, Washington, DC, USA, 132–141. (C1:4, C2:2, C3:2, C4:5, C5:1&3, C6:9)

  70. [70]

    Sapna P.G. and Hrusikesha Mohanty. 2009. Prioritizing Use Cases to Aid Ordering of Scenarios. In Proceedings of the 2009 Third UKSim European Symposium on Computer Modeling and Simulation (EMS ‘09). IEEE Computer Society, Washington, DC, USA, 136–141. (C1:4, C2:1, C3:2, C4:3&7, C5:1, C6:1)

  71. [71]

    Sreedevi Sampath, Renee C. Bryce, Gokulanand Viswanath, Vani Kandimalla, and A. Gunes Koru. 2008. Prioritizing User-Session-Based Test Cases for Web Applications Testing. In Proceedings of the 2008 International Conference on Software Testing, Verification, and Validation. IEEE Computer Society, Washington, DC, USA, 141–150. (C1:4, C2:2, C3:2, C4:1, C5:1, C6:1)

  72. [72]

    Gregg Rothermel and Sebastian Elbaum. 2003. Putting Your Best Tests Forward. IEEE Softw. 20, 5 (September 2003), 74–77. (C1:5, C2:2, C3:2, C4:1, C5:1, C6:1)

  73. [73]

    Shan–Shan Hou, Lu Zhang, Tao Xie, Jiasu Sun. 2008. Quota-Constrained Test-Case Prioritization for Regression Testing of Service-Centric Systems. In Proceedings of International Conference on Software Maintenance 2008, 257–266. (C1:4, C2:1, C3:2, C4:9, C5:1, C6:1)

  74. [74]

    Hema Srikanth, Myra B. Cohen, and Xiao Qu. 2009. Reducing Field Failures in System Configurable Software: Cost-based Prioritization. In Proceedings of the 20th IEEE International Conference on Software Reliability Engineering (ISSRE’09), IEEE Press, Piscataway, NJ, USA, 61–70. (C1:4, C2:2, C3:2, C4:8, C5:1&3, C6:5)

  75. [75]

    Cristian Simons, Emerson Cabrera Paraiso. Regression Test Cases Prioritization using Failure Pursuit Sampling. In Proceedings of the ISDA 2010, 923–928. (C1:4, C2:2, C3:1, C4:2, C5:1, C6:1)

  76. [76]

    Zhihao Zhang, Jun Huang, Bo Zhang, Jinlong Lin, and Xiaolan Chen. 2010. Regression Test Generation Approach Based on Tree-Structured Analysis. In Proceedings of the 2010 International Conference on Computational Science and Its Applications (ICCSA ‘10). IEEE Computer Society, Washington, DC, USA, 244–249. (C1:1, C2:3, C3:4, C4:5, C5:1, C6:8)

  77. [77]

    Nashat Mansour and Wael Statieh. 2009. Regression test selection for c\# programs. Adv. Soft. Eng. 2009, Article 1 (January 2009), 16 pages. (C1:1, C2:2, C3:2, C4:5, C5:1, C6:8)

  78. [78]

    Pavan Kumar Chittimalli and Mary Jean Harrold. 2008. Regression Test Selection on System Requirements. In Proceedings of the 1st India Software Engineering Conference (ISEC ‘08). ACM, New York, NY, USA, 87–96. (C1:4, C2:2, C3:2, C4:6, C5:1, C6:8)

  79. [79]

    S. Yoo, M. Harman. 2012. Regression Testing Minimization, Selection and Prioritization: a Survey. Software Testing, Verification and Reliability 22, 2, 67–120. (C1:6, C2:3, C3:4, C4:10, C5:1, C6:8)

  80. [80]

    Alireza Mahdian, Anneliese Amschler Andrews, and Orest Jacob Pilskalns. 2009. Regression Testing with UML Software Designs: A Survey. J. Softw. Maint. Evol. 21, 4 (July 2009), 253–286. (C1:6, C2:3, C3:4, C4:10, C5:1, C6:8)

  81. [81]

    R. Kavitha, V.R. Kavitha, N. Suresh Kumar. 2010. Requirement Based Test Case Prioritization. In Proceedings of International Conference on Communication Control and Computing Technologies, 826–829. (C1:4, C2:2, C3:2, C4:6, C5:1&3, C6:3)

  82. [82]

    Zheng Li, Mark Harman, and Robert M. Hierons. 2007. Search Algorithms for Regression Test Case Prioritization. IEEE Trans. Softw. Eng. 33, 4 (April 2007), 225–237. (C1:3, C2:2, C3:1, C4:10, C5:1, C6:9)

  83. [83]

    Sebastian Elbaum, Gregg Rothermel, Satya Kanduri, and Alexey G. Malishevsky. 2004. Selecting a Cost-Effective Test Case Prioritization Technique. Software Quality Journal 12, 3 (September 2004), 185–210. (C1:3, C2:2, C3:3, C4:1, C5:4, C6:1)

  84. [84]

    Roberto S. Silva Filho, Christof J. Budnik, William M. Hasling, Monica McKenna, Rajesh Subramanyan. 2010. Supporting Concern-Based Regression Testing and Prioritization in a Model-Driven Environment. In Proceedings of IEEE 34th Annual Computer Software and Applications Conference Workshops, 323–328. (C1:1, C2:3, C3:4, C4:7, C5:1, C6:8)

  85. [85]

    Hema Srikanth, Laurie Williams, Jason Osborne. 2005. System Test Case Prioritization of New and Regression Test Cases. In Proceedings of ISESE’2005, 64–73. (C1:4, C2:1, C3:2, C4:14&6, C5:6&3, C6:9)

  86. [86]

    Debasish Kundu, Monalisa Sarma, Debasis Samanta, and Rajib Mall. 2009. System Testing for Object-Oriented Systems with Test Case Prioritization. Softw. Test. Verif. Reliab. 19, 4 (December 2009), 297–333. (C1:4, C2:2, C3:2, C4:7, C5:1, C6:9)

  87. [87]

    Lijun Mei, W. K. Chan, T. H. Tse, and Robert G. Merkel. 2009. Tag-Based Techniques for Black-Box Test Case Prioritization for Service Testing. In Proceedings of the 2009 Ninth International Conference on Quality Software (QSIC ‘09). IEEE Computer Society, Washington, DC, USA, 21–30. (C1:4, C2:2, C3:2, C4:9, C5:1, C6:1)

  88. [88]

    Ke Zhai, Bo Jiang, W. K. Chan, T. H. Tse. 2010. Taking Advantage of Service Selection: A Study on the Testing of Location-Based Web Services through Test Case Prioritization. In Proceedings of the ICWS 2010, 211–218. (C1:4, C2:2, C3:2, C4:9, C5:1&3, C6:9)

  89. [89]

    P. R. Srivastava, M. Ray, and J. Dermoudy, and B. H. Kang, and T. H. Kim. 2009. Test Case Minimization and Prioritization Using CMIMX Technique. In Proceedings of the International Conference on Advanced Software Engineering and Its Applications. Jeju Island, Korea, 25–33. (C1:4, C2:3, C3:4, C4:1, C5:1, C6:8)

  90. [90]

    Zengkai Ma and Jianjun Zhao. 2008. Test Case Prioritization Based on Analysis of Program Structure. In Proceedings of the 2008 15th Asia–Pacific Software Engineering Conference (APSEC ‘08). IEEE Computer Society, Washington, DC, USA, 471–478. (C1:4, C2:2, C3:1, C4:5, C5:1&3, C6:4)

  91. [91]

    Lucas Lima, Juliano Iyoda, Augusto Sampaio, and Eduardo Aranha. 2009. Test Case Prioritization based on Data Reuse an Experimental Study. In Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement (ESEM ‘09). IEEE Computer Society, Washington, DC, USA, 279–290. (C1:3, C2:2, C3:2, C4:9, C5:1, C6:9)

  92. [92]

    Xiaofang Zhang, Changhai Nie, Baowen Xu, and Bo Qu. 2007. Test Case Prioritization Based on Varying Testing Requirement Priorities and Test Case Costs. In Proceedings of the Seventh International Conference on Quality Software (QSIC ‘07). IEEE Computer Society, Washington, DC, USA, 15–24. (C1:4, C2:3, C3:4, C4:6&8, C5:1&3, C6:9)

  93. [93]

    Bo Qu, Changhai Nie, Baowen Xu, and Xiaofang Zhang. 2007. Test Case Prioritization for Black Box Testing. In Proceedings of the 31st Annual International Computer Software and Applications Conference—Volume 01 (COMPSAC ‘07), Vol. 1. IEEE Computer Society, Washington, DC, USA, 465–474. (C1:4, C2:1, C3:2, C4:5, C5:1&3, C6:9)

  94. [94]

    Bo Qu, Changhai Nie, Baowen Xu. 2008. Test Case Prioritization for Multiple Processing Queues. In Proceedings of International Symposium on Information Science and Engieering, 646–649. (C1:4, C2:1, C3:2, C4:9, C5:1&3, C6:1)

  95. [95]

    Lijun Mei, Zhenyu Zhang, W. K. Chan, and T. H. Tse. 2009. Test Case Prioritization for Regression Testing of Service-Oriented Business Applications. In Proceedings of the 18th International Conference on World Wide Web (WWW ‘09). ACM, New York, NY, USA, 901–910. (C1:4, C2:2, C3:2, C4:1, C5:1, C6:1)

  96. [96]

    Lin Chen, Ziyuan Wang, Lei Xu, Hongmin Lu, and Baowen Xu. 2010. Test Case Prioritization for Web Service Regression Testing. In Proceedings of the 2010 Fifth IEEE International Symposium on Service Oriented System Engineering (SOSE ‘10). IEEE Computer Society, Washington, DC, USA, 173–178. (C1:4, C2:3, C3:4, C4:7, C5:1, C6:8)

  97. [97]

    Dennis Jeffrey and Neelam Gupta. 2006. Test Case Prioritization Using Relevant Slices. In Proceedings of the 30th Annual International Computer Software and Applications Conference—Volume 01 (COMPSAC ‘06), Vol. 1. IEEE Computer Society, Washington, DC, USA, 411–420. (C1:4, C2:2, C3:1, C4:1, C5:1, C6:1)

  98. [98]

    Sebastian Elbaum, Alexey G. Malishevsky, Gregg Rothermel. 2002. Test Case Prioritization: A Family of Empirical Studies. IEEE Transactions on Software Engineering. 28, 2, 159–182. (C1:3, C2:2, C3:1, C4:1, C5:1, C6:1)

  99. [99]

    Jussi Kasurinen, Ossi Taipale, and Kari Smolander. 2010. Test Case Selection and Prioritization: Risk-based or Design-based? In Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ‘10). ACM, New York, NY, USA, Article 10, 10 pages. (C1:2, C2:3, C3:4, C4:10, C5:1&3, C6:8)

  100. [100]

    Farn Wang and Geng-Dian Huang. 2008. Test Plan Generation for Concurrent Real-Time Systems Based on Zone Coverage Analysis. In Proceedings of the 20th IFIP TC 6/WG 6.1 International Conference on Testing of Software and Communicating Systems: 8th International Workshop (TestCom ‘08/FATES ‘08). Springer-Verlag, Berlin, Heidelberg, 234–249. (C1:3, C2:2, C3:2, C4:1, C5:4, C6:9)

  101. [101]

    Bogdan Korel, Luay H. Tahat, and Mark Harman. 2005. Test Prioritization Using System Models. In Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM ‘05). IEEE Computer Society, Washington, DC, USA, 559–568. (C1:4, C2:2, C3:2, C4:7, C5:1&3, C6:6)

  102. [102]

    Renee C. Bryce and Atif M. Memon. 2007. Test Suite Prioritization by Interaction Coverage. In Workshop on Domain Specific Approaches to Software Test Automation: in conjunction with the 6th ESEC/FSE Joint Meeting (DOSTA ‘07). ACM, New York, NY, USA, 1–7. (C1:4, C2:2, C3:2, C4:9, C5:1, C6:1)

  103. [103]

    Adam M. Smith, Joshua Geiger, Gregory M. Kapfhammer, and Mary Lou Soffa. 2007. Test Suite Reduction and Prioritization with Call Trees. In Proceedings of the Twenty-Second IEEE/ACM International Conference on Automated Software Engineering (ASE ‘07). ACM, New York, NY, USA, 539–540. (C1:3, C2:1, C3:2, C4:1, C5:1, C6:7)

  104. [104]

    Marek G. Stochel and Radek Sztando. 2008. Testing Optimization for Mission-Critical, Complex, Distributed Systems. In Proceedings of the 2008 32nd Annual IEEE International Computer Software and Applications Conference (COMPSAC ‘08). IEEE Computer Society, Washington, DC, USA, 847–852. (C1:4, C2:2, C3:2, C4:5, C5:1, C6:9)

  105. [105]

    James A. Jones and Mary Jean Harrold. 2001. Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage. In Proceedings of the IEEE International Conference on Software Maintenance (ICSM’01) (ICSM ‘01). IEEE Computer Society, Washington, DC, USA, 92–101. (C1:4, C2:2, C3:1, C4:1, C5:1, C6:8)

  106. [106]

    James A. Jones and Mary Jean Harrold. 2003. Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage. IEEE Trans. Softw. Eng. 29, 3 (March 2003), 195–209. (C1:4, C2:2, C3:1, C4:1, C5:1, C6:8)

  107. [107]

    Panduka Nagahawatte and Hyunsook Do. 2010. The Effectiveness of Regression Testing Techniques in Reducing the Occurrence of Residual Defects. In Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation (ICST ‘10). IEEE Computer Society, Washington, DC, USA, 79–88. (C1:3, C2:2, C3:1, C4:1, C5:4, C6:9)

  108. [108]

    Hyunsook Do, Siavash Mirarab, Ladan Tahvildari, and Gregg Rothermel. 2010. The Effects of Time Constraints on Test Case Prioritization: A Series of Controlled Experiments. IEEE Trans. Softw. Eng. 36, 5 (September 2010), 593–617. (C1:3, C2:2, C3:1, C4:1&4, C5:4, C6:9)

  109. [109]

    Saif ur Rehman Khan, Inayat ur Rehman, Saif Ur Rehman Malik. 2009. The Impact of Test Case Reduction and Prioritization on Software Testing Effectiveness. In Proceedings of 5th International Conference on Emerging Technologies. Islamabad, Pakistan, 416–421. (C1:3, C2:1, C3:2, C4:1, C5:4, C6:8)

  110. [110]

    K. Walcott, M. Soffa, G. Kapfhammer, and R. Roos. 2006. Time-Aware Test Suite Prioritization. In Proceedings of ISSTA ‘06. Portland, Maine, USA, 1–11. (C1:4, C2:2, C3:3, C4:1, C5:1, C6:1)

  111. [111]

    Lu Zhang, Shan–Shan Hou, Chao Guo, Tao Xie, and Hong Mei. 2009. Time-aware Test-case Prioritization using Integer Linear Programming. In Proceedings of the eighteenth international symposium on Software testing and analysis (ISSTA ‘09). ACM, New York, NY, USA, 213–224. (C1:4, C2:2, C3:1, C4:9, C5:1, C6:9)

  112. [112]

    Matthew J. Rummel, Gregory M. Kapfhammer, and Andrew Thall. 2005. Towards the Prioritization of Regression Test Suites with Data Flow Information. In Proceedings of the 2005 ACM Symposium on Applied Computing (SAC ‘05), ACM, New York, NY, USA, 1499–1504. (C1:4, C2:1, C3:2, C4:7, C5:1, C6:1)

  113. [113]

    Sebastian Elbaum, David Gable, and Gregg Rothermel. 2001. Understanding and Measuring the Sources of Variation in the Prioritization of Regression Test Suites. In Proceedings of the 7th International Symposium on Software Metrics (METRICS ‘01). IEEE Computer Society, Washington, DC, USA, 169-. (C1:3, C2:2, C3:1, C4:1, C5:4, C6:1)

  114. [114]

    Sebastian Elbaum, Praveen Kallakuri, Alexey Malishevsky, Gregg Rothermel, Satya Kanduri. 2003. Understanding the Effects of Changes on the Cost-effectiveness of Regression Testing Techniques. J. Softw. Test. Verif. Reliab. 13, 2, 65–81. (C1:3, C2:2, C3:3, C4:1, C5:4, C6:1)

  115. [115]

    Gregory M. Kapfhammer and Mary Lou Soffa. 2007. Using Coverage Effectiveness to Evaluate Test Suite Prioritizations. In Proceedings of the 1st ACM International Workshop on Empirical Assessment of Software Engineering Languages and Technologies: held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 2007 (WEASELTech ‘07). ACM, New York, NY, USA, 19–20. (C1:1, C2:3, C3:4, C4:10, C5:3, C6:7)

  116. [116]

    Zhi Quan Zhou. 2010. Using Coverage Information to Guide Test Case Selection in Adaptive Random Testing. In Proceedings of 34th Annual Computer Software and Applications Conference Workshops, 208–213. (C1:4, C2:2, C3:1, C4:1, C5:1&3, C6:9)

  117. [117]

    Zachary D. Williams and Gregory M. Kapfhammer. 2010. Using Synthetic Test Suites to Empirically Compare Search-based and Greedy Prioritizers. In Proceedings of the 12th Annual Conference Companion on Genetic and Evolutionary Computation (GECCO ‘10). ACM, New York, NY, USA, 2119–2120. (C1:3, C2:3, C3:4, C4:10, C5:4, C6:7)

  118. [118]

    Paolo Tonella, Paolo Avesani, and Angelo Susi. 2006. Using the Case-Based Ranking Methodology for Test Case Prioritization. In Proceedings of the 22nd IEEE International Conference on Software Maintenance (ICSM ‘06). IEEE Computer Society, Washington, DC, USA, 123–133. (C1:4, C2:2, C3:1, C4:9, C5:1, C6:1)

  119. [119]

    Athira B and Philip Samuel. 2010. Web Services Regression Test Case Prioritization. In Proceedings of CISIM 2010. (C1:4, C2:2, C3:4, C4:7, C5:1, C6:8)

  120. [120]

    Lijun Mei, W. K. Chan, T. H. Tse, and Robert G. Merkel. 2011. XML-manipulating Test Case Prioritization for XML-manipulating Services. J. Syst. Softw. 84, 4 (April 2011), 603–619. (C1:4, C2:2, C3:2, C4:9, C5:1, C6:1)

Appendix 2

1.1 The Numbers of Papers in Each Journal

(The numbers of papers identified in each journal are shown in ().)

ACM Transactions on Software Engineering and Methodology (1)

Advances in Software Engineering (1)

Empirical Software Engineering (1)

IEEE Software (1)

IEEE Transactions on Software Engineering (7)—Rank 1

Information and Software Technology (2)—Rank 3

Journal of Maintenance and Evolution: Research and Practice (1)

Journal of Systems and Software (3)—Rank 2

Software Quality Journal (1)

Software Testing, Verification and Reliability (3)—Rank 2

Appendix 3

Category 1 and Category 2 were obtained from the review paper of Jorgensen and Shepperd (2007). Category 3 was obtained from our review article Catal and Diri (2009). We created the major categories based on the number of studies that used a particular category. For example, if there were just one or two papers about a subcategory, we did not create a new major subcategory for that item. We made multiple passes over the papers to iteratively extract potential properties and categories and, ultimately, we came up with these categories.

1.1 Category 1: Research Approach

  1. 1.

    Theory: Non-empirical research approaches.

  2. 2.

    Survey: Questionnaire and interview-based surveys of industry practice.

  3. 3.

    Experiment: Experiment-based studies.

  4. 4.

    Development of prioritization method: Studies where new prioritization models are developed.

  5. 5.

    Personal experience/lessons learned: Studies where the reference is one’s own personal experience.

  6. 6.

    Review: Studies that review other papers.

1.2 Category 2: Study Context

  1. 1.

    Datasets from student projects: Studies where the subjects are students or student projects. For instance, academic toy examples belong to this category.

  2. 2.

    Datasets from industrial projects: Studies where the subjects are software professionals and/or industrial software projects. They are developed by professional programmers.

  3. 3.

    Not relevant: Studies where the study context is not available or not relevant.

1.3 Category 3: Dataset Type

  1. 1.

    Public: Public datasets are stored in public repositories such as a SIR (Software-artifacts Infrastructure Repository). These datasets are publicly available.

  2. 2.

    Private: Private datasets mostly belong to software companies and are not freely distributed as public datasets.

  3. 3.

    Partial: Partial datasets are datasets that have been created using data from open source projects and have not been distributed to the community.

  4. 4.

    None: If there is no information about the dataset in the paper, it is called “None.”

1.4 Category 4: prioritization approach Yoo and Harman’s (2012) classification

  1. 1.

    Coverage-based: These techniques order test cases based on the coverage of code components. There are different coverage-based techniques. For example, one of them prioritizes test cases in order of coverage of branches, or it prioritizes test cases in order of coverage of statements not yet covered.

  2. 2.

    Distribution-based: These techniques use a multidimensional distribution of test cases based on their multivariable descriptions.

  3. 3.

    Human-based: Prioritization is based on the comparisons made by the human tester.

  4. 4.

    Probabilistic Approach: Probabilistic theory is applied.

  5. 5.

    History-based: Historical information such as faults or execution history is used in this type of technique.

  6. 6.

    Requirements-based: Requirements are taken into account in this type of technique.

  7. 7.

    Model-based: Instead of using code blocks, model-based prioritization techniques utilize different models such as sequence diagram or state charts.

  8. 8.

    Cost-aware approach: The cost of each test case is not equal, and therefore, cost-based techniques provide effective solutions when the cost factor is significant.

  9. 9.

    Other Approaches: The other types of prioritization methods are located in this category.

  10. 10.

    None: Prioritization approach is not used.

1.5 Category 5: Research Topic

  1. 1.

    Prioritization method: A new test case prioritization approach is proposed and evaluated in this type of paper.

  2. 2.

    Comparison of prioritization methods: Several prioritization methods are compared in this type of paper.

  3. 3.

    Measures of prioritization performance: A new evaluation metric for TCP is proposed and evaluated in this type of paper.

  4. 4.

    Others: Change effect identification, combination of prioritization and reduction methods, tool development, use of prioritization method for test case selection, use of prioritization method for software fault localization, time constraint effects, selecting the most effective prioritization method, test suite generation, test plan generation, survey on prioritization methods, review of prioritization methods, residual defects identification, test adequacy criterion, measuring the sources of variation in the prioritization, and effects of test suite granularity.

1.6 Category 6: Evaluation Metric

  1. 1.

    APFD (Average Percentage of Faults Detected): “APFD measures the weighted average of the percentage of faults detected over the life of a test suite” Kapfhammer and Soffa (2007). APFD values are between 0 and 100, with higher values indicating better fault detection.

  2. 2.

    ASFD (Average Severity of Faults Detected): A severity value is assigned to each fault. Total Severity of Faults Detected (TSFD) is the sum of severity values of the faults identified. “The ASFDi for requirement i is the ratio of the sum of severity faults identified for the requirement I, to the TSFD” Srikanth and Williams (2005).

  3. 3.

    TPFD (Total Percentage of Faults Detected): “TPFD metric is the area under the curve when plotting a graph with the fraction of requirement on X-axis, and percentage of TSFD on Y-axis” Krishnamoorthi et al. (2009).

  4. 4.

    APFD(c) (Average Percentage of Faults Detected per Cost): APFD metric assumes that faults have equal severity and test cases have equal costs Ma and Zhao (2008). However, this is not possible in practice and therefore, an APFD(c) metric that takes into account the fault severity and test cost was proposed.

  5. 5.

    NAPFD (Normalized APFD): This metric measures test case prioritization and configuration prioritization. It includes information on fault finding and the time of detection Qu et al. (2007).

  6. 6.

    RP (Most Likely Relative Position): “It represents an average relative position of the first failed test that detects d for a test case prioritization method” Korel and Koutsogiannakis (2009).

  7. 7.

    CE (Coverage Effectiveness): This metric incorporates the cost and the coverage of each test case. CE value is between 0 and 1 with a higher value indicating a better test case. It is the ratio between the reordered suite’s coverage area and the coverage area of the ideal test suite that covers all requirements Kapfhammer and Soffa (2007).

  8. 8.

    None: Evaluation metric was not used.

  9. 9.

    Others: The other types of evaluation metrics were used.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Catal, C., Mishra, D. Test case prioritization: a systematic mapping study. Software Qual J 21, 445–478 (2013). https://doi.org/10.1007/s11219-012-9181-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-012-9181-z

Keywords

Search

Navigation