Jeongho Kim
ABSTRACT
For regression testing in the continuous integration environments, the time and cost should be considered; to satisfy these constraints, it is necessary to improve the test efficiency regarding the achievement of the test goal. It is especially important to identify the problem quickly by first executing a test case with a high probability of failure. This paper therefore proposes the FHD (<u>F</u>ailure <u>H</u>istory <u>D</u>ata)-Prioritization technique for the purpose of effective regression testing. This technique uses the failed test case history, the flipped result of method test case and the correlation data as the prioritization criteria, and the algorithm is designed to calculate the weight through the following two-step classification: 1) The FHD-Prioritization analyzes the failure history data statistically and sequentially arranges the test cases from the highest failure occurrence probability in the current session. 2) If a failure occurs during the test, the FHD-Prioritization reprioritizes in real time based on the correlation data. The performance of the FHD-Prioritization technique is evaluated with Tomcat and Camel, Apache open source software projects that were developed in the continuous integration environment. Because all of these projects are composed of the real faults of real-world projects, it is possible to practically evaluate the efficiency of the proposed approach. The FHD-Prioritization improved the efficiency of test case prioritization by about 5.62+ and 2.17+, respectively, compared to ROCKET and AFSAC.
- Wong, W. E., Horgan, J. R., London, S., and Agrawal, H. A study of effective regression testing in practice. In Proceedings of International Symposium Software Reliability Engineering, 1997, 264--274. Google Scholar
Digital Library
- Yoo, S., and Harman, M. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability, 2012, 22(2), 67--120. Google ScholarCross Ref
- Wong, W. E., Horgan, J. R., Mathur, A. P., and Pasquini, A. Test set size minimization and fault detection effectiveness: A case study in a space application. In Proceedings of Computer Software and Applications Conference, 1997, 522--528. Google ScholarDigital Library
- Rothermel, G. and Harrold, M. J. A safe, efficient regression test selection technique. IEEE Transactions on Software Engineering and Methodology, 1997, 6(2), 173--210. Google ScholarDigital Library
- Rothermel, G., Untch, R. H., Chu, C. and Harrold, M. J., Prioritizing test cases for regression testing. IEEE Transactions on software engineering, 2001, 2, 7(10), 929--948. Google ScholarDigital Library
- Kim, J. M., and Porter, A. A history-based test prioritization technique for regression testing in resource constrained environments. In Proceedings of International Conference on Software Engineering, 2002, 119--129. Google ScholarDigital Library
- Aggrawal, K. K., Singh, Y., and Kaur, A. Code coverage based technique for prioritizing test cases for regression testing. ACM SIGSOFT Software Engineering Notes, 2004, 29(5), 1--4. Google ScholarDigital Library
- Korel, B., Tahat, L. H, and Harman, M. Test prioritization using system models. In Proceedings of International Conference on Software Maintenance, 2005, 559--568. Google ScholarDigital Library
- Arafeen, M. J., and Do, H. Test case prioritization using requirements-based clustering. In Proceedings of International Conference on Software Testing, Verification and Validation, 2013, 312--321. Google ScholarDigital Library
- Di Nardo, D., Alshahwan, N., Briand, L., and Labiche, Y. Coverage-based test case prioritisation: An industrial case study. In Proceedings of International Conference on Software Testing, Verification and Validation, 2013, 302--311. Google ScholarDigital Library
- Saha, R. K., Zhang, L., Khurshid, S., and Perry, D. E. An information retrieval approach for regression test prioritization based on program changes. In Proceedings of International Conference on Software Engineering-Volume 1, 2015, 268--279. Google ScholarDigital Library
- Srikanth, H., and Williams, L. On the economics of requirements-based test case prioritization. In ACM SIGSOFT Software Engineering Notes, 2005, Vol. 30, No. 4, 1--3. Google ScholarDigital Library
- Cho, Y., Kim, J., and Lee, E. History-based Test Case Prioritization for Failure Information. In Proceedings of Asia-Pacific Software Engineering Conference. 2016 (Poster session in press) Google ScholarCross Ref
- Marijan, D., Gotlieb, A., and Sen, S. Test case prioritization for continuous regression testing: An industrial case study. In Proceedings of International Conference on Software Maintenance, 2013, 540--543. Google ScholarDigital Library
- Carlson, R., Do, H., and Denton, A. A clustering approach to improving test case prioritization: An industrial case study. In Proceedings of International Conference on Software Maintenance, 2011, 382--391. Google ScholarDigital Library
- Apache Software Foundation Projects Directory, URL: https://projects.apache.orgGoogle Scholar
- Elbaum, S., Rothermel, G., and Penix, J. Techniques for improving regression testing in continuous integration development environments. In Proceedings of the International Symposium on Foundations of Software Engineering, 2014, 235--245. Google ScholarDigital Library
- Elbaum, S., Malishevsky, A. G., and Rothermel, G. Test case prioritization: A family of empirical studies. IEEE transactions on software engineering, 2002, 28(2), 159--182. Google ScholarDigital Library
Index Terms
- Failure history data-based test case prioritization for effective regression test
Recommendations
History-based dynamic test case prioritization for requirement properties in regression testing
CSED '16: Proceedings of the International Workshop on Continuous Software Evolution and DeliveryRegression testing is an important but extremely costly and time-consuming process. Because of limited resources in practice, test case prioritization focuses on the improvement of testing efficiency. However, traditional test case prioritization ...
Studying Test Case Failure Prediction for Test Case Prioritization
PROMISE: Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software EngineeringBackground: Test case prioritization refers to the process of ranking test cases within a test suite for execution. The goal is ranking fault revealing test cases higher so that in case of limited budget one only executes the top ranked tests and still ...
Test Case Prioritization for Continuous Regression Testing: An Industrial Case Study
ICSM '13: Proceedings of the 2013 IEEE International Conference on Software MaintenanceRegression testing in continuous integration environment is bounded by tight time constraints. To satisfy time constraints and achieve testing goals, test cases must be efficiently ordered in execution. Prioritization techniques are commonly used to ...
Comments