skip to main content
10.1145/3530019.3530023acmotherconferencesArticle/Chapter ViewAbstractPublication PageseaseConference Proceedingsconference-collections
research-article

Black-box Test Case Selection by Relating Code Changes with Previously Fixed Defects

Published: 13 June 2022 Publication History

Abstract

Software continuously changes to address new requirements and to fix defects. Regression testing is performed to ensure that the applied changes do not adversely affect existing functionality. The increasing number of test cases makes it infeasible to execute the whole regression test suite. Test case selection is adopted to select a subset of the test suite, which is associated with the changed parts of the software. These parts are assumed to be error-prone. We present and evaluate a test case selection approach in the context of black-box regression testing of embedded systems. In this context, it is challenging to relate test cases with a set of distinct source code elements to be able to select those test cases associated with the modified parts of the source code. We analyze previously fixed defects for this purpose. We relate test cases that detect these defects with the source files that are previously modified for fixing them. Then, we select test cases related with source code files that are modified in the subsequent revision. The strength of this relation is determined as the number of changes associated with fixed defects previously detected by the same test cases. We conduct a case study on 3 real projects from the consumer electronics domain. Results show that it is possible to detect from 65% up to 85% of the defects detected by the whole test suite by selecting from 30% up to 70% of the test cases.

References

[1]
Y. Chen, D. Rosenblum, and K. Vo. 1994. TestTube: A system for selective regression testing. In Proceedings of the 16th International Conference on Software Engineering. 211–220.
[2]
S. Dirim and H. Sozer. 2020. Prioritization of Test Cases with Varying Test Costs and Fault Severities for Certification Testing. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation Workshops. 386–391.
[3]
E.D. Ekelund and E. Engström. 2015. Efficient regression testing based on test history: An industrial evaluation. In 2015 IEEE International Conference on Software Maintenance and Evolution. 449–457.
[4]
S. Elbaum, P. Kallakuri, A. Malishevsky, G. Rothermel, and S. Kanduri. 2003. Understanding the effects of changes on the cost-effectiveness of regression testing techniques. Software testing, verification and reliability 13, 2(2003), 65–83.
[5]
E. Engström, P. Runeson, and M. Skoglund. 2010a. A systematic review on regression test selection techniques. Information and Software Technology 52, 1 (2010), 14–30.
[6]
E. Engström, P. Runeson, and G. Wikstrand. 2010b. An Empirical Evaluation of Regression Testing Based on Fix-Cache Recommendations. In Proceedings of the International Conference on Software Testing, Verification and Validation. 75–78.
[7]
K. Fischer, F. Raji, and A. Chruscicki. 1981. A methodology for retesting modified software. In Proceedings of the National Telecommunications Conference. 1–6.
[8]
C.S. Gebizli, A. Kirkici, and H. Sozer. 2018. Increasing test efficiency by risk-driven model-based testing. Journal of Systems and Software 144 (2018), 356–365.
[9]
C.S. Gebizli, D. Metin, and H. Sozer. 2015. Combining model-based and risk-based testing for effective test case generation. In Proceedings of the 8th International Conference on Software Testing, Verification and Validation Workshops. 1–4.
[10]
T.L. Graves, M.J. Harrold, J.M. Kim, A. Porter, and G. Rothermel. 2001. An empirical study of regression test selection techniques. ACM Transactions on Software Engineering and Methodology 10, 2(2001), 184–208.
[11]
R. Gupta, M. J. Harrold, and M. Soffa. 1992. An approach to regression testing using slicing. In Proceedings of the Conference on Software Maintenance. 299–308.
[12]
K. Herzig, M. Greiler, J. Czerwonka, and B. Murphy. 2015. The Art of Testing Less Without Sacrificing Quality. In Proceedings of the 37th International Conference on Software Engineering. 483–493.
[13]
R. Kazmi, D.N.A. Jawawi, R. Mohamad, and I. Ghani. 2017. Effective Regression Test Case Selection: A Systematic Literature Review. Comput. Surveys 50, 2 (2017), 1–32.
[14]
S. Kim, T. Zimmermann, E.J. Jr. Whitehead, and A. Zeller.2007. Predicting faults from cached history. In Proceedings of the 29th International Conference on Software Engineering. 489–498.
[15]
X. Qu, M. Acharya, and B. Robinson. 2011. Impact analysis of configuration changes for test case selection. In 2011 IEEE 22nd International Symposium on Software Reliability Engineering. 140–149.
[16]
E. Rogstad, L. Briand, and R. Torkar. 2013. Test case selection for black-box regression testing of database applications. Information and Software Technology 55, 10 (2013), 1781–1795.
[17]
G. Rothermel and M.J. Harrold. 1997. A safe, efficient regression test selection technique. ACM Transactions on Software Engineering and Methodology 6, 2(1997), 173–210.
[18]
P.G. Sapna and H. Mohanty. 2010. Clustering test cases to achieve effective test selection. In Proceedings of the 1st Amrita ACM-W Celebration on Women in Computing in India. 1–8.
[19]
G. Wikstrand, R. Feldt, J.K. Gorantla, W. Zhe, and C. White. 2009. Dynamic regression test selection based on a file cache an industrial evaluation. In Proceedings of the International Conference on Software Testing Verification and Validation. 299–302.
[20]
S. Yoo and M. Harman. 2012. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability 22, 2(2012), 67–120.

Cited By

View all
  • (2025)Test Prioritization Based on the Coverage of Recently Modified Source Code: An Industrial Case StudyTesting Software and Systems10.1007/978-3-031-80889-0_4(46-59)Online publication date: 25-Jan-2025
  • (2024)A Multi-Granularity and Multi-Dimensional Code Knowledge Graph Construction Approach2024 6th International Conference on Communications, Information System and Computer Engineering (CISCE)10.1109/CISCE62493.2024.10653047(754-761)Online publication date: 10-May-2024
  • (2023)State of Practical Applicability of Regression Testing Research: A Live Systematic Literature ReviewACM Computing Surveys10.1145/357985155:13s(1-36)Online publication date: 13-Jul-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
EASE '22: Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering
June 2022
466 pages
ISBN:9781450396134
DOI:10.1145/3530019
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. black-box testing
  2. industrial case study
  3. regression testing
  4. repository mining
  5. test case selection

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

EASE 2022

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)20
  • Downloads (Last 6 weeks)0
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Test Prioritization Based on the Coverage of Recently Modified Source Code: An Industrial Case StudyTesting Software and Systems10.1007/978-3-031-80889-0_4(46-59)Online publication date: 25-Jan-2025
  • (2024)A Multi-Granularity and Multi-Dimensional Code Knowledge Graph Construction Approach2024 6th International Conference on Communications, Information System and Computer Engineering (CISCE)10.1109/CISCE62493.2024.10653047(754-761)Online publication date: 10-May-2024
  • (2023)State of Practical Applicability of Regression Testing Research: A Live Systematic Literature ReviewACM Computing Surveys10.1145/357985155:13s(1-36)Online publication date: 13-Jul-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media