Willian D. F. Mendonça
ABSTRACT
Ensuring the quality of Highly Configurable Systems (HCSs) during its evolution and maintenance is challenging. As an HCS evolves, new features are added, changed, or removed, which makes the test case selection for regression testing a difficult task. The use of test traceability can help in this task, but there is a lack of studies exploring the use of trace links for HCS testing. Existing work is usually based on the variability model, which is not always available or updated. Yet, the few existing approaches rely on links between test cases and files/lines of code, limiting the selection to test cases related to file changes, not considering the whole implementation of features, which can be spread in many files other than the changed ones. Considering this limitation, this work presents a test case selection approach, namely FeaTestSel, that links test cases to features using HCS pre-processor directives. Then, the selection of test cases is based on features affected by changes in each commit. In addition to the selected test cases, the approach also produces the following reports to support the test activity: the lines of code that correspond to each feature, the lines exercised by each test case, and the test cases linked to each feature. To validate the approach, we rely on Libssh, a real open-source HCS in constant evolution. By adding the execution time of the approach to the execution time of the selected test cases, we achieved a reduction of approximately ≈50%, in comparison with the retest-all technique. Furthermore, the approach was able to maintain quality by selecting 100% of failed test files. The traceability and reports produced by our approach can also be used for further work by researchers, analysis of the test quality by engineers, or as a source of information for tool builders.
- Mustafa Al-Hajjaji, Sascha Lity, Remo Lachmann, Thomas Thüm, Ina Schaefer, and Gunter Saake. 2017. Delta-oriented product prioritization for similarity-based product-line testing. In 2nd Intl. Workshop on Variability and Complexity in Software Design (VACE). IEEE, 34--40.Google Scholar
Cross Ref
- David Benavides, Sergio Segura, Pablo Trinidad, and Antonio Ruiz-Cortés. 2006. Using Java CSP solvers in the automated analyses of feature models. Intl. Summer School Generative and Transformational Techniques in Software Engineering (2006), 399--408.Google Scholar
- Antonia Bertolino, Antonio Guerriero, Breno Miranda, Roberto Pietrantuono, and Stefano Russo. 2020. Learning-to-rank vs ranking-to-learn: strategies for regression testing in continuous integration. In ACM/IEEE 42nd Intl. Conference on Software Engineering. 1--12.Google ScholarDigital Library
- Ivan do C. Machado, John D McGregor, Yguaratã Cerqueira Cavalcanti, and Eduardo S. De Almeida. 2014. On strategies for testing software product lines: A systematic literature review. Information and Software Technology 56, 10 (2014), 1183--1199.Google ScholarDigital Library
- Emelie Engström. 2010. Regression Test Selection and Product Line System Testing. In 3rd Intl. Conference on Software Testing, Verification and Validation. IEEE, 512--515.Google Scholar
- Fischer Ferreira, João P Diniz, Cleiton Silva, and Eduardo Figueiredo. 2019. Testing tools for configurable software systems: A review-based empirical study. In 13th Intl. Workshop on Variability Modelling of Software-Intensive Systems. 1--10.Google ScholarDigital Library
- Angelo Gargantini, Justyna Petke, Marco Radavelli, and Paolo Vavassori. 2016. Validation of Constraints Among Configuration Parameters Using Search-Based Combinatorial Interaction Testing. In 8th Intl. Symposium on Search Based Software Engineering. Springer, Cham, 49--63.Google Scholar
- Vahid Garousi and Junji Zhi. 2013. A survey of software testing practices in Canada. Journal of Systems and Software 86, 5 (2013), 1354--1376.Google ScholarDigital Library
- Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. 2015. Practical regression test selection with dynamic file dependencies. In Intl. Symposium on Software Testing and Analysis. 211--222.Google ScholarDigital Library
- Pilsu Jung, Sungwon Kang, and Jihyun Lee. 2019. Automated code-based test selection for software product line regression testing. Journal of Systems and Software 158 (2019), 110419.Google ScholarDigital Library
- Pilsu Jung, Sungwon Kang, and Jihyun Lee. 2020. Efficient regression testing of software product lines by reducing redundant test executions. Applied Sciences 10, 23 (2020), 8686.Google ScholarCross Ref
- C. H. Peter Kim, Don S Batory, and Sarfraz Khurshid. 2011. Reducing combinatorics in testing product lines. In 10th Intl. conference on Aspect-oriented software development. 57--68.Google Scholar
- C. H. Peter Kim, Sarfraz Khurshid, and Don Batory. 2012. Shared execution for efficiently testing product lines. In IEEE 23rd Intl. Symposium on Software Reliability Engineering. IEEE, 221--230.Google Scholar
- Satendra Kumar and Rajkumar. 2016. Test case prioritization techniques for software product line: A survey. In Intl. Conference on Computing, Communication and Automation (ICCCA). IEEE, 884--889.Google ScholarCross Ref
- Remo Lachmann, Simon Beddig, Sascha Lity, Sandro Schulze, and Ina Schaefer. 2017. Risk-based integration testing of software product lines. In 11th Intl. Workshop on Variability Modelling of Software-intensive Systems. 52--59.Google ScholarDigital Library
- Remo Lachmann, Sascha Lity, Sabrina Lischke, Simon Beddig, Sandro Schulze, and Ina Schaefer. 2015. Delta-oriented test case prioritization for integration testing of software product lines. In 19th Intl. Conference on Software Product Line. 81--90.Google ScholarDigital Library
- Jackson A Prado Lima, Willian DF Mendonça, Silvia R Vergilio, and Wesley KG Assunção. 2020. Learning-based prioritization of test cases in continuous integration of highly-configurable software. In 24th ACM conference on systems and software product line. 1--11.Google Scholar
- Jackson A Prado Lima and Silvia R Vergilio. 2020. Test Case Prioritization in Continuous Integration environments: A systematic mapping study. Information and Software Technology 121 (2020), 106268.Google ScholarDigital Library
- Sascha Lity, Manuel Nieke, Thomas Thüm, and Ina Schaefer. 2019. Retest test selection for product-line regression testing of variants and versions of variants. Journal of Systems and Software 147 (2019), 46--63.Google ScholarCross Ref
- Kai Ludwig, Jacob Krüger, and Thomas Leich. 2019. Covert and phantom features in annotations: Do they impact variability analysis?. In 23rd Intl. Systems and Software Product Line Conference-Volume A. 218--230.Google ScholarDigital Library
- Dusica Marijan, Arnaud Gotlieb, and Marius Liaaen. 2019. A learning algorithm for optimizing continuous integration development and testing practice. Software: Practice and Experience 49, 2 (2019), 192--213.Google ScholarCross Ref
- Dusica Marijan and Marius Liaaen. 2018. Practical selective regression testing with effective redundancy in interleaved tests. In 40th Intl. Conference on Software Engineering: Software Engineering in Practice. 153--162.Google ScholarDigital Library
- Dusica Marijan, Marius Liaaen, Arnaud Gotlieb, Sagar Sen, and Carlo Ieva. 2017. TITAN: Test Suite Optimization for Highly Configurable Software. In IEEE Intl. Conference on Software Testing, Verification and Validation (ICST). IEEE, 524--531.Google Scholar
- Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A Comparison of 10 Sampling Algorithms for Configurable Systems. In 38th Intl. Conference on Software Engineering (ICSE). ACM, 643--54.Google ScholarDigital Library
- Flávio Medeiros, Márcio Ribeiro, Rohit Gheyi, Sven Apel, Christian Kästner, Bruno Ferreira, Luiz Carvalho, and Baldoino Fonseca. 2018. Discipline Matters: Refactoring of Preprocessor Directives in the #ifdef Hell. IEEE Transactions on Software Engineering 44, 5 (2018), 453--469.Google ScholarCross Ref
- Jens Meinicke, Chu-Pan Wong, Christian Kästner, Thomas Thüm, and Gunter Saake. 2016. On essential configuration complexity: Measuring interactions in highly-configurable systems. In 31st IEEE/ACM Intl. Conference on Automated Software Engineering. 483--494.Google ScholarDigital Library
- Willian DF Mendonça, Wesley KG Assunção, and Silvia R Vergilio. 2022. Software Product Line Regression Testing: A Research Roadmap. In Intl. Conference on Enterprise Information Systems (ICEIS). SciTePress, 81--89.Google Scholar
- Willian DF Mendonça, Silvia R Vergilio, Gabriela K Michelon, Alexander Egyed, and Wesley KG Assunção. 2022. Test2Feature: feature-based test traceability tool for highly configurable software. In 26th ACM Intl. Systems and Software Product Line Conference-Volume B. 62--65.Google ScholarDigital Library
- Willian Mendonça, Silvia R Vergilio, and Wesley K G Assunçao. 2023. Supplementary Material - Feature-oriented Test Case Selection during Evolution of High-Configurable Systems. https://doi.org/10.17605/OSF.IO/YD2WFGoogle Scholar
- Gabriela K Michelon, Wesley KG Assunção, David Obermann, Lukas Linsbauer, Paul Grünbacher, and Alexander Egyed. 2021. The life cycle of features in highly-configurable software systems evolving in space and time. In 20th ACM SIGPLAN Intl. Conference on Generative Programming: Concepts and Experiences. 2--15.Google ScholarDigital Library
- Gabriela K. Michelon, David Obermann, Wesley KG Assunção, Lukas Linsbauer, Paul Grünbacher, Stefan Fischer, Roberto E Lopez-Herrejon, and Alexander Egyed. 2022. Evolving software system families in space and time with feature revisions. Empirical Software Engineering 27, 5 (2022), 112.Google ScholarDigital Library
- Gabriela K. Michelon, David Obermann, Wesley K. Guez Assunção, Lukas Linsbauer, Paul Grünbacher, and Alexander Egyed. 2020. Mining feature revisions in highly-configurable software systems. In 24th ACM Intl. Systems and Software Product Line Conference-Volume B. 74--78.Google ScholarDigital Library
- Mukelabai Mukelabai, Damir Nešić, Salome Maro, Thorsten Berger, and Jan-Philipp Steghöfer. 2018. Tackling combinatorial explosion: a study of industrial needs and practices for analyzing highly configurable systems. In 33rd ACM/IEEE Intl. Conference on Automated Software Engineering. 155--166.Google ScholarDigital Library
- Raiza Oliveira, Bruno Cafeo, and Andre Hora. 2019. On the Evolution of Feature Dependencies: An Exploratory Study of Preprocessor-Based Systems. In 13th Intl. Workshop on Variability Modelling of Software-Intensive Systems. ACM, 1--9.Google ScholarDigital Library
- Simone Romano, Giuseppe Scanniello, Giuliano Antoniol, and Alessandro Marchetto. 2018. SPIRITuS: A simple information retrieval regression test selection approach. Information and Software Technology 99 (2018), 62--80.Google ScholarDigital Library
- Per Runeson and Emelie Engström. 2012. Chapter 7 - Regression Testing in Software Product Line Engineering. In Advances in Computers, Ali Hurson and Atif Memon (Eds.). Vol. 86. Elsevier, 223--263.Google Scholar
- T. Schiex and S. de Givry. 2019. Principles and Practice of Constraint Programming. In 25th Intl. Conference on Principles and Practice of Constraint Programming, Vol. 11802. Springer.Google ScholarDigital Library
- Paulo Anselmo da Mota Silveira Neto, Ivan do C. Machado, Yguarata Cerqueira Cavalcanti, Eduardo Santana De Almeida, Vinicius Cardoso Garcia, and Silvio Romero de Lemos Meira. 2010. A regression testing approach for software product lines architectures. In 2010 Fourth Brazilian Symposium on Software Components, Architectures and Reuse. IEEE, 41--50.Google ScholarDigital Library
- Davide Spadini, Maurício Aniche, and Alberto Bacchelli. 2018. PyDriller: Python framework for mining software repositories. In 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 908--911.Google ScholarDigital Library
- H. Tufail, M. F. Masood, B. Zeb, F. Azam, and M. W. Anwar. 2017. A systematic review of requirement traceability techniques and tools. In 2nd Intl. Conference on System Reliability and Science. 450--454.Google Scholar
- Tugkan Tuglular and Sercan Şensülün. 2019. SPL-AT Gherkin: A Gherkin Extension for Feature Oriented Testing of Software Product Lines. In 43rd Annual Computer Software and Applications Conference, Vol. 2. IEEE, 344--349.Google Scholar
- Alexander Von Rhein, Alexander Grebhahn, Sven Apel, Norbert Siegmund, Dirk Beyer, and Thorsten Berger. 2015. Presence-condition simplification in highly configurable systems. In IEEE/ACM 37th IEEE Intl. Conference on Software Engineering, Vol. 1. IEEE, 178--188.Google ScholarCross Ref
- Shuai Wang, Shaukat Ali, Arnaud Gotlieb, and Marius Liaaen. 2017. Automated product line test case selection: industrial case study and controlled experiment. Software & Systems Modeling 16 (2017), 417--441.Google ScholarDigital Library
- Shuai Wang, Arnaud Gotlieb, Shaukat Ali, and Marius Liaaen. 2013. Automated Test Case Selection Using Feature Model: An Industrial Case Study. In Model-Driven Engineering Languages and Systems, Ana Moreira, Bernhard Schätz, Jeff Gray, Antonio Vallecillo, and Peter Clarke (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 237--253.Google Scholar
- Shin Yoo and Mark Harman. 2012. Regression testing minimization, selection and prioritization: a survey. Software testing, verification and reliability 22, 2 (2012), 67--120.Google ScholarDigital Library
- Yangyang Zhao, Alexander Serebrenik, Yuming Zhou, Vladimir Filkov, and Bogdan Vasilescu. 2017. The impact of continuous integration on other software development practices: a large-scale empirical study. In 32nd IEEE/ACM Intl. Conference on Automated Software Engineering (ASE). IEEE, 60--71.Google ScholarDigital Library
Index Terms
- Feature-oriented Test Case Selection during Evolution of Highly-Configurable Systems
Recommendations
Combined Source Code Approach for Test Case Prioritization
ICISS '18: Proceedings of the 1st International Conference on Information Science and SystemsRegression testing is an activity in the software testing process to ensure the software is validated and verified after modification occurred on software. It is costly process procedure which has been expected to reach half cost of the software ...
A Static Approach to Prioritizing JUnit Test Cases
Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most ...
Empirical Studies of a Safe Regression Test Selection Technique
Regression testing is an expensive testing procedure utilized to validate modified software. Regression test selection techniques attempt to reduce the cost of regression testing by selecting a subset of a program's existing test suite. Safe regression ...
Comments