ABSTRACT
Dealing with highly configurable systems is generally very complex. Hundreds of different analysis techniques have been conceived to deal with different aspects of configurable systems. One large focal point is the testing of configurable software. This is challenging due to the large number of possible configurations and because tests themselves are rarely configurable and instead built for specific configurations. Existing tests can usually not be reused on other configurations. Therefore, tests need to be adapted for the specific configuration they are supposed to test. In this paper we report on an experiment about reusing tests in a configurable system. We used manually developed tests for specific configurations of Bugzilla and investigated which of them could be reused for other configurations. Moreover, we automatically generated new test variants (by automatically reusing from existing ones) for combinations of previous configurations. Our results showed that we can directly reuse some tests for configurations which they were not intended for. Nonetheless, our automatically generated test variants generally yielded better results. When applying original tests to new configurations we found an average success rate for the tests of 81,84%. In contrast, our generated test variants achieved an average success rate of 98,72%. This is an increase of 16,88%.
- Thorsten Berger, Daniela Lettner, Julia Rubin, Paul Grünbacher, Adeline Silva, Martin Becker, Marsha Chechik, and Krzysztof Czarnecki. 2015. What is a feature?: a qualitative study of features in industrial software product lines. In Proceedings of the 19th International Conference on Software Product Line, SPLC 2015, Nashville, TN, USA, July 20--24, 2015, Douglas C. Schmidt (Ed.). ACM, 16--25. Google ScholarDigital Library
- Thorsten Berger, Steven She, Rafael Lotufo, Andrzej Wasowski, and Krzysztof Czarnecki. 2013. A Study of Variability Models and Languages in the Systems Software Domain. IEEE Trans. Software Eng. 39, 12 (2013), 1611--1640. Google ScholarDigital Library
- Myra B. Cohen, Joshua Snyder, and Gregg Rothermel. 2006. Testing across configurations: implications for combinatorial testing. ACM SIGSOFT Software Engineering Notes 31, 6 (2006), 1--9. Google ScholarDigital Library
- Paulo Anselmo da Mota Silveira Neto, Ivan do Carmo Machado, John D. McGregor, Eduardo Santana de Almeida, and Silvio Romero de Lemos Meira. 2011. A systematic mapping study of software product lines testing. Information & Software Technology 53, 5 (2011), 407--423. Google ScholarDigital Library
- Ivan do Carmo Machado, John D. McGregor, Yguaratã Cerqueira Cavalcanti, and Eduardo Santana de Almeida. 2014. On strategies for testing software product lines: A systematic literature review. Information & Software Technology 56, 10 (2014), 1183--1199. Google ScholarDigital Library
- Ivan do Carmo Machado, John D. McGregor, and Eduardo Santana de Almeida. 2012. Strategies for testing products in software product lines. ACM SIGSOFT Software Engineering Notes 37, 6 (2012), 1--8. Google ScholarDigital Library
- Yael Dubinsky, Julia Rubin, Thorsten Berger, Slawomir Duszynski, Martin Becker, and Krzysztof Czarnecki. 2013. An Exploratory Study of Cloning in Industrial Software Product Lines. In 17th European Conference on Software Maintenance and Reengineering, CSMR 2013, Genova, Italy, March 5--8, 2013, Anthony Cleve, Filippo Ricca, and Maura Cerioli (Eds.). IEEE Computer Society, 25--34. Google ScholarDigital Library
- Emelie Engström and Per Runeson. 2011. Software product line testing - A systematic mapping study. Information & Software Technology 53, 1 (2011), 2--13. Google ScholarDigital Library
- Stefan Fischer, Lukas Linsbauer, Roberto Erick Lopez-Herrejon, and Alexander Egyed. 2014. Enhancing Clone-and-Own with Systematic Reuse for Developing Software Variants. In 30th IEEE International Conference on Software Maintenance and Evolution, Victoria, BC, Canada, September 29 - October 3, 2014. IEEE Computer Society, 391--400. Google ScholarDigital Library
- Stefan Fischer, Lukas Linsbauer, Roberto E. Lopez-Herrejon, and Alexander Egyed. 2015. The ECCO Tool: Extraction and Composition for Clone-and-Own. In 37th IEEE/ACM International Conference on Software Engineering, ICSE 2015, Florence, Italy, May 16--24, 2015, Volume 2, Antonia Bertolino, Gerardo Canfora, and Sebastian G. Elbaum (Eds.). IEEE Computer Society, 665--668. Google ScholarDigital Library
- Charles W. Krueger. 2006. New methods in software product line practice. Commun. ACM 49, 12 (2006), 37--40. Google ScholarDigital Library
- Jacob Krüger, Mustafa Al-Hajjaji, Sandro Schulze, Gunter Saake, and Thomas Leich. 2018. Towards automated test refactoring for software product lines. In Proceeedings of the 22nd International Systems and Software Product Line Conference - Volume 1, SPLC 2018, Gothenburg, Sweden, September 10--14, 2018, Thorsten Berger, Paulo Borba, Goetz Botterweck, Tomi Männistö, David Benavides, Sarah Nadi, Timo Kehrer, Rick Rabiser, Christoph Elsner, and Mukelabai Mukelabai (Eds.). ACM, 143--148. Google ScholarDigital Library
- Roberto Erick Lopez-Herrejon, Stefan Fischer, Rudolf Ramler, and Alexander Egyed. 2015. A first systematic mapping study on combinatorial interaction testing for software product lines. In Eighth IEEE International Conference on Software Testing, Verification and Validation, ICST 2015 Workshops, Graz, Austria, April 13--17, 2015. IEEE Computer Society, 1--10.Google ScholarCross Ref
- Mukelabai Mukelabai, Damir Nesic, Salome Maro, Thorsten Berger, and Jan-Philipp Steghöfer. 2018. Tackling combinatorial explosion: a study of industrial needs and practices for analyzing highly configurable systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3--7, 2018, Marianne Huchard, Christian Kästner, and Gordon Fraser (Eds.). ACM, 155--166. Google ScholarDigital Library
- Rudolf Ramler and Werner Putschögl. 2013. Reusing Automated Regression Tests for Multiple Variants of a Software Product Line. In Sixth IEEE International Conference on Software Testing, Verification and Validation, ICST 2013 Workshops Proceedings, Luxembourg, Luxembourg, March 18--22, 2013. IEEE Computer Society, 122--123. Google ScholarDigital Library
- David J. Sheskin. 2007. Handbook of Parametric and Nonparametric Statistical Procedures (4 ed.). Chapman & Hall/CRC. Google ScholarDigital Library
- Mats Skoglund and Per Runeson. 2004. A case study on regression test suite maintenance in system evolution. In 20th IEEE International Conference on Software Maintenance, 2004. Proceedings, IEEE, 438--442. Google ScholarDigital Library
Index Terms
- Automating Test Reuse for Highly Configurable Software
Recommendations
Automated test reuse for highly configurable software
AbstractDealing with highly configurable systems is generally very complex. Researchers and practitioners have conceived hundreds of different analysis techniques to deal with different aspects of configurable systems. One large focal point is the testing ...
Testing of highly configurable cyber–physical systems — Results from a two-phase multiple case study
AbstractCyber–physical systems are commonly highly configurable. Testing such systems is particularly challenging because they comprise numerous heterogeneous components that can be configured and combined in different ways. Despite a plethora ...
Highlights- Survey of practitioners for configurable cyber–physical industry systems.
- ...
Localizing configurations in highly-configurable systems
SPLC '18: Proceedings of the 22nd International Systems and Software Product Line Conference - Volume 1The complexity of configurable systems has grown immensely, and it is only getting more complex. Such systems are a challenge for software testing and maintenance, because bugs and other defects can and do appear in any configuration. One common ...
Comments