research-article

YASA: yet another sampling algorithm

Authors:

Sebastian Krieter,

Sandro Schulze,

Thomas LeichAuthors Info & Claims

VaMoS '20: Proceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems

Article No.: 4, Pages 1 - 10

https://doi.org/10.1145/3377024.3377042

Published: 06 February 2020 Publication History

Abstract

Configurable systems allow users to derive customized software variants with behavior and functionalities tailored to individual needs. Developers of these configurable systems need to ensure that each configured software variant works as intended. Thus, software testing becomes highly relevant, but also highly expensive due to large configuration spaces that grow exponentially in the number of features. To this end, sampling techniques, such as t-wise interaction sampling, are used to generate a small yet representative subset of configurations, which can be tested even with a limited amount of resources. However, even state-of-the-art t-wise interaction sampling techniques do not scale well for systems with large configuration spaces. In this paper, we introduce the configurable technique YASA that aims to be more efficient than other existing techniques and enables control over trading-off sampling time and sample size. The general algorithm of YASA is based on the existing technique IPOG, but introduces several improvements and options to adapt the sampling procedure to a given configurable system. We evaluate our approach in terms of sampling time and sample size by comparing it to existing t-wise interaction sampling techniques. We find that YASA performs well even for large-scale system and is also able to produce smaller samples than existing techniques.

References

[1]

Iago Abal, Jean Melo, Stefan Stănciulescu, Claus Brabrand, Márcio Ribeiro, and Andrzej Wąsowski. Variability Bugs in Highly Configurable Systems: A Qualitative Analysis. Trans. Software Engineering and Methodology (TOSEM), 26(3):10:1--10:34, 2018.

Digital Library

[2]

Mustafa Al-Hajjaji, Sebastian Krieter, Thomas Thüm, Malte Lochau, and Gunter Saake. IncLing: Efficient Product-line Testing Using Incremental Pairwise Sampling. In Proc. Int'l Conf. on Generative Programming: Concepts & Experiences (GPCE), pages 144--155. ACM, 2016.

[3]

Mustafa Al-Hajjaji, Jens Meinicke, Sebastian Krieter, Reimar Schröter, Thomas Thüm, Thomas Leich, and Gunter Saake. Tool Demo: Testing Configurable Systems with FeatureIDE. In Proc. Int'l Conf. on Generative Programming: Concepts & Experiences (GPCE), pages 173--177. ACM, 2016.

[4]

Mustafa Al-Hajjaji, Thomas Thüm, Malte Lochau, Jens Meinicke, and Gunter Saake. Effective Product-Line Testing Using Similarity-Based Product Prioritization. Software and System Modeling (SoSyM), 2019. To appear.

Digital Library

[5]

Paul Ammann and Jeff Offutt. Introduction to software testing. Cambridge University Press, 2016.

[6]

Sven Apel, Don Batory, Christian Kästner, and Gunter Saake. Feature-Oriented Software Product Lines. Springer, 2013.

[7]

Vasek Chvatal. A Greedy Heuristic for the Set-Covering Problem. Mathematics of operations research, 4(3):233--235, 1979.

[8]

Paul Clements and Linda Northrop. Software Product Lines: Practices and Patterns. Addison-Wesley, 2001.

Digital Library

[9]

Anastasia Cmyrev and Ralf Reissing. Efficient and Effective Testing of Automotive Software Product Lines. Int'l J. Applied Science and Technology (IJAST), 7(2), 2014.

[10]

Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach. IEEE Trans. Software Engineering (TSE), 34(5):633--650, 2008.

Digital Library

[11]

Xavier Devroey, Gilles Perrouin, Axel Legay, Pierre-Yves Schobbens, and Patrick Heymans. Covering SPL Behaviour with Sampled Configurations: An Initial Assessment. In Proc. Int'l Workshop on Variability Modelling of Software-Intensive Systems (VaMoS), pages 59:59--59:66. ACM, 2015.

[12]

Emelie Engström and Per Runeson. Software Product Line Testing - A Systematic Mapping Study. J. Information and Software Technology (IST), 53:2--13, 2011.

[13]

Alireza Ensan, Ebrahim Bagheri, Mohsen Asadi, Dragan Gasevic, and Yevgen Biletskiy. Goal-Oriented Test Case Selection and Prioritization for Product Line Feature Models. In Proc. Inte'l Conf. on Information Technology: New Generations (ITNG), pages 291--298. IEEE, 2011.

[14]

Faezeh Ensan, Ebrahim Bagheri, and Dragan Gasevic. Evolutionary Search-Based Test Generation for Software Product Line Feature Models. In Proc. Int'l Conf. on Advanced Information Systems Engineering (CAiSE), volume 7328, pages 613--628. Springer, 2012.

[15]

Thiago N. Ferreira, Josiel Neumann Kuk, Aurora Pozo, and Silvia Regina Vergilio. Product Selection Based on Upper Confidence Bound MOEA/D-DRA for Testing Software Product Lines. In Proc. Congress Evolutionary Computation (CEC), pages 4135--4142. IEEE, 2016.

[16]

Thiago N. Ferreira, Jackson A. Prado Lima, Andrei Strickler, Josiel N. Kuk, Silvia R. Vergilio, and Aurora Pozo. Hyper-Heuristic Based Product Selection for Software Product Line Testing. Comp. Intell. Mag. (CIM), 12(2):34--45, 2017.

Digital Library

[17]

Helson L. Jakubovski Filho, Jackson A. Prado Lima, and Silvia R. Vergilio. Automatic Generation of Search-Based Algorithms Applied to the Feature Testing of Software Product Lines. In Proc. Brazilian Symposium on Software Engineering (SBES), pages 114--123. ACM, 2017.

Digital Library

[18]

Evelyn Nicole Haslinger, Roberto E. Lopez-Herrejon, and Alexander Egyed. Using Feature Model Knowledge to Speed Up the Generation of Covering Arrays. In Proc. Int'l Workshop on Variability Modelling of Software-Intensive Systems (VaMoS), pages 16:1--16:6. ACM, 2013.

Digital Library

[19]

Christopher Henard, Mike Papadakis, and Yves Le Traon. Mutation-Based Generation of Software Product Line Test Configurations. In Claire Le Goues and Shin Yoo, editors, Search-Based Software Engineering, volume 8636 of Lecture Notes in Computer Science, pages 92--106. Springer International Publishing, 2014.

[20]

Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. Multi-Objective Test Generation for Software Product Lines. In Proc. Int'l Systems and Software Product Line Conf. (SPLC), pages 62--71. ACM, 2013.

[21]

Arnaud Hubaux, Yingfei Xiong, and Krzysztof Czarnecki. A User Survey of Configuration Challenges in Linux and eCos. In Proc. Int'l Workshop on Variability Modelling of Software-Intensive Systems (VaMoS), pages 149--155. ACM, 2012.

Digital Library

[22]

Mikolas Janota. Do SAT Solvers Make Good Configurators? pages 191--195, 2008.

[23]

Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. Properties of Realistic Feature Models Make Combinatorial Testing of Product Lines Feasible. In Proc. Int'l Conf. on Model Driven Engineering Languages and Systems (MODELS), pages 638--652. Springer, 2011.

[24]

Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. An Algorithm for Generating T-Wise Covering Arrays from Large Feature Models. In Proc. Int'l Systems and Software Product Line Conf. (SPLC), pages 46--55. ACM, 2012.

Digital Library

[25]

Martin Fagereng Johansen, Øystein Haugen, Franck Fleurey, Anne Grete Eldegard, and Torbjørn Syversen. Generating Better Partial Covering Arrays by Modeling Weights on Sub-Product Lines. In Proc. Int'l Conf. on Model Driven Engineering Languages and Systems (MODELS), pages 269--284. Springer, 2012.

Digital Library

[26]

Christian Kästner, Sven Apel, Syed Saif ur Rahman, Marko Rosenmüller, Don Batory, and Gunter Saake. On the Impact of the Optional Feature Problem: Analysis and Case Studies. In Proc. Int'l Systems and Software Product Line Conf. (SPLC), pages 181--190. Software Engineering Institute, 2009.

[27]

Chang Hwan Peter Kim, Don Batory, and Sarfraz Khurshid. Reducing Combinatorics in Testing Product Lines. In Proc. Int'l Conf. on Aspect-Oriented Software Development (AOSD), pages 57--68. ACM, 2011.

[28]

Chang Hwan Peter Kim, Eric Bodden, Don Batory, and Sarfraz Khurshid. Reducing Configurations to Monitor in a Software Product Line. In Proc. Int'l Conf. on Runtime Verification (RV), pages 285--299. Springer, 2010.

[29]

Alexander Knüppel, Thomas Thüm, Stephan Mennicke, Jens Meinicke, and Ina Schaefer. Is There a Mismatch between Real-World Feature Models and Product-Line Research? In Matthias Tichy, Eric Bodden, Marco Kuhrmann, Stefan Wagner, and Jan-Philipp Steghöfer, editors, Proc. Software Engineering (SE), pages 53--54. Gesellschaft für Informatik, 2018.

[30]

Matthias Kowal, Sandro Schulze, and Ina Schaefer. Towards Efficient SPL Testing by Variant Reduction. In Proc. Int'l Workshop on Variability and Composition (VariComp), pages 1--6. ACM, 2013.

Digital Library

[31]

Daniel Le Berre and Anne Parrain. The sat4j library, release 2.2, system description. Journal on Satisfiability, Boolean Modeling and Computation, 7:59--64, 2010.

[32]

Jihyun Lee, Sungwon Kang, and Danhyung Lee. A Survey on Software Product Line Testing. In Proc. Int'l Systems and Software Product Line Conf. (SPLC), pages 31--40. ACM, 2012.

[33]

Yu Lei, Raghu N. Kacker, D. Richard Kuhn, Vadim Okun, and James Lawrence. IPOG: A General Strategy for T-Way Software Testing. In Proc. Int'l Conf. on Engineering of Computer-Based Systems (ECBS), pages 549--556. IEEE, 2007.

[34]

Jörg Liebig, Alexander von Rhein, Christian Kästner, Sven Apel, Jens Dörre, and Christian Lengauer. Scalable Analysis of Variable Software. In Proc. Europ. Software Engineering Conf./Foundations of Software Engineering (ESEC/FSE), pages 81--91. ACM, 2013.

[35]

Roberto Erick Lopez-Herrejon, Javier Ferrer, Francisco Chicano, Alexander Egyed, and Enrique Alba. Comparative Analysis of Classical Multi-Objective Evolutionary Algorithms and Seeding Strategies for Pairwise Testing of Software Product Lines. In Proc. Congress Evolutionary Computation (CEC), pages 387--396. IEEE, 2014.

[36]

Dusica Marijan, Arnaud Gotlieb, Sagar Sen, and Aymeric Hervieu. Practical Pairwise Testing for Software Product Lines. In Proc. Int'l Systems and Software Product Line Conf. (SPLC), pages 227--235. ACM, 2013.

[37]

Rui Angelo Matnei Filho and Silvia Regina Vergilio. A Multi-Objective Test Data Generation Approach for Mutation Testing of Feature Models. 4(1), 2016.

[38]

John McGregor. Testing a Software Product Line. In Testing Techniques in Software Engineering, pages 104--140. Springer, 2010.

[39]

Jens Meinicke, Thomas Thüm, Reimar Schröter, Fabian Benduhn, Thomas Leich, and Gunter Saake. Mastering Software Variability with FeatureIDE. Springer, 2017.

[40]

Marcilio Mendonca and Donald Cowan. Decision-Making Coordination and Efficient Reasoning Techniques for Feature-Based Configuration. Science of Computer Programming (SCP), 75(5):311--332, 2010.

[41]

Jeho Oh, Paul Gazzillo, Don Batory, Marijn Heule, and Maggie Myers. Uniform sampling from kconfig feature models. The University of Texas at Austin, Department of Computer Science, Tech. Rep. TR-19-02, 2019.

[42]

Sebastian Oster, Florian Markert, and Philipp Ritter. Automated Incremental Pairwise Testing of Software Product Lines. In Proc. Int'l Systems and Software Product Line Conf. (SPLC), pages 196--210. Springer, 2010.

[43]

Gilles Perrouin, Sagar Sen, Jacques Klein, Benoit Baudry, and Yves Le Traon. Automated and Scalable T-Wise Test Case Generation Strategies for Software Product Lines. In Proc. Int'l Conf. on Software Testing, Verification and Validation (ICST), pages 459--468. IEEE, 2010.

[44]

Tobias Pett, Thomas Thüm, Tobias Runge, Sebastian Krieter, Malte Lochau, and Ina Schaefer. Product sampling for product lines: The scalability challenge. 2019.

Digital Library

[45]

Klaus Pohl, Günter Böckle, and Frank J. van der Linden. Software Product Line Engineering: Foundations, Principles and Techniques. Springer, 2005.

[46]

Dennis Reuling, Johannes Bürdek, Serge Rotärmel, Malte Lochau, and Udo Kelter. Fault-Based Product-Line Testing: Effective Sample Generation Based on Feature-Diagram Mutation. In Proc. Int'l Systems and Software Product Line Conf. (SPLC), pages 131--140. ACM, 2015.

[47]

Steven She, Rafael Lotufo, Thorsten Berger, Andrzej Wąsowski, and Krzysztof Czarnecki. Reverse Engineering Feature Models. In Proc. Int'l Conf. on Software Engineering (ICSE), pages 461--470. ACM, 2011.

[48]

Jiangfan Shi, Myra B. Cohen, and Matthew B. Dwyer. Integration Testing of Software Product Lines Using Compositional Symbolic Execution. In Proc. Int'l Conf. on Fundamental Approaches to Software Engineering (FASE), pages 270--284. Springer, 2012.

Digital Library

[49]

Reinhard Tartler, Christian Dietrich, Julio Sincero, Wolfgang Schröder-Preikschat, and Daniel Lohmann. Static Analysis of Variability in System Software: The 90,000 #Ifdefs Issue. In Proc. USENIX Annual Technical Conference (ATC), pages 421--432. USENIX Association, 2014.

[50]

Thomas Thüm, Sven Apel, Christian Kästner, Ina Schaefer, and Gunter Saake. A Classification and Survey of Analysis Strategies for Software Product Lines. ACM Computing Surveys, 47(1):6:1--6:45, 2014.

Digital Library

Cited By

Ou YWang FFeng BTang LPan J(2024)Spindle Detection Based on Elastic Time Window and Spatial Pyramid PoolingJournal of Integrative Neuroscience10.31083/j.jin230713423:7Online publication date: 17-Jul-2024
https://doi.org/10.31083/j.jin2307134
Luo CLyu SZhao QWu WZhang HHu CChristakis MPradel M(2024)Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable SystemsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680309(641-653)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680309
Stummer AHager ARabiser R(2024)Towards a Flexible Approach for Variability MiningProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3676543(75-81)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3676543
Show More Cited By

Recommendations

t-wise Coverage by Uniform Sampling
SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A

Efficiently testing large configuration spaces of Software Product Lines (SPLs) needs a sampling algorithm that is both scalable and provides good t-wise coverage. The 2019 SPLC Sampling Challenge provides large real-world feature models and asks for a ...
Large-scale T-wise interaction sampling using YASA
SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume A

Testing highly-configurable software systems (i.e., software product lines) is challenging due to their large configuration space. T-wise sampling is one method of finding a representative subset of configurations for a system, which can then be tested. ...
Enabling Efficient Automated Configuration Generation and Management
SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume B

Creating and managing valid configurations is one of the main tasks in software product line engineering. Due to the often complex constraints from a feature model, some kind of automated configuration generation is required to facilitate the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

VaMoS '20: Proceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems

February 2020

184 pages

ISBN:9781450375016

DOI:10.1145/3377024

Editors:
Maxime Cordy
SnT/University of Luxembourg, Luxembourg
,
Mathieu Acher
University of Rennes 1, France
,
Danilo Beuche
pure-systems Gmbh
,
Gunter Saake
Otto-von-Guericke University Magdeburg, Germany

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 February 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Deutsche Forschungsgemeinschaft

Conference

VaMoS '20

VaMoS '20: 14th International Working Conference on Variability Modelling of Software-Intensive Systems

February 5 - 7, 2020

Magdeburg, Germany

Acceptance Rates

Overall Acceptance Rate 66 of 147 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

44
Total Citations
View Citations
377
Total Downloads

Downloads (Last 12 months)81
Downloads (Last 6 weeks)4

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ou YWang FFeng BTang LPan J(2024)Spindle Detection Based on Elastic Time Window and Spatial Pyramid PoolingJournal of Integrative Neuroscience10.31083/j.jin230713423:7Online publication date: 17-Jul-2024
https://doi.org/10.31083/j.jin2307134
Luo CLyu SZhao QWu WZhang HHu CChristakis MPradel M(2024)Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable SystemsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680309(641-653)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680309
Stummer AHager ARabiser R(2024)Towards a Flexible Approach for Variability MiningProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3676543(75-81)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3676543
Sundermann CBrancaccio VKuiter EKrieter SHeß TThüm T(2024)Collecting Feature Models from the Literature: A Comprehensive Dataset for BenchmarkingProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672590(54-65)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672590
Pett TKrieter SThüm TSchaefer I(2024)MulTi-Wise Sampling: Trading Uniform T-Wise Feature Interaction Coverage for Smaller SamplesProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672589(47-53)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672589
Heß TSchmidt TOstheimer LKrieter SThüm T(2024)UnWise: High T-Wise Coverage from Uniform SamplingProceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3634713.3634716(37-45)Online publication date: 7-Feb-2024
https://dl.acm.org/doi/10.1145/3634713.3634716
Böhm SKrieter SHeß TThüm TLochau M(2024)Incremental Identification of T-Wise Feature InteractionsProceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3634713.3634715(27-36)Online publication date: 7-Feb-2024
https://dl.acm.org/doi/10.1145/3634713.3634715
Agh HAzamnouri AWagner S(2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1007/s10664-024-10516-x
Zhao QLuo CCai SWu WLin JZhang HHu CChandra SBlincoe KTonella P(2023)CAmpactor: A Novel and Effective Local Search Algorithm for Optimizing Pairwise Covering ArraysProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616284(81-93)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616284
Luo CSong JZhao QLi YCai SHu C(2023)Generating Pairwise Covering Arrays for Highly Configurable Software SystemsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608998(261-267)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579027.3608998
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten