skip to main content
10.1145/3336294.3336322acmotherconferencesArticle/Chapter ViewAbstractPublication PagessplcConference Proceedingsconference-collections
short-paper

Product Sampling for Product Lines: The Scalability Challenge

Published: 09 September 2019 Publication History

Abstract

Quality assurance for product lines is often infeasible for each product separately. Instead, only a subset of all products (i.e., a sample) is considered during testing such that at least the coverage of certain feature interactions is guaranteed. While pair-wise interaction sampling only covers all interactions between two features, its generalization to t-wise interaction sampling ensures coverage for all interactions among t features. However, sampling large product lines poses a challenge, as today's algorithms tend to run out of memory, do not terminate, or produce samples, which are too large to be tested. To initiate a community effort, we provide a set of large real-world feature models with up-to 19 thousand features, which are supposed to be sampled. The performance of sampling approaches is evaluated based on the CPU time and memory consumed to retrieve a sample, the sample size for a given coverage (i.e. the value of t) and whether the sample achieves full t-wise coverage. A well-performing sampling algorithm achieves full t-wise coverage, while minimizing the other properties as best as possible.

References

[1]
Iago Abal, Jean Melo, Stefan Stănciulescu, Claus Brabrand, Márcio Ribeiro, and Andrzej Wąsowski. 2018. Variability Bugs in Highly Configurable Systems: A Qualitative Analysis. Trans. Software Engineering and Methodology (TOSEM) 26, 3, Article 10 (2018), 10:1--10:34 pages.
[2]
Bestoun S. Ahmed, Kamal Z. Zamli, Wasif Afzal, and Miroslav Bures. 2017. Constrained Interaction Testing: A Systematic Literature Study. IEEE Access 5 (2017), 25706--25730.
[3]
Sven Apel, Don Batory, Christian Kästner, and Gunter Saake. 2013. Feature-Oriented Software Product Lines. Springer.
[4]
Andrea Arcuri and Lionel Briand. 2011. Formal analysis of the probability of interaction fault detection using random testing. TSE 38, 5 (2011), 1088--1099.
[5]
Christian Dietrich, Reinhard Tartler, Wolfgang Schröder-Preikschat, and Daniel Lohmann. 2012. Understanding Linux Feature Distribution. In Proc. of Workshop on Modularity in Systems Software (MISS). ACM, 15--20.
[6]
Nicolas Dintzner, Arie van Deursen, and Martin Pinzger. 2018. FEVER: An Approach to Analyze Feature-Oriented Changes and Artefact Co-Evolution in Highly Configurable Systems. Empirical Software Engineering (EMSE) 23, 2 (2018), 905--952.
[7]
Alireza Ensan, Ebrahim Bagheri, Mohsen Asadi, Dragan Gasevic, and Yevgen Biletskiy. 2011. Goal-Oriented Test Case Selection and Prioritization for Product Line Feature Models. In Proc. Inte'l Conf. on Information Technology: New Generations (ITNG). IEEE, 291--298.
[8]
Mats Grindal, Jeff Offutt, and Sten F Andler. 2005. Combination testing strategies: a survey. STVR 15, 3 (2005), 167--199.
[9]
Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. 2013. Multi-Objective Test Generation for Software Product Lines. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM, 62--71.
[10]
Christian Kästner. 2017. Differential Testing for Variational Analyses: Experience from Developing KConfigReader. arXiv preprint arXiv:1706.09357 (2017).
[11]
Chang Hwan Peter Kim, Don Batory, and Sarfraz Khurshid. 2011. Reducing Combinatorics in Testing Product Lines. In Proc. Int'l Conf. on Aspect-Oriented Software Development (AOSD). ACM, 57--68.
[12]
Alexander Knüppel, Thomas Thüm, Stephan Mennicke, Jens Meinicke, and Ina Schaefer. 2017. Is There a Mismatch Between Real-World Feature Models and Product-Line Research?. In Proc. Europ. Software Engineering Conf./Foundations of Software Engineering (ESEC/FSE). ACM, 291--302.
[13]
Sebastian Krieter, Marcus Pinnecke, Jacob Krüger, Joshua Sprey, Christopher Sontag, Thomas Thüm, Thomas Leich, and Gunter Saake. 2017. FeatureIDE: Empowering Third-Party Developers. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM, 42--45.
[14]
Christian Kästner. 2018. KConfigReader. Website. Available online at https://github.com/ckaestne/kconfigreader; visited on August 29th, 2018.
[15]
Jörg Liebig, Alexander von Rhein, Christian Kästner, Sven Apel, Jens Dörre, and Christian Lengauer. 2013. Scalable Analysis of Variable Software. In Proc. Europ. Software Engineering Conf./Foundations of Software Engineering (ESEC/FSE). ACM, 81--91.
[16]
Roberto E. Lopez-Herrejon and Don Batory. 2001. A Standard Problem for Evaluating Product-Line Methodologies. In Proc. Int'l Symposium on Generative and Component-Based Software Engineering (GCSE). Springer, 10--24.
[17]
Roberto E. Lopez-Herrejon, Stefan Fischer, Rudolf Ramler, and Aalexander Egyed. 2015. A First Systematic Mapping Study on Combinatorial Interaction Testing for Software Product Lines. In Proc. Int'l Workshop on Combinatorial Testing (IWCT). IEEE, 1--10.
[18]
Jens Meinicke, Thomas Thüm, Reimar Schröter, Fabian Benduhn, Thomas Leich, and Gunter Saake. 2017. Mastering Software Variability with FeatureIDE. Springer.
[19]
Paulo Anselmo da Mota Silveira Neto, Ivan do Carmo Machado, Yguarata Cerqueira Cavalcanti, Eduardo Santana de Almeida, Vinicius Cardoso Garcia, and Silvio Romero de Lemos Meira. 2010. A regression testing approach for software product lines architectures. In Software Components, Architectures and Reuse (SBCARS), 2010 Fourth Brazilian Symposium on. IEEE, 41--50.
[20]
Michael Nieke, Jacopo Mauro, Christoph Seidl, Thomas Thüm, Ingrid Chieh Yu, and Felix Franzke. 2018. Anomaly Analyses for Feature-Model Evolution. In Proc. Int'l Conf. on Generative Programming and Component Engineering (GPCE). ACM, 188--201.
[21]
Tobias Pett. 2018. Stability of Product Sampling under Product-Line Evolution. Master's thesis. Braunschweig.
[22]
Reimar Schröter, Sebastian Krieter, Thomas Thüm, Fabian Benduhn, and Gunter Saake. 2016. Feature-Model Interfaces: The Highway to Compositional Analyses of Highly-Configurable Systems. In Proc. Int'l Conf. on Software Engineering (ICSE). ACM, 667--678.
[23]
Jiangfan Shi, Myra B. Cohen, and Matthew B. Dwyer. 2012. Integration Testing of Software Product Lines Using Compositional Symbolic Execution. In Proc. Int'l Conf. on Fundamental Approaches to Software Engineering (FASE). Springer, 270--284.
[24]
Julio Sincero, Horst Schirmeier, Wolfgang Schröder-Preikschat, and Olaf Spinczyk. 2007. Is the Linux Kernel a Software Product Line?. In Proc. Int'l Workshop on Open Source Software and Product Lines (OSSPL). IEEE, 9--12.
[25]
Reinhard Tartler, Christian Dietrich, Julio Sincero, Wolfgang Schröder-Preikschat, and Daniel Lohmann. 2014. Static Analysis of Variability in System Software: The 90,000 #Ifdefs Issue. In Proc. USENIX Annual Technical Conference (ATC). USENIX Association, 421--432.
[26]
Reinhard Tartler, Daniel Lohmann, Julio Sincero, and Wolfgang Schröder-Preikschat. 2011. Feature Consistency in Compile-Time-Configurable System Software: Facing the Linux 10,000 Feature Problem. In Proc. Europ. Conf. on Computer Systems (EuroSys). ACM, 47--60.
[27]
Reinhard Tartler, Julio Sincero, Wolfgang Schröder-Preikschat, and Daniel Lohmann. 2009. Dead or Alive: Finding Zombie Features in the Linux Kernel. In Proc. Int'l Workshop on Feature-Oriented Software Development (FOSD). ACM, 81--86.
[28]
Thomas Thüm, Sven Apel, Christian Kästner, Ina Schaefer, and Gunter Saake. 2014. A Classification and Survey of Analysis Strategies for Software Product Lines. Comput. Surveys 47, 1 (2014), 6:1--6:45.
[29]
Grigori S Tseitin. 1983. On the complexity of derivation in propositional calculus. In Automation of reasoning. Springer, 466--483.
[30]
Mahsa Varshosaz, Mustafa Al-Hajjaji, Thomas Thüm, Tobias Runge, Mohammad Reza Mousavi, and Ina Schaefer. 2018. A Classification of Product Sampling for Software Product Lines. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM, 1--13.
[31]
Roman Zippel. 2017. KConfig Documentation. Website. Available online at http://www.kernel.org/doc/Documentation/kbuild/kconfig-language.txt; visited on June 21st, 2019.

Cited By

View all
  • (2024)Reusing d-DNNFs for Efficient Feature-Model CountingACM Transactions on Software Engineering and Methodology10.1145/368046533:8(1-32)Online publication date: 30-Jul-2024
  • (2024)Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable SystemsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680309(641-653)Online publication date: 11-Sep-2024
  • (2024)On the Use of Multi-valued Decision Diagrams to Count Valid Configurations of Feature ModelsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672594(96-106)Online publication date: 2-Sep-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A
September 2019
356 pages
ISBN:9781450371384
DOI:10.1145/3336294
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2019

Permissions

Request permissions for this article.

Check for updates

Badges

Author Tags

  1. product line testing
  2. product sampling
  3. real-world feature models
  4. software product lines

Qualifiers

  • Short-paper

Conference

SPLC 2019

Acceptance Rates

Overall Acceptance Rate 167 of 463 submissions, 36%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)51
  • Downloads (Last 6 weeks)6
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Reusing d-DNNFs for Efficient Feature-Model CountingACM Transactions on Software Engineering and Methodology10.1145/368046533:8(1-32)Online publication date: 30-Jul-2024
  • (2024)Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable SystemsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680309(641-653)Online publication date: 11-Sep-2024
  • (2024)On the Use of Multi-valued Decision Diagrams to Count Valid Configurations of Feature ModelsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672594(96-106)Online publication date: 2-Sep-2024
  • (2024)UnWise: High T-Wise Coverage from Uniform SamplingProceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3634713.3634716(37-45)Online publication date: 7-Feb-2024
  • (2024)Incremental Identification of T-Wise Feature InteractionsProceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3634713.3634715(27-36)Online publication date: 7-Feb-2024
  • (2024)A Scalable t-Wise Coverage Estimator: Algorithms and ApplicationsIEEE Transactions on Software Engineering10.1109/TSE.2024.341991950:8(2021-2039)Online publication date: Aug-2024
  • (2024)Baital: Sampling Configurable Systems with high t-wise coverageScience of Computer Programming10.1016/j.scico.2024.103209(103209)Online publication date: Sep-2024
  • (2023)CAmpactor: A Novel and Effective Local Search Algorithm for Optimizing Pairwise Covering ArraysProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616284(81-93)Online publication date: 30-Nov-2023
  • (2023)Generating Pairwise Covering Arrays for Highly Configurable Software SystemsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608998(261-267)Online publication date: 28-Aug-2023
  • (2023)Quantum Computing for Feature Model AnalysisProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608971(1-7)Online publication date: 28-Aug-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media