ABSTRACT
Quality assurance for product lines is often infeasible for each product separately. Instead, only a subset of all products (i.e., a sample) is considered during testing such that at least the coverage of certain feature interactions is guaranteed. While pair-wise interaction sampling only covers all interactions between two features, its generalization to t-wise interaction sampling ensures coverage for all interactions among t features. However, sampling large product lines poses a challenge, as today's algorithms tend to run out of memory, do not terminate, or produce samples, which are too large to be tested. To initiate a community effort, we provide a set of large real-world feature models with up-to 19 thousand features, which are supposed to be sampled. The performance of sampling approaches is evaluated based on the CPU time and memory consumed to retrieve a sample, the sample size for a given coverage (i.e. the value of t) and whether the sample achieves full t-wise coverage. A well-performing sampling algorithm achieves full t-wise coverage, while minimizing the other properties as best as possible.
- Iago Abal, Jean Melo, Stefan Stănciulescu, Claus Brabrand, Márcio Ribeiro, and Andrzej Wąsowski. 2018. Variability Bugs in Highly Configurable Systems: A Qualitative Analysis. Trans. Software Engineering and Methodology (TOSEM) 26, 3, Article 10 (2018), 10:1--10:34 pages. Google ScholarDigital Library
- Bestoun S. Ahmed, Kamal Z. Zamli, Wasif Afzal, and Miroslav Bures. 2017. Constrained Interaction Testing: A Systematic Literature Study. IEEE Access 5 (2017), 25706--25730.Google ScholarCross Ref
- Sven Apel, Don Batory, Christian Kästner, and Gunter Saake. 2013. Feature-Oriented Software Product Lines. Springer. Google ScholarDigital Library
- Andrea Arcuri and Lionel Briand. 2011. Formal analysis of the probability of interaction fault detection using random testing. TSE 38, 5 (2011), 1088--1099. Google ScholarDigital Library
- Christian Dietrich, Reinhard Tartler, Wolfgang Schröder-Preikschat, and Daniel Lohmann. 2012. Understanding Linux Feature Distribution. In Proc. of Workshop on Modularity in Systems Software (MISS). ACM, 15--20. Google ScholarDigital Library
- Nicolas Dintzner, Arie van Deursen, and Martin Pinzger. 2018. FEVER: An Approach to Analyze Feature-Oriented Changes and Artefact Co-Evolution in Highly Configurable Systems. Empirical Software Engineering (EMSE) 23, 2 (2018), 905--952. Google ScholarDigital Library
- Alireza Ensan, Ebrahim Bagheri, Mohsen Asadi, Dragan Gasevic, and Yevgen Biletskiy. 2011. Goal-Oriented Test Case Selection and Prioritization for Product Line Feature Models. In Proc. Inte'l Conf. on Information Technology: New Generations (ITNG). IEEE, 291--298. Google ScholarDigital Library
- Mats Grindal, Jeff Offutt, and Sten F Andler. 2005. Combination testing strategies: a survey. STVR 15, 3 (2005), 167--199.Google Scholar
- Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. 2013. Multi-Objective Test Generation for Software Product Lines. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM, 62--71. Google ScholarDigital Library
- Christian Kästner. 2017. Differential Testing for Variational Analyses: Experience from Developing KConfigReader. arXiv preprint arXiv:1706.09357 (2017).Google Scholar
- Chang Hwan Peter Kim, Don Batory, and Sarfraz Khurshid. 2011. Reducing Combinatorics in Testing Product Lines. In Proc. Int'l Conf. on Aspect-Oriented Software Development (AOSD). ACM, 57--68. Google ScholarDigital Library
- Alexander Knüppel, Thomas Thüm, Stephan Mennicke, Jens Meinicke, and Ina Schaefer. 2017. Is There a Mismatch Between Real-World Feature Models and Product-Line Research?. In Proc. Europ. Software Engineering Conf./Foundations of Software Engineering (ESEC/FSE). ACM, 291--302. Google ScholarDigital Library
- Sebastian Krieter, Marcus Pinnecke, Jacob Krüger, Joshua Sprey, Christopher Sontag, Thomas Thüm, Thomas Leich, and Gunter Saake. 2017. FeatureIDE: Empowering Third-Party Developers. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM, 42--45. Google ScholarDigital Library
- Christian Kästner. 2018. KConfigReader. Website. Available online at https://github.com/ckaestne/kconfigreader; visited on August 29th, 2018.Google Scholar
- Jörg Liebig, Alexander von Rhein, Christian Kästner, Sven Apel, Jens Dörre, and Christian Lengauer. 2013. Scalable Analysis of Variable Software. In Proc. Europ. Software Engineering Conf./Foundations of Software Engineering (ESEC/FSE). ACM, 81--91. Google ScholarDigital Library
- Roberto E. Lopez-Herrejon and Don Batory. 2001. A Standard Problem for Evaluating Product-Line Methodologies. In Proc. Int'l Symposium on Generative and Component-Based Software Engineering (GCSE). Springer, 10--24. Google ScholarDigital Library
- Roberto E. Lopez-Herrejon, Stefan Fischer, Rudolf Ramler, and Aalexander Egyed. 2015. A First Systematic Mapping Study on Combinatorial Interaction Testing for Software Product Lines. In Proc. Int'l Workshop on Combinatorial Testing (IWCT). IEEE, 1--10.Google ScholarCross Ref
- Jens Meinicke, Thomas Thüm, Reimar Schröter, Fabian Benduhn, Thomas Leich, and Gunter Saake. 2017. Mastering Software Variability with FeatureIDE. Springer. Google ScholarDigital Library
- Paulo Anselmo da Mota Silveira Neto, Ivan do Carmo Machado, Yguarata Cerqueira Cavalcanti, Eduardo Santana de Almeida, Vinicius Cardoso Garcia, and Silvio Romero de Lemos Meira. 2010. A regression testing approach for software product lines architectures. In Software Components, Architectures and Reuse (SBCARS), 2010 Fourth Brazilian Symposium on. IEEE, 41--50. Google ScholarDigital Library
- Michael Nieke, Jacopo Mauro, Christoph Seidl, Thomas Thüm, Ingrid Chieh Yu, and Felix Franzke. 2018. Anomaly Analyses for Feature-Model Evolution. In Proc. Int'l Conf. on Generative Programming and Component Engineering (GPCE). ACM, 188--201. Google ScholarDigital Library
- Tobias Pett. 2018. Stability of Product Sampling under Product-Line Evolution. Master's thesis. Braunschweig.Google Scholar
- Reimar Schröter, Sebastian Krieter, Thomas Thüm, Fabian Benduhn, and Gunter Saake. 2016. Feature-Model Interfaces: The Highway to Compositional Analyses of Highly-Configurable Systems. In Proc. Int'l Conf. on Software Engineering (ICSE). ACM, 667--678. Google ScholarDigital Library
- Jiangfan Shi, Myra B. Cohen, and Matthew B. Dwyer. 2012. Integration Testing of Software Product Lines Using Compositional Symbolic Execution. In Proc. Int'l Conf. on Fundamental Approaches to Software Engineering (FASE). Springer, 270--284. Google ScholarDigital Library
- Julio Sincero, Horst Schirmeier, Wolfgang Schröder-Preikschat, and Olaf Spinczyk. 2007. Is the Linux Kernel a Software Product Line?. In Proc. Int'l Workshop on Open Source Software and Product Lines (OSSPL). IEEE, 9--12.Google Scholar
- Reinhard Tartler, Christian Dietrich, Julio Sincero, Wolfgang Schröder-Preikschat, and Daniel Lohmann. 2014. Static Analysis of Variability in System Software: The 90,000 #Ifdefs Issue. In Proc. USENIX Annual Technical Conference (ATC). USENIX Association, 421--432. Google ScholarDigital Library
- Reinhard Tartler, Daniel Lohmann, Julio Sincero, and Wolfgang Schröder-Preikschat. 2011. Feature Consistency in Compile-Time-Configurable System Software: Facing the Linux 10,000 Feature Problem. In Proc. Europ. Conf. on Computer Systems (EuroSys). ACM, 47--60. Google ScholarDigital Library
- Reinhard Tartler, Julio Sincero, Wolfgang Schröder-Preikschat, and Daniel Lohmann. 2009. Dead or Alive: Finding Zombie Features in the Linux Kernel. In Proc. Int'l Workshop on Feature-Oriented Software Development (FOSD). ACM, 81--86. Google ScholarDigital Library
- Thomas Thüm, Sven Apel, Christian Kästner, Ina Schaefer, and Gunter Saake. 2014. A Classification and Survey of Analysis Strategies for Software Product Lines. Comput. Surveys 47, 1 (2014), 6:1--6:45. Google ScholarDigital Library
- Grigori S Tseitin. 1983. On the complexity of derivation in propositional calculus. In Automation of reasoning. Springer, 466--483.Google Scholar
- Mahsa Varshosaz, Mustafa Al-Hajjaji, Thomas Thüm, Tobias Runge, Mohammad Reza Mousavi, and Ina Schaefer. 2018. A Classification of Product Sampling for Software Product Lines. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM, 1--13. Google ScholarDigital Library
- Roman Zippel. 2017. KConfig Documentation. Website. Available online at http://www.kernel.org/doc/Documentation/kbuild/kconfig-language.txt; visited on June 21st, 2019.Google Scholar
Index Terms
- Product Sampling for Product Lines: The Scalability Challenge
Recommendations
A classification of product sampling for software product lines
SPLC '18: Proceedings of the 22nd International Systems and Software Product Line Conference - Volume 1The analysis of software product lines is challenging due to the potentially large number of products, which grow exponentially in terms of the number of features. Product sampling is a technique used to avoid exhaustive testing, which is often ...
IncLing: efficient product-line testing using incremental pairwise sampling
GPCE '16A software product line comprises a family of software products that share a common set of features. It enables customers to compose software systems from a managed set of features. Testing every product of a product line individually is often ...
Continuous test suite augmentation in software product lines
SPLC '13: Proceedings of the 17th International Software Product Line ConferenceSoftware Product Line (SPL) engineering offers several advantages in the development of families of software products. There is still a need, however, to generate test cases for individual products in product lines more efficiently. In this paper we ...
Comments