skip to main content
10.1145/3442391.3442410acmotherconferencesArticle/Chapter ViewAbstractPublication PagesvamosConference Proceedingsconference-collections
research-article

Stability of Product-Line Samplingin Continuous Integration

Published: 09 February 2021 Publication History

Abstract

Companies strive to implement continuous integration into their development process to ensure the quality of their systems. Regression testing within the CI process considers the efficient re-test of systems after changes. However, even with regression testing, it is not feasible to test all configurations from a highly-configurable software system due to the combinatorial-explosion problem. Numerous sampling algorithms have been proposed that aim at computing a considerably smaller yet sufficiently representative set of configurations to be tested. Those algorithms are typically evaluated with regard to efficiency (i.e., number of configurations in a sample and computational effort for generating a sample) and effectiveness (i.e., feature-interaction coverage or number of faults detected). In this paper, we argue that a further crucial characteristic of sampling algorithms is their tendency to produce similar configurations when applied consecutively to an evolving configurable system. We propose sampling stability as a new evaluation criterion for sampling algorithms. We present a procedure to compute the sampling stability of sampling algorithms based on the similarity between consecutive samples. In our evaluation, we compare the sampling stability of multiple established t-wise sampling algorithms on large real-world systems.

References

[1]
Iago Abal, Jean Melo, Ştefan Stănciulescu, Claus Brabrand, Márcio Ribeiro, and Andrzej Wąsowski. 2018. Variability bugs in highly configurable systems: A qualitative analysis. ACM Transactions on Software Engineering and Methodology (TOSEM) 26, 3(2018), 1–34.
[2]
Mustafa Al-Hajjaji, Sebastian Krieter, Thomas Thüm, Malte Lochau, and Gunter Saake. 2016. IncLing: efficient product-line testing using incremental pairwise sampling. ACM SIGPLAN Notices 52, 3 (2016), 144–155.
[3]
Mustafa Al-Hajjaji, Jens Meinicke, Sebastian Krieter, Reimar Schröter, Thomas Thüm, Thomas Leich, and Gunter Saake. 2016. Tool demo: testing configurable systems with FeatureIDE. In Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences. 173–177.
[4]
Mustafa Al-Hajjaji, Thomas Thüm, Malte Lochau, Jens Meinicke, and Gunter Saake. 2019. Effective product-line testing using similarity-based product prioritization. Software & Systems Modeling 18, 1 (2019), 499–521.
[5]
Sven Apel, Alexander Von Rhein, Philipp Wendler, Armin Größlinger, and Dirk Beyer. 2013. Strategies for product-line verification: case studies and experiments. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 482–491.
[6]
Eduard Baranov, Axel Legay, and Kuldeep S. Meel. 2020. Baital: an adaptive weighted sampling approach for improved t-wise coverage. In Proc. Int’l Symposium on Foundations of Software Engineering (FSE)(FSE 2020). Association for Computing Machinery, New York, NY, USA, 1114–1126. https://doi.org/10.1145/3368089.3409744
[7]
Jan Bosch. 2004. Software variability management. In Proceedings. 26th International Conference on Software Engineering. IEEE, 720–721.
[8]
Johannes Bürdek, Timo Kehrer, Malte Lochau, Dennis Reuling, Udo Kelter, and Andy Schürr. 2016. Reasoning about product-line evolution using complex feature model differences. Automated Software Engineering 23, 4 (2016), 687–733.
[9]
Lianping Chen, Muhammad Ali Babar, and Nour Ali. 2009. Variability management in software product lines: a systematic review. In 13th International Software Product Line Conference. ACM.
[10]
Vasek Chvatal. 1979. A greedy heuristic for the set-covering problem. Mathematics of operations research 4, 3 (1979), 233–235.
[11]
Myra B Cohen, Matthew B Dwyer, and Jiangfan Shi. 2007. Interaction testing of highly-configurable systems in the presence of constraints. In Proceedings of the 2007 international symposium on Software testing and analysis. ACM, 129–139.
[12]
Myra B Cohen, Matthew B Dwyer, and Jiangfan Shi. 2008. Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Transactions on Software Engineering 34, 5 (2008), 633–650.
[13]
Ivan do Carmo Machado, John D McGregor, Yguaratã Cerqueira Cavalcanti, and Eduardo Santana De Almeida. 2014. On strategies for testing software product lines: A systematic literature review. Information and Software Technology 56, 10 (2014), 1183–1199.
[14]
Paul M Duvall, Steve Matyas, and Andrew Glover. 2007. Continuous integration: improving software quality and reducing risk. Pearson Education.
[15]
Martin Fowler and Matthew Foemmel. 2006. Continuous integration.
[16]
Richard W Hamming. 1950. Error detecting and error correcting codes. The Bell system technical journal 29, 2 (1950), 147–160.
[17]
Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, Patrick Heymans, and Yves Le Traon. 2014. Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Transactions on Software Engineering 40, 7 (2014), 650–670.
[18]
Paul Jaccard. 1912. The distribution of the flora in the alpine zone. 1. New phytologist 11, 2 (1912), 37–50.
[19]
Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2011. Properties of realistic feature models make combinatorial testing of product lines feasible. In International Conference on Model Driven Engineering Languages and Systems. Springer, 638–652.
[20]
Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2012. An algorithm for generating t-wise covering arrays from large feature models. In Proceedings of the 16th International Software Product Line Conference-Volume 1. ACM, 46–55.
[21]
Roy Jonker and Ton Volgenant. 1986. Improving the Hungarian assignment algorithm. Operations Research Letters 5, 4 (1986), 171–175.
[22]
Sebastian Krieter, Thomas Thüm, Sandro Schulze, Gunter Saake, and Thomas Leich. 2020. YASA: yet another sampling algorithm. In Proceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems. ACM, 1–10.
[23]
Harold W Kuhn. 1955. The Hungarian method for the assignment problem. Naval research logistics quarterly 2, 1-2 (1955), 83–97.
[24]
Remo Lachmann, Sascha Lity, Sabrina Lischke, Simon Beddig, Sandro Schulze, and Ina Schaefer. 2015. Delta-oriented test case prioritization for integration testing of software product lines. In Proceedings of the 19th International Conference on Software Product Line. 81–90.
[25]
Jihyun Lee, Sungwon Kang, and Danhyung Lee. 2012. A survey on software product line testing. In Proceedings of the 16th International Software Product Line Conference-Volume 1. ACM, 31–40.
[26]
Jörg Liebig, Alexander Von Rhein, Christian Kästner, Sven Apel, Jens Dörre, and Christian Lengauer. 2013. Scalable analysis of variable software. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 81–91.
[27]
Jackson A. Prado Lima, Willian D. F. Mendonça, Silvia R. Vergilio, and Wesley K. G. Assunção. 2020. Learning-based prioritization of test cases in continuous integration of highly-configurable software. In Proc. Int’l Systems and Software Product Line Conf. (SPLC)(SPLC ’20). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3382025.3414967
[28]
Sascha Lity, Remo Lachmann, Malte Lochau, and Ina Schaefer. 2013. Delta-oriented software product line test models-the body comfort system case study. Technical Report. TU Braunschweig, Germany.
[29]
Sascha Lity, Manuel Nieke, Thomas Thüm, and Ina Schaefer. 2019. Retest test selection for product-line regression testing of variants and versions of variants. JSS 147 (Jan. 2019), 46–63. https://doi.org/10.1016/j.jss.2018.09.090
[30]
Roberto E Lopez-Herrejon, Stefan Fischer, Rudolf Ramler, and Alexander Egyed. 2015. A first systematic mapping study on combinatorial interaction testing for software product lines. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 1–10.
[31]
Dusica Marijan, Arnaud Gotlieb, and Marius Liaaen. 2019. A learning algorithm for optimizing continuous integration development and testing practice. Software: Practice and Experience 49, 2 (2019), 192–213. https://doi.org/10.1002/spe.2661
[32]
Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A comparison of 10 sampling algorithms for configurable systems. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, 643–654.
[33]
Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Sarah Nadi, and Rohit Gheyi. 2015. The Love/Hate Relationship with the C Preprocessor: An Interview Study. In 29th European Conference on Object-Oriented Programming (ECOOP 2015). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, 495–518.
[34]
Mathias Meyer. 2014. Continuous integration and its tools. IEEE Software 31, 3 (2014), 14–16.
[35]
Tobias Müller, Malte Lochau, Stefan Detering, Falko Saust, Henning Garbers, Lukas Märtin, Thomas Form, and Ursula Goltz. 2009. A comprehensive Description of a Model-based, continuous Development Process for AUTOSAR Systems with integrated Quality Assurance. Technical Report. TU Braunschweig, Germany.
[36]
Daniel-Jesus Munoz, Jeho Oh, Mónica Pinto, Lidia Fuentes, and Don Batory. 2019. Uniform random sampling product configurations of feature models that have numerical features. In Proceedings of the 23rd International Systems and Software Product Line Conference-Volume A. ACM, 289–301.
[37]
Sophia Nahrendorf, Sascha Lity, and Ina Schaefer. 2018. Applying Higher-Order Delta Modeling for the Evolution of Delta-Oriented Software Product Lines. Technical Report. TU Braunschweig-ISF.
[38]
Changhai Nie and Hareton Leung. 2011. A survey of combinatorial testing. ACM Computing Surveys (CSUR) 43, 2 (2011), 1–29.
[39]
Michael Nieke, Jacopo Mauro, Christoph Seidl, Thomas Thüm, Ingrid Chieh Yu, and Felix Franzke. 2018. Anomaly analyses for feature-model evolution. ACM SIGPLAN Notices 53, 9 (2018), 188–201.
[40]
Jeho Oh, Paul Gazzillo, Don Batory, Marijn Heule, and Maggie Myers. 2019. Uniform sampling from kconfig feature models. Technical Report. The University of Texas at Austin, Department of Computer Science, Tech. Rep. TR-19-02.
[41]
Ana B Sánchez, Sergio Segura, and Antonio Ruiz-Cortés. 2014. A comparison of test case prioritization criteria for software product lines. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. IEEE, 41–50.
[42]
Sean Stolberg. 2009. Enabling agile testing through continuous integration. In 2009 agile conference. IEEE, 369–374.
[43]
Thomas Thüm, Sven Apel, Christian Kästner, Ina Schaefer, and Gunter Saake. 2014. A classification and survey of analysis strategies for software product lines. ACM Computing Surveys (CSUR) 47, 1 (2014), 1–45.
[44]
Thomas Thum, Don Batory, and Christian Kastner. 2009. Reasoning about edits to feature models. In 2009 IEEE 31st International Conference on Software Engineering. IEEE, 254–264.
[45]
Mahsa Varshosaz, Mustafa Al-Hajjaji, Thomas Thüm, Tobias Runge, Mohammad Reza Mousavi, and Ina Schaefer. 2018. A classification of product sampling for software product lines. In Proceedings of the 22nd International Systems and Software Product Line Conference-Volume 1. ACM, 1–13.
[46]
Alexander Von Rhein, Sven Apel, Christian Kästner, Thomas Thüm, and Ina Schaefer. 2013. The PLA model: on the combination of product-line analyses. In Proceedings of the Seventh International Workshop on Variability Modelling of Software-intensive Systems. ACM, NY, 1–8.

Cited By

View all
  • (2024)Reusing d-DNNFs for Efficient Feature-Model CountingACM Transactions on Software Engineering and Methodology10.1145/368046533:8(1-32)Online publication date: 30-Jul-2024
  • (2024)On the Use of Multi-valued Decision Diagrams to Count Valid Configurations of Feature ModelsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672594(96-106)Online publication date: 2-Sep-2024
  • (2024)Incremental Identification of T-Wise Feature InteractionsProceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3634713.3634715(27-36)Online publication date: 7-Feb-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
VaMoS '21: Proceedings of the 15th International Working Conference on Variability Modelling of Software-Intensive Systems
February 2021
150 pages
ISBN:9781450388245
DOI:10.1145/3442391
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 February 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. product lines
  2. product-line evolution
  3. sampling

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

VaMoS'21

Acceptance Rates

Overall Acceptance Rate 66 of 147 submissions, 45%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)28
  • Downloads (Last 6 weeks)2
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Reusing d-DNNFs for Efficient Feature-Model CountingACM Transactions on Software Engineering and Methodology10.1145/368046533:8(1-32)Online publication date: 30-Jul-2024
  • (2024)On the Use of Multi-valued Decision Diagrams to Count Valid Configurations of Feature ModelsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672594(96-106)Online publication date: 2-Sep-2024
  • (2024)Incremental Identification of T-Wise Feature InteractionsProceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3634713.3634715(27-36)Online publication date: 7-Feb-2024
  • (2024)Testing the Evolution of Feature Models with Specific Combinatorial Tests2024 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW60967.2024.00025(1-10)Online publication date: 27-May-2024
  • (2023)On the Reuse of Existing Configurations for Testing Evolving Feature ModelsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume B10.1145/3579028.3609017(67-76)Online publication date: 28-Aug-2023
  • (2023)Views on Edits to Variational SoftwareProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608985(141-152)Online publication date: 28-Aug-2023
  • (2023)Elimination of constraints for parallel analysis of feature modelsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608981(99-110)Online publication date: 28-Aug-2023
  • (2023)Evaluating state-of-the-art # SAT solvers on industrial configuration spacesEmpirical Software Engineering10.1007/s10664-022-10265-928:2Online publication date: 13-Jan-2023
  • (2023)On the benefits of knowledge compilation for feature-model analysesAnnals of Mathematics and Artificial Intelligence10.1007/s10472-023-09906-692:5(1013-1050)Online publication date: 6-Nov-2023
  • (2022)Tseitin or not Tseitin? The Impact of CNF Transformations on Feature-Model AnalysesProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556938(1-13)Online publication date: 10-Oct-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media