Tobias Pett
Thomas Thüm
ABSTRACT
Companies strive to implement continuous integration into their development process to ensure the quality of their systems. Regression testing within the CI process considers the efficient re-test of systems after changes. However, even with regression testing, it is not feasible to test all configurations from a highly-configurable software system due to the combinatorial-explosion problem. Numerous sampling algorithms have been proposed that aim at computing a considerably smaller yet sufficiently representative set of configurations to be tested. Those algorithms are typically evaluated with regard to efficiency (i.e., number of configurations in a sample and computational effort for generating a sample) and effectiveness (i.e., feature-interaction coverage or number of faults detected). In this paper, we argue that a further crucial characteristic of sampling algorithms is their tendency to produce similar configurations when applied consecutively to an evolving configurable system. We propose sampling stability as a new evaluation criterion for sampling algorithms. We present a procedure to compute the sampling stability of sampling algorithms based on the similarity between consecutive samples. In our evaluation, we compare the sampling stability of multiple established t-wise sampling algorithms on large real-world systems.
- Iago Abal, Jean Melo, Ştefan Stănciulescu, Claus Brabrand, Márcio Ribeiro, and Andrzej Wąsowski. 2018. Variability bugs in highly configurable systems: A qualitative analysis. ACM Transactions on Software Engineering and Methodology (TOSEM) 26, 3(2018), 1–34.Google Scholar
Digital Library
- Mustafa Al-Hajjaji, Sebastian Krieter, Thomas Thüm, Malte Lochau, and Gunter Saake. 2016. IncLing: efficient product-line testing using incremental pairwise sampling. ACM SIGPLAN Notices 52, 3 (2016), 144–155.Google ScholarDigital Library
- Mustafa Al-Hajjaji, Jens Meinicke, Sebastian Krieter, Reimar Schröter, Thomas Thüm, Thomas Leich, and Gunter Saake. 2016. Tool demo: testing configurable systems with FeatureIDE. In Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences. 173–177.Google ScholarDigital Library
- Mustafa Al-Hajjaji, Thomas Thüm, Malte Lochau, Jens Meinicke, and Gunter Saake. 2019. Effective product-line testing using similarity-based product prioritization. Software & Systems Modeling 18, 1 (2019), 499–521.Google ScholarDigital Library
- Sven Apel, Alexander Von Rhein, Philipp Wendler, Armin Größlinger, and Dirk Beyer. 2013. Strategies for product-line verification: case studies and experiments. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 482–491.Google ScholarCross Ref
- Eduard Baranov, Axel Legay, and Kuldeep S. Meel. 2020. Baital: an adaptive weighted sampling approach for improved t-wise coverage. In Proc. Int’l Symposium on Foundations of Software Engineering (FSE)(FSE 2020). Association for Computing Machinery, New York, NY, USA, 1114–1126. https://doi.org/10.1145/3368089.3409744Google ScholarDigital Library
- Jan Bosch. 2004. Software variability management. In Proceedings. 26th International Conference on Software Engineering. IEEE, 720–721.Google ScholarDigital Library
- Johannes Bürdek, Timo Kehrer, Malte Lochau, Dennis Reuling, Udo Kelter, and Andy Schürr. 2016. Reasoning about product-line evolution using complex feature model differences. Automated Software Engineering 23, 4 (2016), 687–733.Google ScholarDigital Library
- Lianping Chen, Muhammad Ali Babar, and Nour Ali. 2009. Variability management in software product lines: a systematic review. In 13th International Software Product Line Conference. ACM.Google ScholarDigital Library
- Vasek Chvatal. 1979. A greedy heuristic for the set-covering problem. Mathematics of operations research 4, 3 (1979), 233–235.Google Scholar
- Myra B Cohen, Matthew B Dwyer, and Jiangfan Shi. 2007. Interaction testing of highly-configurable systems in the presence of constraints. In Proceedings of the 2007 international symposium on Software testing and analysis. ACM, 129–139.Google ScholarDigital Library
- Myra B Cohen, Matthew B Dwyer, and Jiangfan Shi. 2008. Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Transactions on Software Engineering 34, 5 (2008), 633–650.Google ScholarDigital Library
- Ivan do Carmo Machado, John D McGregor, Yguaratã Cerqueira Cavalcanti, and Eduardo Santana De Almeida. 2014. On strategies for testing software product lines: A systematic literature review. Information and Software Technology 56, 10 (2014), 1183–1199.Google ScholarDigital Library
- Paul M Duvall, Steve Matyas, and Andrew Glover. 2007. Continuous integration: improving software quality and reducing risk. Pearson Education.Google Scholar
- Martin Fowler and Matthew Foemmel. 2006. Continuous integration.Google Scholar
- Richard W Hamming. 1950. Error detecting and error correcting codes. The Bell system technical journal 29, 2 (1950), 147–160.Google Scholar
- Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, Patrick Heymans, and Yves Le Traon. 2014. Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Transactions on Software Engineering 40, 7 (2014), 650–670.Google ScholarDigital Library
- Paul Jaccard. 1912. The distribution of the flora in the alpine zone. 1. New phytologist 11, 2 (1912), 37–50.Google Scholar
- Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2011. Properties of realistic feature models make combinatorial testing of product lines feasible. In International Conference on Model Driven Engineering Languages and Systems. Springer, 638–652.Google ScholarCross Ref
- Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2012. An algorithm for generating t-wise covering arrays from large feature models. In Proceedings of the 16th International Software Product Line Conference-Volume 1. ACM, 46–55.Google ScholarDigital Library
- Roy Jonker and Ton Volgenant. 1986. Improving the Hungarian assignment algorithm. Operations Research Letters 5, 4 (1986), 171–175.Google ScholarDigital Library
- Sebastian Krieter, Thomas Thüm, Sandro Schulze, Gunter Saake, and Thomas Leich. 2020. YASA: yet another sampling algorithm. In Proceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems. ACM, 1–10.Google ScholarDigital Library
- Harold W Kuhn. 1955. The Hungarian method for the assignment problem. Naval research logistics quarterly 2, 1-2 (1955), 83–97.Google Scholar
- Remo Lachmann, Sascha Lity, Sabrina Lischke, Simon Beddig, Sandro Schulze, and Ina Schaefer. 2015. Delta-oriented test case prioritization for integration testing of software product lines. In Proceedings of the 19th International Conference on Software Product Line. 81–90.Google ScholarDigital Library
- Jihyun Lee, Sungwon Kang, and Danhyung Lee. 2012. A survey on software product line testing. In Proceedings of the 16th International Software Product Line Conference-Volume 1. ACM, 31–40.Google ScholarDigital Library
- Jörg Liebig, Alexander Von Rhein, Christian Kästner, Sven Apel, Jens Dörre, and Christian Lengauer. 2013. Scalable analysis of variable software. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 81–91.Google ScholarDigital Library
- Jackson A. Prado Lima, Willian D. F. Mendonça, Silvia R. Vergilio, and Wesley K. G. Assunção. 2020. Learning-based prioritization of test cases in continuous integration of highly-configurable software. In Proc. Int’l Systems and Software Product Line Conf. (SPLC)(SPLC ’20). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3382025.3414967Google ScholarDigital Library
- Sascha Lity, Remo Lachmann, Malte Lochau, and Ina Schaefer. 2013. Delta-oriented software product line test models-the body comfort system case study. Technical Report. TU Braunschweig, Germany.Google Scholar
- Sascha Lity, Manuel Nieke, Thomas Thüm, and Ina Schaefer. 2019. Retest test selection for product-line regression testing of variants and versions of variants. JSS 147 (Jan. 2019), 46–63. https://doi.org/10.1016/j.jss.2018.09.090Google Scholar
- Roberto E Lopez-Herrejon, Stefan Fischer, Rudolf Ramler, and Alexander Egyed. 2015. A first systematic mapping study on combinatorial interaction testing for software product lines. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 1–10.Google ScholarCross Ref
- Dusica Marijan, Arnaud Gotlieb, and Marius Liaaen. 2019. A learning algorithm for optimizing continuous integration development and testing practice. Software: Practice and Experience 49, 2 (2019), 192–213. https://doi.org/10.1002/spe.2661Google ScholarCross Ref
- Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A comparison of 10 sampling algorithms for configurable systems. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, 643–654.Google ScholarDigital Library
- Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Sarah Nadi, and Rohit Gheyi. 2015. The Love/Hate Relationship with the C Preprocessor: An Interview Study. In 29th European Conference on Object-Oriented Programming (ECOOP 2015). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, 495–518.Google Scholar
- Mathias Meyer. 2014. Continuous integration and its tools. IEEE Software 31, 3 (2014), 14–16.Google ScholarCross Ref
- Tobias Müller, Malte Lochau, Stefan Detering, Falko Saust, Henning Garbers, Lukas Märtin, Thomas Form, and Ursula Goltz. 2009. A comprehensive Description of a Model-based, continuous Development Process for AUTOSAR Systems with integrated Quality Assurance. Technical Report. TU Braunschweig, Germany.Google Scholar
- Daniel-Jesus Munoz, Jeho Oh, Mónica Pinto, Lidia Fuentes, and Don Batory. 2019. Uniform random sampling product configurations of feature models that have numerical features. In Proceedings of the 23rd International Systems and Software Product Line Conference-Volume A. ACM, 289–301.Google ScholarDigital Library
- Sophia Nahrendorf, Sascha Lity, and Ina Schaefer. 2018. Applying Higher-Order Delta Modeling for the Evolution of Delta-Oriented Software Product Lines. Technical Report. TU Braunschweig-ISF.Google Scholar
- Changhai Nie and Hareton Leung. 2011. A survey of combinatorial testing. ACM Computing Surveys (CSUR) 43, 2 (2011), 1–29.Google ScholarDigital Library
- Michael Nieke, Jacopo Mauro, Christoph Seidl, Thomas Thüm, Ingrid Chieh Yu, and Felix Franzke. 2018. Anomaly analyses for feature-model evolution. ACM SIGPLAN Notices 53, 9 (2018), 188–201.Google ScholarDigital Library
- Jeho Oh, Paul Gazzillo, Don Batory, Marijn Heule, and Maggie Myers. 2019. Uniform sampling from kconfig feature models. Technical Report. The University of Texas at Austin, Department of Computer Science, Tech. Rep. TR-19-02.Google Scholar
- Ana B Sánchez, Sergio Segura, and Antonio Ruiz-Cortés. 2014. A comparison of test case prioritization criteria for software product lines. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. IEEE, 41–50.Google ScholarDigital Library
- Sean Stolberg. 2009. Enabling agile testing through continuous integration. In 2009 agile conference. IEEE, 369–374.Google Scholar
- Thomas Thüm, Sven Apel, Christian Kästner, Ina Schaefer, and Gunter Saake. 2014. A classification and survey of analysis strategies for software product lines. ACM Computing Surveys (CSUR) 47, 1 (2014), 1–45.Google ScholarDigital Library
- Thomas Thum, Don Batory, and Christian Kastner. 2009. Reasoning about edits to feature models. In 2009 IEEE 31st International Conference on Software Engineering. IEEE, 254–264.Google ScholarDigital Library
- Mahsa Varshosaz, Mustafa Al-Hajjaji, Thomas Thüm, Tobias Runge, Mohammad Reza Mousavi, and Ina Schaefer. 2018. A classification of product sampling for software product lines. In Proceedings of the 22nd International Systems and Software Product Line Conference-Volume 1. ACM, 1–13.Google ScholarDigital Library
- Alexander Von Rhein, Sven Apel, Christian Kästner, Thomas Thüm, and Ina Schaefer. 2013. The PLA model: on the combination of product-line analyses. In Proceedings of the Seventh International Workshop on Variability Modelling of Software-intensive Systems. ACM, NY, 1–8.Google ScholarDigital Library
Recommendations
AutoSMP: an evaluation platform for sampling algorithms
SPLC '21: Proceedings of the 25th ACM International Systems and Software Product Line Conference - Volume BTesting configurable systems is a challenging task due to the combinatorial explosion problem. Sampling is a promising approach to reduce the testing effort for product-based systems by finding a small but still representative subset (i.e., a sample) of ...
IncLing: efficient product-line testing using incremental pairwise sampling
GPCE '16A software product line comprises a family of software products that share a common set of features. It enables customers to compose software systems from a managed set of features. Testing every product of a product line individually is often ...
Line segment sampling with blue-noise properties
Line segment sampling has recently been adopted in many rendering algorithms for better handling of a wide range of effects such as motion blur, defocus blur and scattering media. A question naturally raised is how to generate line segment samples with ...
Comments