research-article

Automated Test Suite Generation for Software Product Lines Based on Quality-Diversity Optimization

Authors:

Xiaowei YangAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology, Volume 33, Issue 2

Article No.: 46, Pages 1 - 52

https://doi.org/10.1145/3628158

Published: 22 December 2023 Publication History

Abstract

A Software Product Line (SPL) is a set of software products that are built from a variability model. Real-world SPLs typically involve a vast number of valid products, making it impossible to individually test each of them. This arises the need for automated test suite generation, which was previously modeled as either a single-objective or a multi-objective optimization problem considering only objective functions. This article provides a completely different mathematical model by exploiting the benefits of Quality-Diversity (QD) optimization that is composed of not only an objective function (e.g., t-wise coverage or test suite diversity) but also a user-defined behavior space (e.g., the space with test suite size as its dimension). We argue that the new model is more suitable and generic than the two alternatives because it provides at a time a large set of diverse (measured in the behavior space) and high-performing solutions that can ease the decision-making process. We apply MAP-Elites, one of the most popular QD algorithms, to solve the model. The results of the evaluation, on both realistic and artificial SPLs, are promising, with MAP-Elites significantly and substantially outperforming both single- and multi-objective approaches, and also several state-of-the-art SPL testing tools. In summary, this article provides a new and promising perspective on the test suite generation for SPLs.

References

[1]

Mustafa Al-Hajjaji, Sebastian Krieter, Thomas Thüm, Malte Lochau, and Gunter Saake. 2016. IncLing: Efficient product-line testing using incremental pairwise sampling. In Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE’16). ACM, New York, NY, 144–155.

Digital Library

[2]

Mustafa Al-Hajjaji, Thomas Thüm, Malte Lochau, Jens Meinicke, and Gunter Saake. 2019. Effective product-line testing using similarity-based product prioritization. Software & Systems Modeling 18 (2019), 499–521.

Digital Library

[3]

Andrea Arcuri and Lionel Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering (ICSE’11). ACM, New York, NY, 1–10.

Digital Library

[4]

Andrea Arcuri and Lionel Briand. 2014. A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability 24, 3 (2014), 219–250.

Digital Library

[5]

Aitor Arrieta, Sergio Segura, Urtzi Markiegi, Goiuria Sagardui, and Leire Etxeberria. 2018. Spectrum-based fault localization in software product lines. Information and Software Technology 100 (2018), 18–31. DOI:http://dx.doi.org/DOI:

[6]

Ebrahim Bagheri, Faezeh Ensan, and Dragan Gasevic. 2012. Grammar-based test generation for software product line feature models. In Proceedings of the 2012 Conference of the Center for Advanced Studies on Collaborative Research (CASCON’12). 87–101.

Digital Library

[7]

Eduard Baranov, Axel Legay, and Kuldeep S. Meel. 2020. Baital: An adaptive weighted sampling approach for improved t-wise coverage. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’20). ACM, New York, NY, 1114–1126.

Digital Library

[8]

Don Batory. 2005. Feature models, grammars, and propositional formulas. In Proceedings of the 9th International Conference Software Product Lines (SPLC’05). 7–20.

Digital Library

[9]

David Benavides, Sergio Segura, and Antonio Ruiz-Cortés. 2010. Automated analysis of feature models 20 years later: A literature review. Information Systems 35, 6 (2010), 615–636.

Digital Library

[10]

Daniel Le Berre and Anne Parrain. 2010. The Sat4j library, release 2.2, system description. Journal on Satisfiability, Boolean Modeling and Computation 7 (2010), 59–64.

[11]

Mohamed Boussaa, Olivier Barais, Gerson Sunyé, and Benoît Baudry. 2015. A novelty search approach for automatic test data generation. In Proceedings of the 8th International Workshop on Search-Based Software Testing (SBST’15). IEEE, Los Alamitos, CA, 40–43.

Digital Library

[12]

Rodrigo Canaan, Julian Togelius, Andy Nealen, and Stefan Menzel. 2019. Diverse agents for ad-hoc cooperation in Hanabi. In Proceedings of the 2019 IEEE Conference on Games (CoG’19). 1–8. DOI:

Digital Library

[13]

Konstantinos Chatzilygeroudis, Antoine Cully, Vassilis Vassiliades, and Jean-Baptiste Mouret. 2021. Quality-diversity optimization: A novel branch of stochastic optimization. In Black Box Optimization, Machine Learning, and No-Free Lunch Theorems, Panos M. Pardalos, Varvara Rasskazova, and Michael N. Vrahatis (Eds.). Springer, 109–135. DOI:

[14]

Konstantinos Chatzilygeroudis, Antoine Cully, Vassilis Vassiliades, and Jean-Baptiste Mouret. 2021. Quality-diversity optimization: A novel branch of stochastic optimization. In Black Box Optimization, Machine Learning, and No-Free Lunch Theorems. Springer, 109–135.

[15]

Vaclav Chvatal. 1979. A greedy heuristic for the set-covering problem. Mathematics of Operations Research 4, 3 (1979), 233–235.

Digital Library

[16]

Paul Clements and Linda Northrop. 2001. Software Product Lines: Practices and Patterns. Addison Wesley Longman.

Digital Library

[17]

Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. 2007. Interaction testing of highly-configurable systems in the presence of constraints. In Proceedings of the 2007 International Symposium on Software Testing and Analysis (ISSTA’07). ACM, New York, NY, 129–139.

Digital Library

[18]

Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. 2015. Robots that can adapt like animals. Nature 521, 7553 (2015), 503–507.

[19]

Antoine Cully and Yiannis Demiris. 2018. Hierarchical behavioral repertoires with unsupervised descriptors. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’18). ACM, New York, NY, 69–76. DOI:

Digital Library

[20]

Antoine Cully and Yiannis Demiris. 2018. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation 22, 2 (2018), 245–259. DOI:

[21]

A. Cully and J.-B. Mouret. 2016. Evolving a behavioral repertoire for a walking robot. Evolutionary Computation 24, 1 (2016), 59–88. DOI:

Digital Library

[22]

Paulo Anselmo da Mota Silveira Neto, Ivan do Carmo Machado, John D. McGregor, Eduardo Santana de Almeida, and Silvio Romero de Lemos Meira. 2011. A systematic mapping study of software product lines testing. Information and Software Technology 53, 5 (2011), 407–423. DOI:

Digital Library

[23]

Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and Tamt Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation 6, 2 (2002), 182–197.

Digital Library

[24]

Xavier Devroey, Gilles Perrouin, Axel Legay, “P.-Y.” Schobbens, and Patrick Heymans. 2016. Search-based similarity-driven behavioural SPL testing. In Proceedings of the 10th International Workshop on Variability Modelling of Software-Intensive Systems (VaMoS’16). ACM, New York, NY, 89–96. DOI:

Digital Library

[25]

Ivan do Carmo Machado, John D. McGregor, Yguarata Cerqueira Cavalcanti, and Eduardo Santana de Almeida. 2014. On strategies for testing software product lines: A systematic literature review. Information and Software Technology 56, 10 (2014), 1183–1199. DOI:

Digital Library

[26]

Stephane Doncieux, Alban Laflaquière, and Alexandre Coninx. 2019. Novelty search: A theoretical perspective. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’19). ACM, New York, NY, 99–106.

Digital Library

[27]

Miguel Duarte, Jorge Gomes, Sancho Moura Oliveira, and Anders Lyhne Christensen. 2018. Evolution of repertoire-based control for robots with complex locomotor systems. IEEE Transactions on Evolutionary Computation 22, 2 (2018), 314–328. DOI:

[28]

Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. 2021. First return, then explore. Nature 590 (2021), 580–586.

[29]

Faezeh Ensan, Ebrahim Bagheri, and Dragan Gašević. 2012. Evolutionary search-based test generation for software product line feature models. In Advanced Information Systems Engineering. Lecture Notes in Computer Science, Vol. 7328. Springer, 613–628.

[30]

Robert Feldt and Simon M. Poulding. 2017. Searching for test data with feature diversity. CoRR abs/1709.06017 (2017).

[31]

Fischer Ferreira, João P. Diniz, Cleiton Silva, and Eduardo Figueiredo. 2019. Testing tools for configurable software systems: A review-based empirical study. In Proceedings of the 13th International Workshop on Variability Modelling of Software-Intensive Systems (VAMOS’19). ACM, New York, NY, Article 6, 10 pages. DOI:

Digital Library

[32]

Fischer Ferreira, Gustavo Vale, Joao P. Diniz, and Eduardo Figueiredo. 2021. Evaluating t-wise testing strategies in a community-wide dataset of configurable software systems. Journal of Systems and Software 179 (2021), 110990. DOI:

[33]

Thiago N. Ferreira, Jackson A. Prado Lima, Andrei Strickler, Josiel N. Kuk, Silvia R. Vergilio, and Aurora Pozo. 2017. Hyper-heuristic based product selection for software product line testing. IEEE Computational Intelligence Magazine 12, 2 (2017), 34–45. DOI:

Digital Library

[34]

Thiago Nascimento Ferreira, Silvia Regina Vergilio, and Mauroane Kessentini. 2020. Many-objective search-based selection of software product line test products with Nautilus. In Proceedings of the 24th ACM International Systems and Software Product Line Conference—Volume B (SPLC’20). ACM, New York, NY, 1–4. DOI:

Digital Library

[35]

Javier Ferrer, Francisco Chicano, and Enrique Alba. 2017. Hybrid algorithms based on integer programming for the search of prioritized test data in software product lines. In Applications of Evolutionary Computation, Giovanni Squillero and Kevin Sim (Eds.). Springer International, Cham, Switzerland, 3–19.

[36]

Alan M. Frisch and Timothy J. Peugniez. 2001. Solving non-Boolean satisfiability problems with stochastic local search. In Proceedings of the 17th International Joint Conference on Artificial Intelligence—Volume 1 (IJCAI’01). 282–288.

Digital Library

[37]

Brady J. Garvin, Myra B. Cohen, and Matthew B. Dwyer. 2011. Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Software Engineering 16, 1 (2011), 61–102.

Digital Library

[38]

Daniele Gravina, Ahmed Khalifa, Antonios Liapis, Julian Togelius, and Georgios N. Yannakakis. 2019. Procedural content generation through quality diversity. In Proceedings of the 2019 IEEE Conference on Games (CoG’19). 1–8. DOI:

Digital Library

[39]

Luca Grillotti and Antoine Cully. 2022. Unsupervised behavior discovery with quality-diversity optimization. IEEE Transactions on Evolutionary Computation 26, 6 (2022), 1539–1552. DOI:

[40]

Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based software engineering: Trends, techniques and applications. ACM Computing Surveys 45, 1 (Dec. 2012), Article 11, 61 pages. DOI:

Digital Library

[41]

Christopher Henard, Mike Papadakis, and Yves Le Traon. 2014. Mutation-based generation of software product line test configurations. In Proceedings of the 6th International Symposium on Search-Based Software Engineering (SSBSE’14). 92–106. DOI:

[42]

Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, Patrick Heymans, and Yves Le Traon. 2014. Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Transactions on Software Engineering 40, 7 (July2014), 650–670.

Digital Library

[43]

Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. 2013. Multi-objective test generation for software product lines. In Proceedings of the 17th International Software Product Line Conference (SPLC’13). ACM, New York, NY, 62–71. DOI:

Digital Library

[44]

Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. 2013. PLEDGE: A product line editor and test generation tool. In Proceedings of the 17th International Software Product Line Conference Co-Located Workshops (SPLC’13 Workshops). ACM, New York, NY, 126–129.

Digital Library

[45]

Robert M. Hierons, Miqing Li, Xiaohui Liu, Jose Antonio Parejo, Sergio Segura, and Xin Yao. 2020. Many-objective test suite generation for software product lines. ACM Transactions on Software Engineering and Methodology 29, 1 (Jan. 2020), Article 2, 46 pages. DOI:

Digital Library

[46]

Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2011. Properties of realistic feature models make combinatorial testing of product lines feasible. In Proceedings of the 14th International Conference on Model Driven Engineering Languages and Systems (MODELS’11). 638–652.

Digital Library

[47]

Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2012. An algorithm for generating t-wise covering arrays from large feature models. In Proceedings of the 16th International Software Product Line Conference—Volume 1 (SPLC’12). ACM, New York, NY, 46–55.

Digital Library

[48]

Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2012. An algorithm for generating t-wise covering arrays from large feature models. In Proceedings of the 16th International Software Product Line Conference—Volume 1. ACM, New York, NY, 46–55.

Digital Library

[49]

Alexander Knüppel, Thomas Thüm, Stephan Mennicke, Jens Meinicke, and Ina Schaefer. 2017. Is there a mismatch between real-world feature models and product-line research? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’17). ACM, New York, NY, 291–302. DOI:

Digital Library

[50]

Sebastian Krieter, Thomas Thüm, Sandro Schulze, Gunter Saake, and Thomas Leich. 2020. YASA: Yet another sampling algorithm. In Proceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems (VaMoS’20). ACM, New York, NY, Article 4, 10 pages.

Digital Library

[51]

D. R. Kuhn, D. R. Wallace, and A. M. Gallo. 2004. Software fault interactions and implications for software testing. IEEE Transactions on Software Engineering 30, 6 (2004), 418–421.

Digital Library

[52]

Jihyun Lee, Sungwon Kang, and Pilsu Jung. 2020. Test coverage criteria for software product line testing: Systematic literature review. Information and Software Technology 122 (2020), 106272. DOI:

[53]

Jihyun Lee, Sungwon Kang, and Danhyung Lee. 2012. A survey on software product line testing. In Proceedings of the 16th International Software Product Line Conference—Volume 1 (SPLC’12). ACM, New York, NY, 31–40. DOI:

Digital Library

[54]

Joel Lehman and O. Stanley Kenneth. 2011. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation 19, 2 (2011), 189–223.

Digital Library

[55]

Joel Lehman and Kenneth O. Stanley. 2011. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation (GECCO’11). ACM, New York, NY, 211–218. DOI:

Digital Library

[56]

Bingdong Li, Jinlong Li, Ke Tang, and Xin Yao. 2015. Many-objective evolutionary algorithms: A survey. ACM Computing Surveys 48, 1 (Sept.2015), 1–35.

Digital Library

[57]

Miqing Li, Shengxiang Yang, and Xiaohui Liu. 2014. Shift-based density estimation for Pareto-based algorithms in many-objective optimization. IEEE Transactions on Evolutionary Computation 18, 3 (June2014), 348–365.

[58]

Antonios Liapis, Georgios N. Yannakakis, and Julian Togelius. 2015. Constrained novelty search: A study on game content generation. Evolutionary Computation 23, 1 (2015), 101–129. DOI:

Digital Library

[59]

Roberto E. Lopez-Herrejon, Francisco Chicano, Javier Ferrer, Alexander Egyed, and Enrique Alba. 2013. Multi-objective optimal test suite computation for software product line pairwise testing. In Proceedings of the IEEE International Conference on Software Maintenance. 404–407.

Digital Library

[60]

Roberto E. Lopez-Herrejon, Javier Ferrer, Francisco Chicano, Alexander Egyed, and Enrique Alba. 2014. Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of software product lines. In Proceedings of the 2014 IEEE Congress on Evolutionary Computation (CEC’14). 387–396. DOI:

[61]

Roberto E. Lopez-Herrejon, Javier Ferrer, Francisco Chicano, Alexander Egyed, and Enrique Alba. 2016. Evolutionary Computation for Software Product Line Testing: An Overview and Open Challenges. Springer International, Cham, Switzerland, 59–87. DOI:

[62]

Roberto E. Lopez-Herrejon, Stefan Fischer, Rudolf Ramler, and Alexander Egyed. 2015. A first systematic mapping study on combinatorial interaction testing for software product lines. In Proceedings of the IEEE 8th International Conference on Software Testing, Verification, and Validation Workshops (ICSTW’15). IEEE, Los Alamitos, CA, 1–10.

[63]

Roberto Erick Lopez-Herrejon, Javier Javier Ferrer, Francisco Chicano, Evelyn Nicole Haslinger, Alexander Egyed, and Enrique Alba. 2014. A parallel evolutionary algorithm for prioritized pairwise testing of software product lines. In Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation (GECCO’14). ACM, New York, NY, 1255–1262. DOI:

Digital Library

[64]

Roberto E. Lopez-Herrejon, Lukas Linsbauer, and Alexander Egyed. 2015. A systematic mapping study of search-based software engineering for software product lines. Information & Software Technology 61 (2015), 33–51.

Digital Library

[65]

Víctor R. López-López, Leonardo Trujillo, and Pierrick Legrand. 2018. Novelty search for software improvement of a SLAM system. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO’18). ACM, New York, NY, 1598–1605. DOI:

Digital Library

[66]

Chuan Luo, Binqi Sun, Bo Qiao, Junjie Chen, Hongyu Zhang, Jinkun Lin, Qingwei Lin, and Dongmei Zhang. 2021. LS-Sampling: An effective local search based sampling approach for achieving high t-wise coverage. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’21). ACM, New York, NY, 1081–1092.

Digital Library

[67]

Chuan Luo, Qiyuan Zhao, Shaowei Cai, and Hongyu Zhangand Chunming Hu. 2022. SamplingCA: Effective and efficient sampling-based pairwise testing for highly configurable software systems. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on theFoundations of Software Engineering (ESEC/FSE’22). ACM, New York, NY, 1–13.

[68]

Bogdan Marculescu, Robert Feldt, and Richard Torkar. 2016. Using exploration focused techniques to augment search-based software testing: An experimental evaluation. In Proceedings of the 2016 IEEE International Conference on Software Testing, Verification, and Validation (ICST’16). 69–79. DOI:

[69]

Urtzi Markiegi, Aitor Arrieta, Goiuria Sagardui, and Leire Etxeberria. 2017. Search-based product line fault detection allocating test cases iteratively. In Proceedings of the 21st International Systems and Software Product Line Conference—Volume A (SPLC’17). ACM, New York, NY, 123–132. DOI:

Digital Library

[70]

Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A comparison of 10 sampling algorithms for configurable systems. In Proceedings of the 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE’16). 643–654.

Digital Library

[71]

Marcilio Mendonca, Moises Branco, and Donald Cowan. 2009. SPLOT: Software product lines online tools. In Proceedings of the 24th ACM SIGPLAN Conference Companion on Object Oriented Programming Systems Languages and Applications. ACM, New York, NY, 761–762.

Digital Library

[72]

Jean-Baptiste Mouret and Jeff Clune. 2015. Illuminating search spaces by Mapping Elites. arXiv preprint arXiv:1504.04909 (2015).

[73]

Jean-Baptiste Mouret and Glenn Maguire. 2020. Quality diversity for multi-task optimization. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference (GECCO’20). ACM, New York, NY, 121–129. DOI:

Digital Library

[74]

Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. 2015. Innovation engines: Automated creativity and improved stochastic optimization via deep learning. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation (GECCO’15). ACM, New York, NY, 959–966. DOI:

Digital Library

[75]

Giuseppe Paolo, Alban Laflaquire, Alexandre Coninx, and Stephane Doncieux. 2020. Unsupervised learning and exploration of reachable outcome space. In Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA’20). 2379–2385. DOI:

[76]

Mike Papadakis, Christopher Henard, and Yves Le Traon. 2014. Sampling program inputs with mutation analysis: Going beyond combinatorial interaction testing. In Proceedings of the 2014 IEEE 7th International Conference on Software Testing, Verification, and Validation. 1–10. DOI:

Digital Library

[77]

José A. Parejo, Ana B. Sánchez, Sergio Segura, Antonio Ruiz-Cortés, Roberto E. Lopez-Herrejon, and Alexander Egyed. 2016. Multi-objective test case prioritization in highly configurable systems: A case study. Journal of Systems & Software 122, C (2016), 287–310.

Digital Library

[78]

Tobias Pett, Thomas Thüm, Tobias Runge, Sebastian Krieter, Malte Lochau, and Ina Schaefer. 2019. Product sampling for product lines: The scalability challenge. In Proceedings of the 23rd International Systems and Software Product Line Conference—Volume A (SPLC’19). ACM, New York, NY, 78–83.

Digital Library

[79]

Thomas Pierrot, Guillaume Richard, Karim Beguir, and Antoine Cully. 2022. Multi-objective quality diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference(GECCO’22). ACM, New York, NY, 139–147. DOI:

Digital Library

[80]

Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. 2016. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI 3 (2016), Article 40, 17 pages. DOI:

[81]

Justin K. Pugh, L. B. Soros, Paul A. Szerlip, and Kenneth O. Stanley. 2015. Confronting the challenge of quality diversity. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation (GECCO’15). ACM, New York, NY, 967–974. DOI:

Digital Library

[82]

Vincenzo Riccio and Paolo Tonella. 2020. Model-based exploration of the frontier of behaviours for deep learning system testing. In Proceedings of the 28th ACM Joint Meeting on the European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’20). ACM, New York, NY, 876–888. DOI:

Digital Library

[83]

José Raúl Romero, Aurora Ramírez, and Christopher L. Simons. 2019. Looking for novelty in SBSE problems. In Proceedings of the Spanish Conference on Software Engineering and Databases (JISBD’19). 1–4.

[84]

Ana B. Sánchez, Sergio Segura, José A. Parejo, and Antonio Ruiz-Cortés. 2017. Variability testing in the wild: The Drupal case study. Software & Systems Modeling 16, 1 (Feb. 2017), 173–194.

Digital Library

[85]

Ana B. Sánchez, Sergio Segura, and Antonio Ruiz-Cortés. 2014. A comparison of test case prioritization criteria for software product lines. In Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation (ICST’14). IEEE, Los Alamitos, CA, 41–50. DOI:

Digital Library

[86]

Sergio Segura, José A. Galindo, David Benavides, José A. Parejo, and Antonio Ruiz-Cortés. 2012. BeTTy: Benchmarking and testing on the automated analysis of feature models. In Proceedings of the 6th International Workshop on Variability Modeling of Software-Intensive Systems (VaMoS’12). ACM, New York, NY, 63–71. DOI:

Digital Library

[87]

Konstantinos Sfikas, Antonios Liapis, and Georgios N. Yannakakis. 2021. Monte Carlo elites: Quality-diversity selection as a multi-armed bandit problem. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’21). ACM, New York, NY, 180–188. DOI:

Digital Library

[88]

Steven She, Rafael Lotufo, Thorsten Berger, Andrzej Wasowski, and Krzysztof Czarnecki. 2011. Reverse engineering feature models. In Proceedings of the 33rd International Conference on Software Engineering (ICSE’11). ACM, New York, NY, 461–470. DOI:

Digital Library

[89]

Andrei Strickler, Jackson A. Prado Lima, Silvia R. Vergilio, and Aurora T. R. Pozo. 2016. Deriving products for variability test of feature models with a hyper-heuristic approach. Applied Soft Computing 49 (2016), 1232–1242. DOI:

Digital Library

[90]

Thomas Thüm, Christian Kästner, Fabian Benduhn, Jensand Meinicke, Gunter Saake, and Thomas Leich. 2014. FeatureIDE: An extensible framework for feature-oriented software development. Science of Computer Programming 79 (2014), 70–85. DOI:

Digital Library

[91]

A. Vargha and H. D. Delaney. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101–132.

[92]

Omar M. Villanueva, Leonardo Trujillo, and Daniel E. Hernandez. 2020. Novelty search for automatic bug repair. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference (GECCO’20). ACM, New York, NY, 1021–1028. DOI:

Digital Library

[93]

Shuai Wang, Shaukat Ali, and Arnaud Gotlieb. 2013. Minimizing test suites in software product lines using weight-based genetic algorithms. In Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation (GECCO’13). ACM, New York, NY, 1493–1500. DOI:

Digital Library

[94]

Shuai Wang, Shaukat Ali, and Arnaud Gotlieb. 2015. Cost-effective test suite minimization in product lines using search techniques. Journal of Systems and Software 103 (2015), 370–391. DOI:

Digital Library

[95]

Shuai Wang, David Buchmann, Shaukat Ali, Arnaud Gotlieb, Dipesh Pradhan, and Marius Liaaen. 2014. Multi-objective test prioritization in software product line testing: An industrial case study. In Proceedings of the 18th International Software Product Line Conference—Volume 1 (SPLC’14). ACM, New York, NY, 32–41. DOI:

Digital Library

[96]

Yi Xiang, Han Huang, Miqing Li, Sizhe Li, and Xiaowei Yang. 2022. Looking for novelty in search-based software product line testing. IEEE Transactions on Software Engineering 48, 7 (2022), 2317–2338. DOI:

[97]

Yi Xiang, Xiaowei Yang, Han Huang, Zhengxin Huang, and Miqing Li. 2022. Sampling configurations from software product lines via probability-aware diversification and SAT solving. Automated Software Engineering 29, 2 (2022), 54.

Digital Library

[98]

Zhihong Xu, Myra B. Cohen, Wayne Motycka, and Gregg Rothermel. 2013. Continuous test suite augmentation in software product lines. In Proceedings of the 17th International Software Product Line Conference (SPLC’13). ACM, New York, NY, 52–61. DOI:

Digital Library

[99]

Qingfu Zhang and Hui Li. 2007. MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation 11, 6 (2007), 712–731.

Digital Library

[100]

Yulun Zhang, Matthew C. Fontaine, Amy K. Hoover, and Stefanos Nikolaidis. 2022. Deep surrogate assisted MAP-elites for automated hearthstone deckbuilding. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’22). ACM, New York, NY, 158–167. DOI:

Digital Library

[101]

Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, and Paolo Tonella. 2021. DeepHyperion: Exploring the feature space of deep learning-based systems through illumination search. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’21). ACM, New York, NY, 79–90. DOI:

Digital Library

[102]

Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, and Paolo Tonella. 2023. Efficient and effective feature space exploration for testing deep learning systems. ACM Transactions on Software Engineering and Methodology 32, 2 (2023), Article 49, 38 pages. DOI:

Digital Library

Cited By

Elgendy IHierons RMcMinn P(2025)A Systematic Mapping Study of the Metrics, Uses and Subjects of Diversity‐Based Testing TechniquesSoftware Testing, Verification and Reliability10.1002/stvr.191435:2Online publication date: 17-Jan-2025
https://doi.org/10.1002/stvr.1914
Luo CSong JZhao QSun BChen JZhang HLin JHu C(2024)Solving the t-wise Coverage Maximum Problem via Effective and Efficient Local Search-based SamplingACM Transactions on Software Engineering and Methodology10.1145/3688836Online publication date: 16-Aug-2024
https://dl.acm.org/doi/10.1145/3688836
Luo CLyu SZhao QWu WZhang HHu CChristakis MPradel M(2024)Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable SystemsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680309(641-653)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680309

Index Terms

Automated Test Suite Generation for Software Product Lines Based on Quality-Diversity Optimization
1. Software and its engineering
  1. Software creation and management
    1. Search-based software engineering
    2. Software verification and validation
      1. Empirical software validation

Recommendations

Continuous test suite augmentation in software product lines
SPLC '13: Proceedings of the 17th International Software Product Line Conference

Software Product Line (SPL) engineering offers several advantages in the development of families of software products. There is still a need, however, to generate test cases for individual products in product lines more efficiently. In this paper we ...
Automated code-based test case reuse for software product line testing
Abstract Context
A software product line (SPL) grows in size as a new product is developed. A new product in an SPL should be tested extensively for quality assurance. For the efficient testing, previous studies suggested reusing the existing test cases ...
Model-based coverage-driven test suite generation for software product lines
MODELS'11: Proceedings of the 14th international conference on Model driven engineering languages and systems

Software Product Line (SPL) engineering is a popular approach for the systematic reuse of software artifacts across a large number of similar products. Unfortunately, testing each product of an SPL separately is often unfeasible. Consequently, SPL ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 33, Issue 2

February 2024

947 pages

EISSN:1557-7392

DOI:10.1145/3618077

Editor:
Mauro Pezzè
USI Universitá della Svizzera italiana and SIT Schaffhausen Institute of Technology, Switzerland

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 December 2023

Online AM: 16 October 2023

Accepted: 18 September 2023

Revised: 25 July 2023

Received: 20 November 2022

Published in TOSEM Volume 33, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
Science and Technology Program of Guangzhou
Guangdong Basic and Applied Basic Research Foundation
Guangdong Province Key Area R&D Program
Fundamental Research Funds for the Central Universities
Natural Science Research Project of Education Department of Guizhou Province

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
556
Total Downloads

Downloads (Last 12 months)350
Downloads (Last 6 weeks)28

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Elgendy IHierons RMcMinn P(2025)A Systematic Mapping Study of the Metrics, Uses and Subjects of Diversity‐Based Testing TechniquesSoftware Testing, Verification and Reliability10.1002/stvr.191435:2Online publication date: 17-Jan-2025
https://doi.org/10.1002/stvr.1914
Luo CSong JZhao QSun BChen JZhang HLin JHu C(2024)Solving the t-wise Coverage Maximum Problem via Effective and Efficient Local Search-based SamplingACM Transactions on Software Engineering and Methodology10.1145/3688836Online publication date: 16-Aug-2024
https://dl.acm.org/doi/10.1145/3688836
Luo CLyu SZhao QWu WZhang HHu CChristakis MPradel M(2024)Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable SystemsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680309(641-653)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680309

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents