skip to main content
10.1145/3106195.3106210acmotherconferencesArticle/Chapter ViewAbstractPublication PagessplcConference Proceedingsconference-collections
research-article

Search-based product line fault detection allocating test cases iteratively

Published: 25 September 2017 Publication History

Abstract

The large number of possible configurations makes it unfeasible to test every single system variant in a product line. Consequently, a small subset of the product line products must be selected, typically following combinatorial interaction testing approaches. Recently, many product line engineering approaches have considered the selection and prioritization of relevant products within the product line. In a further step, these products are thoroughly tested individually. However, the test cases that must be executed in each of the products are not always insignificant, and in systems such as Cyber-Physical System Product Lines (CPSPLs), their test execution time can vary from tens to thousands of seconds. This issue leads to spending a lot of time testing each individual product. To solve this problem we propose a search-based approach to perform the testing of product lines by allocating small number of test cases in each of the products. This approach increases the probability of detecting faults faster. Specifically, our search-based approach obtains a set of products, which are derived from using any state-of-the-art approach as inputs, and a set of attributed test cases. As an output a list of allocated test cases for each product is obtained. We also define a novel fitness function to guide the search and we propose corresponding crossover and mutation operators. The search and test process is iteratively repeated until the time budget is consumed. We performed an evaluation with a CPSPL as a case study. Results suggest that our approach can reduce the fault detection time by 61% and 65% on average when compared with the traditional test process and the Random Search algorithm respectively.

References

[1]
Mustafa Al-Hajjaji, Thomas Thüm, Malte Lochau, Jens Meinicke, and Gunter Saake. 2016. Effective product-line testing using similarity-based product prioritization. Software & Systems Modeling (2016), 1--23.
[2]
Mustafa Al-Hajjaji, Thomas Thüm, Jens Meinicke, Malte Lochau, and Gunter Saake. 2014. Similarity-based Prioritization in Software Product-line Testing. In Proceedings of the 18th International Software Product Line Conference - Volume 1 (SPLC '14). 197--206.
[3]
Andrea Arcuri and Lionel Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Software Engineering (ICSE), 2011 33rd International Conference on. IEEE, 1--10.
[4]
Andrea Arcuri and Gordon Fraser. 2011. On Parameter Tuning in Search Based Software Engineering. Springer-Verlag, 33--47.
[5]
Aitor Arrieta, Urtzi Markiegi, and Leire Etxeberria. 2017. Towards Mutation Testing of Configurable Simulink Models: a Product Line Engineering Perspective. In Jornadas de Ingeniera del Software y Bases de Datos (JISBD).
[6]
Aitor Arrieta, Goiuria Sagardui, Leire Etxeberria, and Justyna Zander. 2016. Automatic generation of test system instances for configurable cyber-physical systems. Software Quality Journal (2016), 1--43.
[7]
Aitor Arrieta, Shuai Wang, Urtzi Markiegi, Goiuria Sagardui, and Leire Etxeberria. 2017. Search-Based Test Case Generation for Cyber-Physical Systems. In 2017 IEEE Congress on Evolutionary Computation (CEC). 688--697.
[8]
Aitor Arrieta, Shuai Wang, Goiuria Sagardui, and Leire Etxeberria. 2016. Search-based Test Case Selection of Cyber-physical System Product Lines for Simulation-based Validation. In Proceedings of the 20th International Systems and Software Product Line Conference (SPLC '16). 297--306.
[9]
Aitor Arrieta, Shuai Wang, Goiuria Sagardui, and Leire Etxeberria. 2016. Test Case Prioritization of Configurable Cyber-Physical Systems with Weight-Based Search Algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference 2016 (GECCO '16). 1053--1060.
[10]
David Benavides, Sergio Segura, and Antonio Ruiz-Cortés. 2010. Automated Analysis of Feature Models 20 Years Later: A Literature Review. Inf. Syst. 35, 6 (2010), 615--636.
[11]
Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. 2008. Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach. IEEE Trans. Softw. Eng. 34, 5 (2008), 633--650.
[12]
Paulo Anselmo da Mota Silveira Neto, Ivan do Carmo Machado, John D. McGregor, Eduardo Santana de Almeida, and Silvio Romero de Lemos Meira. 2011. A systematic mapping study of software product lines testing. Information & Software Technology 53, 5 (2011), 407--423.
[13]
Ivan do Carmo Machado, John D. McGregor, Yguaratfl Cerqueira Cavalcanti, and Eduardo Santana de Almeida. 2014. On strategies for testing software product lines: A systematic literature review. Information and Software Technology 56, 10 (2014), 1183--1199.
[14]
Emelie Engström and Per Runeson. 2011. Software product line testing - A systematic mapping study. Information and Software Technology 53, 1 (2011), 2--13.
[15]
Mark Harman, Yue Jia, Jens Krinke, William B. Langdon, Justyna Petke, and Yuanyuan Zhang. 2014. Search Based Software Engineering for Software Product Line Engineering: A Survey and Directions for Future Work. In Proceedings of the 18th International Software Product Line Conference - Volume 1 (SPLC '14). 5--18.
[16]
Christopher Henard, Mike Papadakis, Gilles Perrouin, Klein, Patrick Heymans, and Yves Le Traon. 2014. Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines. IEEE Transactions on Software Engineering 40, 7 (2014), 650--670.
[17]
Robert M. Hierons, Miqing Li, Xiaohui Liu, Sergio Segura, and Wei Zheng. 2016. SIP: Optimal Product Selection from Feature Models Using Many-Objective Evolutionary Optimization. ACM Trans. Softw. Eng. Methodol. 25, 2 (2016), 17:1--17:39.
[18]
Martin-Fagereng Johansen, Oystein Haugen, and Franck Fleurey. 2012. An Algorithm for Generating T-wise Covering Arrays from Large Feature Models (SPLC '12). 46--55.
[19]
René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing? (SIGSOFT '14). ACM, 654--665.
[20]
Remo Lachmann, Simon Beddig, Sascha Lity, Sandro Schulze, and Ina Schaefer. 2017. Risk-based Integration Testing of Software Product Lines (VAMOS '17). 52--59.
[21]
Sascha Lity, Mustafa Al-Hajjaji, Thomas Thüm, and Ina Schaefer. 2017. Optimizing Product Orders Using Graph Algorithms for Improving Incremental Product-line Analysis (VAMOS '17). 60--67.
[22]
Roberto E Lopez-Herrejon, Javier Ferrer, Francisco Chicano, Alexander Egyed, and Enrique Alba. 2016. Evolutionary computation for software product line testing: an overview and open challenges. In Computational Intelligence and Qantitative Software Engineering. Springer International Publishing, 59--87.
[23]
Roberto E Lopez-Herrejon, Javier Ferrer, Francisco Chicano, Lukas Linsbauer, Alexander Egyed, and Enrique Alba. 2014. A hitchhiker's guide to search-based software engineering for software product lines. Computer Research Repositori (CoRR) (2014).
[24]
Roberto Erick Lopez-Herrejon, Stefan Fischer, Rudolf Ramler, and Alexander Egyed. 2015. A first systematic mapping study on combinatorial interaction testing for software product lines. In Eighth IEEE International Conference on Software Testing, Verification and Validation, ICST 2015 Workshops, Graz, Austria, April 13-17, 2015. 1--10.
[25]
Roberto E. Lopez-Herrejon, Lukas Linsbauer, and Alexander Egyed. 2015. A systematic mapping study of search-based software engineering for software product lines. Information and Software Technology 61 (2015), 33--51.
[26]
Dusica Marijan, Arnaud Gotlieb, Sagar Sen, and Aymeric Hervieu. 2013. Practical Pairwise Testing for Software Product Lines (SPLC '13). 227--235.
[27]
Reza Matinnejad, Shiva Nejati, Lionel C. Briand, and Thomas Bruckmann. 2016. Automated Test Suite Generation for Time-continuous Simulink Models (ICSE '16). 595--606.
[28]
Pieter J. Mosterman, David Escobar Sanabria, Enes Bilgin, Kun Zhang, and Justyna Zander. 2014. Automating humanitarian missions with a heterogeneous fleet of vehicles. Annual Reviews in Control 38, 2 (2014), 259--270.
[29]
Sebastian Oster, Florian Markert, and Philipp Ritter. 2010. Automated Incremental Pairwise Testing of Software Product Lines (SPLC '10). 196--210.
[30]
Sebastian Oster, Ivan Zorcic, Florian Markert, and Malte Lochau. 2011. MoSo-PoLiTe - Tool Support for Pairwise and Model-Based Software Product Line Testing (VaMoS '11). 79--82.
[31]
Mike Papadakis, Christopher Henard, Mark Harman, Yue Jia, and Yves Le Traon. 2016. Threats to the Validity of Mutation-based Test Assessment (ISSTA 2016). 354--365.
[32]
José A. Parejo, Ana B Sánchez, Sergio Segura, Antonio Ruiz-Cortés, Roberto E. Lopez-Herrejon, and Alexander Egyed. 2016. Multi-Objective Test Case Prioritization in Highly Configurable Systems: A Case Study. Journal of Systems and Software (2016), 287--310.
[33]
Gilles Perrouin, Sebastian Oster, Sagar Sen, Jacques Klein, Benoit Baudry, and Yves Traon. 2012. Pairwise Testing for Software Product Lines: Comparison of Two Approaches. Software Quality Journal 20, 3--4 (2012), 605--643.
[34]
Gilles Perrouin, Sagar Sen, Jacques Klein, Benoit Baudry, and Yves Le Traon. 2010. Automated and scalable t-wise test case generation strategies for software product lines (ICST '10). IEEE, 459--468.
[35]
K. Pohl, G. Bckle, and F. van der Linden. 2005. Software Product Line Engineering: Foundations, Principles, and Techniques. Springer-Verlag.
[36]
Ana B Sánchez, Sergio Segura, José A Parejo, and Antonio Ruiz-Cortés. 2015. Variability testing in the wild: the Drupal case study. Software & Systems Modeling (2015), 1--22.
[37]
Ana B. Sánchez, S. Segura, and Antonio Ruiz-Cortés. 2014. A Comparison of Test Case Prioritization Criteria for Software Product Lines. In IEEE International Conference on Software Testing, Verification, and Validation. 41--50.
[38]
Vanessa Stricker, Andreas Metzger, and Klaus Pohl. 2010. Avoiding redundant testing in application engineering. In Software Product Lines: Going Beyond. Springer, 226--240.
[39]
Thomas Thüm, Sven Apel, Christian Kästner, Ina Schaefer, and Gunter Saake. 2014. A classification and survey of analysis strategies for software product lines. ACM Computing Surveys (CSUR) 47, 1 (2014), 6.
[40]
Shuai Wang, Shaukat Ali, and Arnaud Gotlieb. 2013. Minimizing test suites in software product lines using weight-based genetic algorithms (GECCO '13). 1493--1500.
[41]
Shuai Wang, Shaukat Ali, and Arnaud Gotlieb. 2015. Cost-effective test suite minimization in product lines using search techniques. Journal of Systems and Software 103 (2015), 370--391.
[42]
Shuai Wang, Shaukat Ali, Arnaud Gotlieb, and Marius Liaaen. 2015. Automated product line test case selection: industrial case study and controlled experiment. Software & Systems Modeling (2015), 1--25.
[43]
ShuaiWang, Shaukat Ali, Arnaud Gotlieb, and Marius Liaaen. 2016. A systematic test case selection methodology for product lines: results and insights from an industrial case study. Empirical Software Engineering 21, 4 (2016), 1586--1622.
[44]
Shuai Wang, David Buchmann, Shaukat Ali, Arnaud Gotlieb, Dipesh Pradhan, and Marius Liaaen. 2014. Multi-objective Test Prioritization in Software Product Line Testing: An Industrial Case Study. In Proceedings of the 18th International Software Product Line Conference - Volume 1 (SPLC '14). 32--41.

Cited By

View all
  • (2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
  • (2023)Automated Test Suite Generation for Software Product Lines Based on Quality-Diversity OptimizationACM Transactions on Software Engineering and Methodology10.1145/362815833:2(1-52)Online publication date: 22-Dec-2023
  • (2023)Search-based Test Case Selection for PLC Systems using Functional Block Diagram Programs2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00040(228-239)Online publication date: 9-Oct-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
SPLC '17: Proceedings of the 21st International Systems and Software Product Line Conference - Volume A
September 2017
253 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

  • Fidetia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 September 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Fault Detection
  2. Product Line Testing
  3. Search-based Software Engineering

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SPLC '17

Acceptance Rates

Overall Acceptance Rate 167 of 463 submissions, 36%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)6
  • Downloads (Last 6 weeks)1
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
  • (2023)Automated Test Suite Generation for Software Product Lines Based on Quality-Diversity OptimizationACM Transactions on Software Engineering and Methodology10.1145/362815833:2(1-52)Online publication date: 22-Dec-2023
  • (2023)Search-based Test Case Selection for PLC Systems using Functional Block Diagram Programs2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00040(228-239)Online publication date: 9-Oct-2023
  • (2022)Reducing Redundant Test Executions in Software Product Line Testing—A Case StudyElectronics10.3390/electronics1107116511:7(1165)Online publication date: 6-Apr-2022
  • (2022)Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithmsJournal of Software: Evolution and Process10.1002/smr.246534:11Online publication date: 25-May-2022
  • (2021)Dynamic test prioritization of product lines: An application on configurable simulation modelsSoftware Quality Journal10.1007/s11219-021-09571-0Online publication date: 20-Oct-2021
  • (2021)Survey on test case generation, selection and prioritization for cyber‐physical systemsSoftware Testing, Verification and Reliability10.1002/stvr.179432:1Online publication date: 15-Sep-2021
  • (2019)Test case selection using structural coverage in software product lines for time-budget constrained scenariosProceedings of the 34th ACM/SIGAPP Symposium on Applied Computing10.1145/3297280.3297512(2362-2371)Online publication date: 8-Apr-2019
  • (2019)Pareto efficient multi-objective black-box test case selection for simulation-based testingInformation and Software Technology10.1016/j.infsof.2019.06.009114:C(137-154)Online publication date: 1-Oct-2019
  • (2017)Test optimisation for Highly-Configurable Cyber-Physical SystemsProceedings of the 21st International Systems and Software Product Line Conference - Volume B10.1145/3109729.3109745(139-144)Online publication date: 25-Sep-2017

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media