skip to main content
10.1145/3297280.3297512acmconferencesArticle/Chapter ViewAbstractPublication PagessacConference Proceedingsconference-collections
research-article

Test case selection using structural coverage in software product lines for time-budget constrained scenarios

Published: 08 April 2019 Publication History

Abstract

Testing product lines is a challenging activity due to the large number of products to be tested. Many approaches focus on reducing the time for testing a product line by reducing the number of products to be tested, by employing, for instance, combinatorial approaches. However, even if the number of derived products by a combinatorial approach is limited, testing can still be time consuming. In this paper, we propose three different test case selection methods that consider a given time budget to test product lines in an efficient manner using structural coverage information. We analyze the three methods with three white-box coverage criteria (i.e., Decision Coverage, Condition Coverage and Modified Condition/Decision Coverage). We evaluate the different approaches with a case study from the automotive domain and mutation testing. The results suggest that considering coverage information at the domain engineering level helps on detecting more faults, particularly when time budgets are low.

References

[1]
M. Al-Hajjaji, S. Lity, R. Lachmann, T. Thüm, I. Schaefer, and G. Saake. 2017. Delta-oriented product prioritization for similarity-based product-line testing. In VACE '17. 34--40.
[2]
M. Al-Hajjaji, T. Thüm, M. Lochau, J. Meinicke, and G. Saake. 2016. Effective product-line testing using similarity-based product prioritization. Software & Systems Modeling (2016), 1--23.
[3]
M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake. 2014. Similarity-based Prioritization in Software Product-line Testing. In SPLC '14. 197--206.
[4]
P. Ammann and J. Offutt. 2008. Introduction to Software Testing.
[5]
A. Arrieta, S. Wang, A. Arruabarrena, U. Markiegi, G. Sagardui, and L. Etxeberria. 2018. Multi-Objective Black-Box Test Case Selection for Cost-Effectively Testing Simulation Models. In GECCO '18. 1411--1418.
[6]
A. Arrieta, S. Wang, G. Sagardui, and L. Etxeberria. 2016. Search-based Test Case Selection of Cyber-physical System Product Lines for Simulation-based Validation. In SPLC '16. 297--306.
[7]
A. Arrieta, S. Wang, G. Sagardui, and L. Etxeberria. 2016. Test Case Prioritization of Configurable Cyber-Physical Systems with Weight-Based Search Algorithms. In GECCO '16. 1053--1060.
[8]
D. Benavides, S. Segura, and A. Ruiz-Cortés. 2010. Automated analysis of feature models 20 years later: A literature review. Information Systems 35 (2010), 615 -- 636.
[9]
T. Y. Chen, F.-C. Kuo, R. G. Merkel, and T. H. Tse. 2010. Adaptive Random Testing: The ART of test case diversity. Journal of Systems and Software 83 (2010), 60 -- 66.
[10]
M. B. Cohen, M. B. Dwyer, and J. Shi. 2008. Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach. IEEE Trans. Softw. Eng. 34 (2008), 633--650.
[11]
X. Devroey, G. Perrouin, M. Cordy, H. Samih, A. Legay, P.-Y.s Schobbens, and P. Heymans. 2015. Statistical prioritization for software product line testing: an experience report. Software & Systems Modeling (2015), 1--19.
[12]
X. Devroey, G. Perrouin, A. Legay, M. Cordy, P.-Y. Schobbens, and P. Heymans. 2014. Coverage Criteria for Behavioural Testing of Software Product Lines. In Leveraging Appl. Form. Methods, Verif. Validation. Technol. Mastering Chang. 336--350.
[13]
I. do Carmo Machado, J. Mcgregor, Y. C. Cavalcanti, and E. S. De Almeida. 2014. On strategies for testing software product lines: A systematic literature review. Information and Software Technology 56 (2014), 1183--1199.
[14]
E. Engström and P. Runeson. 2011. Software product line testing - A systematic mapping study. Information and Software Technology 53 (2011), 2--13.
[15]
F. Ensan, E. Bagheri, and D. Gašević. 2012. Evolutionary Search-based Test Generation for Software Product Line Feature Models. In CAISE '12. 613--628.
[16]
L. T. M. Hanh, N. T. Binh, and K. T. Tung. 2016. A Novel Fitness function of metaheuristic algorithms for test data generation for simulink models based on mutation analysis. Journal of Systems and Software 120 (2016), 17--30.
[17]
K. J. Hayhurst, D. S. Veerhusen, J. J. Chilenski, and L. K. Rierson. 2001. A practical tutorial on modified condition/decision coverage. Technical Report. NASA.
[18]
C. Henard, M. Papadakis, M. Harmany, and Y. L. Traon. 2015. Combining Multi-Objective Search and Constraint Solving for Configuring Large Scale Software Product Lines. In ICSE'15. 517--528.
[19]
C. Henard, M. Papadakis, G. Perrouin, J. Klein, and Y. L. Traon. 2013. Assessing software product line testing via model-based mutation: An application to similarity testing. In ICSTW '13. 188--197.
[20]
C. Henard, M. Papadakis, G. Perrouin, J. Klein, and Y. L. Traon. 2013. Multi-objective Test Generation for Software Product Lines. In SPLC '13. 62--71.
[21]
M. F. Johansen, Ø. Haugen, and F. Fleurey. 2012. An Algorithm for Generating T-wise Covering Arrays from Large Feature Models. In SPLC '12. 46--55.
[22]
R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In FSE '14. 654--665.
[23]
C. H. P. Kim, D. S. Batory, and S. Khurshid. 2011. Reducing combinatorics in testing product lines. In AOSD '11. 57--68.
[24]
R. Lachmann, M. Felderer, M. Nieke, S. Schulze, C. Seidl, and I. Schaefer. 2017. Multi-objective black-box test case selection for system testing. In GECCO '17. ACM, 1311--1318.
[25]
R. Lachmann, S. Lity, S. Lischke, S. Beddig, S. Schulze, and I. Schaefer. 2015. Delta-oriented Test Case Prioritization for Integration Testing of Software Product Lines. In SPLC '15. 81--90.
[26]
R. E. Lopez-Herrejon, S. Fischer, R. Ramler, and A. Egyed. 2015. A first systematic mapping study on combinatorial interaction testing for software product lines. In ICSTW '15. 1--10.
[27]
U. Markiegi, A. Arrieta, G. Sagardui, and L. Etxeberria. 2017. Search-based Product Line Fault Detection Allocating Test Cases Iteratively. In SPLC '17. 123--132.
[28]
M. Marré and A. Bertolino. 2003. Using spanning sets for coverage testing. IEEE Transactions on Software Engineering 29 (2003), 974--984.
[29]
R. Matinnejad, S. Nejati, L. C. Briand, and T. Bruckmann. 2016. Automated Test Suite Generation for Time-continuous Simulink Models. In ICSE '16. 595--606.
[30]
J. C. Miller and C. J. Maloney. 1963. Systematic mistake analysis of digital computer programs. Commun. ACM 6 (1963), 58--63.
[31]
S. Oster, F. Markert, and P. Ritter. 2010. Automated Incremental Pairwise Testing of Software Product Lines. In SPLC '10. 196--210.
[32]
M. Papadakis, Y. Jia, M. Harman, and Y. L. Traon. 2015. Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In ICSE '15. 936--946.
[33]
J. A. Parejo, A. B. Sánchez, S. Segura, A. Ruiz-Cortés, R. E. Lopez-Herrejon, and A. Egyed. 2016. Multi-Objective Test Case Prioritization in Highly Configurable Systems: A Case Study. Journal of Systems and Software (2016), -.
[34]
G. Perrouin, S. Oster, S. Sen, J. Klein, B. Baudry, and Y. L. Traon. 2012. Pairwise testing for software product lines: Comparison of two approaches. Software Quality Journal 20 (2012), 605--643.
[35]
A. Rajan, M. Whalen, M. Staats, and M. Heimdahl. 2008. Requirements coverage as an adequacy measure for conformance testing. Formal Methods and Software Engineering (2008), 86--104.
[36]
A. B. Sánchez, S. Segura, and A. Ruiz-Cortés. 2014. A Comparison of Test Case Prioritization Criteria for Software Product Lines. In ICST '14. 41--50.
[37]
V. Stricker, A. Metzger, and K. Pohl. 2010. Avoiding redundant testing in application engineering. In SPLC '10. 226--240.
[38]
M. Utting and B. Legeard. 2010. Practical model-based testing: a tools approach. Elsevier.
[39]
M. Utting, A. Pretschner, and B. Legeard. 2012. A taxonomy of model-based testing approaches. Software Testing, Verification and Reliability 22 (2012), 297--312.
[40]
S. Wang, S. Ali, and A. Gotlieb. 2013. Minimizing test suites in software product lines using weight-based genetic algorithms. In GECCO '13. 1493 -- 1500.
[41]
S. Wang, S. Ali, and A. Gotlieb. 2015. Cost-effective test suite minimization in product lines using search techniques. Journal of Systems and Software 103 (2015), 370 -- 391.
[42]
S. Wang, S. Ali, A. Gotlieb, and M. Liaaen. 2016. A systematic test case selection methodology for product lines: results and insights from an industrial case study. Empirical Software Engineering 21 (2016), 1--37.
[43]
S. Wang, D. Buchmann, S. Ali, A. Gotlieb, D. Pradhan, and M. Liaaen. 2014. Multi-objective Test Prioritization in Software Product Line Testing: An Industrial Case Study. In SPLC '14. 32--41.
[44]
S. Weißleder and H. Lackner. 2013. Top-Down and Bottom-Up Approach for Model-Based Testing of Product Lines. In MBT. 82--94.
[45]
H. Wu, C. Nie, and F.-C. Kuo. 2014. Test Suite Prioritization by Switching Cost. In IWCT '14.
[46]
S. Yoo and M. Harman. 2007. Pareto efficient multi-objective test case selection. In ISSTA '07. 140--150.
[47]
S. Yoo and M. Harman. 2012. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability 22 (2012), 67--120.

Cited By

View all
  • (2024)Towards Efficient Build Ordering for Incremental Builds with Multiple ConfigurationsProceedings of the ACM on Software Engineering10.1145/36607741:FSE(1494-1517)Online publication date: 12-Jul-2024
  • (2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
  • (2022)Reducing Redundant Test Executions in Software Product Line Testing—A Case StudyElectronics10.3390/electronics1107116511:7(1165)Online publication date: 6-Apr-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SAC '19: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing
April 2019
2682 pages
ISBN:9781450359337
DOI:10.1145/3297280
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 April 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. product line testing
  2. structural coverage
  3. test case selection

Qualifiers

  • Research-article

Funding Sources

  • Eusko Jaurlaritza: Ikerketa Taldeak -

Conference

SAC '19
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25
The 40th ACM/SIGAPP Symposium on Applied Computing
March 31 - April 4, 2025
Catania , Italy

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)5
  • Downloads (Last 6 weeks)1
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Towards Efficient Build Ordering for Incremental Builds with Multiple ConfigurationsProceedings of the ACM on Software Engineering10.1145/36607741:FSE(1494-1517)Online publication date: 12-Jul-2024
  • (2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
  • (2022)Reducing Redundant Test Executions in Software Product Line Testing—A Case StudyElectronics10.3390/electronics1107116511:7(1165)Online publication date: 6-Apr-2022
  • (2022)Combining Genetic Programming and Model Checking to Generate Environment AssumptionsIEEE Transactions on Software Engineering10.1109/TSE.2021.310181848:9(3664-3685)Online publication date: 1-Sep-2022
  • (2021)Dynamic test prioritization of product lines: An application on configurable simulation modelsSoftware Quality Journal10.1007/s11219-021-09571-029:4(943-988)Online publication date: 1-Dec-2021
  • (2019)White-Box and Black-Box Test Quality Metrics for Configurable Simulation ModelsProceedings of the 23rd International Systems and Software Product Line Conference - Volume B10.1145/3307630.3342396(211-214)Online publication date: 9-Sep-2019

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media