skip to main content
10.1145/2648511.2648515acmotherconferencesArticle/Chapter ViewAbstractPublication PagessplcConference Proceedingsconference-collections
research-article

Multi-objective test prioritization in software product line testing: an industrial case study

Published: 15 September 2014 Publication History

Abstract

Test prioritization is crucial for testing products in a product line considering limited budget in terms of available time and resources. In general, it is not practically feasible to execute all the possible test cases and so, ordering test case execution permits test engineers to discover faults earlier in the testing process. An efficient prioritization of test cases for one or more products requires a clear consideration of the tradeoff among various costs (e.g., time, required resources) and effectiveness (e.g., feature coverage) objectives. As an integral part of the future Cisco's test scheduling system for validating video conferencing products, we introduce a search-based multi-objective test prioritization technique, considering multiple cost and effectiveness measures. In particular, our multi-objective optimization setup includes the minimization of execution cost (e.g., time), and the maximization of number of prioritized test cases, feature pairwise coverage and fault detection capability. Based on cost-effectiveness measures, a novel fitness function is defined for such test prioritization problem. The fitness function is empirically evaluated together with three commonly used search algorithms (e.g., (1+1) Evolutionary algorithm (EA)) and Random Search as a comparison baseline based on the Cisco's industrial case study and 500 artificial designed problems. The results show that (1+1) EA achieves the best performance for solving the test prioritization problem and it scales up to solve the problems of varying complexity.

References

[1]
McGregor, J. Testing a Software Product Line. 2001. Technical Report CMU/SEI-2001-TR-022, Software Engineering Institute, Carnegie Mellon University.
[2]
Benavides, D., Segura, S., and Cortés, A. R. Automated Analysis of Feature Models 20 Years Later: A Literature Review. 2010. Information Systems (35). pp. 615--636.
[3]
Software engineering institute: Hall of Fame. Available at: www.splc.net/fame.html
[4]
Wang, S., Gotlieb, A., Ali, S., and Liaaen, M. Automated Selection of Test Cases using Feature Model: An Industrial Case Study. 2013. Proceedings of the ACM International Conference of Model-Driven Engineering Languages and Systems (MODELS). pp. 237--253.
[5]
Harman, M., Mansouri, S. A., and Zhang, Y. Search Based Software Engineering: A Comprehensive Analysis and Review of Trends Techniques and Applications. 2009. Technical Report TR-09-03, King College London.
[6]
Konak, A., Coit, D. W., and Smith, A. E. 2006. Multi-objective optimization using genetic algorithms: A tutorial. Reliability Engineering & System Safety. (91). pp. 992--1007.
[7]
Ali, S., Briand, L. C., Hemmati, H., and Panesar-Walawege, R. K. A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation. 2010. IEEE Transactions on Software Engineering 36(6). pp. 742--762.
[8]
Wang, S., Ali, S., and Gotlieb, A. Minimizing Test Suites in Software Product Lines Using Weighted-based Genetic Algorithms. 2013. Proceedings of the Genetic and Evolutionary Computation Conference (GECCO). pp. 1493--1500.
[9]
Korel, B. Automated Software Test Data Generation. 1990. IEEE Transactions on Software Engineering. 16(8). pp. 870--879.
[10]
McMinn, P. Search-based software test data generation: a survey: Research Articles. 2004. Software Testing, Verification and Reliability. 14(2). pp. 105--156.
[11]
Droste, S., Jansen, T., and Wegener, I. On the analysis of the (1+1) evolutionary algorithm. 2002. Theoretical Computer Science 276 (1-2). pp. 51--81.
[12]
Arcuri, A. It really does matter how you normalize the branch distance in search-based software testing. 2013. Software Testing, Verification and Reliability 23(2). pp. 119--147.
[13]
Cisco Systems. TelePresence codec c90. 2010.
[14]
Marler, R. T. and Arora, J. S. Survey of multi-objective optimization methods for engineering. 2004. Structural and Multidisciplinary Optimization, 26(6). pp. 369--395.
[15]
Henard, C., Papadakis, M., Perrouin, G., Klein, J. and Traon Y. L. Multi-objective Test Generation for Software Product Lines. 2013. In Proceedings of Software Product Line Conference (SPLC). pp. 62--71.
[16]
Arcuri, A., and Briand, L. C. A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering. 2011. Proceedings of the International Conference on Software Engineering. pp. 21--28.
[17]
Sheskin, D. J. Handbook of Parametric and Nonparametric Statistical Procedures. 2003.
[18]
Arcuri, A., and Fraser, G. On Parameter Tuning in Search Based Software Engineering. 2011. Proceedings of the International Symposium on Search Based Software Engineering (SSBSE). pp. 33--47.
[19]
Yue, T., Briand, L. C., and Labiche, Y. Facilitating the transition from use case models to analysis models: Approach and experiments. 2013. ACM Transactions on Software Engineering and Methodology (TOSEM) 22(1).
[20]
Yoo, S., and Harman, M. Regression testing minimization, selection and prioritization: a survey. 2012. Software Testing, Verification and Reliability, 22(2). pp. 67--120.
[21]
Malishevsky, A., Rothermel, G., and Elbaum, S. Modeling the cost-benefits tradeoffs for regression testing techniques. 2002. Proceedings of the International Conference on Software Maintenance (ICSM). pp. 230--240.
[22]
Rothermel, G., Elbaum, S., Malishevsky, A., Kallakuri, P., and Davia, B. The impact of test suite granularity on the cost- effectiveness of regression testing. 2002. Proceedings of the 24th International Conference on Software Engineering (ICSE). pp. 130--140.
[23]
Rothermel, G., Untch, R. H., Chu, C., and Harrold, M. J. Test case prioritization: An empirical study. Proceedings of International Conference on Software Maintenance (ICSM 1999), pp. 179--188.
[24]
Tonella, P., Avesani, P., and Susi, A. Using the case-based ranking methodology for test case prioritization. 2006. Proceedings of the 22nd International Conference on Software Maintenance (ICSM) pp. 123--133.
[25]
Korel, B., Tahat, L., and Harman, M. Test prioritization using system models. 2005. Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM). pp. 559--568.
[26]
Korel, B., Koutsogiannakis, G., and Tahat, L. Application of system models in regression test suite prioritization. 2008. Proceedings of the IEEE International Conference on Software Maintenance (ICSM). pp. 247--256.
[27]
Elbaum, S. G, Malishevsky, A. G, and Rothermel, G. Incorporating varying test costs and fault severities into test case prioritization. 2001. Proceedings of the International Conference on Software Engineering (ICSE). pp. 329--338.
[28]
Do, H., Mirarab, S. M., Tahvildari, L., and Rothermel, G. An empirical study of the effect of time constraints on the cost-benefits of regression testing. 2008. Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). pp. 71--82.
[29]
Harman, M. Making the Case for MORTO: Multi Objective Regression Test Optimization. 2011. Proceedings of the Fourth International Conference on Software Testing, Verification and Validation Workshops. pp. 111--114.
[30]
Walcott, K. R., Soffa, M. L., Kapfhammer, G. M., and Roos, R. S. Time-Aware Test Suite Prioritization. 2006. Proceedings of the International Symposium on Software Testing and Analysis (ISSTA). pp.1--12.

Cited By

View all
  • (2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
  • (2023)Software Product Line Maintenance Using Multi-Objective Optimization TechniquesApplied Sciences10.3390/app1315901013:15(9010)Online publication date: 6-Aug-2023
  • (2023)Automated Test Suite Generation for Software Product Lines Based on Quality-Diversity OptimizationACM Transactions on Software Engineering and Methodology10.1145/362815833:2(1-52)Online publication date: 22-Dec-2023
  • Show More Cited By

Index Terms

  1. Multi-objective test prioritization in software product line testing: an industrial case study

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    SPLC '14: Proceedings of the 18th International Software Product Line Conference - Volume 1
    September 2014
    377 pages
    ISBN:9781450327404
    DOI:10.1145/2648511
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    • University of Florence: University of Florence
    • CNR: Istituto di Scienza e Tecnologie dell Informazione

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 15 September 2014

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. multi-objective optimization
    2. search algorithms
    3. software product lines
    4. test prioritization

    Qualifiers

    • Research-article

    Conference

    SPLC '14
    Sponsor:
    • University of Florence
    • CNR

    Acceptance Rates

    SPLC '14 Paper Acceptance Rate 36 of 97 submissions, 37%;
    Overall Acceptance Rate 167 of 463 submissions, 36%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)11
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 19 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
    • (2023)Software Product Line Maintenance Using Multi-Objective Optimization TechniquesApplied Sciences10.3390/app1315901013:15(9010)Online publication date: 6-Aug-2023
    • (2023)Automated Test Suite Generation for Software Product Lines Based on Quality-Diversity OptimizationACM Transactions on Software Engineering and Methodology10.1145/362815833:2(1-52)Online publication date: 22-Dec-2023
    • (2023)Some Seeds Are Strong: Seeding Strategies for Search-based Test Case SelectionACM Transactions on Software Engineering and Methodology10.1145/353218232:1(1-47)Online publication date: 13-Feb-2023
    • (2023)A Case Study on the “Jungle” Search for Industry-Relevant Regression Testing2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS)10.1109/QRS60937.2023.00045(382-393)Online publication date: 22-Oct-2023
    • (2023)A Systematic Literature Review on Test Case Prioritization TechniquesAgile Software Development10.1002/9781119896838.ch7(101-159)Online publication date: 8-Feb-2023
    • (2022)Reducing Redundant Test Executions in Software Product Line Testing—A Case StudyElectronics10.3390/electronics1107116511:7(1165)Online publication date: 6-Apr-2022
    • (2021)Systematic Literature Review on Test Case Selection and Prioritization: A Tertiary StudyApplied Sciences10.3390/app11241212111:24(12121)Online publication date: 20-Dec-2021
    • (2021)A Tag-based Recommender System for Regression Test Case Prioritization2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW52544.2021.00035(146-157)Online publication date: Apr-2021
    • (2021)Dynamic test prioritization of product lines: An application on configurable simulation modelsSoftware Quality Journal10.1007/s11219-021-09571-0Online publication date: 20-Oct-2021
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media