skip to main content
10.1145/2908812.2908850acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
research-article

Search-Based Cost-Effective Test Case Selection within a Time Budget: An Empirical Study

Published: 20 July 2016 Publication History

Abstract

Due to limited time and resources available for execution, test case selection always remains crucial for cost-effective testing. It is even more prominent when test cases require manual steps, e.g., operating physical equipment. Thus, test case selection must consider complicated trade-offs between cost (e.g., execution time) and effectiveness (e.g., fault detection capability). Based on our industrial collaboration within the Maritime domain, we identified a real-world and multi-objective test case selection problem in the context of robustness testing, where test case execution requires human involvement in certain steps, such as turning on the power supply to a device. The high-level goal is to select test cases for execution within a given time budget, where test engineers provide weights for a set of objectives, depending on testing requirements, standards, and regulations. To address the identified test case selection problem, we defined a fitness function including one cost measure, i.e., Time Difference (TD) and three effectiveness measures, i.e., Mean Priority (MPR), Mean Probability (MPO) and Mean Consequence (MC) that were identified together with test engineers. We further empirically evaluated eight multi-objective search algorithms, which include three weight-based search algorithms (e.g., Alternating Variable Method) and five Pareto-based search algorithms (e.g., Strength Pareto Evolutionary Algorithm 2 (SPEA2)) using two weight assignment strategies (WASs). Notice that Random Search (RS) was used as a comparison baseline. We conducted two sets of empirical evaluations: 1) Using a real world case study that was developed based on our industrial collaboration; 2) Simulating the real world case study to a larger scale to assess the scalability of the search algorithms. Results show that SPEA2 with either of the WASs performed the best for both the studies. Overall, SPEA2 managed to improve on average 32.7%, 39% and 33% in terms of MPR, MPO and MC respectively as compared to RS.

References

[1]
Elbaum, S., Malishevsky, A. G. and Rothermel, G. 2002. Test case prioritization: A family of empirical studies. IEEE Transactions on Software Engineering, 28 (2): 159--182, 2002.
[2]
Harman, M., Mansouri, S. A. and Zhang, Y. 2009. Search based software engineering: A comprehensive analysis and review of trends techniques and applications. Department of Computer Science, King's College London.
[3]
Konak, A., Coit, D. W. and Smith, A. E. 2006. Multi-objective optimization using genetic algorithms: A tutorial. Reliability Engineering & System Safety, 91 (9): 992--1007, 2006.
[4]
Korel, B. 1990. Automated software test data generation. IEEE Transactions on Software Engineering, 16 (8): 870--879, 1990.
[5]
Droste, S., Jansen, T. and Wegener, I. 2002. On the analysis of the (1+ 1) evolutionary algorithm. Theoretical Computer Science, 276 (1): 51--81.
[6]
Coello, C. A. C., Van Veldhuizen, D. A. and Lamont, G. B. 2002. Evolutionary algorithms for solving multi-objective problems.
[7]
Deb, K. 2001. Multi-objective optimization using evolutionary algorithms.
[8]
Deb, K., Pratap, A., Agarwal, S. and Meyarivan, T. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutioary Computation, 6 (2): 182--197, 2002.
[9]
Nebro, A. J., Durillo, J. J., Luna, F., Dorronsoro, B. and Alba, E. 2007. Design issues in a multiobjective cellular genetic algorithm. Evolutionary multi-criterion optimization.
[10]
Zitzler, E., Laumanns, M., Thiele, L., Zitzler, E., Zitzler, E., Thiele, L. and Thiele, L. 2001. SPEA2: Improving the strength Pareto evolutionary algorithm. Eidgenössische Technische Hochschule Zürich (ETH).
[11]
Durillo, J. J., Nebro, A. J., Luna, F. and Alba, E. 2008. Solving three-objective optimization problems using a new hybrid cellular genetic algorithm. Parallel problem solving from Nature-PPSN X.
[12]
Zitzler, E. and Künzli, S. 2004. Indicator-based selection in multiobjective search. Parallel Problem Solving from Nature-PPSN VIII.
[13]
Zitzler, E. and Thiele, L. 1999. Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach. IEEE transactions on Evolutionary Computation, 3 (4): 257--271, 1999.
[14]
Ali, S. and Yue, T. 2014. Evaluating Normalization Functions with Search Algorithms for Solving OCL Constraints. Testing Software and Systems.
[15]
OREDA Offshore Reliability Data Handbook 2002, 4th edition Høvik, Norway : OREDA Participants : Distributed by Der Norske Veritas.
[16]
Wang, S., Ali, S., Yue, T., Li, Y. and Liaaen, M. 2016. A Practical Guide to Select Quality Indicators for Assessing Pareto-based Search Algorithms in Search-Based Software Engineering. In Proceedings of the 38th International Conference on Software Engineering (ICSE), 2016.
[17]
Vargha, A. and Delaney, H. D. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25 (2): 101--132, 2000.
[18]
Mann, H. B. and Whitney, D. R. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, 50--60.
[19]
Arcuri, A. and Briand, L. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In the 33rd International Conference on Software Engineering (ICSE), 2011.
[20]
Durillo, J. J. and Nebro, A. J. 2011. jMetal: A Java framework for multi-objective optimization. Advances in Engineering Software,42(10): 760--771.
[21]
Results of the experiments available publicly in website. http://zen-tools.com/conference/GECCO2016.html.
[22]
The Abel computer cluster. http://www.uio.no/english/services/it/research/hpc/abel/.
[23]
De Oliveira Barros, M. and Neto, A. 2011. Threats to Validity in Search-based Software Engineering Empirical Studies. UNIRIO-Universidade Federal do Estado do Rio de Janeiro, techreport 6.
[24]
Arcuri, A. and Fraser, G. 2011. On parameter tuning in search based software engineering. Search Based Software Engineering, 2011: 33--47.
[25]
Statistical Sciences, I. 1995. S-PLUS Guide to Statistical and Mathematical Analysis, Version 3.3. StatSci, a division of MathSoft, Incorporated.
[26]
Kim, J.-M. and Porter, A. 2002. A history-based test prioritization technique for regression testing in resource constrained environments. In Proceedings of the 24rd ICSE, 2002.
[27]
Walcott, K. R., Soffa, M. L., Kapfhammer, G. M. and Roos, R. S. 2006. Timeaware test suite prioritization. In Proceedings of the 2006 international symposium on Software testing and analysis (ISSTA), 2006.
[28]
Yoo, S. and Harman, M. 2007. Pareto efficient multi-objective test case selection. In Proceedings of the 2007 ISSTA.
[29]
Wang, S., Ali, S. and Gotlieb, A. 2015. Cost-effective test suite minimization in product lines using search techniques.
[30]
Wang, S., Ali, S., Gotlieb, A. and Liaaen, M. 2014. A systematic test case selection methodology for product lines: results and insights from an industrial case study. Empirical Software Engineering, 2014: 1--37.
[31]
Wang, S., Ali, S., Yue, T., Bakkeli, Ø. and Liaaen, M. 2016. Enhancing Test Case Prioritization in an Industrial Setting with Resource Awareness and Multi-Objective Search. In Proceedings of the 38th ICSE, Software Engineering in Practice track (SEIP), 2016.
[32]
Wang, S., Ali, S. and Gotlieb, A. 2013. Minimizing test suites in software product lines using weight-based genetic algorithms. In Proceedings of the fifteenth annual conference on Genetic and evolutionary computation conference (GECCO), 2013.
[33]
Henard, C., Papadakis, M., Perrouin, G., Klein, J. and Traon, Y. L. 2013. Multi-objective test generation for software product lines. In Proceedings of the 17th International Software Product Line Conference (SPLC), 2013.
[34]
Wang, S., Buchmann, D., Ali, S., Gotlieb, A., Pradhan, D. and Liaaen, M. 2014. Multi-objective test prioritization in software product line testing: an industrial case study. In Proceedings of the 18th International SPLC, 2014.
[35]
Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C. M. and Da Fonseca, V. G. 2003. Performance assessment of multiobjective optimizers: an analysis and review. IEEE Transactions on Evolutionary Computation, 7(2).

Cited By

View all
  • (2024)Improving agility in projects using machine learning algorithmMultimedia Tools and Applications10.1007/s11042-024-19909-y83:38(85987-86005)Online publication date: 17-Sep-2024
  • (2023)State of Practical Applicability of Regression Testing Research: A Live Systematic Literature ReviewACM Computing Surveys10.1145/357985155:13s(1-36)Online publication date: 13-Jul-2023
  • (2023)Some Seeds Are Strong: Seeding Strategies for Search-based Test Case SelectionACM Transactions on Software Engineering and Methodology10.1145/353218232:1(1-47)Online publication date: 13-Feb-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GECCO '16: Proceedings of the Genetic and Evolutionary Computation Conference 2016
July 2016
1196 pages
ISBN:9781450342063
DOI:10.1145/2908812
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2016

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. multi-objective optimization
  2. search
  3. test case selection

Qualifiers

  • Research-article

Conference

GECCO '16
Sponsor:
GECCO '16: Genetic and Evolutionary Computation Conference
July 20 - 24, 2016
Colorado, Denver, USA

Acceptance Rates

GECCO '16 Paper Acceptance Rate 137 of 381 submissions, 36%;
Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)6
  • Downloads (Last 6 weeks)0
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Improving agility in projects using machine learning algorithmMultimedia Tools and Applications10.1007/s11042-024-19909-y83:38(85987-86005)Online publication date: 17-Sep-2024
  • (2023)State of Practical Applicability of Regression Testing Research: A Live Systematic Literature ReviewACM Computing Surveys10.1145/357985155:13s(1-36)Online publication date: 13-Jul-2023
  • (2023)Some Seeds Are Strong: Seeding Strategies for Search-based Test Case SelectionACM Transactions on Software Engineering and Methodology10.1145/353218232:1(1-47)Online publication date: 13-Feb-2023
  • (2023)What Not to Test (For Cyber-Physical Systems)IEEE Transactions on Software Engineering10.1109/TSE.2023.327230949:7(3811-3826)Online publication date: Jul-2023
  • (2023)Search-based Test Case Selection for PLC Systems using Functional Block Diagram Programs2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00040(228-239)Online publication date: 9-Oct-2023
  • (2022)Multi-Objective Metamorphic Test Case Selection: an Industrial Case Study (Practical Experience Report)2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE55969.2022.00058(541-552)Online publication date: Oct-2022
  • (2022)Multi-objective test suite optimization for detection and localization of software faultsJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2020.01.00934:6(2897-2909)Online publication date: Jun-2022
  • (2022)On the preferences of quality indicators for multi-objective search algorithms in search-based software engineeringEmpirical Software Engineering10.1007/s10664-022-10127-427:6Online publication date: 1-Nov-2022
  • (2021)Recommending Faulty Configurations for Interacting Systems Under Test Using Multi-objective SearchACM Transactions on Software Engineering and Methodology10.1145/346493930:4(1-36)Online publication date: 3-Aug-2021
  • (2021) CBGA-ES + : A Cluster-Based Genetic Algorithm with Non-Dominated Elitist Selection for Supporting Multi-Objective Test Optimization IEEE Transactions on Software Engineering10.1109/TSE.2018.288217647:1(86-107)Online publication date: 1-Jan-2021
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media