skip to main content
10.1145/3205455.3205490acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
research-article

Multi-objective black-box test case selection for cost-effectively testing simulation models

Published: 02 July 2018 Publication History

Abstract

In many domains, engineers build simulation models (e.g., Simulink) before developing code to simulate the behavior of complex systems (e.g., Cyber-Physical Systems). Those models are commonly heavy to simulate which makes it difficult to execute the entire test suite. Furthermore, it is often difficult to measure white-box coverage of test cases when employing such models. In addition, the historical data related to failures might not be available. This paper proposes a cost-effective approach for test case selection that relies on black-box data related to inputs and outputs of the system. The approach defines in total five effectiveness measures and one cost measure followed by deriving in total 15 objective combinations and integrating them within Non-Dominated Sorting Genetic Algorithm-II (NSGA-II). We empirically evaluated our approach with all these 15 combinations using four case studies by employing mutation testing to assess the fault revealing capability. The results demonstrated that our approach managed to improve Random Search by 26% on average in terms of the Hypervolume quality indicator.

Supplemental Material

ZIP File
Supplemental files.

References

[1]
2018. FMI Standard. (January 2018). http://fmi-standard.org/
[2]
Andrea Arcuri and Lionel Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Software Engineering (ICSE), 2011 33rd International Conference on. IEEE, 1--10.
[3]
Andrea Arcuri, Muhammad Zohaib Iqbal, and Lionel Briand. 2010. Black-box System Testing of Real-time Embedded Systems Using Random and Search-based Testing. In Proceedings of the 22Nd IFIP WG 6.1 International Conference on Testing Software and Systems (ICTSS'10). Springer-Verlag, Berlin, Heidelberg, 95--110.
[4]
Aitor Arrieta, Shuai Wang, Goiuria Sagardui, and Leire Etxeberria. 2016. Search-based Test Case Selection of Cyber-physical System Product Lines for Simulation-based Validation. In Proceedings of the 20th International Systems and Software Product Line Conference. 297--306.
[5]
Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2016. Testing Advanced Driver Assistance Systems Using Multi-objective Search and Neural Networks. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. 63--74.
[6]
Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2018. Testing Vision-Based Control Systems Using Learnable Evolutionary Algorithms. In Proceedings of the 40th International Conference on Software Engineering (ICSE '18). 12.
[7]
Lionel Briand, Shiva Nejati, Mehrdad Sabetzadeh, and Domenico Bianculli. 2016. Testing the Untestable: Model Testing of Complex Software-intensive Systems. In Proceedings of the 38th International Conference on Software Engineering Companion (ICSE '16). ACM, 789--792.
[8]
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation 6, 2 (2002), 182--197.
[9]
Emelie Engström, Per Runeson, and Mats Skoglund. 2010. A systematic review on regression test selection techniques. Information and Software Technology 52, 1 (2010), 14--30.
[10]
Robert Feldt, Simon M. Poulding, David Clark, and Shin Yoo. 2016. Test Set Diameter: Quantifying the Diversity of Sets of Test Cases. In 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, Chicago, IL, USA, April 11--15, 2016. 223--233.
[11]
Le Thi My Hanh, Nguyen Thanh Binh, and Khuat Thanh Tung. 2016. A Novel Fitness function of metaheuristic algorithms for test data generation for simulink models based on mutation analysis. Journal of Systems and Software 120, C (2016), 17--30.
[12]
Hadi Hemmati, Andrea Arcuri, and Lionel Briand. 2013. Achieving scalable model-based testing through test case diversity. ACM Transactions on Software Engineering and Methodology 22, 1 (2013), 6:1--6:42.
[13]
Christopher Henard, Mike Papadakis, Mark Harman, Yue Jia, and Yves Le Traon. 2016. Comparing white-box and black-box test prioritization. In Proceedings of the 38th International Conference on Software Engineering. ACM, 523--534.
[14]
René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 654--665.
[15]
Remo Lachmann, Michael Felderer, Manuel Nieke, Sandra Schulze, Christoph Seidl, and Ina Schaefer. 2017. Multi-objective black-box test case selection for system testing. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 1311--1318.
[16]
Reza Matinnejad, Shiva Nejati, and Lionel C. Briand. 2017. Automated Testing of Hybrid Simulink/Stateflow Controllers: Industrial Case Studies. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017). ACM, New York, NY, USA, 938--943.
[17]
Reza Matinnejad, Shiva Nejati, Lionel C Briand, and Thomas Bruckmann. 2015. Effective test suites for mixed discrete-continuous stateflow controllers. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 84--95.
[18]
Reza Matinnejad, Shiva Nejati, Lionel C. Briand, and Thomas Bruckmann. 2016. Automated Test Suite Generation for Time-continuous Simulink Models. In Proceedings of the 38th International Conference on Software Engineering (ICSE '16). ACM, New York, NY, USA, 595--606.
[19]
Annibale Panichella, Fitsum Kifetew, and Paolo Tonella. 2017. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering (2017).
[20]
Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In Proceedings of the 37th International Conference on Software Engineering-Volume 1. IEEE Press, 936--946.
[21]
Dipesh Pradhan, Shuai Wang, Shaukat Ali, and Tao Yue. 2016. Search-Based Cost-Effective Test Case Selection Within a Time Budget: An Empirical Study. In Proceedings of the Genetic and Evolutionary Computation Conference 2016 (GECCO '16). ACM, New York, NY, USA, 1085--1092.
[22]
Goiuria Sagardui, Leire Etxeberria, Joseba A Agirre, Aitor Arrieta, Carlos Fernando Nicolas, and Jose Maria Martin. 2017. A Configurable Validation Environment for Refactored Embedded Software: An Application to the Vertical Transport Domain. In 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 16--19.
[23]
T. Strathmann and J. Oehlerking. 2015. Verifying Properties of an Electro-Mechanical Braking System. In In 1st and 2nd International Workshop on Applied veRification for Continuous and Hybrid Systems. 49--56.
[24]
Shuai Wang, Shaukat Ali, and Arnaud Gotlieb. 2013. Minimizing test suites in software product lines using weight-based genetic algorithms, In Proceedings of the 2013 Genetic and Evolutionary Computation Conference. GECCO 2013 - Proceedings of the 2013 Genetic and Evolutionary Computation Conference, 1493 -- 1500.
[25]
Shuai Wang, Shaukat Ali, Tao Yue, Yan Li, and Marius Liaaen. 2016. A Practical Guide to Select Quality Indicators for Assessing Pareto-based Search Algorithms in Search-based Software Engineering. In Proceedings of the 38th International Conference on Software Engineering (ICSE '16). 631--642.
[26]
Shin Yoo and Mark Harman. 2007. Pareto efficient multi-objective test case selection. In Proceedings of the 2007 international symposium on Software testing and analysis. ACM, 140--150.
[27]
Yuan Zhan and John A Clark. 2008. A search-based framework for automatic testing of MATLAB/Simulink models. Journal of Systems and Software 81, 2 (2008), 262--285.

Cited By

View all
  • (2025)Enhancing multi-objective test case selection through the mutation operatorAutomated Software Engineering10.1007/s10515-025-00489-632:1Online publication date: 30-Jan-2025
  • (2025)Defining and generating multi-level and uncertainty-wise test oracles for cyber-physical systemsSoftware and Systems Modeling10.1007/s10270-025-01271-8Online publication date: 14-Feb-2025
  • (2024)SensoDat: Simulation-based Sensor Dataset of Self-driving CarsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644891(510-514)Online publication date: 15-Apr-2024
  • Show More Cited By

Index Terms

  1. Multi-objective black-box test case selection for cost-effectively testing simulation models

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference
    July 2018
    1578 pages
    ISBN:9781450356183
    DOI:10.1145/3205455
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 July 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. search-based testing
    2. simulation models
    3. test selection

    Qualifiers

    • Research-article

    Conference

    GECCO '18
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)15
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Enhancing multi-objective test case selection through the mutation operatorAutomated Software Engineering10.1007/s10515-025-00489-632:1Online publication date: 30-Jan-2025
    • (2025)Defining and generating multi-level and uncertainty-wise test oracles for cyber-physical systemsSoftware and Systems Modeling10.1007/s10270-025-01271-8Online publication date: 14-Feb-2025
    • (2024)SensoDat: Simulation-based Sensor Dataset of Self-driving CarsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644891(510-514)Online publication date: 15-Apr-2024
    • (2024)Identifying and Explaining Safety-critical Scenarios for Autonomous Vehicles via Key FeaturesACM Transactions on Software Engineering and Methodology10.1145/364033533:4(1-32)Online publication date: 11-Jan-2024
    • (2024)Replicability of experimental tool evaluations in model-based software and systems engineering with MATLAB/SimulinkInnovations in Systems and Software Engineering10.1007/s11334-022-00442-w20:3(209-224)Online publication date: 1-Sep-2024
    • (2024)Generating Test Suites for an Embedded ESP8266 System by Using Black-Box Testing MethodsAdvanced Technologies, Systems, and Applications IX10.1007/978-3-031-71694-2_61(879-891)Online publication date: 1-Oct-2024
    • (2023)Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual EnvironmentsACM Transactions on Software Engineering and Methodology10.1145/353381832:2(1-30)Online publication date: 4-Apr-2023
    • (2023)Some Seeds Are Strong: Seeding Strategies for Search-based Test Case SelectionACM Transactions on Software Engineering and Methodology10.1145/353218232:1(1-47)Online publication date: 13-Feb-2023
    • (2023)RoadSign at the SBFT 2023 Tool Competition Cyber-Physical Systems Track2023 IEEE/ACM International Workshop on Search-Based and Fuzz Testing (SBFT)10.1109/SBFT59156.2023.00006(37-38)Online publication date: May-2023
    • (2023)Search-based Test Case Selection for PLC Systems using Functional Block Diagram Programs2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00040(228-239)Online publication date: 9-Oct-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media