skip to main content
10.1145/3650212.3685552acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article
Open access

From Fault Injection to Formal Verification: A Holistic Approach to Fault Diagnosis in Cyber-Physical Systems

Published: 11 September 2024 Publication History

Abstract

Cyber-Physical Systems (CPSs) face growing complexity, especially in safety-critical areas. Ensuring their correctness is vital to maintain full operational capacity, as undetected failures can be both costly and life-threatening. Therefore, advanced fault diagnosis procedures are essential for thorough CPS testing, enabling accurate fault detection, explanation, and rectification. This doctoral research contributes to the field by developing novel tools and techniques to enhance fault-based testing and diagnosis of CPSs. Our research focuses on testing of CPS dataflow models created in Simulink, validated against strict formal specifications. Our contributions include (i) an automated tool for systematic fault injection, (ii) a bio-inspired global optimization algorithm, (iii) a robust fault localization method, (iv) a novel approach to mutation testing for evaluating test suites against formal properties, and (v) a new coverage criterion tailored for CPS dataflow models. This comprehensive approach offers significant improvements over existing methods, ensuring thorough testing across various scenarios. We validate the effectiveness of our solutions using publicly available benchmarks from various domains. Our findings open new perspectives on CPS testing, laying the foundation for more robust CPSs

References

[1]
Houssam Abbas and Georgios Fainekos. 2012. Convergence proofs for Simulated Annealing falsification of safety properties. In 50th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2012, Allerton Park & Retreat Center, Monticello, IL, USA, October 1-5, 2012. IEEE, USA. 1594–1601. https://doi.org/10.1109/ALLERTON.2012.6483411
[2]
Allen Troy Acree, Timothy Alan Budd, Richard A. DeMillo, Richard J. Lipton, and Frederick Gerald Sayward. 1979. Mutation Analysis. Georgia Institute of Technology, Atlanta, Georgia.
[3]
Ezio Bartocci, Thomas Ferrère, Niveditha Manjunath, and Dejan Nickovic. 2018. Localizing Faults in Simulink/Stateflow Models with STL. In Proceedings of the 21st International Conference on Hybrid Systems: Computation and Control (part of CPS Week), HSCC 2018, Porto, Portugal, April 11-13, 2018. ACM, USA. 197–206. https://doi.org/10.1145/3178126.3178131
[4]
Ezio Bartocci, Niveditha Manjunath, Leonardo Mariani, Cristinel Mateis, and Dejan Nickovic. 2019. Automatic Failure Explanation in CPS Models. In Software Engineering and Formal Methods - 17th International Conference, SEFM 2019, Oslo, Norway, September 18-20, 2019, Proceedings (Lecture Notes in Computer Science, Vol. 11724). Springer, USA. 69–86. https://doi.org/10.1007/978-3-030-30446-1_4
[5]
Ezio Bartocci, Niveditha Manjunath, Leonardo Mariani, Cristinel Mateis, and Dejan Nickovic. 2021. CPSDebug: Automatic failure explanation in CPS models. Int. J. Softw. Tools Technol. Transf., 23, 5 (2021), 783–796. https://doi.org/10.1007/S10009-020-00599-4
[6]
Ezio Bartocci, Leonardo Mariani, Dejan Nickovic, and Drishti Yadav. 2022. FIM: fault injection and mutation for Simulink. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022. ACM, USA. 1716–1720. https://doi.org/10.1145/3540250.3558932
[7]
Ezio Bartocci, Leonardo Mariani, Dejan Nickovic, and Drishti Yadav. 2022. Search-based Testing for Accurate Fault Localization in CPS. In IEEE 33rd International Symposium on Software Reliability Engineering, ISSRE 2022, Charlotte, NC, USA, October 31 - Nov. 3, 2022. IEEE, USA. 145–156. https://doi.org/10.1109/ISSRE55969.2022.00024
[8]
Ezio Bartocci, Leonardo Mariani, Dejan Nickovic, and Drishti Yadav. 2023. Property-Based Mutation Testing. In IEEE Conference on Software Testing, Verification and Validation, ICST 2023, Dublin, Ireland, April 16-20, 2023. IEEE, USA. 222–233. https://doi.org/10.1109/ICST57152.2023.00029
[9]
Marco Bozzano and Adolfo Villafiorita. 2007. The FSAP/NuSMV-SA Safety Analysis Platform. Int. J. Softw. Tools Technol. Transf., 9, 1 (2007), 5–24. https://doi.org/10.1007/S10009-006-0001-2
[10]
Johan Cederbladh, Romina Eramo, Vittoriano Muttillo, and Per Erik Strandberg. 2024. Experiences and challenges from developing cyber-physical systems in industry-academia collaboration. Software: Practice and Experience, 56, 6 (2024), 1193–1212. https://doi.org/10.1002/spe.3312
[11]
Oscar Cornejo, Fabrizio Pastore, and Lionel C. Briand. 2022. Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results in the Space Domain. IEEE Trans. Software Eng., 48, 10 (2022), 3913–3939. https://doi.org/10.1109/TSE.2021.3107680
[12]
Richard A. DeMillo, Richard J. Lipton, and Frederick Gerald Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer, 11, 4 (1978), April, 34–41.
[13]
Jyotirmoy V. Deshmukh, Xiaoqing Jin, James Kapinski, and Oded Maler. 2015. Stochastic Local Search for Falsification of Hybrid Systems. In Automated Technology for Verification and Analysis - 13th International Symposium, ATVA 2015, Shanghai, China, October 12-15, 2015, Proceedings (Lecture Notes in Computer Science, Vol. 9364). Springer, Switzerland. 500–517. https://doi.org/10.1007/978-3-319-24953-7_35
[14]
Tagir Fabarisov, Ilshat Mamaev, Andrey Morozov, and Klaus Janschek. 2021. Model-based Fault Injection Experiments for the Safety Analysis of Exoskeleton System. CoRR, abs/2101.01283 (2021), arXiv:2101.01283. arxiv:2101.01283
[15]
Daniel Fortunato, Jose Campos, and Rui Abreu. 2022. Mutation Testing of Quantum Programs: A Case Study With Qiskit. IEEE Transactions on Quantum Engineering, 3 (2022), 1–17.
[16]
Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In SIGSOFT/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-19) and ESEC’11: 13th European Software Engineering Conference (ESEC-13), Szeged, Hungary, September 5-9, 2011. ACM, USA. 416–419. https://doi.org/10.1145/2025113.2025179
[17]
Gregory Gay, Ajitha Rajan, Matt Staats, Michael W. Whalen, and Mats Per Erik Heimdahl. 2016. The Effect of Program and Model Structure on the Effectiveness of MC/DC Test Adequacy Coverage. ACM Trans. Softw. Eng. Methodol., 25, 3 (2016), 25:1–25:34. https://doi.org/10.1145/2934672
[18]
Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In 36th International Conference on Software Engineering, ICSE ’14, Hyderabad, India - May 31 - June 07, 2014. ACM, USA. 435–445. https://doi.org/10.1145/2568225.2568271
[19]
Marko Ivankovic, Goran Petrovic, René Just, and Gordon Fraser. 2019. Code coverage at Google. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019, Tallinn, Estonia, August 26-30, 2019. ACM, USA. 955–963. https://doi.org/10.1145/3338906.3340459
[20]
Bing Liu, Lucia, Shiva Nejati, and Lionel C. Briand. 2017. Improving fault localization for Simulink models using search-based testing and prediction models. In IEEE 24th International Conference on Software Analysis, Evolution and Reengineering, SANER 2017, Klagenfurt, Austria, February 20-24, 2017. IEEE Computer Society, USA. 359–370. https://doi.org/10.1109/SANER.2017.7884636
[21]
Bing Liu, Shiva Nejati, Lucia, and Lionel C. Briand. 2019. Effective fault localization of automotive Simulink models: achieving the trade-off between test oracle effort and fault localization accuracy. Empir. Softw. Eng., 24, 1 (2019), 444–490. https://doi.org/10.1007/s10664-018-9611-z
[22]
Oded Maler and Dejan Nickovic. 2004. Monitoring Temporal Properties of Continuous Signals. In Formal Techniques, Modelling and Analysis of Timed and Fault-Tolerant Systems, Joint International Conferences on Formal Modelling and Analysis of Timed Systems, FORMATS 2004 and Formal Techniques in Real-Time and Fault-Tolerant Systems, FTRTFT 2004, Grenoble, France, September 22-24, 2004, Proceedings (Lecture Notes in Computer Science, Vol. 3253). Springer, Berlin, Heidelberg. 152–166. https://doi.org/10.1007/978-3-540-30206-3_12
[23]
Mathworks. 2024. Simulink Documentation. https://in.mathworks.com/help/simulink/
[24]
Reza Matinnejad, Shiva Nejati, Lionel C. Briand, and Thomas Bruckmann. 2019. Test Generation and Test Prioritization for Simulink Models with Dynamic Behavior. IEEE Trans. Software Eng., 45, 9 (2019), 919–944. https://doi.org/10.1109/TSE.2018.2811489
[25]
Akbar Siami Namin and James H. Andrews. 2009. The influence of size and coverage on test suite effectiveness. In Proceedings of the Eighteenth International Symposium on Software Testing and Analysis, ISSTA 2009, Chicago, IL, USA, July 19-23, 2009, Gregg Rothermel and Laura K. Dillon (Eds.). ACM, USA. 57–68. https://doi.org/10.1145/1572272.1572280
[26]
Ingo Pill, Ivan Rubil, Franz Wotawa, and Mihai Nica. 2016. Simultate: A toolset for fault injection and mutation testing of simulink models. In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, USA. 168–173. https://doi.org/10.1109/ICSTW.2016.21
[27]
Ludovic Pintard, Jean-Charles Fabre, Karama Kanoun, Michel Leeman, and Matthieu Roy. 2013. Fault Injection in the Automotive Standard ISO 26262: An Initial Approach. In Dependable Computing - 14th European Workshop, EWDC 2013, Coimbra, Portugal, May 15-16, 2013. Proceedings (Lecture Notes in Computer Science, Vol. 7869). Springer, Berlin, Heidelberg. 126–133. https://doi.org/10.1007/978-3-642-38789-0_11
[28]
Mustafa Saraoğlu, Andrey Morozov, Mehmet Turan Söylemez, and Klaus Janschek. 2017. ErrorSim: A tool for error propagation analysis of simulink models. In International Conference on Computer Safety, Reliability, and Security. Springer, Cham. 245–254. https://doi.org/10.1007/978-3-319-66266-4_16
[29]
Nikhil Kumar Singh and Indranil Saha. 2020. Specification-Guided Automated Debugging of CPS Models. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst., 39, 11 (2020), 4142–4153. https://doi.org/10.1109/TCAD.2020.3012862
[30]
Matt Staats and Corina S. Pasareanu. 2010. Parallel symbolic execution for structural test generation. In Proceedings of the Nineteenth International Symposium on Software Testing and Analysis, ISSTA 2010, Trento, Italy, July 12-16, 2010. ACM, USA. 183–194. https://doi.org/10.1145/1831708.1831732
[31]
Rickard Svenningsson, Jonny Vinter, Henrik Eriksson, and Martin Törngren. 2010. MODIFI: a MODel-implemented fault injection tool. In International Conference on Computer Safety, Reliability, and Security. Springer, Berlin, Heidelberg. 210–222. isbn:978-3-642-15651-9 https://doi.org/10.1007/978-3-642-15651-9_16
[32]
Drishti Yadav. 2021. Blood coagulation algorithm: A novel bio-inspired meta-heuristic algorithm for global optimization. Mathematics, 9, 23 (2021), 3011. https://doi.org/10.3390/math9233011
[33]
Qian Yang, J. Jenny Li, and David M. Weiss. 2006. A Survey of Coverage Based Testing Tools. In Proceedings of the 2006 International Workshop on Automation of Software Test, AST 2006, Shanghai, China, May 23, 2006. ACM, USA. 99–103. https://doi.org/10.1145/1138929.1138949
[34]
Justyna Zander-Nowicka. 2009. Model-based testing of real-time embedded systems in the automotive domain. Technical University Berlin, Berlin, Germany.
[35]
Zhenya Zhang, Deyun Lyu, Paolo Arcaini, Lei Ma, Ichiro Hasuo, and Jianjun Zhao. 2021. Effective Hybrid System Falsification Using Monte Carlo Tree Search Guided by QB-Robustness. In Computer Aided Verification - 33rd International Conference, CAV 2021, Virtual Event, July 20-23, 2021, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 12759). Springer, Switzerland. 595–618. https://doi.org/10.1007/978-3-030-81685-8_29

Index Terms

  1. From Fault Injection to Formal Verification: A Holistic Approach to Fault Diagnosis in Cyber-Physical Systems

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ISSTA 2024: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis
    September 2024
    1928 pages
    ISBN:9798400706127
    DOI:10.1145/3650212
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 11 September 2024

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Coverage Criteria
    2. Cyber-Physical Systems
    3. Debugging
    4. Fault Injection
    5. Fault Localization
    6. Model-Based Development
    7. Mutation Testing
    8. Optimization
    9. Signal Temporal Logic (STL)
    10. Simulink Models
    11. Software Testing

    Qualifiers

    • Research-article

    Conference

    ISSTA '24
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 58 of 213 submissions, 27%

    Upcoming Conference

    ISSTA '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 253
      Total Downloads
    • Downloads (Last 12 months)253
    • Downloads (Last 6 weeks)41
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media