Skip to main content
Log in

Fail-safe testing of safety-critical systems: a case study and efficiency analysis

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

This paper proposes an approach for testing of safety-critical systems. It is based on a behavioral and a fault model. The two models are analyzed for compatibility, and necessary changes are identified to make them compatible. Then, transformation rules are used to transform the fault model into the same model type as the behavioral model. Integration rules define how to combine them. This approach results in an integrated model which then can be used to generate tests using a variety of testing criteria. The paper illustrates this general framework using a CEFSM for the behavioral model and a fault tree for the fault model. We apply the technique to an aerospace launch system. We also investigate the scalability of the approach and compare its efficiency with integrating a state chart and a fault tree.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32
Fig. 33
Fig. 34
Fig. 35
Fig. 36
Fig. 37
Fig. 38
Fig. 39

Similar content being viewed by others

Notes

  1. For readability, we omit some of the variables for the gates.

  2. This front-end tool is a collaboration between the University of Denver and the University of North Dakota.

References

  • Amberkar, S., Murray, M. T., Demerly, J. D., D’Ambrosio, J. G., & Czerny, B. J. (2001). A comprehensive hazard analysis technique for safety-critical automotive systems.

  • Angeletti, D., Giunchiglia, E., Narizzano, M., Puddu, A., & Sabina, S. (2009). Automatic test generation for coverage analysis of ERTMS software. In International Conference on Software Testing Verification and Validation, 2009. ICST ’09 (pp. 303–306). Washington, DC, USA.

  • Bobbio, A., Portinale, L., Minichino, M., & Ciancamerla, E. (2001). Improving the analysis of dependable systems by mapping fault trees into bayesian networks. Reliability Engineering and System Safety, 71(3), 249–260.

    Article  Google Scholar 

  • Boroday, S., Petrenko, A., Groz, R., & Quemener, Y. M. (2002). Test generation for CEFSM combining specification and fault coverage. In Proceedings of the IFIP 14th International Conference on Testing Communicating Systems XIV, TestCom ’02 (pp. 355–372). Deventer: Kluwer, B.V.

  • Boudali, H., & Dugan, J. B. (2005). A discrete-time bayesian network reliability modeling and analysis framework. Reliability Engineering and System Safety, 87, 337–349.

    Article  Google Scholar 

  • Bourhfir, C., Aboulhamid, E., Dssouli, R., & Rico, N. (2001). A test case generation approach for conformance testing of SDL systems. Computer Communications, 24(3–4), 319–333.

    Article  Google Scholar 

  • Bourhfir, C., Dssouli, R., Aboulhamid, E. M., & Rico, N. (1998). A guided incremental test case generation procedure for conformance testing for CEFSM specified protocols. In Proceedings of the IFIP TC6 11th International Workshop on Testing Communicating Systems, IWTCS (pp. 275–290). Deventer: Kluwer, B.V.

  • Bourhfir, C., Dssouli, R., Aboulhamid, M., & Rico, N. (1999). A test case generation tool for conformance testing of SDL systems. In SDL forum (pp. 405–420).

  • Brand, D., & Zafiropulo, P. (1983). On communicating finite-state machines. Journal of ACM, 30(2), 323–342.

    Article  MathSciNet  MATH  Google Scholar 

  • Buchacker, K., & Friedrich Alexander Universitht, I. (1999). Combining fault trees and petri nets to model safety-critical systems. In Society for Computer Simulation International (pp. 439–444).

  • Byun, Y. (2003). Pattern-based design and validation of communication protocols. Ph.D. thesis, University of Florida, Gainesville, FL, USA.

  • Byun, Y., Beverly, S., & Chung, K. (2002). A pattern language for communication protocols. In Proceedings of the 9th Conference on Pattern Languages of Programs (PLoP).

  • Byun, Y., & Sanders, B. A. (2005). A pattern-based development methodology for communication protocols. In Hisham Haddad, Lorie M. Liebrock, Andrea Omicini, & Roger L. Wainwright, (Eds.) SAC (pp. 1524–1528). ACM.

  • Byun, Y., & Sanders, B. A. (2006). A pattern-based development methodology for communication protocols. Journal of Information Science and Engineering, 22(2), 315–335.

    Google Scholar 

  • Byun, Y., Sanders, B. A., & Keum, C. (2001). Design patterns of communicating extended finite state machines in sdl. In In proceedings of the 8th Conference on Pattern Languages if Programs.

  • Cheng, K. T., & Krishnakumar, K. S. (1993). Automatic functional test generation using the extended finite state machine model. In 30th Conference on Design Automation (pp. 86–91).

  • Czerny, B. J., Ambrosio, J. G., Murray, B. T., & Sundaram, P. (2005) Effective application of software safety techniques for automotive embedded control systems. Engineering, 1(724).

  • Dalal, S. R., Jain, A., Karunanithi, N., Leaton, J. M., Lott, C. M., Patton, G.C., et al. (1999). Model-based testing in practice. In ICSE (pp. 285–294).

  • Ek, A., Grabowski, J., Hogrefe, D., Jerome, R., Koch, B., & Schmitt II, M. (1997). Towards the industrial use of validation techniques and automatic test generation methods for SDL specifications. In SDL forum (pp. 245–260).

  • El Ariss, O., Xu, D., & Wong, W. E. (2011). Integrating safety analysis with functional modeling. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 41(4), 610–624.

    Article  Google Scholar 

  • Ericson, C. A. (2005). Hazard analysis techniques for system safety. New Jersey: wiley-interscience.

    Book  Google Scholar 

  • Flammini, F., Marrone, S., Iacono, M., Mazzocca, N., & Vittorini, V. (2014). A multiformalism modular approach to ERTMS/ETCS failure modeling. International Journal of Reliability, Quality and Safety Engineering, 21(01), 1–29.

    Article  Google Scholar 

  • Flammini, F., Mazzocca, N., Iacono, M., & Marrone, S. (2005). Using repairable fault trees for the evaluation of design choices for critical repairable systems. In Proceedings of the Ninth IEEE International Symposium on High-Assurance Systems Engineering, HASE ’05 (pp. 163–172). Washington, DC, USA, 2005. IEEE Computer Society.

  • France, R., & Rumpe, B. (2007). Model-driven development of complex software: A research roadmap. In 2007 future of software engineering, FOSE ’07 (pp. 37–54). Washington, DC: IEEE Computer Society.

  • Garavel, H., Helmstetter, C., Ponsini, O., & Serwe, W. (2009). Verification of an industrial systemC/TLM model using LOTOS and CADP. In MEMOCODE (pp. 46–55).

  • Garavel, H., Lang, F., Mateescu, R., & Serwe, W. (2013). CADP 2011: a toolbox for the construction and analysis of distributed processes. The International Journal on Software Tools for Technology Transfer (STTT), 15(2), 89–107.

    Article  MATH  Google Scholar 

  • Garavel, H., Mateescu, R., & Serwe, W. (2013). Large-scale distributed verification using CADP: Beyond clusters to grids. Electronic Notes Theory Computer Science, 296, 145–161.

    Article  Google Scholar 

  • Gario, A. (2014). Fail-Safe testing of safety-critical systems. PhD thesis, University of Denver, Denver, CO, USA, 11.

  • Gario, A., & Andrews, A. (2014). Fail-safe testing of safety-critical systems. In Software Engineering Conference (ASWEC), 2014 23rd Australian (pp. 190–199). IEEE.

  • Gario, A., Andrews, A., & Hagerman, S. (2014). Testing of safety-critical systems: An aerospace launch application. In Aerospace Conference, 2014 IEEE (pp. 1–17). IEEE.

  • Ghazel, M. (2014). Formalizing a subset of ERTMS/ETCS specifications for verification purposes. Transportation Research Part C: Emerging Technologies, 42, 60–75.

    Article  Google Scholar 

  • Di Giorgio, A., & Liberati, F. (2011). Interdependency modeling and analysis of critical infrastructures based on dynamic bayesian networks. In 19th Mediterranean Conference on Control Automation (MED), 2011 (pp. 791–797).

  • Henniger, O., Lu, M., & Ural, H. (2004). Automatic generation of test purposes for testing distributed systems. In Alexandre Petrenko & Andreas Ulrich (Eds.), Formal approaches to software testing (Vol. 2931, pp. 1105–1105). Lecture Notes in Computer Science Berlin/Heidelberg: Springer.

  • Hessel, A., & Pettersson, P. (2007). A global algorithm for model-based test suite generation. Electronic Notes in Theoretical Computer Science, 190(2), 47–59.

    Article  Google Scholar 

  • Kaiser, B. (2003). A fault-tree semantics to model software-controlled systems. Softwaretechnik-Trends, 23(3), 33–39.

    Google Scholar 

  • Kaiser, B. (2005). Extending the expressive power of fault trees. In Proceedings on Reliability and Maintainability Symposium, 2005 (pp. 468–474). Alexandria, Virginia.

  • Kaiser, B., Gramlich, C., & Förster, M. (2007). State/event fault trees—A safety analysis model for software-controlled systems. Reliability Engineering and System Safety, 92(11), 1521–1537.

    Article  Google Scholar 

  • Kaiser, B., Liggesmeyer, P., & Mäckel, O. (2003). A new component concept for fault trees. In Proceedings of the 8th Australian workshop on Safety critical systems and software, volume 33 of SCS ’03 (pp. 37–46). Darlinghurst: Australian Computer Society Inc.

  • Keller, R. M. (1976). Formal verification of parallel programs. Communications of the ACM, 19(7), 371–384.

    Article  MathSciNet  MATH  Google Scholar 

  • Kim, H., Wong, W. E., Debroy, V., & Bae, D. (2010). Bridging the gap between fault trees and UML state machine diagrams for safety analysis. In 17th Asia Pacific Software Engineering Conference (APSEC) (pp. 196–205).

  • Kloos, J., Hussain, T., & Eschbach, R. (2011). Risk-based testing of safety-critical embedded systems driven by fault tree analysis. In IEEE International Conference on Software Testing Verification and Validation Workshop (ICSTW 2011) (pp. 26–33). Los Alamitos, CA: IEEE Computer Society.

  • Kovács, G., Pap, Z., & Csopaki, G. (2002). Automatic test selection based on CEFSM specifications. Acta Cybernet, 15(4), 583–599.

    MATH  Google Scholar 

  • Leaphart, E. G., Czerny, B. J., Ambrosio, J. G. D., Denlinger, C. L., & Littlejohn, D. (2005). Survey of software failsafe techniques for safety-critical automotive applications. Engineering, 1(724).

  • Lee, D., & Yannakakis, M. (1996). Principles and methods of testing finite state machines- a survey. Proceedings of the IEEE, 84(8), 1090–1123.

    Article  Google Scholar 

  • Leveson, N. G., & Harvey, P. R. (1983). Analyzing software safety. IEEE Transactions on Software Engineering, SE–9(5), 569–579.

    Article  Google Scholar 

  • Li, J. J., & Wong, W. E. (2002). Automatic test generation from communicating extended finite state machine (CEFSM)-based models. In Fifth IEEE International Symposium on Object-Oriented Real-Time Distributed Computing (ISORC 2002) Proceedings (pp. 181–185).

  • Marrone, S., Flammini, F., Mazzocca, N., Nardone, R., Vittorini, V. (2014). Towards model-driven V&V assessment of railway control systems. International Journal on Software Tools for Technology Transfer (pp. 669–683).

  • Medikonda, B. S., Ramaiah, P. S., & Gokhale, A. A. (2011). FMEA and fault tree based software safety analysis of a railroad crossing critical system. Global Journal of Computer Science and Technology GJCST, 11, 57–62.

    Google Scholar 

  • Montani, S., Portinale, L., Bobbio, A., & Codetta-Raiteri, D. (2008). Radyban: A tool for reliability analysis of dynamic fault trees through conversion into dynamic bayesian networks. Reliability Engineering and System Safety, 93(7), 922–932.

    Article  Google Scholar 

  • Nazier, R., & Bauer, T. (2012). Automated risk-based testing by integrating safety analysis information into system behavior models. In IEEE 23rd International Symposium on Software Reliability Engineering Workshops (ISSREW) (pp. 213–218).

  • Ortmeier, F., Güdemann, M., & Wolfgang, R. (2007). Formal failure models. In Proceedings of the 1st IFAC Workshop on Dependable Control of Discrete Systems (DCDS 07). Elsevier.

  • Petricic, A., Crnkovic, I., & Zagar, M. (2008). Models transformation between UML and a domain specific language. In Eight Conference on Software Engineering Research and Practice in Sweden (SERPS 08).

  • Petricic, A., Lednicki, L., & Crnkovic, I. (2009). Using UML for domain-specific component models. In Proceedings of the 14th International Workshop on Component-Oriented Programming.

  • Raiteri, D. C., Franceschinis, G., Iacono, M., & Vittorini, V. (2004). Repairable fault tree for the automatic evaluation of repair policies. In 2004 International Conference on Dependable Systems and Networks (pp. 659–668).

  • Sánchez, M., & Felder, M. (2003). A systematic approach to generate test cases based on faults. In Argentine Symposium in Software Engineering, Buenos Aires, Argentina.

  • Savage, P., Walters, S., & Stephenson, M. (1997). Automated test methodology for operational flight programs. In Aerospace Conference, 1997. Proceedings, IEEE (Vol. 4, pp. 293–304).

  • Sinha, A., & Smidts, C. (2006). An experimental evaluation of a higher-ordered-typed-functional specification-based test-generation technique. Empirical Software Engineering, 11(2), 173–202.

    Article  Google Scholar 

  • Teradyne Software and Systems Test, (1999). Testmaster: User’s guide. New Hampshire: Empirix Inc.

    Google Scholar 

  • Tretmans, J. (2008). Model based testing with labeled transition systems. In Formal methods and testing (pp. 1–38).

  • Tribble, A. C., & Miller, S. P. (2004). Software intensive systems safety analysis. IEEE Aerospace and Electronic Systems Magazine, 19(10), 21–26.

    Article  Google Scholar 

  • Utting, M., & Legeard, B. (2007). Practical model-based testing: A tools approach. San Francisco, CA: Morgan Kaufmann Publishers Inc.

    Google Scholar 

  • VASY. CADP (Caesar/Aldebaran Development Package). http://cadp.inria.fr/

  • Vesely, W., Dugan, J., Fragola, J., Minarick, & Railsback, J. (2002). Fault tree handbook with aerospace applications. Washington, DC: Handbook, National Aeronautics and Space Administration.

  • Vaos, J. M., & McGraw, G. (1998). Software fault injection: Inoculating programs against errors. New Jersey: Wiley Computer Pub.

    Google Scholar 

  • Wada, H., Suzuki, J., & Takada, S. (2005). A model transformation framework for domain specific languages: An approach using UML and attribute-Oriented programming. In In Proceedings of the 9th World Multi-Conference on Systemics, Cybernetics and Informatics.

  • Wang, D., & Pan, J. (2010). An optimization to automatic fault tree analysis and failure mode and effect analysis approaches for processes. In 2010 International Conference on Computer Design and Applications (ICCDA) (Vol. 3, pp. 153–157).

  • Xiang, J., Futatsugi, K., & He, Y. (2004). Formal fault tree construction and system safety analysis. In IASTED Conference on Software Engineering (pp. 378–384).

Download references

Acknowledgments

This work was partially supported by NSF Grant Numbers 0934413, 1127947, and 1332078.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ahmed Gario.

Appendix

Appendix

1.1 INHIBIT gate

INHIBIT is similar to the AND gate. They have the same states and transitions. The only difference is that the predicate for the transitions \(\hbox {T}_{2}\) and \(\hbox {T}_{3}\) should include the enabling condition. We do not need to have a separate gate representation for NOT gate since we can express it in any predicate. If we want to negate any event, we can use the NOT logical operator inside the gate that the negated event is one of its inputs.

Fig. 40
figure 40

XOR gate representation in FT and GCEFSM

1.2 XOR gate

This gate is slightly different from the AND gate, although it has the same structure and same number of transitions and states. At this gate, it is necessary to distinguish between the event that has not occurred in the first place and the one whose status is false. The representation of GCEFSM XOR gate is shown in Fig. 40. T\(_{0}\) to T\(_{3}\) are the possible transitions that may be taken based on their predicates.

  • \( T_{0}:(S_{0},[NoOfOccurredEvents=0 \& e_{j}.eOccurrence=true],get(m_{j}))/(S_{0},\, update(events))\)

  • \( T_{1}:(S_{0},[NoOfOccurredEvents=1 \& e_{j}.eOccurrence=true \& xor(events)=false],get(m_{j}))/(S_{0},update(events),-)\)

  • \( T_{2}:(S_{0},[NoOfOccurredEvents=1 \& e_{j}.eOccurrence=true \& xor(events)=true],get(m_{j}))/(S_{1},update(events),Send(gateOccurred))\)

  • \(T_{3}:(S_{1},[inputStatusChanged(e_{j})=true],get(m_{j}))/(S_{0},update(events), Send(gate not occurred))\)

1.3 Timing an event gate

FT gates such as AND, OR, and INHIBIT are well defined and can be syntactically represented. Events in FT can be simple or composed. A composed event can be decomposed further to simple events or a timed simple event. A timed simple event is the simple event that should occur for a specific period of time to contribute to a hazard. However, FT has no timing gates. Therefore, we need to have a representation that can handle the timing issue (either a minimum or maximum timing). A CEFSM can be supplemented with timers and timer-related operations. A timer is set with a time value during a transition. If the timer is not canceled by the CEFSM, the timer will generate a time expiration signal after the time period has been exceeded (Byun et al. 2001, 2002; Byun 2003; Byun and Sanders 2005, 2006). Thus, we introduce this gate that can time an event and the gate in the subsection 7 that deals with the timing intervals. This gate works as follows. Upon receiving a message that indicates the occurrence of the event, the transition \(\hbox {T}_{0}\) takes place which starts the timer. When the time expires and no further “gate not occurred” message was received that indicates that the event is no longer happening, the transition \(T_{2}\) is taken and sends a “gate occurred” message. Otherwise, the gate does not occur. \(T_{2}\) is taken when the event status \(e_{i}\).eStatus changes to false.

Fig. 41
figure 41

Event timer GCEFSM

  • \(T_0:(S_0,[e_j.eStatus=true],get(m_{j}))/(S_{1},setTimer(v,Timer_{i}),-)\)

  • \(T_{1}:(S_{1},time-out)/(S_{2},-,Send(GateOccurred))\)

  • \(T_{2}:(S_{1},e_{j}.eStatus= false],get(m_{j}))/(S_{0},reset(Timer_{i});update(events))\)

  • \(T_{3}:(S_{2},e_{i}.eStatus= false],get(m_{i}))/(S_{0},reset(Timer_{i});update(events),Send(GateNotOccurred))\)

Fig. 42
figure 42

Timing continuous intervals GCEFSM

1.3.1 Timing an event for continuous intervals gate

Some event may need to be timed for continuous intervals. For example, we may need to observe an occurrence of an event every consecutive 5 sec as long as the system is operational. Fig. 42 shows that as long as the transition \(\hbox {T}_{0}\) is fired and \(\hbox {T}_{2}\) was not, the event will be timed for fixed consecutive amounts of time and it keeps timing until the status of the event \(e_{i}.eStatus\) changes to false. Upon receiving this event change, the transition \(\hbox {T}_{3}\) to the state \(S_{1}\) is fired sending out a “gate not occurred” message.

  • \(T_{0}:(S_{0},[e_j.eStatus=true],get(m_j))/(S_1,setTimer(v,Timer_i),Send(Gate Occurred))\)

  • \(T_1:(S_1,time-out)/(S_2,-,Send(Gate not Occurred))\)

  • \(T_2:(S_1,[e_j.eStatus=false],get(m_j))/(S_0,reset(Timer_i),Send(Gate Not Occurred))\)

  • \(T_3:(S_2,setTimer(v,Timer_i))/(S_1,-,Send(Gate Occurred))\)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gario, A., Andrews, A. & Hagerman, S. Fail-safe testing of safety-critical systems: a case study and efficiency analysis. Software Qual J 26, 3–48 (2018). https://doi.org/10.1007/s11219-015-9283-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-015-9283-5

Keywords

Navigation