Skip to main content
SpringerLink
Log in

Comprehensive analysis of FBD test coverage criteria using mutants

  • Regular Paper
  • Published:
Software & Systems Modeling Aims and scope Submit manuscript

Abstract

Function block diagram (FBD), a graphical modeling language for programmable logic controllers, has been widely used to implement safety critical system software such as nuclear reactor protection systems. With the growing importance of structural testing for FBD models, structural test coverage criteria for FBD models have been proposed and evaluated using mutation analysis in our previous work. We extend the previous work by comprehensively analyzing the relationships among fault detection effectiveness, test suite size, and coverage level through several research questions. We generate a large number of test suites achieving an FBD test coverage ranging from 0 to 100 %, and we also generate many artificial faults (i.e. mutants) for the FBD models. Our analysis results show that the fault detection effectiveness of the FBD coverage criteria increases with increasing coverage levels, and the coverage criteria are highly effective at detecting faults in all subject models. Furthermore, the test suites generated with the FBD coverage criteria are more effective and efficient than the randomly generated test suites. The FBD coverage criteria are strong at detecting faults in Boolean edges, while relatively weak at detecting wrong constants in FBD models. Empirical knowledge regarding our experiments provide the validity of using the FBD coverage criteria, and therefore, of FBD model-based testing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. The AND block is negated because of the inverter, represented by a small bubble attached to the output of the AND block.

  2. We used an SMT solver to verify whether or not each test requirement is feasible.

    Table 1 Size information for the subject models
    Full size table

  3. We performed F-test to verify the same variance assumption before performing t-test.

  4. We only show the result for CCC-suites since those for BC-suites and ICC-suites have similar characteristics to the CCC-suites.

References

  1. Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing. Wiley, New York (2011)

    Google Scholar 

  2. Ould, M.A., Unwin, C.: Testing in Software Development. Cambridge University Press, Cambridge (1986)

    Google Scholar 

  3. Ammann, P., Offutt, J.: Introduction to Software Testing. Cambridge University Press, UK (2008)

    Book  MATH  Google Scholar 

  4. Beizer, B.: Software Testing Techniques. Dreamtech Press, India (2003)

  5. IEC Standard 1131–3, Part3: Programming Languages (1993)

  6. Doosan Heavy Industry and Construction: Software Design Specification for the Bistable Processor of the Reactor Protection System. KNICS-RPS-SDS231-01, Rev. 01 (2006)

  7. NRC. Regulatory guide 1.171, software unit testing for digital computer software used in safety systems of nuclear power plants. Tech. rep. (1997)

  8. NRC, Draft regulatory guide dg-1208, proposed revision 1 of regulatory guide 1.171. Tech. rep. (2012)

  9. Jee, E., Yoo, J., Cha, S., Bae, D.: A data flow-based structural testing technique for fbd programs. Inf. Softw. Technol. 51(7), 1131–1139 (2009)

    Article  Google Scholar 

  10. Andrews, J.H., Briand, L.C., Labiche, Y., Namin, A.S.: Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans. Softw. Eng. 32(8), 608–624 (2006)

    Article  Google Scholar 

  11. Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011)

    Article  Google Scholar 

  12. King, K.N., Offutt, A.J.: A fortran language system for mutation-based software testing. Softw. Pract. Exp. 21(7), 685–718 (1991)

    Article  Google Scholar 

  13. Shin, D., Jee, E., Bae, D.-H.: Empirical evaluation on fbd model-based test coverage criteria using mutation analysis. In: Model Driven Engineering Languages and Systems, pp. 465–479. Springer, Berlin (2012)

  14. Hutchins, M., Foster, H., Goradia, T., Ostrand, T.: Experiments of the effectiveness of dataflow-and controlflow-based test adequacy criteria. In: Proceedings of the 16th international conference on Software engineering, pp. 191–200. IEEE Computer Society Press, Silver Spring (1994)

  15. Li, N., Praphamontripong, U., Offutt, J.: An experimental comparison of four unit test criteria: Mutation, edge-pair, all-uses and prime path coverage. In: International Conference on Software Testing, Verification and Validation Workshops, 2009. ICSTW’09, pp. 220–229. IEEE, New York (2009)

  16. Offutt, J., Abdurazik, A.: Generating tests from uml specifications. In: UML99-The Unified Modeling Language, pp. 416–429. Springer, Berlin (1999)

  17. Kim, Y.G., Hong, H.S., Bae, D.-H., Cha, S.-D.: Test cases generation from uml state diagrams. In: Software, IEE Proceedings-, vol. 146, pp. 187–192. IET (1999)

  18. Rountev, A., Kagan, S., Sawin, J.: Coverage criteria for testing of object interactions in sequence diagrams. In: Fundamental Approaches to Software Engineering, pp. 289–304. Springer, Berlin (2005)

  19. Briand, L., Labiche, Y., Wang, Y.: Using simulation to empirically investigate test coverage criteria based on statechart. In: Proceedings of the 26th International Conference on Software Engineering, pp. 86–95. IEEE Computer Society, Berlin (2004)

  20. Lakehal, A., Parissis, I.: Structural coverage criteria for lustre/scade programs. Softw. Test. Verif. Reliab. 19(2), 133–154 (2009)

    Article  Google Scholar 

  21. Papailiopoulou, V., Rajan, A., Parissis, I.: Structural test coverage criteria for integration testing of lustre/scade programs. In: Salaün, G., Schätz, B. (eds.) Formal Methods for Industrial Critical Systems, pp. 85–101. Springer, Berlin (2011)

  22. Alyokhin, V., Elbel, B., Rothfelder, M., Pretschner, A.: Coverage metrics for continuous function charts. In: 15th International Symposium on Software Reliability Engineering, 2004. ISSRE 2004, pp. 257–268. IEEE, New York (2004)

  23. Frankl, P.G., Weyuker, E.J.: A formal analysis of the fault-detecting ability of testing methods. IEEE Trans. Softw. Eng. 19(3), 202–213 (1993)

    Article  Google Scholar 

  24. Zhu, H.: A formal analysis of the subsume relation between software test adequacy criteria. IEEE Trans. Softw. Eng. 22(4), 248–255 (1996)

    Article  Google Scholar 

  25. Zhu, H., Hall, P.A., May, J.H.: Software unit test coverage and adequacy. ACM Comput. Surv. (CSUR) 29(4), 366–427 (1997)

    Article  Google Scholar 

  26. Gargantini, A., Heitmeyer, C.: Using model checking to generate tests from requirements specifications. In Software Engineering? ESEC/FSE99, pp. 146–162. Springer, Berlin (1999)

  27. Rayadurgam, S., Heimdahl, M.P.E.: Coverage based test-case generation using model checkers. In: Engineering of Computer Based Systems, 2001. ECBS 2001. Proceedings. Eighth Annual IEEE International Conference and Workshop on the, pp. 83–91. IEEE, New York (2001)

  28. Staats, M., Gay, G., Whalen, M., Heimdahl, M.: On the danger of coverage directed test case generation. In: Fundamental Approaches to Software Engineering, pp. 409–424. Springer, Berlin (2012)

  29. Heimdahl, M.P., George, D.: Test-suite reduction for model based tests: Effects on test quality and implications for testing. In Proceedings of the 19th IEEE International Conference on Automated Software Engineering, pp. 176–185. IEEE Computer Society, Silver Spring (2004)

  30. Arcuri, A.: A theoretical and empirical analysis of the role of test sequence length in software testing for structural coverage. IEEE Trans. Softw. Eng. 38(3), 497–519 (2012)

    Article  Google Scholar 

  31. Mader, A.: A classification of plc models and applications. In: Boel, R., Stremersch, G. (eds.) Discrete Event Systems, pp. 239–246. Springer, Berlin (2000)

  32. Oh, Y., Yoo, J., Cha, S., Seong Son, H.: Software safety analysis of function block diagrams using fault trees. Reliab. Eng. Syst. Saf. 88(3), 215–228 (2005)

    Article  Google Scholar 

  33. Offutt, A.J., Lee, A., Rothermel, G., Untch, R.H., Zapf, C.: An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Methodol. (TOSEM) 5(2), 99–118 (1996)

    Article  Google Scholar 

  34. De Moura, L., Bjørner, N.: Z3: An efficient smt solver. In: Tools and Algorithms for the Construction and Analysis of Systems, pp. 337–340. Springer, Berlin (2008)

Download references

Acknowledgments

The authors thank the anonymous reviewers for their valuable comments and suggestions to improve the quality of the paper. This research was partially supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2013R1A1A2058618) and the IT R&D program of MSIP/KEIT [10044457, Development of Autonomous Intelligent Collaboration Framework for Knowledge Bases and Smart Devices].

Author information

Authors and Affiliations

  1. Department of Computer Science, KAIST, Daejeon, Republic of Korea

    Donghwan Shin, Eunkyoung Jee & Doo-Hwan Bae

Authors
  1. Donghwan Shin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eunkyoung Jee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Doo-Hwan Bae
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Donghwan Shin.

Additional information

Communicated by Dr. Perdita Stevens.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shin, D., Jee, E. & Bae, DH. Comprehensive analysis of FBD test coverage criteria using mutants. Softw Syst Model 15, 631–645 (2016). https://doi.org/10.1007/s10270-014-0428-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10270-014-0428-y

Keywords

Search

Navigation