Abstract
We present a novel approach to complete property-oriented white box module testing: a finite test suite, created and extended online (that is, during test execution), in combination with model learning and model checking allows to prove or disprove that a software module fulfils an arbitrary LTL property. The approach is applicable for modules with possibly infinite input and output domains. The testing strategy is based on the concept of black box checking proposed by other authors and on a complete model-based equivalence testing strategy developed previously by the authors of this paper. Since the white box approach allows for static analyses, basic information about internal states, guards and assignment expressions can be extracted from the module code. With this information at hand, the approach effectively performs a proof whether the implementation satisfies the specified property. The “classical” black box checking method is accelerated by means of coverage-guided fuzzing, in combination with effective methods for learning, failure monitoring, and conformance testing. This combination allows to reduce the overall effort for proving that the software fulfils the desired property in a considerable way.
Niklas Krafczyk is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number 407708394. Felix Brüning, Wen-ling Huang, and Jan Peleska are funded by the German Ministry of Economics, Grant Agreement 20X1908E.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
Note that this theorem has only been formulated for finite traces \(\pi _i\) in [14]. The proof, however, holds for infinite traces \(\pi _i\in (D^ V )^\omega \) as well, because \(\pi _1, \pi _2\in (D^ V )^\omega \) are equivalent if and only if all finite prefixes of \(\pi _1,\pi _2\) with identical length are equivalent.
- 4.
Recall that LTL formulae over free variables from \( V \) have infinite sequences of valuations in \(D^ V \) as models [10].
- 5.
Traces \(\alpha ,\beta \) are distinguishable in T if there exists \(\alpha .(\bar{x}/\bar{y}),\beta .(\bar{x}/\bar{y}') \in T\) with \(\bar{y} \ne \bar{y}'\).
- 6.
Note that finding the largest such set is equivalent to finding the largest clique [6] in an undirected graph with vertexes T where traces are adjacent if and only if they are distinguishable. This constitutes a computationally expensive problem, so that we currently apply a greedy heuristic.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
ABS, AOR, LCR, ROR, UOI.
- 13.
To reproduce these results, our implementations of Algorithm 2 and of the ABS experiment can be accessed at https://doi.org/10.5281/zenodo.8143283.
References
Angluin, D.: Learning regular sets from queries and counterexamples. Inf. Comput. 75(2), 87–106 (1987)
Baier, C., Katoen, J.: Principles of model checking. MIT Press (2008)
Bauer, A., Leucker, M., Schallhart, C.: Runtime verification for LTL and TLTL. ACM Trans. Softw. Eng. Methodol. 20(4), 14:1–14:64 (2011). https://doi.org/10.1145/2000799.2000800
Biere, A., Heljanko, K., Junttila, T., Latvala, T., Schuppan, V.: Linear encodings of bounded LTL model checking. Logical Methods Comput. Sci. 2(5), November 2006. https://doi.org/10.2168/LMCS-2(5:5)2006, http://arxiv.org/abs/cs/0611029, arXiv: cs/0611029
Böhme, M., Pham, V., Roychoudhury, A.: Coverage-based greybox fuzzing as markov chain. In: Weippl, E.R., Katzenbeisser, S., Kruegel, C., Myers, A.C., Halevi, S. (eds.) Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24–28, 2016, pp. 1032–1043. ACM (2016). https://doi.org/10.1145/2976749.2978428
Bomze, I.M., Budinich, M., Pardalos, P.M., Pelillo, M.: The maximum clique problem. In: Du, D., Pardalos, P.M. (eds.) Handbook of Combinatorial Optimization, pp. 1–74. Springer (1999). https://doi.org/10.1007/978-1-4757-3023-4_1
Brauer, J., Peleska, J., Schulze, U.: Efficient and Trustworthy Tool Qualification for Model-Based Testing Tools. In: Nielsen, B., Weise, C. (eds.) ICTSS 2012. LNCS, vol. 7641, pp. 8–23. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-34691-0_3
CENELEC: EN 50128:2011 Railway applications - Communication, signalling and processing systems - Software for railway control and protection systems (2011)
Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. SE-4(3), 178–186 (1978)
Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. The MIT Press, Cambridge (1999)
Dietsche, K.H., Reif, K.: Kraftfahrtechnisches Taschenbuch, 2nd edn. Springer Vieweg (2018)
Dorofeeva, R., El-Fakih, K., Yevtushenko, N.: An improved conformance testing method. In: Wang, F. (ed.) FORTE 2005. LNCS, vol. 3731, pp. 204–218. Springer, Heidelberg (2005). https://doi.org/10.1007/11562436_16
Endo, A.T., da Silva Simão, A.: Evaluating test suite characteristics, cost, and effectiveness of FSM-based testing methods. Inf. Softw. Technol. 55(6), 1045–1062 (2013). https://doi.org/10.1016/j.infsof.2013.01.001
Huang, W.l., Krafczyk, N., Peleska, J.: Model-Based Conformance Testing and Property Testing With Symbolic Finite State Machines - Technical Report. Zenodo, October 2022. https://doi.org/10.5281/zenodo.7267975. https://zenodo.org/record/7267975, to appear in Science of Computer Programming SCP (Part I) and Proceedings of the 10th IPM International Conference on Fundamentals of Software Engineering FSEN 2023 (Part II)
Isberner, M., Howar, F., Steffen, B.: The Open-Source LearnLib. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 487–495. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21690-4_32
ISO/DIS 26262–6: Road vehicles - functional safety - Part 6: Product development: software level (2009)
Manès, V.J.M., Han, H., Han, C., Cha, S.K., Egele, M., Schwartz, E.J., Woo, M.: The art, science, and engineering of fuzzing: a survey. IEEE Trans. Software Eng. (2019). https://doi.org/10.1109/TSE.2019.2946563
Meng, R., Dong, Z., Li, J., Beschastnikh, I., Roychoudhury, A.: Linear-time temporal logic guided greybox fuzzing. In: Proceedings of the 44th International Conference on Software Engineering, ICSE 2022, pp. 1343–1355. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3510003.3510082
Muskardin, E., Aichernig, B.K., Pill, I., Pferscher, A., Tappler, M.: Aalpy: an active automata learning library. Innov. Syst. Softw. Eng. 18(3), 417–426 (2022). https://doi.org/10.1007/s11334-022-00449-3
Offutt, A.J., Lee, A., Rothermel, G., Untch, R.H., Zapf, C.: An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Methodol. (TOSEM) 5(2), 99–118 (1996)
Peled, D., Vardi, M.Y., Yannakakis, M.: Black box checking. In: Wu, J., Chanson, S.T., Gao, Q. (eds.) Formal Methods for Protocol Engineering and Distributed Systems. IAICT, vol. 28, pp. 225–240. Springer, Boston (1999). https://doi.org/10.1007/978-0-387-35578-8_13
Peled, D., Vardi, M.Y., Yannakakis, M.: Black box checking. J. Automata Lang. Combinatorics 7(2), 225–246 (2002). https://doi.org/10.25596/jalc-2002-225
Pferscher, A., Aichernig, B.K.: Stateful black-box fuzzing of bluetooth devices using automata learning. In: Deshmukh, J.V., Havelund, K., Perez, I. (eds.) NFM 2022. LNCS, vol. 13260, pp. 373–392. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-06773-0_20
Sistla, A.P.: Safety, liveness and fairness in temporal logic. Formal Aspects Comput. 6(5), 495–511 (1994). https://doi.org/10.1007/BF01211865. http://link.springer.com/article/10.1007/BF01211865
Vaandrager, F., Garhewal, B., Rot, J., Wißmann, T.: A new approach for active automata learning based on apartness. In: TACAS 2022. LNCS, vol. 13243, pp. 223–243. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99524-9_12
Vasilevskii, M.P.: Failure diagnosis of automata. Kibernetika (Transl.) 4, 98–108 (July-August 1973)
Waga, M.: Falsification of cyber-physical systems with robustness-guided black-box checking. In: Proceedings of the 23rd International Conference on Hybrid Systems: Computation and Contro, HSCC 2020. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3365365.3382193, https://doi.org/10.1145/3365365.3382193
WG-71, R.S.E.: RTCA DO-178C - Software Considerations in Airborne Systems and Equipment Certification. 1140 Connecticut Avenue, N.W., Suite 1020, Washington, D.C. 20036, December 2011
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 IFIP International Federation for Information Processing
About this paper
Cite this paper
Brüning, F., Gleirscher, M., Huang, Wl., Krafczyk, N., Peleska, J., Sachtleben, R. (2023). Complete Property-Oriented Module Testing. In: Bonfanti, S., Gargantini, A., Salvaneschi, P. (eds) Testing Software and Systems. ICTSS 2023. Lecture Notes in Computer Science, vol 14131. Springer, Cham. https://doi.org/10.1007/978-3-031-43240-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-43240-8_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43239-2
Online ISBN: 978-3-031-43240-8
eBook Packages: Computer ScienceComputer Science (R0)
