Abstract
Active automata learning comprises techniques for learning automata models of black-box systems by testing such systems. While this form of learning enables model-based analysis and verification, it may also require a substantial amount of interactions with considered systems to learn adequate models, which capture the systems’ behaviour.
The test cases executed during learning can be divided into two categories: (1) test cases to gain knowledge about a system and (2) test cases to falsify a learned hypothesis automaton. The former are selected by learning algorithms, whereas the latter are selected by conformance-testing algorithms. There exist various options for both types of algorithms and there are dependencies between them. In this paper, we investigate the performance of combinations of four different learning algorithms and seven different testing algorithms. For this purpose, we perform learning experiments using 39 benchmark models. Based on experimental results, we discuss insights regarding the performance of different configurations for various types of systems. These insights may serve as guidance for future users of active automata learning.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Available online at http://automata.cs.ru.nl/, accessed: February 2, 2020.
- 2.
Available online at http://automata.cs.ru.nl/, accessed: February 2, 2020.
References
Aichernig, B.K., et al.: Learning a behavior model of hybrid systems through combining model-based testing and machine learning. In: Gaston, C., Kosmatov, N., Le Gall, P. (eds.) ICTSS 2019. LNCS, vol. 11812, pp. 3–21. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31280-0_1
Aichernig, B.K., Bloem, R., Ebrahimi, M., Tappler, M., Winter, J.: Automata learning for symbolic execution. In: Bjørner, N., Gurfinkel, A. (eds.) 2018 Formal Methods in Computer Aided Design, FMCAD 2018, Austin, TX, USA, October 30–November 2, 2018, pp. 1–9. IEEE (2018). https://doi.org/10.23919/FMCAD.2018.8602991
Aichernig, B.K., Mostowski, W., Mousavi, M.R., Tappler, M., Taromirad, M.: Model learning and model-based testing. In: Bennaceur, A., Hähnle, R., Meinke, K. (eds.) Machine Learning for Dynamic Software Analysis: Potentials and Limits. LNCS, vol. 11026, pp. 74–100. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-96562-8_3
Aichernig, B.K., Tappler, M.: Efficient active automata learning via mutation testing. J. Autom. Reason. 63(4), 1103–1134 (2019). https://doi.org/10.1007/s10817-018-9486-0
Angluin, D.: Learning regular sets from queries and counter examples. Inf. Comput. 75(2), 87–106 (1987). https://doi.org/10.1016/0890-5401(87)90052-6
Berg, T., Jonsson, B., Leucker, M., Saksena, M.: Insights to Angluin’s learning. Electron. Notes Theor. Comput. Sci. 118, 3–18 (2005). https://doi.org/10.1016/j.entcs.2004.12.015
Brémond, N., Groz, R.: Case studies in learning models and testing without reset. In: 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICST Workshops 2019, Xi’an, China, April 22–23, 2019, pp. 40–45. IEEE (2019). https://doi.org/10.1109/ICSTW.2019.00030
Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. 4(3), 178–187 (1978). https://doi.org/10.1109/TSE.1978.231496
Combe, D., de la Higuera, C., Janodet, J.-C.: Zulu: an interactive learning competition. In: Yli-Jyrä, A., Kornai, A., Sakarovitch, J., Watson, B. (eds.) FSMNLP 2009. LNCS (LNAI), vol. 6062, pp. 139–146. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14684-8_15
Fiterău-Broştean, P., Janssen, R., Vaandrager, F.: Combining model learning and model checking to analyze TCP implementations. In: Chaudhuri, S., Farzan, A. (eds.) CAV 2016. LNCS, vol. 9780, pp. 454–471. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-41540-6_25
Fiterău-Broştean, P., Lenaerts, T., Poll, E., de Ruiter, J., Vaandrager, F.W., Verleg, P.: Model learning and model checking of SSH implementations. In: Erdogmus, H., Havelund, K. (eds.) Proceedings of the 24th ACM SIGSOFT International SPIN Symposium on Model Checking of Software, Santa Barbara, CA, USA, July 10–14, 2017, pp. 142–151. ACM (2017). https://doi.org/10.1145/3092282.3092289
Fujiwara, S., von Bochmann, G., Khendek, F., Amalou, M., Ghedamsi, A.: Test selection based on finite state models. IEEE Trans. Softw. Eng. 17(6), 591–603 (1991). https://doi.org/10.1109/32.87284
Groz, R., Brémond, N., Simão, A.: Using adaptive sequences for learning non-resettable FSMs. In: Unold, O., Dyrka, W., Wieczorek, W. (eds.) Proceedings of the 14th International Conference on Grammatical Inference, ICGI 2018, Wrocław, Poland, September 5–7, 2018. Proceedings of Machine Learning Research, vol. 93, pp. 30–43. PMLR (2018). http://proceedings.mlr.press/v93/groz19a.html
Howar, F., Steffen, B., Merten, M.: From ZULU to RERS. In: Margaria, T., Steffen, B. (eds.) ISoLA 2010. LNCS, vol. 6415, pp. 687–704. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16558-0_55
Isberner, M., Howar, F., Steffen, B.: The TTT algorithm: a redundancy-free approach to active automata learning. In: Bonakdarpour, B., Smolka, S.A. (eds.) RV 2014. LNCS, vol. 8734, pp. 307–322. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11164-3_26
Isberner, M., Howar, F., Steffen, B.: The open-source LearnLib - a framework for active automata learning. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 487–495. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21690-4_32
Kearns, M.J., Vazirani, U.V.: An Introduction to Computational Learning Theory. MIT Press, Cambridge (1994). https://mitpress.mit.edu/books/introduction-computational-learning-theory
Khosrowjerdi, H., Meinke, K.: Learning-based testing for autonomous systems using spatial and temporal requirements. In: Perrouin, G., Acher, M., Cordy, M., Devroey, X. (eds.) Proceedings of the 1st International Workshop on Machine Learning and Software Engineering in Symbiosis, MASES@ASE 2018, Montpellier, France, September 3, 2018, pp. 6–15. ACM (2018). https://doi.org/10.1145/3243127.3243129
Margaria, T., Niese, O., Raffelt, H., Steffen, B.: Efficient test-based model generation for legacy reactive systems. In: Ninth IEEE International High-Level Design Validation and Test Workshop 2004, Sonoma Valley, CA, USA, November 10–12, 2004, pp. 95–100. IEEE Computer Society (2004). https://doi.org/10.1109/HLDVT.2004.1431246
Neider, D., Smetsers, R., Vaandrager, F., Kuppens, H.: Benchmarks for automata learning and conformance testing. In: Margaria, T., Graf, S., Larsen, K.G. (eds.) Models, Mindsets, Meta: The What, the How, and the Why Not?. LNCS, vol. 11200, pp. 390–416. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22348-9_23
Niese, O.: An integrated approach to testing complex systems. Ph.D. thesis, Dortmund University of Technology (2003). https://d-nb.info/969717474/34
Rivest, R.L., Schapire, R.E.: Inference of finite automata using homing sequences. Inf. Comput. 103(2), 299–347 (1993). https://doi.org/10.1006/inco.1993.1021
de Ruiter, J., Poll, E.: Protocol state fuzzing of TLS implementations. In: Jung, J., Holz, T. (eds.) 24th USENIX Security Symposium, USENIX Security 15, Washington, D.C., USA, August 12–14, 2015, pp. 193–206. USENIX Association (2015). https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/de-ruiter
Shahbaz, M., Groz, R.: Inferring Mealy machines. In: Cavalcanti, A., Dams, D.R. (eds.) FM 2009. LNCS, vol. 5850, pp. 207–222. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-05089-3_14
Smeenk, W., Moerman, J., Vaandrager, F., Jansen, D.N.: Applying automata learning to embedded control software. In: Butler, M., Conchon, S., Zaïdi, F. (eds.) ICFEM 2015. LNCS, vol. 9407, pp. 67–83. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-25423-4_5
Smetsers, R., Moerman, J., Janssen, M., Verwer, S.: Complementing model learning with mutation-based fuzzing. CoRR abs/1611.02429 (2016). http://arxiv.org/abs/1611.02429
Steffen, B., Howar, F., Merten, M.: Introduction to active automata learning from a practical perspective. In: Bernardo, M., Issarny, V. (eds.) SFM 2011. LNCS, vol. 6659, pp. 256–296. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21455-4_8
Tappler, M.: Learning-based testing in networked environments in the presence of timed and stochastic behaviour. Ph.D. thesis, Graz University of Technology (2019)
Tappler, M., Aichernig, B.K., Bloem, R.: Model-based testing IoT communication via active automata learning. In: 2017 IEEE International Conference on Software Testing, Verification and Validation, ICST 2017, Tokyo, Japan, March 13–17, 2017, pp. 276–287. IEEE Computer Society (2017). https://doi.org/10.1109/ICST.2017.32
Vaandrager, F.W.: Model learning. Commun. ACM 60(2), 86–95 (2017). https://doi.org/10.1145/2967606
Vasilevskii, M.P.: Failure diagnosis of automata. Cybernetics 9(4), 653–665 (1973). https://doi.org/10.1007/BF01068590
Wallner, F.: Benchmarking active automata learning configurations. Bachelor’s thesis, Graz University of Technology (2019)
Wallner, F.: Learn-combinations: evaluation framework for combinations of learning and testing algorithms (2019). https://gitlab.com/felixwallner/learn-combinations. Accessed 2 Feb 2020
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Aichernig, B.K., Tappler, M., Wallner, F. (2020). Benchmarking Combinations of Learning and Testing Algorithms for Active Automata Learning. In: Ahrendt, W., Wehrheim, H. (eds) Tests and Proofs. TAP 2020. Lecture Notes in Computer Science(), vol 12165. Springer, Cham. https://doi.org/10.1007/978-3-030-50995-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-50995-8_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-50994-1
Online ISBN: 978-3-030-50995-8
eBook Packages: Computer ScienceComputer Science (R0)