Skip to main content
SpringerLink
Log in

Learning Finite State Models from Recurrent Neural Networks

  • Conference paper
  • First Online:
Integrated Formal Methods (IFM 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13274))

Included in the following conference series:

Abstract

Explaining and verifying the behavior of recurrent neural networks (RNNs) is an important step towards achieving confidence in machine learning. The extraction of finite state models, like deterministic automata, has been shown to be a promising concept for analyzing RNNs. In this paper, we apply a black-box approach based on active automata learning combined with model-guided conformance testing to learn finite state machines (FSMs) from RNNs. The technique efficiently infers a formal model of an RNN classifier’s input-output behavior, regardless of its inner structure. In several experiments, we compare this approach to other state-of-the-art FSM extraction methods. By detecting imprecise generalizations in RNNs that other techniques miss, model-guided conformance testing learns FSMs that more accurately model the RNNs under examination. We demonstrate this by identifying counterexamples with this testing approach that falsifies wrong hypothesis models learned by other techniques. This entails that testing guided by learned automata can be a useful method for finding adversarial inputs, that is, inputs incorrectly classified due to improper generalization.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Source code, experiments, and interactive examples can be found at: https://github.com/DES-Lab/Extracting-FSM-From-RNNs.

  2. 2.

    DOI of the artifact: https://doi.org/10.5281/zenodo.6412571.

  3. 3.

    https://github.com/tech-srl/lstar_extraction.

References

  1. Aichernig, B.K., et al.: Learning a behavior model of hybrid systems through combining model-based testing and machine learning. In: Gaston, C., Kosmatov, N., Gall, P.L. (eds.) Testing Software and Systems - 31st IFIP WG 6.1 International Conference, ICTSS 2019, Paris, France, 15–17 October 2019, Proceedings. LNPSE, vol. 11812, pp. 3–21. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31280-0_1

  2. Aichernig, B.K., Mostowski, W., Mousavi, M.R., Tappler, M., Taromirad, M.: Model learning and model-based testing. In: Bennaceur, A., Hähnle, R., Meinke, K. (eds.) Machine Learning for Dynamic Software Analysis: Potentials and Limits. LNCS, vol. 11026, pp. 74–100. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-96562-8_3

  3. Aichernig, B.K., Tappler, M., Wallner, F.: benchmarking combinations of learning and testing algorithms for active automata learning. In: Ahrendt, W., Wehrheim, H. (eds.) TAP 2020. LNCS, vol. 12165, pp. 3–22. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50995-8_1

  4. Angluin, D.: Learning regular sets from queries and counterexamples. Inf. Comput. 75(2), 87–106 (1987). https://doi.org/10.1016/0890-5401(87)90052-6

  5. Carr, S., Jansen, N., Topcu, U.: Verifiable RNN-based policies for POMDPs under temporal logic constraints. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 4121–4127. ijcai.org (2020). https://doi.org/10.24963/ijcai.2020/570

  6. Cho, K., van Merrienboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: Encoder-decoder approaches. CoRR abs/1409.1259 (2014). http://arxiv.org/abs/1409.1259

  7. Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Software Eng. 4(3), 178–187 (1978). https://doi.org/10.1109/TSE.1978.231496

  8. Dong, G., Wang, J., Sun, J., Zhang, Y., Wang, X., Dai, T., Dong, J.S., Wang, X.: Towards interpreting recurrent neural networks through probabilistic abstraction. In: 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, 21–25 September 2020, pp. 499–510. IEEE, Melbourne, Australia (2020). https://doi.org/10.1145/3324884.3416592

  9. Gargantini, A.: 4 conformance testing. In: Broy, M., Jonsson, B., Katoen, J.-P., Leucker, M., Pretschner, A. (eds.) Model-Based Testing of Reactive Systems. LNCS, vol. 3472, pp. 87–111. Springer, Heidelberg (2005). https://doi.org/10.1007/11498490_5

  10. Ghosh, B., Neider, D.: A formal language approach to explaining RNNs. CoRR abs/2006.07292 (2020). https://arxiv.org/abs/2006.07292

  11. Gopinath, D., Katz, G., Pasareanu, C.S., Barrett, C.W.: Deepsafe: a data-driven approach for checking adversarial robustness in neural networks. CoRR abs/1710.00486 (2017). http://arxiv.org/abs/1710.00486

  12. Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5), 93:1–93:42 (2019). https://doi.org/10.1145/3236009

  13. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

  14. Howar, F., Steffen, B., Merten, M.: From ZULU to RERS - lessons learned in the ZULU challenge. In: ISoLA 2010. LNCS, vol. 6415, pp. 687–704 (2010)

    Google Scholar 

  15. Huang, X., Kwiatkowska, M., Wang, S., Wu, M.: Safety verification of deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 3–29. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_1

  16. Isberner, M., Howar, F., Steffen, B.: The open-source LearnLib. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 487–495. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21690-4_32

  17. Khmelnitsky, I., et al.: Property-directed verification and robustness certification of recurrent neural networks. In: Hou, Z., Ganesh, V. (eds.) ATVA 2021. LNCS, vol. 12971, pp. 364–380. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88885-5_24

  18. Kleene, S.C.: Representation of Events in Nerve Nets and Finite Automata. RAND Corporation, Santa Monica, CA (1951)

    Google Scholar 

  19. Koul, A., Fern, A., Greydanus, S.: Learning finite state representations of recurrent policy networks. In: 7th International Conference on Learning Representations, ICLR 2019, 6–9 May 2019. OpenReview.net, New Orleans, LA, USA (2019). https://openreview.net/forum?id=S1gOpsCctm

  20. Mayr, F., Yovine, S.: regular inference on artificial neural networks. In: Holzinger, A., Kieseberg, P., Tjoa, A.M., Weippl, E. (eds.) CD-MAKE 2018. LNCS, vol. 11015, pp. 350–369. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99740-7_25

  21. Mohri, M., Rostamizadeh, A., Talwalkar, A.: Foundations of Machine Learning. The MIT Press (2012)

    Google Scholar 

  22. Muškardin, E., Aichernig, B.K., Pill, I., Pferscher, A., Tappler, M.: AALpy: an active automata learning library. In: Hou, Z., Ganesh, V. (eds.) ATVA 2021. LNCS, vol. 12971, pp. 67–73. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88885-5_5

  23. Neubig, G., et al.: DyNet: The dynamic neural network toolkit. CoRR abs/1701.03980 (2017). http://arxiv.org/abs/1701.03980

  24. Oliva, C., Lago-Fernández, L.F.: On the interpretation of recurrent neural networks as finite state machines. In: Tetko, I.V., Kurková, V., Karpov, P., Theis, F. (eds.) ICANN 2019. LNCS, vol. 11727, pp. 312–323. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30487-4_25

  25. Oliva, C., Lago-Fernández, L.F.: Stability of internal states in recurrent neural networks trained on regular languages. Neurocomputing 452, 212–223 (2021). https://doi.org/10.1016/j.neucom.2021.04.058

  26. Omlin, C.W., Giles, C.L.: Extraction of rules from discrete-time recurrent neural networks. Neural Networks 9(1), 41–52 (1996). https://doi.org/10.1016/0893-6080(95)00086-0

  27. Ribeiro, M.T., Singh, S., Guestrin, C.: “why should I trust you?": explaining the predictions of any classifier. In: Krishnapuram, B., Shah, M., Smola, A.J., Aggarwal, C.C., Shen, D., Rastogi, R. (eds.) Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 13–17 August 2016. pp. 1135–1144. ACM,San Francisco, CA, USA (2016). https://doi.org/10.1145/2939672.2939778

  28. Shih, A., Darwiche, A., Choi, A.: Verifying binarized neural networks by angluin-style learning. In: Janota, M., Lynce, I. (eds.) SAT 2019. LNCS, vol. 11628, pp. 354–370. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24258-9_25

  29. Siegelmann, H.T., Sontag, E.D.: Turing computability with neural nets. Appl. Math. Lett. 4(6), 77–80 (1991). https://doi.org/10.1016/0893-9659(91)90080-F, https://www.sciencedirect.com/science/article/pii/089396599190080F

  30. Smeenk, W., Moerman, J., Vaandrager, F., Jansen, D.N.: Applying automata learning to embedded control software. In: Butler, M., Conchon, S., Zaïdi, F. (eds.) ICFEM 2015. LNCS, vol. 9407, pp. 67–83. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-25423-4_5

  31. Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., Kroening, D.: Concolic testing for deep neural networks. In: Huchard, M., Kästner, C., Fraser, G. (eds.) Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, 3–7 September 2018, pp. 109–119. ACM, Montpellier, France (2018). https://doi.org/10.1145/3238147.3238172

  32. Tomita, M.: Dynamic construction of finite automata from examples using hill-climbing. In: Conference of the Cognitive Science Society, pp. 105–108 (1982)

    Google Scholar 

  33. Valiant, L.G.: A theory of the learnable. Commun. ACM 27(11), 1134–1142 (1984). https://doi.org/10.1145/1968.1972

  34. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017, pp. 5998–6008. Long Beach, CA, USA 2017), https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

  35. Walkinshaw, N., Bogdanov, K.: Automated comparison of state-based software models in terms of their language and structure. ACM Trans. Softw. Eng. Methodol. 22(2), 13:1–13:37 (2013). https://doi.org/10.1145/2430545.2430549

  36. Wang, C., Niepert, M.: State-regularized recurrent neural networks. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9–15 June 2019. Proceedings of Machine Learning Research, vol. 97, pp. 6596–6606. PMLR,Long Beach, California, USA (2019). http://proceedings.mlr.press/v97/wang19j.html

  37. Wang, Q., Zhang, K., II, A.G.O., Xing, X., Liu, X., Giles, C.L.: A comparison of rule extraction for different recurrent neural network models and grammatical complexity. CoRR abs/1801.05420 (2018). http://arxiv.org/abs/1801.05420

  38. Wang, Q., Zhang, K., Liu, X., Giles, C.L.: Verification of recurrent neural networks through rule extraction. CoRR abs/1811.06029 (2018). http://arxiv.org/abs/1811.06029

  39. Weiss, G., Goldberg, Y., Yahav, E.: Extracting automata from recurrent neural networks using queries and counterexamples. In: Dy, J.G., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, 10–15 July 2018. Proceedings of Machine Learning Research, vol. 80, pp. 5244–5253. PMLR, Stockholm, Sweden (2018). http://proceedings.mlr.press/v80/weiss18a.html

  40. Weiss, G., Goldberg, Y., Yahav, E.: Learning deterministic weighted automata with queries and counterexamples. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8–14 December 2019, pp. 8558–8569. Vancouver, BC, Canada (2019). https://proceedings.neurips.cc/paper/2019/hash/d3f93e7766e8e1b7ef66dfdd9a8be93b-Abstract.html

  41. Yellin, D.M., Weiss, G.: Synthesizing context-free grammars from recurrent neural networks. In: TACAS 2021. LNCS, vol. 12651, pp. 351–369. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72016-2_19

  42. Zachary Chase Lipton, John Berkowitz, C.E.: A critical review of recurrent neural networks for sequence learning. CoRR abs/1506.00019 (2015). http://arxiv.org/abs/1506.00019

Download references

Acknowledgments

This work has been supported by the “University SAL Labs" initiative of Silicon Austria Labs (SAL) and its Austrian partner universities for applied fundamental research for electronic based systems.

Author information

Authors and Affiliations

  1. Silicon Austria Labs, TU Graz - SAL DES Lab, Graz, Austria

    Edi Muškardin, Ingo Pill & Martin Tappler

  2. Institute of Software Technology, Graz University of Technology, Graz, Austria

    Edi Muškardin, Bernhard K. Aichernig & Martin Tappler

Authors
  1. Edi Muškardin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bernhard K. Aichernig
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ingo Pill
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Martin Tappler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Edi Muškardin .

Editor information

Editors and Affiliations

  1. ISTI-CNR, Pisa, Italy

    Maurice H. ter Beek

  2. Maynooth University, Maynooth, Ireland

    Rosemary Monahan

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Muškardin, E., Aichernig, B.K., Pill, I., Tappler, M. (2022). Learning Finite State Models from Recurrent Neural Networks. In: ter Beek, M.H., Monahan, R. (eds) Integrated Formal Methods. IFM 2022. Lecture Notes in Computer Science, vol 13274. Springer, Cham. https://doi.org/10.1007/978-3-031-07727-2_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-07727-2_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-07726-5

  • Online ISBN: 978-3-031-07727-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation