Abstract
Noisy or adverse input is a threat to the safe deployment of neural networks in production. To ensure the safe operations of such networks they need to be hardened to work under such conditions. Abstract interpretation, as a tool to formally verify properties of computations, can be used for this task. But, to date, this has mostly been studied for feed-forward networks, but not so for recurrent neural networks. For a subclass of recurrent neural networks, called echo state networks, we propose a new training algorithm using abstract interpretation and convex programming to increase the robustness against noisy inputs. Our empirical results show that the new training regime improves the performance of echo state networks in an open loop setup under high noise and generally improves their performance in closed loop setups.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chatfield, C., Weigend, A.S.: Time series prediction: forecasting the future and understanding the past. Int. J. Forecast. 10(1), 161–163 (1994). https://doi.org/10.1016/0169-2070(94)90058-2
Cousot, P., Cousot, R.: Abstract interpretation. In: Proceedings of the 4th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages - POPL 1977. ACM Press (1977). https://doi.org/10.1145/512950.512973
Diamond, S., Boyd, S.: CVXPY: a Python-embedded modeling language for convex optimization. J. Mach. Learn. Res. 17(83), 1–5 (2016)
Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., Vechev, M.: AI2: safety and robustness certification of neural networks with abstract interpretation. In: 2018 IEEE Symposium on Security and Privacy (SP). IEEE, May 2018. https://doi.org/10.1109/sp.2018.00058
Ghorbal, K., Goubault, E., Putot, S.: The zonotope abstract Domain Taylor1+. In: Bouajjani, A., Maler, O. (eds.) CAV 2009. LNCS, vol. 5643, pp. 627–633. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02658-4_47
Glass, L., Mackey, M.C., Zweifel, P.F.: From clocks to chaos: the rhythms of life. Phys. Today 42(7), 72–72 (1989). https://doi.org/10.1063/1.2811091
Gurfinkel, A., Chaki, S.: Boxes: a symbolic abstract domain of boxes. In: Cousot, R., Martel, M. (eds.) SAS 2010. LNCS, vol. 6337, pp. 287–303. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15769-1_18
Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks. Technical report, GMD Report 148, German National Research Center for Information Technology (2001). http://www.faculty.iu-bremen.de/hjaeger/pubs/EchoStatesTechRep.pdf
Mirman, M., Gehr, T., Vechev, M.: Differentiable abstract interpretation for provably robust neural networks. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 3578–3586. PMLR, Stockholmsmässan, Stockholm Sweden, 10–15 July 2018. http://proceedings.mlr.press/v80/mirman18b.html
O’Donoghue, B., Chu, E., Parikh, N., Boyd, S.: Conic optimization via operator splitting and homogeneous self-dual embedding. J. Optim. Theory Appl. 169(3), 1042–1068 (2016). http://stanford.edu/~boyd/papers/scs.html
O’Donoghue, B., Chu, E., Parikh, N., Boyd, S.: SCS: splitting conic solver, version 2.1.2, November 2019. https://github.com/cvxgrp/scs
Senn, C.W., Kumazawa, I.: Robust echo state networks. In: Proceedings of the 29th Annual Conference of Japanese Neural Network Society (JNNS 2019), September 2019
Singh, G., Gehr, T., Mirman, M., Püschel, M., Vechev, M.: Fast and effective robustness certification. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS 2018NIPS 2018, pp. 10825–10836. Curran Associates Inc., Red Hook (2018)
Singh, G., Gehr, T., Püschel, M., Vechev, M.: An abstract domain for certifying neural networks. Proc. ACM Program. Lang. 3(POPL), 1–30 (2019). https://doi.org/10.1145/3290354
Singh, G., Gehr, T., Püschel, M., Vechev, M.: Boosting robustness certification of neural networks. In: International Conference on Learning Representations (ICLR) (2019)
Van Der Hoeven, J.: Ball arithmetic, November 2009, 33 p. https://hal.archives-ouvertes.fr/hal-00432152
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Senn, C.W., Kumazawa, I. (2020). Abstract Echo State Networks. In: Schilling, FP., Stadelmann, T. (eds) Artificial Neural Networks in Pattern Recognition. ANNPR 2020. Lecture Notes in Computer Science(), vol 12294. Springer, Cham. https://doi.org/10.1007/978-3-030-58309-5_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-58309-5_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58308-8
Online ISBN: 978-3-030-58309-5
eBook Packages: Computer ScienceComputer Science (R0)
