Abstract
With the rapid development and wide application of artificial neural networks, formal modeling and verification of their security become more and more significant. As a basic step towards the direction, this work proposes a comprehensive modeling framework for back propagation (BP) neural networks based on the formal language MSVL. In this framework, the structure and behavior of a BP neural network are formalized as specifications of data structures and operations, and they are in turn implemented as MSVL structs and functions, respectively. Based on the formalization, models of BP neural networks can be constructed and trained according to the requirements of users. Experimental results show that these models have good performance in terms of metrics concerning training and prediction such as loss and accuracy.
This research is supported by National Natural Science Foundation of China Grant Nos. 61972301, 61672403 and 61732013, National Natural Science Foundation of Shaanxi Province under Grant No. 2020GY-043, and Shaanxi Key Science and Technology Innovation Team Project Grant No. 2019TD-001.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Data set acquisition address: http://yann.lecun.com/exdb/mnist/.
References
Chen, Y., Chi, Y., Fan, J., Ma, C.: Gradient descent with random initialization: fast global convergence for nonconvex phase retrieval. Math. Program. 176(1–2), 5–37 (2019). https://doi.org/10.1007/s10107-019-01363-6
Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. The MIT Press, Hoboken (2000)
Williams, R.J., Rumelhart, D.E., Hinton, G.E.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986)
Duan, Z.: Temporal Logic and Temporal Logic Programming. Science Press, Beijing (2006)
Duan, Z., Tian, C.: A unified model checking approach with projection temporal logic. In: Liu, S., Maibaum, T., Araki, K. (eds.) ICFEM 2008. LNCS, vol. 5256, pp. 167–186. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88194-0_12
Duan, Z., Yang, X., Koutny, M.: Framed temporal logic programming. Sci. Comput. Program. 70(1), 31–61 (2008)
Ghodsi, Z., Gu, T., Garg, S.: Safetynets: verifiable execution of deep neural networks on an untrusted cloud. In: NIPS, pp. 4672–4681 (2017)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Whye Teh, Y., Titterington, M. (eds.) Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Chia Laguna Resort, Sardinia, vol. 9, pp. 249–256. PMLR (2010)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778. IEEE Computer Society (2016)
Huang, X., Kwiatkowska, M., Wang, S., Wu, M.: Safety verification of deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 3–29. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_1
Kanso, K., Setzer, A.: A light-weight integration of automated and interactive theorem proving. Math. Struct. Comput. Sci. 26(01), 129–153 (2016)
Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: an efficient SMT solver for verifying deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 97–117. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_5
Khirirat, S., Feyzmahdavian, H.R., Johansson, M.: Mini-batch gradient descent: faster convergence under data sparsity. In: CDC, pp. 2880–2887. IEEE (2017)
Kingma, D.P., Adam, J.B.: A method for stochastic optimization. In: ICLR (Poster) (2015)
Pang, T., Xu, K., Dong, Y., Du, C., Chen, N., Zhu, J.: Rethinking softmax cross-entropy loss for adversarial robustness. CoRR, abs/1905.10626 (2019)
Singh, G., Gehr, T., Püschel, M., Vechev, M.: An abstract domain for certifying neural networks. In: Proceedings ACM Programming Language 3(POPL) (2019). Article 41
Tian, C., Duan, Z.: Expressiveness of propositional projection temporal logic with star. Theoretical Comput. Sci. 412, 1729–1744 (2011)
Wang, X., Tian, C., Duan, Z., Zhao, L.: MSVL: a typed language for temporal logic programming. Front. Comput. Sci. 11(5), 762–785 (2017). https://doi.org/10.1007/s11704-016-6059-4
Randall Wilson, D., Martinez, T.R.: The general inefficiency of batch training for gradient descent learning. Neural Networks 16(10), 1429–1451 (2003)
Yang, K., Duan, Z., Tian, C., Zhang, N.: A compiler for MSVL and its applications. Theor. Comput. Sci. 749, 2–16 (2018)
Zhang, H., Cheng, X., Zang, H., Park, D.H.: Compiler-level matrix multiplication optimization for deep learning. CoRR, abs/1909.10616 (2019)
Zhang, N., Duan, Z., Tian, C.: Model checking concurrent systems with MSVL. Sci. China Inf. Sci. 59(11), 101–118 (2016). https://doi.org/10.1007/s11432-015-0882-6
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhao, L., Feng, Z., Wang, X., Shu, X. (2021). An MSVL-Based Modeling Framework for Back Propagation Neural Networks. In: Xue, J., Nagoya, F., Liu, S., Duan, Z. (eds) Structured Object-Oriented Formal Language and Method. SOFL+MSVL 2020. Lecture Notes in Computer Science(), vol 12723. Springer, Cham. https://doi.org/10.1007/978-3-030-77474-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-77474-5_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77473-8
Online ISBN: 978-3-030-77474-5
eBook Packages: Computer ScienceComputer Science (R0)