Abstract
The adoption of Artificial Neural Networks (ANNs) in safety-related applications is often avoided because it is difficult to rule out possible misbehaviors with traditional analytical or probabilistic techniques. In this paper we present NeVer, our tool for checking safety of ANNs. NeVer encodes the problem of verifying safety of ANNs into the problem of satisfying corresponding Boolean combinations of linear arithmetic constraints. We describe the main verification algorithm and the structure of NeVer. We present also empirical results confirming the effectiveness of NeVer on realistic case studies.
Article PDF
Similar content being viewed by others
References
Zhang, G.P.: Neural networks for classification: a survey. IEEE Trans. Syst. Man Cybern., Part C Appl. Rev. 30(4), 451–462 (2000)
Smith, D.J., Simpson, K.G.L.: Functional Safety – A Straightforward Guide to Applying IEC 61505 and Related Standards (2nd edn.). Elsevier (2004)
Schumann, J., Gupta, P., Nelson, S.: On verification & validation of neural network based controllers. In: Proc. of International Conf. on Engineering Applications of Neural Networks (EANN’03) (2003)
Kurd, Z., Kelly, T., Austin, J.: Developing artificial neural networks for safety critical systems. Neural Comput. Appl. 16(1), 11–19 (2007)
Clarke, E.M., Emerson, E.A., Sistla, A.P.: Automatic verification of finite-state concurrent systems using temporal logic specifications. ACM Trans. Program. Lang. Syst. (TOPLAS) 8(2), 263 (1986)
Queille, J., Sifakis, J.: Specification and verification of concurrent systems in CESAR. In: International Symposium on Programming, pp. 337–351. Springer (1982)
Schubert, T.: High level formal verification of next-generation microprocessors. In: Proceedings of the 40th annual Design Automation Conference. ACM (2003)
Ball, T., Cook, B., Levin, V., Rajamani, S.K.: SLAM and static driver verifier: Technology transfer of formal methods inside Microsoft. In: Integrated Formal Methods, pp. 1–20. Springer (2004)
Armando, A., Carbone, R., Compagna, L.: LTL model checking for security protocols. In: 20th IEEE Computer Security Foundations Symposium, pp. 385–396 (2007)
Alur, R., Henzinger, T.A., Ho, P.: Automatic symbolic verification of embedded systems. In: IEEE Real-Time Systems Symposium, pp. 2–11 (1993)
Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. Springer (1999)
Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw 2(5), 359–366 (1989)
Pulina, L., Tacchella, A.: An abstraction-refinement approach to verification of artificial neural networks. In: 22nd International Conference on Computer Aided Verification (CAV 2010). Lecture Notes in Computer Science, vol. 6174, pp. 243–257. Springer (2010)
Solar-Lezama, A., Jones, C.G., Bodik, R.: Sketching concurrent data structures. In: 2008 ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 136–148. ACM (2008)
Vechev, M., Yahav, E., Yorsh, G.G.: Abstraction-guided synthesis of synchronization. In: 37th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pp. 327–338. ACM (2010)
Igel, C., Glasmachers, T., Heidrich-Meisner, V.: Shark. J. Mach. Learn. Res. 9, 993–996 (2008)
Franzle, M., Herde, C., Teige, T., Ratschan, S., Schubert, T.: Efficient solving of large non-linear arithmetic constraint systems with complex boolean structure. JSAT, Boolean Modeling and Computation 1, 209–236 (2007)
Yeh, I.C.: Modeling of strength of high-performance concrete using artificial neural networks. Cem. Concr. Res. 28(12), 1797–1808 (1998)
Haykin, S.: Neural Networks: a Comprehensive Foundation. Prentice Hall (2008)
Mackworth, A.K.: Consistency in networks of relations. Artif. Intell. 8(1), 99–118 (1977)
Van Hentenryck, P.: Numerica: a modeling language for global optimization. In: Fifteenth International Joint Conference on Artificial Intelligence (IJCAI), pp. 1642–1650 (1997)
Rossi, F., Van Beek, P., Walsh, T.: Handbook of Constraint Programming. Elsevier Science Ltd (2006)
Barichard, V., Hao, J.K.: A population and interval constraint propagation algorithm. In: Evolutionary Multi-Criterion Optimization, Second International Conference (EMO 2003), pp. 88–101. Springer (2003)
Marques-Silva, J., Lynce, I., Malik, S.: Conflict-driven Clause Learning SAT Solvers. Handbook of Satisfiability, pp. 131–153. IOS Press, Amsterdam (2009)
Barrett, C., Sebastiani, R., Seshia, S.A., Tinelli, C.: Satisfiability Modulo Theories. Handbook of Satisfiability, pp. 825–885. IOS Press, Amsterdam (2009)
Jermann, C., Sam-Haroud, D., Trombettoni, G. (eds.): CP Workshop on Interval Analysis, Constraint Propagation, Applications (IntCP 2009) (2009)
Cousot, P., Cousot, R.: Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: 4th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages, pp. 238–252 (1977)
Clarke, E., Grumberg, O., Jha, S., Lu, Y., Veith, H.: Counterexample-guided abstraction refinement for symbolic model checking. J. ACM (JACM) 50(5), 794 (2003)
Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., Euler, T.: Yale: rapid prototyping for complex data mining tasks. In: 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’06), pp. 935–940. ACM, New York (2006)
Gordeau, R.: Roboop – a robotics object oriented package in C++. http://www.cours.polymtl.ca/roboop (2005)
Rabunal, J.R., Dorrado, J.: Artificial Neural Networks in Real-life Applications. Idea Group Pub (2006)
Witten, I.H., Frank, E.: Data Mining (2nd edn.). Morgan Kaufmann (2005)
Gordon, D.F.: Asimovian adaptive agents. J. Artif. Intell. Res. 13(1), 95–153 (2000)
Pappas, G., Kress-Gazit, H. (eds.): ICRA Workshop on Formal Methods in Robotics and Automation (2009)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Pulina, L., Tacchella, A. NeVer: a tool for artificial neural networks verification. Ann Math Artif Intell 62, 403–425 (2011). https://doi.org/10.1007/s10472-011-9243-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-011-9243-0