Abstract
Neural network robustness measurement is a critical step before deploying neural network applications. However, existing methods, such as neural network verification and validation, do not fully meet our criteria for robustness measurement. From the industrial point-of-view, this paper proposes to use statistical robustness certificates (SRC) for measuring the robustness of neural networks against random noises as well as semantic perturbations and tries to bridge between verification and validation methods through Hoeffding Inequality. Our experiments show that our method is accurate in comparing robustness of different neural networks and has polynomial time complexity which leads to 3x-30x boost in efficiency compared to related methods. Together with the intrinsic statistical guarantee, the issued certificates are considered practical in comparing the robustness of various commercial neural networks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bastani, O., Ioannou, Y., Lampropoulos, L., Vytiniotis, D., Nori, A., Criminisi, A.: Measuring neural net robustness with constraints. In: Advances in Neural Information Processing Systems, pp. 2613–2621 (2016)
Yu, F., Qin, Z., Liu, C., Zhao, L., Wang, Y., Chen, X.: Interpreting and evaluating neural network robustness (2019). arXiv preprint arXiv:1905.04270
Hendrycks, D., Dietterich, T.: Benchmarking neural networks robustness to common corruptions and perturbations (2019). arXiv preprint arXiv:1903.12261
Carlini N., Wagner, D.: Towards evaluating the robustness of neural networks. In: IEEE Symposium on Security and Privacy, pp. 39–57 (2017)
Szegedy, C., et al.: Intriguing properties of neural networks (2013). arXiv preprint arXiv:1312.6199
Liu, C., Arnon, T., Lazarus, C., Barrett, C., Kochenderfer, M.J., Algorithms for verifying deep neural networks (2019). arXiv preprint arXiv:1903.06758
Rauber, J., Brendel, W., Bethge, M.: Foolbox: a python toolbox to benchmark the robustness of machine learning models (2017). arXiv preprint arXiv:1707.04131
Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: an efficient SMT solver for verifying deep neural networks. In: International Conference on Computer Aided Verification, pp. 97–117 (2017)
Boopathy, A., Weng, T.W., Chen, P.Y., Liu, S., Daniel, L.: CNN-Cert: an efficient framework for certifying robustness of convolutional neural networks. Proc. AAAI Conf. Artif. Intell. 33, 3240–3247 (2019)
Singh, G., Gehr, T., Püschel, M., Vechev, M.: An abstract domain for certifying neural networks. Proc. ACM Program. Lang. 3, 1–30 (2019)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications (2017). arXiv preprint arXiv:1704.04861
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples (2014). arXiv preprint arXiv:1412.6572
Singh, G., Gehr, T., Mirman, M., Püschel, M., Vechev, M.: Fast and effective robustness certification. In: Advances in Neural Information Processing Systems, pp. 10802–10813 (2018)
Dutta, S., Jha, S., Sanakaranarayanan, S., Tiwari, A.: Output range analysis for deep neural networks (2017). arXiv preprint arXiv:1709.09130
Huang, X., Kwiatkowska, M., Wang, S., Wu, M.: Safety verification of deep neural networks. In: International Conference on Computer Aided Verification, pp. 3–29 (2017)
Papernot, N., et al.: Technical report on the cleverhans v2. 1.0 adversarial examples library (2016). arXiv preprint arXiv:1610.00768
Baidu (2019).https://github.com/advboxes/perceptron-benchmark
Narodytska, N., Kasiviswanathan, S.P.: Simple black-box adversarial perturbations for deep networks (2016). arXiv preprint arXiv:1612.06299
Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58, 13–30 (1963)
Serfling, R.: Probability inequalities for the sum in sampling without replacement. Ann. Stat. 38, 39–48 (1973)
Papernot, N., McDaniel, P., Wu, X., Jha, S., Swami, A.: Distillation as a defense to adversarial perturbations against deep neural networks. In: IEEE Symposium on Security and Privacy, pp. 582–597 (2016)
Mohapatra, J., Chen, P.Y., Liu, S., Daniel, L.: Towards verifying robustness of neural networks against semantic perturbations (2019). arXiv preprint arXiv:1912.09533
FAA: System Safety Handbook, Washington, DC (2000)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Huang, C., Hu, Z., Huang, X., Pei, K. (2021). Statistical Certification of Acceptable Robustness for Neural Networks. In: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2021. ICANN 2021. Lecture Notes in Computer Science(), vol 12891. Springer, Cham. https://doi.org/10.1007/978-3-030-86362-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-86362-3_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86361-6
Online ISBN: 978-3-030-86362-3
eBook Packages: Computer ScienceComputer Science (R0)