Skip to main content
SpringerLink
Log in

Neural Network Precision Tuning

  • Conference paper
  • First Online:
Quantitative Evaluation of Systems (QEST 2019)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11785))

Included in the following conference series:

Abstract

Minimizing the precision in which the neurons of a neural network compute is a desirable objective to limit the resources needed to execute it. This is specially important for neural networks used in embedded systems. Unfortunately, neural networks are very sensitive to the precision in which they have been trained and changing this precision generally degrades the quality of their answers. In this article, we introduce a new technique to tune the precision of neural networks in such a way that the optimized network computes in a lower precision without modifying the quality of the outputs of more than a percentage chosen by the user. From a technical point of view, we generate a system of linear constraints among integer variables that we can solve by linear programming. The solution to this system is the new precision of the neurons. We present experimental results obtained by using our method.

This work is supported by La Region Occitanie under Grant GRAINE - Syfi. https://www.laregion.fr.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. ANSI/IEEE: IEEE Standard for Binary Floating-point Arithmetic (2008)

    Google Scholar 

  2. Chiang, W., Baranowski, M., Briggs, I., Solovyev, A., Gopalakrishnan, G., Rakamaric, Z.: Rigorous floating-point mixed-precision tuning. In: POPL, pp. 300–315. ACM (2017)

    Article  Google Scholar 

  3. Damouche, N., Martel, M.: Mixed precision tuning with salsa. In: Proceedings of the 8th International Joint Conference on Pervasive and Embedded Computing and Communication Systems, PECCS 2018, pp. 185–194. SciTePress (2018)

    Google Scholar 

  4. Damouche, N., Martel, M., Chapoutot, A.: Improving the numerical accuracy of programs by automatic transformation. STTT 19(4), 427–448 (2017). https://doi.org/10.1007/s10009-016-0435-0

    Article  Google Scholar 

  5. Darulova, E., Horn, E., Sharma, S.: Sound mixed-precision optimization with rewriting. In: Proceedings of the 9th ACM/IEEE International Conference on Cyber-Physical Systems, ICCPS, pp. 208–219. IEEE Computer Society/ACM (2018)

    Google Scholar 

  6. Darulova, E., Kuncak, V.: Sound compilation of reals. In: POPL 2014, pp. 235–248. ACM (2014)

    Google Scholar 

  7. Dutta, S., Jha, S., Sankaranarayanan, S., Tiwari, A.: Output range analysis for deep feedforward neural networks. In: Dutle, A., Muñoz, C., Narkawicz, A. (eds.) NFM 2018. LNCS, vol. 10811, pp. 121–138. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77935-5_9

    Chapter  Google Scholar 

  8. Gao, X., Bayliss, S., Constantinides, G.A.: SOAP: structural optimization of arithmetic expressions for high-level synthesis. In: International Conference on Field-Programmable Technology, pp. 112–119. IEEE (2013)

    Google Scholar 

  9. Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., Vechev, M.T.: AI2: safety and robustness certification of neural networks with abstract interpretation. In: 2018 IEEE Symposium on Security and Privacy, SP, pp. 3–18. IEEE (2018)

    Google Scholar 

  10. Goubault, E.: Static analysis by abstract interpretation of numerical programs and systems, and FLUCTUAT. In: Logozzo, F., Fähndrich, M. (eds.) SAS 2013. LNCS, vol. 7935, pp. 1–3. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38856-9_1

    Chapter  Google Scholar 

  11. Graphics, M.: Algorithmic C Datatypes, software version 2.6 edn. (2011). http://www.mentor.com/esl/catapult/algorithmic

  12. Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: an efficient SMT solver for verifying deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 97–117. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_5

    Chapter  Google Scholar 

  13. Lam, M.O., Hollingsworth, J.K., de Supinski, B.R., LeGendre, M.P.: Automatically adapting programs for mixed-precision floating-point computation. In: Supercomputing, ICS 2013, pp. 369–378. ACM (2013)

    Google Scholar 

  14. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: 6th International Conference on Learning Representations, ICLR 2018 (2018). OpenReview.net

  15. Martel, M.: Floating-point format inference in mixed-precision. In: Barrett, C., Davies, M., Kahsai, T. (eds.) NFM 2017. LNCS, vol. 10227, pp. 230–246. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-57288-8_16

    Chapter  Google Scholar 

  16. Martel, M., Najahi, A., Revy, G.: Code size and accuracy-aware synthesis of fixed-point programs for matrix multiplication. In: Pervasive and Embedded Computing and Communication Systems, pp. 204–214. SciTePress (2014)

    Google Scholar 

  17. Muller, J.M., et al.: Handbook of Floating-Point Arithmetic. Birkhäuser, Boston (2010)

    Book  Google Scholar 

  18. Nguyen, C., et al.: Floating-point precision tuning using blame analysis. In: International Conference on Software Engineering (ICSE). ACM (2016)

    Google Scholar 

  19. Panchekha, P., Sanchez-Stern, A., Wilcox, J.R., Tatlock, Z.: Automatically improving accuracy for floating point expressions. In: PLDI 2015, pp. 1–11. ACM (2015)

    Google Scholar 

  20. Rubio-Gonzalez, C., et al.: Precimonious: tuning assistant for floating-point precision. In: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 27:1–27:12. ACM (2013)

    Google Scholar 

  21. Singh, G., Gehr, T., Mirman, M., Püschel, M., Vechev, M.T.: Fast and effective robustness certification. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, pp. 10825–10836 (2018)

    Google Scholar 

  22. Singh, G., Gehr, T., Püschel, M., Vechev, M.T.: An abstract domain for certifying neural networks. PACMPL 3(POPL), 41:1–41:30 (2019)

    Google Scholar 

  23. Solovyev, A., Jacobsen, C., Rakamarić, Z., Gopalakrishnan, G.: Rigorous estimation of floating-point round-off errors with symbolic taylor expansions. In: Bjørner, N., de Boer, F. (eds.) FM 2015. LNCS, vol. 9109, pp. 532–550. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19249-9_33

    Chapter  Google Scholar 

  24. Wang, S., Pei, K., Whitehouse, J., Yang, J., Jana, S.: Formal security analysis of neural networks using symbolic intervals. In: 27th USENIX Security Symposium, USENIX Security 2018, pp. 1599–1614. USENIX Association (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Numalis, Cap Omega, Rond-Point Benjamin Franklin, 34960, Montpellier, France

    Arnault Ioualalen & Matthieu Martel

  2. Laboratoire de Mathématiques et Physique (LAMPS), Université de Perpignan Via Domitia, Perpignan, France

    Matthieu Martel

Authors
  1. Arnault Ioualalen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matthieu Martel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matthieu Martel .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    David Parker

  2. Saarland University, Saarbrücken, Germany

    Verena Wolf

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ioualalen, A., Martel, M. (2019). Neural Network Precision Tuning. In: Parker, D., Wolf, V. (eds) Quantitative Evaluation of Systems. QEST 2019. Lecture Notes in Computer Science(), vol 11785. Springer, Cham. https://doi.org/10.1007/978-3-030-30281-8_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-30281-8_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-30280-1

  • Online ISBN: 978-3-030-30281-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation