Skip to main content
SpringerLink
Log in

ProxyBNN: Learning Binarized Neural Networks via Proxy Matrices

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12348))

Included in the following conference series:

Abstract

Training Binarized Neural Networks (BNNs) is challenging due to the discreteness. In order to efficiently optimize BNNs through backward propagations, real-valued auxiliary variables are commonly used to accumulate gradient updates. Those auxiliary variables are then directly quantized to binary weights in the forward pass, which brings about large quantization errors. In this paper, by introducing an appropriate proxy matrix, we reduce the weights quantization error while circumventing explicit binary regularizations on the full-precision auxiliary variables. Specifically, we regard pre-binarization weights as a linear combination of the basis vectors. The matrix composed of basis vectors is referred to as the proxy matrix, and auxiliary variables serve as the coefficients of this linear combination. We are the first to empirically identify and study the effectiveness of learning both basis and coefficients to construct the pre-binarization weights. This new proxy learning contributes to new leading performances on benchmark datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    h, w, n and c are kernel height, width, kernel number and input channel number, respectively. For \(1\times 1\) convolutions and FC layers, \([h\times w\times n]\times c\) degrades into \(n\times c\).

  2. 2.

    Further details in appendix 1.

References

  1. Alizadeh, M., Fernández-Marqués, J., Lane, N.D., Gal, Y.: A systematic study of binary neural networks’ optimisation. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=rJfUCoR5KX

  2. Anderson, A.G., Berg, C.P.: The high-dimensional geometry of binary neural networks. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=B1IDRdeCW

  3. Andoni, A., Indyk, P.: Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Commun. ACM 51(1), 117–122 (2008). http://doi.acm.org/10.1145/1327452.1327494

  4. Bahou, A.A., Karunaratne, G., Andri, R., Cavigelli, L., Benini, L.: XNORBIN: A 95 top/s/w hardware accelerator for binary convolutional neural networks. In: 2018 IEEE Symposium in Low-Power and High-Speed Chips, COOL CHIPS 2018, Yokohama, Japan, 18–20 Apr 2018, pp. 1–3. IEEE Computer Society (2018). https://doi.org/10.1109/CoolChips.2018.8373076

  5. Bethge, J., Yang, H., Bornstein, M., Meinel, C.: Back to simplicity: How to train accurate BNNs from scratch? CoRR abs/1906.08637 (2019). http://arxiv.org/abs/1906.08637

  6. Bulat, A., Tzimiropoulos, G.: XNOR-Net++: Improved binary neural networks. In: British Machine Vision Conference, BMVC 2019 (2019)

    Google Scholar 

  7. Bulat, A., Tzimiropoulos, G., Kossaifi, J., Pantic, M.: Improved training of binary networks for human pose estimation and image recognition. CoRR abs/1904.05868 (2019). http://arxiv.org/abs/1904.05868

  8. Cheng, J., Leng, C., Wu, J., Cui, H., Lu, H.: Fast and accurate image matching with cascade hashing for 3d reconstruction. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, 23–28 Jun 2014, pp. 1–8 (2014). https://doi.org/10.1109/CVPR.2014.8

  9. Courbariaux, M., Bengio, Y., David, J.: Binaryconnect: Training deep neural networks with binary weights during propagations. In: Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, 7–12 Dec 2015, Montreal, Quebec, Canada, pp. 3123–3131 (2015). http://papers.nips.cc/paper/5647-binaryconnect-training-deep-neural-networks-with-binary-weights-during-propagations

  10. Darabi, S., Belbahri, M., Courbariaux, M., Nia, V.P.: BNN+: improved binary network training. CoRR abs/1812.11800 (2018). http://arxiv.org/abs/1812.11800

  11. Ding, R., Chin, T., Liu, Z., Marculescu, D.: Regularizing activation distribution for training binarized deep networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 11408–11417 (2019)

    Google Scholar 

  12. Gong, Y., Lazebnik, S., Gordo, A., Perronnin, F.: Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2916–2929 (2013). https://doi.org/10.1109/TPAMI.2012.193

  13. Gu, J., et al.: Projection convolutional neural networks for 1-bit CNNs via discrete back propagation. In: The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, 27 Jan–1 Feb 2019, pp. 8344–8351 (2019). https://doi.org/10.1609/aaai.v33i01.33018344

  14. Gu, J., et al.: Bayesian optimized 1-bit CNNs. In: IEEE Proceedings of the IEEE International Conference on Computer Vision ICCV 2019, Seoul, South Korea (2019)

    Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 Jun 2016, pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  16. He, X., Wang, P., Cheng, J.: K-nearest neighbors hashing. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 Jun 2019, pp. 2839–2848 (2019). http://openaccess.thecvf.com/content_CVPR_2019/html/He_K-Nearest_Neighbors_Hashing_CVPR_2019_paper.html

  17. Helwegen, K., Widdicombe, J., Geiger, L., Liu, Z., Cheng, K., Nusselder, R.: Latent weights do not exist: Rethinking binarized neural network optimization. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8–14 Dec 2019, Vancouver, BC, Canada, pp. 7531–7542 (2019). http://papers.nips.cc/paper/8971-latent-weights-do-not-exist-rethinking-binarized-neural-network-optimization

  18. Hu, Q., Wang, P., Cheng, J.: From hashing to CNNs: Training binary weight networks via hashing. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, 2–7 Feb 2018, pp. 3247–3254 (2018). https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16466

  19. Hu, Q., Wu, J., Bai, L., Zhang, Y., Cheng, J.: Fast k-means for large scale clustering. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM 2017, Singapore, 06–10 Nov 2017, pp. 2099–2102 (2017). https://doi.org/10.1145/3132847.3133091

  20. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks. In: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, 5–10 Dec 2016, Barcelona, Spain, pp. 4107–4115 (2016). http://papers.nips.cc/paper/6573-binarized-neural-networks

  21. Ji, C., Psaltis, D.: Capacity of two-layer feedforward neural networks with binary weights. IEEE Trans. Inf. Theory 44(1), 256–268 (1998). https://doi.org/10.1109/18.651033

  22. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1412.6980

  23. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017). http://doi.acm.org/10.1145/3065386

  24. Leng, C., Dou, Z., Li, H., Zhu, S., Jin, R.: Extremely low bit neural network: Squeeze the last bit out with ADMM. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, 2–7 Feb 2018, pp. 3466–3473 (2018). https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16767

  25. Lin, X., Zhao, C., Pan, W.: Towards accurate binary convolutional neural network. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 Dec 2017, Long Beach, CA, USA, pp. 345–353 (2017). http://papers.nips.cc/paper/6638-towards-accurate-binary-convolutional-neural-network

  26. Lin, Z., Courbariaux, M., Memisevic, R., Bengio, Y.: Neural networks with few multiplications. In: 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2–4 May 2016, Conference Track Proceedings (2016). http://arxiv.org/abs/1510.03009

  27. Liu, C., Ding, W., Hu, Y., Zhang, B., Liu, J., Guo, G.: GBCNs: Genetic binary convolutional networks for enhancing the performance of 1-bit DCNNs. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) (February 2020)

    Google Scholar 

  28. Liu, Z., Wu, B., Luo, W., Yang, X., Liu, W., Cheng, K.: Bi-real net: Enhancing the performance of 1-bit CNNs with improved representational capability and advanced training algorithm. In: Computer Vision - ECCV 2018–15th European Conference, Munich, Germany, 8–14 Sept 2018, Proceedings, Part XV, pp. 747–763 (2018). https://doi.org/10.1007/978-3-030-01267-0_44

  29. Martinez, B., Yang, J., Bulat, A., Tzimiropoulos, G.: Training binary neural networks with real-to-binary convolutions. In: International Conference on Learning Representations (2020). https://openreview.net/forum?id=BJg4NgBKvH

  30. Mayoraz, E., Aviolat, F.: Constructive training methods for feedforward neural networks with binary weights. Int. J. Neural Syst. 7(2), 149–66 (1995)

    Article  Google Scholar 

  31. Mishra, A.K., Nurvitadhi, E., Cook, J.J., Marr, D.: WRPN: wide reduced-precision networks. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 30 Apr–3 May 2018, Conference Track Proceedings (2018). https://openreview.net/forum?id=B1ZvaaeAZ

  32. Oliveira, A.L., Sangiovanni-Vincentelli, A.L.: Learning complex Boolean functions: Algorithms and applications. In: Advances in Neural Information Processing Systems 6, [7th NIPS Conference, Denver, Colorado, USA, 1993], pp. 911–918 (1993). http://papers.nips.cc/paper/857-learning-complex-boolean-functions-algorithms-and-applications

  33. Pagallo, G., Haussler, D.: Boolean feature discovery in empirical learning. Mach. Learn. 5, 71–99 (1990). https://doi.org/10.1007/BF00115895

  34. Paszke, A., et al.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8–14 Dec 2019, Vancouver, BC, Canada, pp. 8024–8035 (2019). http://papers.nips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library

  35. Peters, J.W., Genewein, T., Welling, M.: Probabilistic binary neural networks (2019). https://openreview.net/forum?id=B1fysiAqK7

  36. Qin, H., Gong, R., Liu, X., Wei, Z., Yu, F., Song, J.: IR-Net: Forward and backward information retention for highly accurate binary neural networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle Wastington, USA (June 2020)

    Google Scholar 

  37. Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-Net: Imagenet classification using binary convolutional neural networks. In: Computer Vision - ECCV 2016–14th European Conference, Amsterdam, The Netherlands, 11–14 Oct 2016, Proceedings, Part IV, pp. 525–542 (2016). https://doi.org/10.1007/978-3-319-46493-0_32

  38. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.R.: ORB: An efficient alternative to SIFT or SURF. In: IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, 6–13 Nov 2011, pp. 2564–2571 (2011). https://doi.org/10.1109/ICCV.2011.6126544

  39. Santurkar, S., Tsipras, D., Ilyas, A., Madry, A.: How does batch normalization help optimization? In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 Dec 2018, Montréal, Canada, pp. 2488–2498 (2018). http://papers.nips.cc/paper/7515-how-does-batch-normalization-help-optimization

  40. Schonemann, P.H.: A generalized solution of the orthogonal procrustes problem. Psychometrika 31(1), 1–10 (1966)

    Article  MathSciNet  Google Scholar 

  41. Shayer, O., Levi, D., Fetaya, E.: Learning discrete weights using the local reparameterization trick. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 30 Apr–3 May 2018, Conference Track Proceedings. OpenReview.net (2018). https://openreview.net/forum?id=BySRH6CpW

  42. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1409.1556

  43. Soudry, D., Hubara, I., Meir, R.: Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights. In: Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, 8–13 Dec 2014, Montreal, Quebec, Canada, pp. 963–971 (2014). http://papers.nips.cc/paper/5269-expectation-backpropagation-parameter-free-training-of-multilayer-neural-networks-with-continuous-or-discrete-weights

  44. Soudry, D., Meir, R.: Mean field Bayes backpropagation: scalable training of multilayer neural networks with binary weights (2013)

    Google Scholar 

  45. Tang, W., Hua, G., Wang, L.: How to train a compact binary neural network with high accuracy? In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 4–9 Feb 2017, San Francisco, California, USA, pp. 2625–2631 (2017). http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14619

  46. Wang, P., He, X., Li, G., Zhao, T., Cheng, J.: Sparsity-inducing binarized neural networks. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) (February 2020)

    Google Scholar 

  47. Wen, Y., Zhang, K., Li, Z., Qiao, Y.: A discriminative feature learning approach for deep face recognition. In: Computer Vision-ECCV 2016–14th European Conference, Amsterdam, The Netherlands, 11–14 Oct 2016, Proceedings, Part VII, pp. 499–515 (2016). https://doi.org/10.1007/978-3-319-46478-7_31

  48. Yang, J., et al.: Quantization networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach CA, USA (June 2019)

    Google Scholar 

  49. Yazdani, M.: Linear backprop in non-linear networks. In: Compact Deep Neural Network Representation with Industrial Applications Workshop, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 Dec 2018, Montréal, Canada (2018)

    Google Scholar 

  50. Zagoruyko, S., Komodakis, N.: Wide residual networks. In: Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, 19–22 Sept 2016 (2016). http://www.bmva.org/bmvc/2016/papers/paper087/index.html

  51. Zhao, T., He, X., Cheng, J., Hu, J.: Bitstream: Efficient computing architecture for real-time low-power inference of binary neural networks on CPUs. In: Boll, S., Lee, K.M., Luo, J., Zhu, W., Byun, H., Chen, C.W., Lienhart, R., Mei, T. (eds.) 2018 ACM Multimedia Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea, 22–26 Oct 2018, pp. 1545–1552. ACM (2018). https://doi.org/10.1145/3240508.3240673

  52. Zhou, S., Ni, Z., Zhou, X., Wen, H., Wu, Y., Zou, Y.: DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR abs/1606.06160 (2016). http://arxiv.org/abs/1606.06160

Download references

Acknowledgement

This work was supported in part by National Natural Science Foundation of China (No.61972396, 61876182, 61906193), National Key Research and Development Program of China (No. 2019AAA0103402), the Strategic Priority Research Program of Chinese Academy of Science(No.XDB32050200), the Advance Research Program (No. 31511130301), and Jiangsu Frontier Technology Basic Research Project (No. BK20192004).

Author information

Authors and Affiliations

  1. NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Xiangyu He, Zitao Mo, Ke Cheng, Weixiang Xu, Qinghao Hu, Peisong Wang & Jian Cheng

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

    Xiangyu He, Ke Cheng, Weixiang Xu & Jian Cheng

  3. Center for Excellence in Brain Science and Intelligence Technology, Beijing, China

    Jian Cheng

  4. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

Authors
  1. Xiangyu He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zitao Mo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ke Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weixiang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qinghao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Peisong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Qingshan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jian Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian Cheng .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 193 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

He, X. et al. (2020). ProxyBNN: Learning Binarized Neural Networks via Proxy Matrices. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12348. Springer, Cham. https://doi.org/10.1007/978-3-030-58580-8_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58580-8_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58579-2

  • Online ISBN: 978-3-030-58580-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation