skip to main content
research-article

Logic Shrinkage: Learned Connectivity Sparsification for LUT-Based Neural Networks

Published: 01 September 2023 Publication History

Abstract

Field-programmable gate array (FPGA)–specific deep neural network (DNN) architectures using native lookup tables (LUTs) as independently trainable inference operators have been shown to achieve favorable area-accuracy and energy-accuracy trade-offs. The first work in this area, LUTNet, exhibited state-of-the-art performance for standard DNN benchmarks. In this article, we propose the learned optimization of such LUT-based topologies, resulting in higher-efficiency designs than via the direct use of off-the-shelf, hand-designed networks. Existing implementations of this class of architecture require the manual specification of the number of inputs per LUT, K. Choosing appropriate K a priori is challenging. Doing so at even high granularity, for example, per layer, is a time-consuming and error-prone process that leaves FPGAs’ spatial flexibility underexploited. Furthermore, prior works see LUT inputs connected randomly, which does not guarantee a good choice of network topology. To address these issues, we propose logic shrinkage, a fine-grained netlist pruning methodology enabling K to be automatically learned for every LUT in a neural network targeted for FPGA inference. By removing LUT inputs determined to be of low importance, our method increases the efficiency of the resultant accelerators. Our GPU-friendly solution to LUT input removal is capable of processing large topologies during their training with negligible slowdown. With logic shrinkage, we improve the area and energy efficiency of the best-performing LUTNet implementation of the CNV network classifying CIFAR-10 by 1.54× and 1.31×, respectively, while matching its accuracy. This implementation also reaches 2.71× the area efficiency of an equally accurate, heavily pruned binary neural network (BNN). On ImageNet, with the Bi-Real Net architecture, employment of logic shrinkage results in a post-synthesis area reduction of 2.67× vs. LUTNet, allowing for implementation that was previously impossible on today’s largest FPGAs. We validate the benefits of logic shrinkage in the context of real application deployment by implementing a face mask detection DNN using a BNN, LUTNet, and logic-shrunk layers. Our results show that logic shrinkage results in area gains versus LUTNet (up to 1.20×) and equally pruned BNNs (up to 1.08×), along with accuracy improvements.

References

[1]
Tushar Agrawal, K. Imran, Matteo Figus, and C. Kirkpatrick. 2020. Automatically Detecting Personal Protective Equipment on Persons in Images Using Amazon Rekognition. https://aws.amazon.com/cn/blogs/machine-learning/automaticallydetecting-personal-protective-equipment-on-persons-in-images-using-amazon-rekognition/.
[2]
Adnane Cabani, Karim Hammoudi, Halim Benhabiles, and Mahmoud Melkemi. 2021. MaskedFace-Net–A dataset of correctly/incorrectly masked face images in the context of COVID-19. Smart Health 19 (2021), 100144.
[3]
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. 2018. Model compression and acceleration for deep neural networks: the principles, progress, and challenges. IEEE Signal Processing Magazine 35, 1 (2018).
[4]
Jia Deng, Wei Dong, Richard Socher, Jia Li, Kai Li, and Feifei Li. 2009. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248–255.
[5]
Nael Fasfous, Manoj-Rohit Vemparala, Alexander Frickenstein, Lukas Frickenstein, Mohamed Badawy, and Walter Stechele. 2021. BinaryCoP: Binary neural network-based COVID-19 face-mask wear and positioning predictor on edge devices. In IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW’21). IEEE, 108–115.
[6]
Shiming Ge, Jia Li, Qiting Ye, and Zhao Luo. 2017. Detecting masked faces in the wild with LLE-CNNs. In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2682–2690.
[7]
Mohammad Ghasemzadeh, Mohammad Samragh, and Farinaz Koushanfar. 2018. ReBNet: Residual binarized neural network. In IEEE International Symposium on Field-Programmable Custom Computing Machines. IEEE, 57–64.
[8]
Karim Hammoudi, Adnane Cabani, Halim Benhabiles, and Mahmoud Melkemi. 2020. Validating the correct wearing of protection mask by taking a selfie: Design of a mobile application “CheckYourMask” to limit the spread of COVID-19. HAL preprint hal-02614790 (2020).
[9]
Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural network. In Conference on Neural Information Processing Systems, Vol. 28. IEEE.
[10]
Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. Master’s thesis. University of Toronto.
[11]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84–90.
[12]
Amey Kulkarni, Amulya Vishwanath, and Chintan Shah. 2020. Implementing a real-time, AI-based, face mask detector application for COVID-19. NVIDIA Developer Blog 13 (2020).
[13]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11 (1998).
[14]
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. 2019. SNIP: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations.
[15]
Shuang Liang, Shouyi Yin, Leibo Liu, Wayne Luk, and Shaojun Wei. 2018. FP-BNN: Binarized neural network on FPGA. Neurocomputing 275, C (2018).
[16]
Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. DARTS: Differentiable architecture search. In International Conference on Learning Representations.
[17]
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. 2018. Bi-Real Net: Enhancing the performance of 1-bit CNNs with improved representational capability and advanced training algorithm. In European Conference on Computer Vision, Springer, 722–737.
[18]
Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang, Alan Yuille, and Jianchao Yang. 2019. AtomNAS: Fine-grained end-to-end neural architecture search. In International Conference on Learning Representations.
[19]
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2017. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations.
[20]
Mahdi Nazemi, Ghasem Pasandi, and Massoud Pedram. 2018. NullaNet: Training deep neural networks for reduced-memory-access inference. arXiv preprint arXiv:1807.08716 (2018).
[21]
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. 2011. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning.
[22]
World Health Organization. 2020. Archived: WHO Timeline - COVID-19. https://www.who.int/news/item/27-04-2020-who-timeline---covid-19.
[23]
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. 2019. Regularized evolution for image classifier architecture search. In AAAI Conference on Artificial Intelligence, Vol. 33, 4780–4789.
[24]
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. 2021. A comprehensive survey of neural architecture search: Challenges and solutions. Computing Surveys 54, 4 (2021).
[25]
Abigail See, Minh-Thang Luong, and Christopher D. Manning. 2016. Compression of neural machine translation models via pruning. In SIGNLL Conference on Computational Natural Language Learning. 291–301.
[26]
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer. 2017. Efficient processing of deep neural networks: a tutorial and survey. Proceedings of the IEEE 105, 12 (2017).
[27]
Yaman Umuroglu, Yash Akhauri, Nicholas J. Fraser, and Michaela Blott. 2020. LogicNets: Co-designed neural networks and circuits for extreme-throughput applications. In International Conference on Field-Programmable Logic and Applications. IEEE, 291–297.
[28]
Yaman Umuroglu, Nicholas J. Fraser, Giulio Gambardella, Michaela Blott, Philip H. W. Leong, Magnus Jahre, and Kees Vissers. 2017. FINN: A framework for fast, scalable binarized neural network inference. In ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. ACM, 65–74.
[29]
Erwei Wang, James J. Davis, Peter Y. K. Cheung, and George A. Constantinides. 2019. LUTNet: Rethinking inference in FPGA soft logic. In IEEE International Symposium on Field-Programmable Custom Computing Machines. IEEE, 26–34.
[30]
Erwei Wang, James J. Davis, Peter Y. K. Cheung, and George A. Constantinides. 2020. LUTNet: Learning FPGA configurations for highly efficient neural network inference. IEEE Transactions on Computers 69, 12 (2020).
[31]
Erwei Wang, James J. Davis, Georgios-Ilias Stavrou, Peter Y. K. Cheung, George A. Constantinides, and Mohamed Abdelfattah. 2022. Logic shrinkage: Learned FPGA netlist sparsity for efficient neural network inference. In ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. ACM, 101–111.
[32]
Erwei Wang, James J. Davis, Ruizhe Zhao, Ho-Cheung Ng, Xinyu Niu, Wayne Luk, Peter Y. K. Cheung, and George A. Constantinides. 2019. Deep neural network approximation for custom hardware: Where we’ve been, where we’re going. Computing Surveys 52, 2 (2019).
[33]
Zhongyuan Wang, Guangcheng Wang, Baojin Huang, Zhangyang Xiong, Qi Hong, Hao Wu, Peng Yi, Kui Jiang, Nanxi Wang, Yingjiao Pei, et al. 2020. Masked face recognition dataset and application. arXiv preprint arXiv:2003.09093 (2020).
[34]
Zekun Wang, Pengwei Wang, Peter C. Louis, Lee E. Wheless, and Yuankai Huo. 2021. Wearmask: Fast in-browser face mask detection with serverless edge computing for COVID-19. arXiv preprint arXiv:2101.00784 (2021).
[35]
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2018. Learning transferable architectures for scalable image recognition. In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 8697–8710.

Cited By

View all
  • (2023)Recognition and Analysis of Scene-Emotion in Photographic Works Based on AI TechnologyInternational Journal of Information Technologies and Systems Approach10.4018/IJITSA.32605516:2(1-15)Online publication date: 20-Jul-2023

Index Terms

  1. Logic Shrinkage: Learned Connectivity Sparsification for LUT-Based Neural Networks

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Transactions on Reconfigurable Technology and Systems
        ACM Transactions on Reconfigurable Technology and Systems  Volume 16, Issue 4
        December 2023
        343 pages
        ISSN:1936-7406
        EISSN:1936-7414
        DOI:10.1145/3615981
        • Editor:
        • Deming Chen
        Issue’s Table of Contents

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 01 September 2023
        Online AM: 10 February 2023
        Accepted: 26 January 2023
        Revised: 23 December 2022
        Received: 25 September 2022
        Published in TRETS Volume 16, Issue 4

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. LUT-based neural networks
        2. binary neural networks
        3. pruning
        4. neural architecture search

        Qualifiers

        • Research-article

        Funding Sources

        • United Kingdom EPSRC

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)148
        • Downloads (Last 6 weeks)12
        Reflects downloads up to 18 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2023)Recognition and Analysis of Scene-Emotion in Photographic Works Based on AI TechnologyInternational Journal of Information Technologies and Systems Approach10.4018/IJITSA.32605516:2(1-15)Online publication date: 20-Jul-2023

        View Options

        Login options

        Full Access

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        Full Text

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media