Cecilia De la Parra
Akash Kumar
ABSTRACT
Approximate Computing is a promising paradigm for mitigating the computational demands of Deep Neural Networks (DNNs), by leveraging DNN performance and area, throughput or power. The DNN accuracy, affected by such approximations, can be then effectively improved through retraining. In this paper, we present a novel methodology for modelling the approximation error introduced by approximate hardware in DNNs, which accelerates retraining and achieves negligible accuracy loss. To this end, we implement the behavioral simulation of several approximate multipliers and model the error generated by such approximations on pre-trained DNNs for image classification on CIFAR10 and ImageNet. Finally, we optimize the DNN parameters by applying our error model during DNN retraining, to recover the accuracy lost due to approximations. Experimental results demonstrate the efficiency of our proposed method for accelerated retraining (11 x faster for CIFAR10 and 8x faster for ImageNet) for full DNN approximation, which allows us to deploy approximate multipliers with energy savings of up to 36% for 8-bit precision DNNs with an accuracy loss lower than 1%.
- Martín Abadi, Ashish Agarwal, Paul Barham, et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/Google Scholar
- Chris M. Bishop. 1995. Training with Noise is Equivalent to Tikhonov Regularization. Neural Comput. (1995).Google Scholar
- C. De la Parra, A. Guntoro, and A. Kumar. 2020. ProxSim: GPU-based Simulation Framework for Cross-Layer Approximate DNN Optimization. In DATE '20.Google Scholar
- Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. 2015. Deep Learning with Limited Numerical Precision. ICML '15 (2015).Google Scholar
- I. Hammad et al. 2019. Deep Learning Training with Simulated Approximate Multipliers. In ROBIO.Google Scholar
- Issam Hammad and Kamal El-Sankary. 2018. Impact of Approximate Multipliers on VGG Deep Learning Network. IEEE Access (2018).Google ScholarCross Ref
- Song Han et al. 2016. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. In ICLR '16.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. CVPR '15 (2015).Google Scholar
- Xin He et al. 2018. AxTrain: Hardware-Oriented Neural Network Training for Approximate Inference. ISLPED '18 (2018).Google Scholar
- Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel Pruning for Accelerating Very Deep Neural Networks. ICCV '17 (2017).Google Scholar
- Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. University of Toronto (2009).Google Scholar
- Yann LeCun and Corinna Cortes. 2010. MNIST handwritten digit database. (2010). http://yann.lecun.com/exdb/mnist/Google Scholar
- Darryl Dexu Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy. 2016. Fixed Point Quantization of Deep Convolutional Networks. ICML 16 (2016).Google Scholar
- M.A.Hanif et al. 2018. Error resilience analysis for systematically employing approximate computing in convolutional neural networks. DATE 18 (2018).Google Scholar
- Alberto Marchisio, Muhammad Hanif, Maurizio Martina, and Muhammad Shafique. [n.d.].Google Scholar
- Vojtech Mrazek et al. 2016. Design of Power-efficient Approximate Multipliers for Approximate Artificial Neural Networks. In ICCAD '16.Google Scholar
- Vojtěch Mrázek, Radek Hrbáček, Zdeněk Vašíček, and Lukáš Sekanina. 2017. EvoApprox8B: Library of Approximate Adders and Multipliers for Circuit Design and Benchmarking of Approximation Methods. In DATE '17.Google ScholarCross Ref
- Vojtech Mrazek, Zdenek Vasícek, Lukás Sekanina, Muhammad Abdullah Hanif, and Muhammad Shafique. 2019. ALWANN: Automatic Layer-Wise Approximation of Deep Neural Network Accelerators without Retraining. ICCAD '19 (2019).Google ScholarCross Ref
- John Nickolls, Ian Buck, Michael Garland, and Kevin Skadron. 2008. Scalable Parallel Programming with CUDA. Queue (2008).Google Scholar
- E. Park, J. Ahn, and S. Yoo. 2017. Weighted-Entropy-Based Quantization for Deep Neural Networks. In CVPR '17.Google Scholar
- Olga Russakovsky et al. 2015. ImageNet Large Scale Visual Recognition Challenge. IJCV '15 (2015).Google Scholar
- Syed Shakib Sarwar et al. 2018. Energy-Efficient Neural Computing with Approximate Multipliers. J. Emerg. Technol. Comput. Syst. (2018).Google Scholar
- Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. 2014. Striving for Simplicity: The All Convolutional Net. (2014).Google Scholar
- Salim Ullah, Sanjeev Sripadraj Murthy, and Akash Kumar. 2018. SMApproxLib: Library of FPGA-based Approximate Multipliers. In DAC 18.Google Scholar
- Swagath Venkataramani et al. 2014. AxNN: Energy-efficient neuromorphic systems using approximate computing. ISLPED '14 (2014).Google Scholar
- Qian Zhang et al. 2015. ApproxANN: An Approximate Computing Framework for Artificial Neural Network. In DATE '15.Google Scholar
Index Terms
- Efficient Accuracy Recovery in Approximate Neural Networks by Systematic Error Modelling
Recommendations
Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
Deep Neural Networks (DNNs) are very popular because of their high performance in various cognitive tasks in Machine Learning (ML). Recent advancements in DNNs have brought levels beyond human accuracy in many tasks, but at the cost of high computational ...
Is approximation universally defensive against adversarial attacks in deep neural networks?
DATE '22: Proceedings of the 2022 Conference & Exhibition on Design, Automation & Test in EuropeApproximate computing is known for its effectiveness in improvising the energy efficiency of deep neural network (DNN) accelerators at the cost of slight accuracy loss. Very recently, the inexact nature of approximate components, such as approximate ...
LightNN: Filling the Gap between Conventional Deep Neural Networks and Binarized Networks
GLSVLSI '17: Proceedings of the on Great Lakes Symposium on VLSI 2017Application-specific integrated circuit (ASIC) implementations for Deep Neural Networks (DNNs) have been adopted in many systems because of their higher classification speed. However, although they may be characterized by better accuracy, larger DNNs ...
Comments