Abstract
The compulsive nonlinearity in neural networks (NN) is introduced by well-known nonlinear functions called activation functions. Performing AI-inferences on edge devices calls for efficient approximation of those complex functions on highly restrictive hardware (HW) platforms. When designing such systems, balancing area, footprint and power consumption at an application appropriate latency is key. To address those challenges, we propose a HW-based interpolation component capable of approximating arbitrary mathematical functions. A combinatorial search-based optimization algorithm is employed to find the optimal set of interpolation points for a set of functions while also considering non-uniform distributions. The proposed solution is accompanied by a Python-based HW generator, that facilitates the process of deploying software-computed search results on HW and provides room for generating different flavors of application-optimized HW. In an effort to reduce area footprint and delay, the proposed approach exploits symmetry and biased symmetry properties of functions and applies bit width optimizations to reduce the size of the utilized computational units. Additionally, property-aware reprogrammable solutions for multifunctional use cases are incorporated into our design. Experimental analyses show that our proposed method permits achieving better area utilization and deviation error results than state-of-the-art implementations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P.: Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830 (2014)
Amin, H., Curtis, K., Hayes-Gill, B.: Piecewise linear approximation applied to nonlinear function of a neural network. IEEE Proc. Circuits Dev. Syst. 144(6), 313 (1997). https://doi.org/10.1049/ip-cds:19971587
Chandra, M.: Hardware implementation of hyperbolic tangent function using catmull-rom spline interpolation. arXiv preprint arXiv:2007.13516 (2020)
Ciregan, D., Meier, U., Schmidhuber, J.: Multi-column deep neural networks for image classification. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3642–3649. IEEE (2012)
Duren, R., Marks, R., Reynolds, P., Trumbo, M.: Real-time neural network inversion on the SRC-6e reconfigurable computer. IEEE Trans. Neural Networks 18(3), 889–901 (2007). https://doi.org/10.1109/tnn.2007.891679
Ecker, W., Schreiner, J.: Introducing model-of-things (mot) and model-of-design (mod) for simpler and more efficient hardware generators. In: 2016 IFIP/IEEE International Conference on Very Large Scale Integration (VLSI-SoC), pp. 1–6 (2016). https://doi.org/10.1109/VLSI-SoC.2016.7753576
Hu, J., Shen, L., Albanie, S., Sun, G., Wu, E.: Squeeze-and-excitation networks, September 2017
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Quantized neural networks: training neural networks with low precision weights and activations. J. Mach. Learn. Res. 18(1), 6869–6898 (2017)
Kumar, P.A.: FPGA implementation of the trigonometric functions using the CORDIC algorithm. In: 2019 5th International Conference on Advanced Computing & Communication Systems (ICACCS). IEEE, March 2019. https://doi.org/10.1109/icaccs.2019.8728315
Li, Z., Zhang, Y., Sui, B., Xing, Z., Wang, Q.: Fpga implementation for the sigmoid with piecewise linear fitting method based on curvature analysis. Electronics 11(9), 1365 (2022)
Lin, M.H., Carlsson, J.G., Ge, D., Shi, J., Tsai, J.F.: A review of piecewise linearization methods. Math. Probl. Eng. 2013, 1–8 (2013). https://doi.org/10.1155/2013/101376
Muscedere, R., Dimitrov, V., Jullien, G., Miller, W.: Efficient techniques for binary-to-multidigit multidimensional logarithmic number system conversion using range-addressable look-up tables. IEEE Trans. Comput. 54(3), 257–271 (2005). https://doi.org/10.1109/tc.2005.48
Namin, A.H., Leboeuf, K., Muscedere, R., Wu, H., Ahmadi, M.: Efficient hardware implementation of the hyperbolic tangent sigmoid function. In: 2009 IEEE International Symposium on Circuits and Systems. IEEE, May 2009. https://doi.org/10.1109/iscas.2009.5118213
Piazza, F., Uncini, A., Zenobi, M.: Neural networks with digital LUT activation functions. In: Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan). IEEE (1993). https://doi.org/10.1109/ijcnn.1993.716806
Raut, G., Rai, S., Vishvakarma, S.K., Kumar, A.: A CORDIC based configurable activation function for ANN applications. In: 2020 IEEE Computer Society Annual Symposium on VLSI (ISVLSI). IEEE, July 2020. https://doi.org/10.1109/isvlsi49217.2020.00024
Tommiska, M.: Efficient digital implementation of the sigmoid function for reprogrammable logic. IET Proc. Comput. Digital Tech. 150(6), 403 (2003). https://doi.org/10.1049/ip-cdt:20030965
Tsmots, I., Skorokhoda, O., Rabyk, V.: Hardware implementation of sigmoid activation functions using FPGA. In: 2019 IEEE 15th International Conference on the Experience of Designing and Application of CAD Systems (CADSM). IEEE, February 2019. https://doi.org/10.1109/cadsm.2019.8779253
Volder, J.E.: The CORDIC trigonometric computing technique. IRE Trans. Electron. Comput. EC-8(3), 330–334 (1959). https://doi.org/10.1109/tec.1959.5222693
Zhang, M., Vassiliadis, S., Delgado-Frias, J.: Sigmoid generators for neural computing using piecewise approximations. IEEE Trans. Comput. 45(9), 1045–1049 (1996). https://doi.org/10.1109/12.537127
Acknowledgements
Part of the work has been funded by the German Federal Ministry of Education and Research (BMBF) as part of the research project Scale4Edge (16ME0122K).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Prebeck, S., Lawand, W., Vaddeboina, M., Ecker, W. (2022). A Smart HW-Accelerator for Non-uniform Linear Interpolation of ML-Activation Functions. In: Orailoglu, A., Reichenbach, M., Jung, M. (eds) Embedded Computer Systems: Architectures, Modeling, and Simulation. SAMOS 2022. Lecture Notes in Computer Science, vol 13511. Springer, Cham. https://doi.org/10.1007/978-3-031-15074-6_17
Download citation
DOI: https://doi.org/10.1007/978-3-031-15074-6_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15073-9
Online ISBN: 978-3-031-15074-6
eBook Packages: Computer ScienceComputer Science (R0)