Abstract
Deep Neural Networks have become an increasingly favourite choice for a variety of machine learning tasks. Two important components are largely responsible for this success, improved neural network functionalities, and availability of suitable hardware for training large complex networks. Using these types of novel networks and functions, Deep Neural Networks have been shown to be very highly efficient for various classification tasks. As the next level of optimization, dedicated ASIC and FPGA ICs are being developed, to realize Deep Neural Networks. This provides an additional level of performance optimization beyond traditional software-based implementations. Towards this direction, in this work, we have developed CMOS circuits for realizing the highly popular Rectified Linear (ReLu) activation function. The ReLu activation function has largely replaced the traditional sigmoid activation function due to better learning rates and reduced computational requirements. With dedicated CMOS implementations of such functions, we get better operating speed with lower power consumption, leading to improved real-time implementations of classification tasks.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 5, 1106–1114 (2012)
Chen, Y.-H., Emer, J., Sze, V.: Using dataflow to optimize energy efficiency of deep neural network accelerators. IEEE Micro 37(3), 12–21 (2017)
Douglas, R., Mahowald, M., Mead, C.: Neuromorphic analogue VLSI. Annu. Rev. Neurosci. 18(1), 255–281 (1995)
Vinod, N., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10) (2010)
He, K., et al.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision (2015)
Chen, Y.-H., Joel, E., Vivienne, S.: Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks. In: 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA). IEEE (2016)
Merolla, P.A., et al.: A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345(6197), 668–673 (2014)
Yilmaz, M., Tunkar, B.A., Park, S., Elrayes, K., Mahmoud, M.A.E.: High-efficiency passive full wave rectification for electromagnetic harvesters. J. Appl. Phys. 116(13), 134902 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Priyanka, P., Nisarga, G.K., Raghuram, S. (2019). CMOS Implementations of Rectified Linear Activation Function. In: Rajaram, S., Balamurugan, N., Gracia Nirmala Rani, D., Singh, V. (eds) VLSI Design and Test. VDAT 2018. Communications in Computer and Information Science, vol 892. Springer, Singapore. https://doi.org/10.1007/978-981-13-5950-7_11
Download citation
DOI: https://doi.org/10.1007/978-981-13-5950-7_11
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-5949-1
Online ISBN: 978-981-13-5950-7
eBook Packages: Computer ScienceComputer Science (R0)