Optimization of Softmax Layer in Deep Neural Network Using Integral Stochastic Computation

Deep neural network (DNN), as a very important machine learning technique in classification and detection tasks for images, video, speech as well as audio, has recently received tremendous attention. Integral Stochastic Computation (Integral SC), on the other hand, has proved its extraordinary ability in hardware implementation of DNNs. The softmax layer is generally used in multiclassification tasks as a very basic and important network layer in DNNs. However, the hardware implementation of softmax layer is expensive because of the exponentiation and division computation. In this paper, we designed an efficient way to simulate softmax layer in DNNs based on Integral stochastic computing, filling the vacancy of previous academic works. Compared to conventional softmax hardware implementation, our method achieves reduction in power and area by 63% and 68%, respectively.