Logarithm-approximate floating-point multiplier is applicable to power-efficient neural network training

Highlights

• Edge computing and wide-dynamic range instinct of gradients facilitate the demand for developing power-efficient floating-point (FP) neural network (NN) training.

• We present LAM incorporating BWS to significantly reduce primary MAC computation in FP NN training up to 4-hidden layer NN. Besides, solo-LAM training is proved to be sufficiently accurate and there is no need to rely on the exact multiplier for our datasets.

• Benchmarking results through our dedicated training hardware show that compared to the exact multiplier, LAM/LAM + BWS can reduce 2.5X/4.9X power while sustaining the accuracy.

• We deploy an open-source GPGPU design with a programmable compiler to train LAM and LAM + BWS based NNs, and attain at 1.32X and 1.54X power efficiency through real-time measurement.

Abstract

Recently, emerging “edge computing” moves data and services from the cloud to nearby edge servers to achieve short latency and wide bandwidth, and solve privacy concerns. However, edge servers, often embedded with GPU processors, highly demand a solution for power-efficient neural network (NN) training due to the limitation of power and size. Besides, according to the nature of the broad dynamic range of gradient values computed in NN training, floating-point representation is more suitable. This paper proposes to adopt a logarithm-approximate multiplier (LAM) for multiply-accumulate (MAC) computation in neural network (NN) training engines, where LAM approximates a floating-point multiplication as a fixed-point addition, resulting in smaller delay, fewer gates, and lower power consumption. We demonstrate the efficiency of LAM in two platforms, which are dedicated NN training hardware, and open-source GPU design. Compared to the NN training applying the exact multiplier, our implementation of the NN training engine for a 2-D classification dataset achieves 10% speed-up and 2.3X efficiency improvement in power and area, respectively. LAM is also highly compatible with conventional bit-width scaling (BWS). When BWS is applied with LAM in five test datasets, the implemented training engines achieve more than 4.9X power efficiency improvement, with at most 1% accuracy degradation, where 2.2X improvement originates from LAM. Also, the advantage of LAM can be exploited in processors. A GPU design embedded with LAM executing an NN-training workload, which is implemented in an FPGA, presents 1.32X power efficiency improvement, and the improvement reaches 1.54X with LAM + BWS. Finally, LAM-based training in deeper NN is evaluated. Up to 4-hidden layer NN, LAM-based training achieves highly comparable accuracy as that of the accurate multiplier, even with aggressive BWS.

Introduction

For enhancing our daily life with artificial intelligence (AI), machine learning (ML) is currently adopted and executed everywhere, even on end devices such as smartphones, Internet-of-Things (IoT) sensors, and cameras. The generated data from devices need to be collected and aggregated as the samples for learning processing and analysis. Conventionally, the data is transferred to the cloud since the learning and analysis processes require high computational capability and large memory capacity. However, the data moving to the cloud involves challenges in terms of latency, bandwidth, and privacy concerns. Some applications like computer vision and natural language processing highly demand local and real-time services, and then edge computing emerges as a solution. In edge computing, the edge servers, typically embedded with GPU processors, are placed near the end devices, and they process and analyze the data independently from the cloud or before sending the data to the cloud [1,2]. Compared with ML training in the cloud, training in the edge servers may provide tailored ML models without security risk [1]. On the other hand, due to the size and power limitations, power-efficient training is demanded and explored [3,4].

Neural network (NN) is one of the most widely-used techniques in machine learning [5]. A feedforward NN model is composed of a few to hundreds of layers, each of which includes a number of neurons. The neurons are connected layer by layer through synaptic weights. The synaptic weights are optimized to provide sufficiently high accuracy through the computationally expensive training phase. Hardware NN system is mainly categorized into two types. The first one is the inference engine that processes a network with given pre-trained weights, and the latter is the training engine that has the additional capability of weight optimization in the training phase. Regardless of the inference or training engine, multiply-accumulate (MAC) arithmetic computation is the primary operation. The rapidly increasing trend of NN size to deal with more intricate and sophisticated problems explodes the amount of MAC computation, resulting in a strong demand for dedicated hardware engines.

Inference engines in several studies exploit inherent error-tolerant property in machine learning and introduce approximate computing (AC) techniques for gaining performance and reducing cost [[6], [7], [8], [9]]. Among various AC techniques proposed for inference engines, bit-width scaling (BWS), which reduces the bit width of data representation, is the most popular and powerful way that trades computation reduction with accuracy degradation [6,[10], [11], [12]]. Even binarized neural networks are studied [13].

In contrast to inference, training engines need to perform more arithmetic computation with a wider dynamic range since the gradient, which is numerically computed and used to guide the weight update, spreads in a broad dynamic range [10]. A simple example can help us understand this property. Fig. 1 plots the distribution density of the gradient values found when training a NN for MNIST dataset [24]. The gradient values spread from 2⁻⁴⁷ to 2⁶. If adopting fixed-point expression, more than 50 bits are required to cover this range, while floating-point expression spends only a few bits for exponents and some extra bits for fraction parts to cover this wide range. Thus, adopting floating-point units (FPUs) is beneficial for training engines to accommodate such gradient computation.

Nevertheless, FPUs are known as power-hungry and area-expensive units [14]. Specifically, in the neural network algorithm that mainly consists of MAC computations, floating-point multiplication is the most power-hungry and space-demanding arithmetic operators. Fig. 2 exemplifies the power and area benchmarking between a 32-bit floating-point multiplier and an adder synthesized for the same clock frequency. The figure shows that the floating-point multiplier consumes 3.01X power and 1.75X area. The MAC operation requires the same number of multiplication and addition, indicating that the power for MAC operation is mostly consumed by the multiplier. Consequently, massive MAC computations in NN training deteriorate the area and power efficiency. Therefore, power-efficient floating-point multiplication is highly demanded in training engine development.

In this paper, we perform that the logarithm-approximate multiplier (LAM), which approximates floating-point multiplication to fixed-point addition, benefits to NN training and improves the power efficiency of massive MAC computation involved in NN training under floating-point format. We also show that LAM is useful even when the BWS is already implemented in training, and hence power efficiency is further enhanced. These advantages are quantitatively evaluated through the experiments with dedicated training hardware. Experimental results show 2.5X power reduction by LAM and 4.9X reduction by LAM + BWS while sustaining the accuracy, where 2.2X reduction originates from LAM. These results are reported in our preliminary work [15].

In this work, we newly evaluate whether solo or hybrid usages of exact and approximate multipliers in training and testing phases affect the classification accuracy. Experimental results reveal that adopting approximate multipliers (LAMs) in both training and testing phases induces no significant accuracy degradation, and then there is no need to rely on accurate multipliers. Next, we conduct additional experiments for evaluating the applicability of LAM and LAM + BWS to open-source GPU design, on which NN training programs are executed. The power reductions thanks to LAM and LAM + BWS are measured with an FPGA implementation of the GPU design, and they are 1.32X and 1.54X compared to the original design. Finally, we extend the NN depth, up to 4-hidden layers, and show that solo-LAM training achieves highly comparable results with solo-EFM training. This trend sustains even when BWS is aggressively adopted as long as the acceptable training accuracy is obtained.

The rest of the paper is organized as follows. Section 2 reviews the NN with its training and related works. Section 3 introduces LAM and discusses its approximation error. Experimental results of adopting LAM in dedicated training engines are presented in Section 4. Section 5 provides the environmental setup and measurement results of LAM-based NN training with FPGA implementation of an open-source GPU design. Section 6 applies LAM-based training to deeper NNs and provides evaluation results. Finally, Section 7 concludes this paper.