A fine-grained mixed precision DNN accelerator using a two-stage big–little core RISC-V MCU

Highlights

• A mixed precision structure to process different effective bit-width weights separately.

• An MCU contains two big–little RISC-V cores to control the accelerator and peripherals.

• A fine-grain control flow to control the queuing operations with low hardware amount.

Abstract

Deep neural networks (DNNs) are widely used in modern AI systems, and their dedicated accelerators have become a promising option for edge scenarios due to the energy efficiency and high performance. Since the DNN model requires significant storage and computation resources, various energy efficient DNN accelerators or algorithms have been proposed for edge devices. Many quantization algorithms for efficient DNN training have been proposed, where the weights in the DNN layers are quantified to small/zero values, therefore requiring much fewer effective bits, i.e., low-precision bits in both arithmetic and storage units. Such sparsity can be leveraged to remove non-effective bits and reduce the design cost, however at the cost of accuracy degradation, as some key operations still demand a higher precision. Therefore, in this paper, we propose a universal mixed-precision DNN accelerator architecture that can simultaneously support mixed-precision DNN arithmetic operations. A big–little core controller based on RISC-V is implemented to effectively control the datapath, and assign the arithmetic operations to full precision and low precision process units, respectively. Experimental results show that, with the proposed designs, we can save 16% chip area and 45.2% DRAM access compared with the state-of-the-art design.

Introduction

Deep neural networks (DNNs) have been recently demonstrated to be highly effective for computer vision and speech recognition, and hence are particularly promising for Internet of Things (IoT) devices at the edge [1], [2], [3], [4]. Such intelligent edge devices are typically constrained by the available computational and storage resources, it is thus highly desired to design efficient hardware implementations to support efficient DNN models with high speed and low power. The general purpose GPU and CPU are possible solutions for DNN deployments but are not applicable to edge scenarios due to their significant resource requirements. Many recent works have developed energy efficient DNN acceleration implementations using Field Programmable Gate Array (FPGA) [5], [6], [7], [8], [9] and Application Specific Integrated Circuit (ASIC) [10], [11], [12], [13], [14]. With awareness of the complexity of DNN algorithms featuring millions of Multiply-Accumulate (MAC) operations and frequent data transmissions, many prior researches have placed focus on effective performance and accuracy trade-off using limited computational resources [15], [16], [17], [18]. Among these efforts, quantization and sparsification are the two most commonly used techniques [19], [20], [21]. Through careful training procedure, many weights in a DNN can be zeroed out or quantified to fewer bits while maintaining a similar accuracy [22], [23]. The weight representation using lower bit-width in hardware can bring down the demanded resources for both computation and storage. It is then an appealing option to design the processing element (PE) in a DNN accelerator, i.e., the basic arithmetic unit, only with demanded bit-width to reduce the design overhead in both cache and MAC.

For example, prior work [11], [14], [24], [25], [26], [27], [28] have already proposed to reduce the bit-widths of inputs and weights to 8-bit, and even 4- or 2- bits. Moreover, binary neural networks are also very popular to have aggressive energy reduction [5], [11]. While the quantization provides non-trivial memory bandwidth reduction, the quantified networks have to be implemented in hardware with fixed precision, which is often the largest bit-width needed for the network. Thus, the applications have been constrained to particular scenarios without flexibility to support varying demands [5], [6], [11]. To have more flexibility, some general purpose accelerator architectures have been proposed with configurability, which can be used to deploy different DNN algorithms [10], [12], [13], [14]. The PE in [13] has two 8-bit multipliers, which can be switched between 8- and 16-bit precision modes. ENVISION in [12] uses a booth multiplier that can be configured to 4-, 8- or 16-bits. Dynamic voltage and frequency scaling technology is also adopted to achieve more power-performance trade-off. UNPU in [14] proposes serial lookup table-based multipliers to implement the PE that supports precision from 1 to 16 bits. In other word, at the expense of more complex control logic for PEs, the configurability can provide various precision for different neural networks or different layers in one neural network, thus improving the overall energy efficiency [12], [13], [14].

In most of the previous work, the configuration of precision existed only in the time dimension, that is, one precision for one layer. In the same layer, the fixed precision is used even if the data values are vastly different. We note that the effective precision of weights within a layer may actually significantly vary, which refers to the minimum number of bits to represent a weight without accuracy loss. Assuming that in a one layer of a network, most weights have values less than 128 (normalized to integers for ease of description), these weights can actually be represented by int8. But the other part of weights are greater than 128, and they need to be represented by int16. In prior accelerators, this layer would need to be calculated with 16 bits precision, even though the weights greater than 128 are quite few. We found that the above phenomenon is very common. Fig. 1 shows the effective weight precision distributions of different layers in AlexNet [29] and VGG16 [1] from [21]. The effective weight precision indicates the minimum bits that a weight can be represented by. While the highest effective precision is up to 16 bits, the majority of weights within the same layer is only 5 bits or even lower. This observation is due to the fact that the neural network tends to converge to more distributed and sparse weights in the process of quantization-driven training. Previous designs had to calculate with the highest precision,to ensure the accuracy, this inevitably induce non-trivial computation and storage resources waste. A lot of storage and calculation that could be done with 8 bits had to be done with 16 bits. Actually the higher order bits of these low effective precision weights can be skipped without any change in value.

An ideal approach is then to analyze the weights using only necessarily minimal bit-widths and then have fine-granular mixed-precision support to process data with different precision in different PEs. For example, weights greater than 128 can be processed with 16-bit precision, and the rest are processed with 8-bit precision or even finer granularity. Obviously, per-PE precision is impractical in terms of resource consumption [30], [31]. Moreover, even with 2 different precision, its control logic can be too complex to actually reduce the overall hardware resources. Thus, in order to design a fine-grained mixed-precision DNN accelerator, there are a few challenges that need to be addressed:

• It is not efficiency to process all data with uniform precision, especially in scenarios where the data distribution has a distinct pattern. It is highly desired to have a universal accelerator architecture that filters the data and processes them separately.

• Because of the complexity of neural networks operation, processing data separately will further increase the complexity of hardware. How to design an efficient controller that can use as few hardware resources as possible and achieve precise control?

• A effective datapath control flow is needed to fetch, process and merge the results from the mixed-precision PEs and ensure its cost can be covered by the savings from the mixed-precision support.

To address the above challenges, this paper proposes a general-purpose mixed-precision DNN accelerator with a two-stage RISC-V micro-controller (MCU). The main contributions are summarized as the following:

• Mixed-precision accelerator architecture: The weights in DNN were divided into two groups according to the effective bit width, Full Precision (FP) and Low Precision (LP). A small set of full precision PEs (FPPE) and a low precision PE (LPPE) array are used to process the two groups of weights separately. Such an architecture is applicable to different DNNs to support simultaneous computations.

• Two-stage big–little core Micro-controller Unit (MCU): We propose a RISC-V-based two-stage MCU that consists of a big core and little one to control the accelerator. The big core controls the complete accelerator operation as well as the small core, while the small one micro-manages the FPPE array in the accelerator. Such an MCU can effectively manage global and local operation for the proposed DNN accelerator.

• Unique datapath control flow: A unique control flow is designed to control the queuing operations. The FP data fetch and merging are implemented with minimal hardware cost through the control flow for fast execution.

The proposed architecture is implemented in both FPGA and UMC 40 nm library for ASIC and then compared with a prior state-of-the-art accelerator [10]. The experimental results show that the proposed implementation can save almost half of the weights storage and MAC area when compared with the single precision accelerator in [10]. The proposed two-stage big–little core MCU only consumes 11k look-up tables (LUTs) in FPGA or 91k $\mathrm{\mu }{\mathrm{m}}^2$ chip area in ASIC. Due to the mixed-precision configuration, DRAM access is also reduced by more than 40%, resulting in overall power consumption reduction.

The rest of this paper is organized as follows. Section 2 introduces the preliminaries of our design. Details of the proposed architecture are shown in Section 3. We show the experimental results in Section 4, with Section 5 concluding this paper.