A systematic study on benchmarking AI inference accelerators

Abstract

AI inference accelerators have drawn extensive attention. But none of the previous work performs a holistic and systematic benchmarking on AI inference accelerators. First, an end-to-end AI inference pipeline consists of six stages on both host and accelerators. However, previous work mainly evaluates hardware execution performance, which is only one stage on accelerators. Second, there is a lack of a systematic evaluation of different optimizations on AI inference accelerators. Along with six representative AI workloads and a typical AI inference accelerator–Diannao based on Cambricon ISA, we implement five frequently-used AI inference optimizations as user-configurable hyper-parameters. We explore the optimization space by sweeping the hyper-parameters and quantifying each optimization’s effect on the chosen metrics. We also provide cross-platform comparisons between Diannao and traditional platforms (Intel CPUs and Nvidia GPUs). Our evaluation provides several new observations and insights, which sheds light on the comprehensive understanding of AI inference accelerators’ performance and instructs the co-design of the upper-level optimizations and underlying hardware architecture.

Appendix A

1.1 A.1 Implementation Details on Diannao

Considering the diversity of the network architecture, there is no-one-size-fits-all algorithm for the quantization and pruning. Research Jain (2020) shows some networks need to tailor the dedicated algorithm or retrain-based training method to make a compensate for the drop in model quality. Studying more general pruning and quantization algorithms Mishra et al. (2020) is still an open problem and beyond the scope of this paper. Here we briefly introduce our implementation of pruning and quantization.

1.1.1 A.1.1 Quantization

Diannao is equipped with large numbers of INT8-based ALUs. We implement INT8 quantization, which means that parameters of the model are stored using 8-bit fix-point integers instead of original floating-point numbers (Diannao use FP16 as its floating-point numbers). These model parameters are usually composed of three parts: weights, activations and bias. Considering that the proportion of bias in the overall parameters is small, we only quantify weights and activations. The computation process of quantified parameters can be summarized by the following formula:

$$\begin{aligned} real\_number = stored\_integers * scaling\_factor \end{aligned}$$

(A1)

where $real\_number$ refers to the parameters before quantization and $stored\_integers$ refers to the parameters after quantization. And $scaling\_factor$ aims to prevent over or underflows when computing the lower precision results.

1.1.2 A.1.2 Weight Pruning

Inside Diannao, there are also large numbers of sparse computing units. We only prune the weight of convolutional and fully-connected layers, because the weights of these two types of layers occupy most of the parameters of the entire model. Sparsity is a decimal between 0 and 1, referring to the percentage of zero-valued weights in the model. We use sparsity to reflect the effects of weight pruning optimization. Motivated by Deep Compression Han et al. (2016), in each convolutional and fully-connected layer, we sort the weights and then zero out the weights that with the lowest magnitude based on the sparsity. To show the effect of weight punning on model quality and inference throughput, we gradually increase the sparsity from 0.01 to 0.9 with the increment step of 0.01 while keeping other optimizations fixed.

1.2 A.2 An Example of Efficient Network Deployment

Table 6 presents the best configuration candidates in terms of the end-to-end throughput. We get these configurations by looking up the database (discussed in Sect. 6.2). To illustrate the trade-off process between the throughput and model quality, we present 4 configurations for each workload. The pre-defined target quality as the minimum requirement for model quality.

For DenseNet121, the target quality is 0.73, it achieves the highest end-to-end throughput at the configuration (sparse, INT8, 1, 4, 1, 8). However, the accuracy of the model does not meet the requirements, so this configuration will still be discarded. Then the configuration (Dense, FP16, 1, 4, 1, 8) that reaches the second highest end-to-end throughput is chosen since the accuracy requirement is satisfied. We followed the same method to select the best configuration for the remaining workload.

Table 6 Best optimization configurations in terms of end-to-end throughput for each DNNs on Diannao