Efficient depthwise separable convolution accelerator for classification and UAV object detection

Abstract

Depthwise separable convolutions (DSC) have been widely deployed in lightweight convolutional neural networks due to high efficiency. But the acceleration performance of the Graphics Processing Unit for DSC was not as well as in theory. In this paper, some approaches were proposed for accelerating DSC based on Field-Programmable Gate Array (FPGA). For the preceding layers, S2C (spatial to channel) was proposed to accelerate computing and improve the utilization rate of computational resources and bandwidth. An efficient SharePE was proposed to accelerate the DSC, which can improve the efficiency of the computing resource. The regulable parallelism approach was proposed to compute efficiently the different pointwise convolutional layers. P2D&D2P approach is proposed to reduce the external memory access. For the entire accelerating system, the pre-load workflow was proposed to reduce the waiting time of the accelerator between two images. We demonstrated our approaches on the SkyNet using the Ultra96V2 development board. Results indicated that our proposed accelerator obtained 80.030 frames per second and 0.072 Joule per image for UAV object detection, which achieved the state-of-the-art results for SkyNet. Besides, the MobileNetV2 model was implemented on a larger XC7Z100 FPGA, and the results showed our accelerator classified each picture from ImageNet in 2.69 ms. Code is available at https://github.com/AILearnerLi/DAC-SDC-2020-SEUer.

Introduction

Convolutional neural networks (CNNs) have become the mainstream approach in the computational vision field [1], [2], [3], [4] because of their high accuracy. Generally, CNN is a computing-intensive model, which has abundant parameters and floating-point operations (FLOPs). For instance, VGG-19 [5] had 16 convolutional layers and 3 fully connected layers with 144 million parameters and 19.6 billion FLOPs for one $224\times 224$ image. ResNet-101 [6] and DenseNet-121 [7] had 25 million parameters and 8.1 million parameters, respectively. These CNNs were difficult to be deployed to resource and power limited devices and obtain real-time performance. Therefore, lightweight CNNs [8], [9], [10] caught the attention of researchers. Some lightweight convolution operations were proposed to reduce the parameters and FLOPs.

The group convolution is a popular lightweight convolution. ResNeXt [11] adopted group convolution and obtained higher accuracy than ResNet with a similar computational cost. Chen et al. [2] reduced the parameters of group convolution by sharing weights via Bayesian Learning, which obtained higher accuracy than ResNeXt with similar model size. Dense2Net [12] obtained higher accuracy than DenseNet [7] with fewer parameters by using group convolution. ShuffleNet [13], a famous lightweight CNN, used group convolution to reduce the parameters and FLOPs of $1\times 1$ convolutional layer, which achieved about $13\times$ actual speedup over AlexNet while maintaining comparable accuracy.

Depthwise convolution (DWC) is an extreme case of group convolution, in which one group only contains a feature map channel. DWC is more efficient than group convolution due to fewer parameters and FLOPs. DWC has been applied in MobileNetV1 [14] and later MobileNetV2 [15], and achieved comparable results with much fewer FLOPs and parameters. Now, DWC is a very popular approach for designing lightweight CNNs. Many CNNs [16], [17] based on network architecture search also adopted DWC to improve the efficiencies.

DWC has a high memory access cost (MAC)/FLOPs ratio because of the large feature maps and few FLOPs. The acceleration performance of Graphics Processing Unit (GPU) for depthwise convolution cannot reach the theoretical value because of its high MAC/FLOPs ratio [16], [18]. FPGAs excel at low-precision computation, and their adaptability to new algorithms lends themselves to supporting rapidly changing CNN architectures.

Many hardware accelerators [19], [20], [21], [22] have been developed to improve the speed and power performance of the compute-intensive CNNs. Moini et al. [23] exploited the inherent parallelism in CNNs to reduce the required bandwidth, resource usage, and power consumption of highly computationally complex convolution operations, and the proposed hardware accelerator only used 391 DSP48 and obtained 19.2 Giga multiply accumulation operations per second while consuming less than 10 watts (W) in power. Wang et al. [24] implemented the VGG16 model on Xilinx Virtex VC707 platforms and achieved a frame rate of 33.80 frames per second (FPS) with average performances of 1250.21 Giga operations per second (GOPS). Ma et al. [25] studied the convolution loop optimization before the hardware design phase and proposed a specific dataflow. The proposed accelerator implemented NiN, VGG-16, and ResNet-50/ResNet152 and it achieved 707.2 GOPS for ResNet-152. Azizimazreah et al. [26] exploited cross-layer shortcut reuse in CNN accelerators, and experiment results showed that the proposed Shortcut Mining achieves 53.3%, 58%, and 43% reduction in off-chip feature map traffic for SqueezeNet, ResNet-34, and ResNet152. However, most hardware accelerators were designed for large CNNs, which were not very suitable for depthwise separable convolution (DSC) because of the special operation and low MAC/FLOPs of the DWC.

Currently, with the wide application of DSC in lightweight CNNs [16], [27], the hardware accelerator for DSC has been very urgent. The existing accelerators [17], [28] for DSC have a low utilization rate of computational resources during the entire running phase. In this paper, we gave a roofline model analysis for DSC, which helps to design an efficient accelerator. Then, spatial to channel (S2C), D2P&P2D, sharing processing element (SharePE) between DWC and pointwise convolution (PWC), regulable parallelism (R-Parallel) in computing unit, and pre-load workflow are proposed to improve the resource utilization rate, reduce the external memory access and speed up the accelerator. Based on iSmart3 (the champion model for IEEE/ACM Design Automation Conference System Design Contest, DAC’2019-SDC) [17], we designed an accelerator for SkyNet by using partial methods, which obtained the $6^{\mathit{th}}$ place in the DAC’2020-SDC. Using all of the proposed approaches on Skrkr-SkyNet (the $2^{\mathit{nd}}$ place in the DAC’2020-SDC) [28], we obtained the state-of-the-art results for SkyNet.

The Energy score (according to the evaluation of DAC’2020-SDC, $\mathit{ES}=\mathit{max}\{0,1+0.2\times {\mathit{log}}_2\frac{\bar{E}}{E}\},\bar{E}$ is the average energy usage across all teams.) and inference accuracy (IOU) on FPGA of our accelerators and DAC-SDC Top-3 design for the last two years are compared in Fig. 1. The higher energy score corresponds to lower energy consumption. Our accelerator based on the Skrskr achieves 80.030 FPS with 0.731 IOU, which score will surpass the 1st place solution in the DAC’2020-SDC. Furthermore, we implemented the MobileNetV2 model on Xilinx XC7Z100 FPGA platform and achieved a frame rate of 371.4 FPS and 0.19 GOPS per DSP (GOPS/DSP). The contribution of this work can be summarized as follows.

• An efficient SharePE was proposed to compute depthwise separable convolution, which could compute efficiently DWC and PWC, and had a high computing resource utilization rate for DWC and PWC.

• The regulable parallelism for computing unit was proposed to compute the different PWC layers in the residual block, which could improve the utilization rate of the computational resource.

• A spatial to channel approach was proposed to accelerate the computing of preceding DWC layers in CNNs, which could improve the utilization rate of computational resources and bandwidth.

• D2P&P2D was proposed to reduce the external memory access. When feature map channels are few, D2P is adopted. On the contrary, P2D is adopted when the number of feature map channels are large.

• The pre-load workflow was proposed for accelerating the entire system, which could reduce the waiting times between computing two images.

• SkyNet was implemented by using the proposed approach on the resource-constraint Ultra96V2 FPGA platform for object detection, which achieved state-of-the-art results.

• MobileNetV2 was implemented on the XC7Z100 FPGA platform based on the proposed improvement approaches, which obtained a high FPS and throughput per DSP.

The remainder of this paper are organized as follows. Section 2 reviews two typical DSC CNNs, MobileNetV2 for image classification and SkyNet for object detection. Then, some related works about the hardware accelerator for depthwise separable convolution and roofline model are introduced. In Section 3, the roofline model analysis of DSC is presented, and Skynet is analyzed by the roofline model. In Section 4, the system architectures, including dedicated accelerator architecture and improvement approaches for accelerating depthwise separable convolution are described. The experiment results of our proposed accelerators for the classification task (MobileNetV2) and object detection (SkyNet) are analyzed and discussed in Section 5. The conclusions are given in Section 6.