RFPruning: A retraining-free pruning method for accelerating convolutional neural networks

Highlights

• A retraining-free pruning framework is proposed to compress CNN models.

• An ADMM-based sparse-learning method is introduced to make CNN models prunable.

• A GA-based pruning strategy is designed to obtain the optimal pruned CNN models.

• The proposed RFPruning works as well as the methods with retraining, but faster.

Abstract

Network pruning has been developed as a remedy for accelerating the inference of deep convolutional neural networks (DCNNs). The mainstream methods retrain the pruned models, which maintain the performance of the pruned models but consume a great deal of time. While the other methods reduce the time consumption by omitting to retrain, they lose the performance. To resolve the above conflicts, we propose a two-stage Retraining-Free pruning method, named RFPruning, which embeds the rough screening of channels into training and fine-tunes the structures during pruning, to achieve both good performance and low time consumption. In the first stage, the network training is reformulated as an optimization problem with constraints and solved by a sparse learning approach for rough channel selection. In the second stage, the pruning process is regarded as a multiobjective optimization problem, where the genetic algorithm is applied to carefully select channels for a trade-off between the performance and model size. The proposed method is evaluated against several DCNNs on CIFAR-10 and ImageNet datasets. Extensive experiments demonstrate that such a retraining-free pruning method obtains 43.0% $\sim$ 88.4% compression on model size and maintains the accuracy as the methods with retraining while achieving $3\times$ speed up in pruning.

Introduction

Benefited from the powerful capacity of feature extraction, deep convolutional neural networks (DCNNs) have achieved significant advances in various computer vision tasks. However, the powerful learning capacity of DCNNs relies on expensive storage and computational resources, which restrict applications of DCNNs on mobile devices. This issue drives the research of network acceleration algorithms [1], [2], [3], [4], where network pruning is one of the most effective methods.

In general, network pruning is aimed to remove the unimportant network parameters and can be divided into two categories, i.e., unstructured pruning [5], [6] and structured pruning [7], [8], [9]. The unstructured pruning methods achieve high compression ratios because arbitrary weights in the network may be pruned. However, they destroy the form of the weight matrix and need special hardware to store the pruned weights, which limits their application in real hardware implementation. The structured pruning methods reduce the number of parameters by pruning whole channels or filters of the network. By this means, the size of the weight matrix is compressed and the structure of the matrix is preserved. Therefore, the structured pruning methods are more flexible for most hardware implementations and receive increased attention.

The framework of most existing structured pruning methods [10], [11] contains three stages: training, pruning, and retraining. As revealed by Z. Liu et al. [12] and X. Ding et al. [13], such a framework exists two major drawbacks: (1) The optimization process, which requires iterative pruning and retraining, is computation-intensive and time-consuming. (2) The pruned models, at the retraining stage, are easily trapped into bad local minima. For overcoming the two drawbacks, some works (e.g., GAL [14] and VCNNP [15]) have proposed to omit the retraining stage and directly obtain the target pruned models from the trained models. However, the compression ratio and accuracy they achieved are inferior to those of the methods with retraining. Thus, it is a challenge for both good performance and fast deployment.

In this paper, we propose a retraining-free structured pruning method, named RFPruning, for fast and efficient network pruning. To directly achieve the good-performing pruned models from the original models, we embed the rough screening of channels into training and fine-tune the structures during pruning. In the training stage, the network training is reformulated as an optimization problem with the sparse constraint, which is abstracted as $l_0$ norm in the loss function. Considering that $l_0$ norm is not differentiable, a sparse-learning approach based on the Alternating Direction Method of Multipliers (ADMM) algorithm is proposed to minimize the loss function. Through such an optimization, the unimportant channels are roughly identified and invalidated, which prevents the significant channels from being pruned by mistake and makes the models prunable. In the pruning stage, the task of obtaining the optimal pruned model from the trained model is regarded as a multiobjective optimization problem, where the targets of optimization are the accuracy and compression ratios. For a trade-off between performance and model size, the genetic algorithm is chosen to fine-tune the structure of pruned models, which is implemented by searching suitable pruning rates for each layer. The contributions of this work can be summarized as follows:

• A retraining-free pruning framework, which contains sparse-learning and automatic searching, is proposed for both the fast pruning process and good model performance.

• An ADMM-based sparse-learning technique is embedded into training to roughly identify and invalidate insignificant channels during training, which prevents important channels from being removed by mistake and makes the DCNN models prunable.

• A genetic algorithm-based pruning strategy is developed to fine-tune the network structure for a trade-off between performance and model size. This strategy automatically obtains the optimal pruning rates for models, while has no requirement of additional optimized networks like other automatic strategies (e.g., GAL [14] and MetaPruning [16]).

• Extensive experiments on CIFAR-10 and ImageNet demonstrate that the proposed retraining-free pruning method works as well as the competing methods with retraining, on several popular DCNN models, including VGGNet [17], ResNet [18], GoogLeNet [19], and DenseNet [20].

The organization of this paper is as follows. The relevant literature is reviewed and discussed in Section 2. The proposed method is introduced in detail in Section 3. In Section 4, we report and analyze the experimental results. Finally, this paper is concluded in Section 5.