Group Pruning Using a Bounded-$\ell _p$ Norm for Group Gating and Regularization

Abstract

Deep neural networks achieve state-of-the-art results on several tasks while increasing in complexity. It has been shown that neural networks can be pruned during training by imposing sparsity inducing regularizers. In this paper, we investigate two techniques for group-wise pruning during training in order to improve network efficiency. We propose a gating factor after every convolutional layer to induce channel level sparsity, encouraging insignificant channels to become exactly zero. Further, we introduce and analyse a bounded variant of the $\ell _1$ regularizer, which interpolates between $\ell _1$ and $\ell _0$-norms to retain performance of the network at higher pruning rates. To underline effectiveness of the proposed methods, we show that the number of parameters of ResNet-164, DenseNet-40 and MobileNetV2 can be reduced down by $30\%$, $69\%$, and $75\%$ on CIFAR100 respectively without a significant drop in accuracy. We achieve state-of-the-art pruning results for ResNet-50 with higher accuracy on ImageNet. Furthermore, we show that the light weight MobileNetV2 can further be compressed on ImageNet without a significant drop in performance .

T. Genewein—Currently at DeepMind.

Appendices

Supplementary material

A1 Proof of Lemma 1 (Lemma 1):

To improve readability, we will restate Lemma 1 from the main text:

The mapping $\Vert .\Vert _{\text {bound-}p,\sigma }$ has the following properties:

• For $\sigma \rightarrow 0^{+}$ the bounded-norm converges towards the $0$-norm:

  $$\begin{aligned} \lim \limits _{\sigma \rightarrow 0^{+}} \Vert x\Vert _{\text {bound-}p,\sigma } = \Vert x\Vert _{0}. \end{aligned}$$

  (1)

• In case $|x_i| \approx 0$ for all coefficients of $x$, the bounded-norm of $x$ is approximately equal to the $p$-norm of $x$ weighted by $1/\sigma$:

  $$\begin{aligned} \Vert x\Vert _{\text {bound-}p,\sigma } \approx \left||\frac{x}{\sigma }\right||_{p}^{p} \end{aligned}$$

  (2)

Proof

The first statement Eq. (1) can easily be seen using:

$$ \lim \limits _{\sigma \rightarrow 0} \text {exp}(-\frac{|x_i|^p}{\sigma ^p}) = \mathbf 1 _0(x_i) $$

For the second statement Eq. (2) we use the taylor expansion of $\text {exp}$ around zero to get:

$$\begin{aligned} \begin{aligned} \Vert x\Vert _{\text {bound-}p,\sigma }&= \sum \limits _{i=1}^{n} 1 - \text {exp}\left( -\frac{|x_i|^p}{\sigma ^p}\right) \\&= \sum \limits _{i=1}^{n} 1 - \sum \limits _{j=0}^{\infty }\left( -\frac{|x_i|^p}{\sigma ^p}\right) ^j \frac{1}{j!} \end{aligned} \end{aligned}$$

(3)

For $|x_i| \approx 0$ we keep only the leading coefficient $j = 1$ yielding:

$$ \Vert x\Vert _{\text {bound-}p,\sigma } \approx \sum \limits _{i=1}^{n} \frac{|x_i|^p}{\sigma ^p} = \left||\frac{x}{\sigma }\right||_{p}^{p}. $$

A2 Experiment Details

Both CIFAR100 and ImageNet datasets are augmented with standard techniques like random horizontal flip and random crop of the zero-padded input image and further processed with mean-std normalization. The architecture MobileNetV2 is originally designed for the task of classification on ImageNet dataset. We adapt the network¹ to fit the input resolution $32\times 32$ of CIFAR100. ResNet-164 is a pre-activation ResNet architecture containing 164 layers with bottleneck structure while DenseNet with 40 layer network and growth rate 12 has been used. All the networks are trained from scratch (weights with random initialization and bias is disabled for all the convolutional and fully connected layers) with a hypeparameter search on regularization strengths $\lambda _1$ for $\ell _1$ or bounded-$\ell _1$ regularizers and weight decay $\lambda _2$ on each dataset. The scaling factor $\gamma$ of BN is initialized with 1.0 in case of exponential gate while it is initialized with 0.5 for linear gate as described in [24] and bias $\beta$ to be zero. The hyperparameter $\sigma$ in bounded-$\ell _1$ regularizer is set to be 1.0 when the scheduling of this parameter is disabled. All the gating parameters g are initialized with 1.0.

We use the standard categorical cross-entropy loss and an additional penalty is added to the loss objective in the form of weight decay and sparsity induced $\ell _1$ or bounded-$\ell _1$ regularizers. Note that $\ell _1$ and bounded-$\ell _1$ regularization acts only on the gating parameters g whereas weight decay regularizes all the network parameters including the gating parameters g. We reimplemented the technique proposed in  [24] which impose $\ell _1$ regularization on scaling factor $\gamma$ of Batch Normalization layers to induce channel level sparsity. We refer this method as $\ell _1$ on linear gate and compare it against our methods bounded-$\ell _1$ on linear gate, $\ell _1$ on exponential gate and bounded-$\ell _1$ on exponential gate. We train ResNet-164, DenseNet-40 and ResNet-50 for $240$, $130$ and $100$ epochs respectively. Furthermore, learning rate of ResNet-164, DenseNet-40 and ResNet-50 is dropped by a factor of $10$ after $(30, 60, 90)$, $(120, 200, 220)$, $(100, 110, 120)$ epochs. The networks are trained with batch size 128 using the SGD optimizer with initial learning rate 0.1 and momentum 0.9 unless specified. Below, we present the training details of each architecture individually.

LeNet5-Caffe: Since this architecture does not contain Batch Normalization layers, we do not compare our results with the method $\ell _1$ on linear gate. We train the network with exponential gating layers that are added after every convolution/fully connected layer except the output layer and apply different regularizers like $\ell _1$, bounded-$\ell _1$ and weight decay separately to evaluate their pruning results. We set the weight decay to zero when training with $\ell _1$ or bounded-$\ell _1$ regularizers. The network is trained for 200 epochs with the weight decay and 60 epochs in case of other regularizers.

ResNet-50: We train the network with exponential gating layers that are added after every convolutional layer on ImageNet dataset. We evaluate performance of the network on different values of regularization strength $\lambda _1$ like $10^{-5}$, $5\times 10^{-5}$ and $10^{-4}$. The weight decay $\lambda _2$ is enabled for all the settings of $\lambda _1$ and set to be $10^{-4}$. We analyzed the influence of exponential gate and compared against the existing methods.

ResNet-164: We use a dropout rate of $0.1$ after the first Batch Normalization layer in every Bottleneck structure. Here, every convolutional layer in the network is followed by an exponential gating layer.

DenseNet-40: We use a dropout of 0.05 after every convolutional layer in the Dense block. Here, the exponential gating layer is added after every convolutional layer in the network except the first convolutional layer.

MobileNetV2: On CIFAR100, we train the network for 240 epochs where learning rate drops by 0.1 at 200 and 220 epochs. A dropout of 0.3 is applied after the global average pooling layer. On ImageNet, we train this network for 100 epochs which is in contrast to the standard training of 400 epochs. We start with learning rate 0.045 and reduced it by 0.1 at 30, 60 and 90 epochs. We evaluate performance of the network on exponential gate over the linear gate with $\ell _1$ regularizer and also tested the significance of bounded-$\ell _1$ on linear gate. Exponential gating layer is added after every standard convolutional/depthwise separable convolutional layer in the network.

On CIFAR100, we investigate the influence of weight decay, $\ell _1$ and bounded-$\ell _1$ regularizers, the role of linear and exponential gates on every architecture. We also study the influence of scheduling $\sigma$ in both $\ell _1$ and bounded-$\ell _1$ regularizers on this dataset. For MobileNetV2, we initialize $\sigma$ with 2.0 and decay it at a rate of 0.99 after every epoch. In case of ResNet-164 and DenseNet-40, we initialize the hyperparameters $\lambda _1$ and $\lambda _2$ with $10^{-4}$ and $5\times 10^{-4}$ respectively and $\sigma$ with 2.0. We increase the $\lambda _1$ to $5\times 10^{-4}$ after 120 epochs and $\sigma$ drops by 0.02 after every epoch until the value of $\sigma$ reaches to 0.2 and later decays at a rate of 0.99.