Margin-Aware Binarized Weight Networks for Image Classification

Abstract

Deep neural networks (DNNs) have achieved remarkable successes in many vision tasks. However, due to the dependence on large memory and high-performance GPUs, it is extremely hard to deploy DNNs on low-power devices. For compressing and accelerating deep neural networks, many techniques have been proposed recently. Particularly, binarized weight networks, which store one weight using only one bit and replace complex floating operations with simple calculations, are attractive from the perspective of hardware implementation. In this paper, we propose a simple strategy to learn better binarized weight networks. Motivated by the phenomenon that the stochastic binarization approach usually converges with real-valued weights close to two boundaries $\{-1, +1\}$ and gives better performance than deterministic binarization, we construct a margin-aware binarization strategy by adding a weight constraint into the objective function of deterministic scheme to minimize the margins between real-valued weights and boundaries. This constraint can be easily realized by a Binary-L2 regularization without suffering from the complex random number generation. Experimental results on MNIST and CIFAR-10 datasets show that the proposed method yields better performance than recent network binarization schemes and the full precision network counterpart.

1 Introduction

Deep Neural Networks (DNNs) have been successfully applied to a wide range of vision tasks, such as object recognition [1,2,3,4], object detection [5,6,7] and semantic segmentation [8, 9]. Despite their huge power and success, DNNs require large memory overhead and high-performance hardware for implementation, which makes it difficult to run DNNs on some resource-limited devices (e.g., Smartphone, Smart wearable devices and Embedded devices) or dedicated Deep Learning hardware directly. For instance, the famous AlexNet [1] requires about 61 $\times$ 10$^{6}$ parameters (232 MB¹) with 725 million FLoating-point OPerations (FLOPs), and VGG-16 [2] involves 138 $\times$ 10$^{6}$ parameters (527 MB) with 15.3 billion FLOPs. Therefore, reducing the parameters of DNNs and removing most arithmetic operations are crucial problems for many real-time deployments.

Fig. 1.

Binarized weight networks and weight constraint terms.

To break the above bottleneck of DNNs’ application on resource-limited devices, various deep model compression methods have been proposed recently. The first strategy proposes new architectures to reduce the parameters, especially in the fully-connected (fc) layer, such as GoogLenet [3], ResNet [4], Network in Network [10], SqueezeNet [11]. They replace fc layers with global average pooling layer to reduce parameters dramatically, and design diversified compact blocks (e.g., Inception module [3], “bottleneck” building block of ResNet [4], mlpconv layer [10] and Fire module [11]) to reduce parameters of convolutional layers further. Finally, the deeper and light networks are constructed for higher performance. However, such kind of light networks do not take care of the number of FLOPs (speed) adequately. For instance, the ResNet-18 [4] network still needs about 1.8 billion FLOPs to classify an image of size 224 $\times$ 224.

The second strategy utilizes compression techniques to remove redundant parameters [12] of the classical DNNs for less storage and faster inference, including tensor decomposition with low-rank approximation [13,14,15], sparse constraints [16], hash function [17], threshold based reduction with retraining [18], integration of hash trick and Huffman coding [19], etc. Though these approaches achieve significant compression ratio for fc layers, they are not well suitable for the new DNN models because the new DNNs structures (e.g., ResNet [4], SqueezeNet [11]) have discarded fc layers. While the training process is also complicated since the repetitional fine-tuned operations and difficultly recover the original accuracy produced by information loss of pre-trained models.

To obtain better compression and acceleration, the third strategy which uses lower bits to store the weight is proposed. For example, Vanhoucke et al. [20] directly replaced the 32-bit “float” with 8-bit “char” to shrink total memory footprint by 4$\times$. Later, binarized weight networks were proposed, where the weights were approximated with binary values $\{-1, +1\}$ (in Fig. 1(a)) resulting in $32\times$ memory saving and simple addition or subtraction operations. Soudry et al. [21] were the first to show that the binarized DNNs can obtain good accuracy by the Expectation BackPropagation (EBP) algorithm which updates the posterior distributions over real-valued weights. Courbariaux et al. [22, 23] designed the BinaryConnect algorithm and BinarizedNet method by extending the probabilistic idea behind EBP algorithm, where they directly utilized binarized parameters to train Binarized Neural Networks using standard back propagation. For better performance, Rastegari et al. [24] proposed Binary-Weight-Networks and XNOR-Networks by combining optimized scale factors with binarized weights or activations. However, the overall compression rate of XNOR-Network was severely degraded because both the first and last layers were of full precision to guarantee reasonable accuracy. To address this problem, Tang et al. [25] adopted a scalar layer and new regularization term to mitigate severe accuracy drop when the last layer was also binarized, but the compression rate is still limited since the first layer is still of full precision.

In this paper, we focus on binarized weight networks which are attractive from the perspective of hardware, and study to achieve both better performance and simplicity of implementation. In the experiments of BinaryConnect [22], the stochastic binarization scheme is of better performance than deterministic binarization, but harder to implement as it needs the hardware to generate random bits. To analyse the difference between the two schemes, we carefully examine the weight distribution after the algorithm converges and find an interesting phenomenon. For the stochastic approach, all the weights gather close to the boundaries $\{-1, +1\}$ by a very small margin, while for the deterministic approach, there are considerable amount of weights scattered over the interval and some even stuck around 0, leaving a large margin to the boundaries.

This phenomenon motivates our proposed margin-aware binarization strategy which is constructed by introducing a weight constraint into the objective function of deterministic binarization to minimize the loss between real-valued weights and corresponding binarized weights (boundaries). The constraint formulated as a Binary-L2 term gives the weights awareness of the margins to boundaries, and the proposed method can effectively improve the performance of deterministic binarization approach, which can be easily realized without random number generation. Extensive experiments demonstrate that our Binarized Weight Networks algorithm yields better performance than recent network binarization approaches and the corresponding full precision networks.

It’s worth noting that although the regularization used in our method is similar to the one in [25] to some extent, the contributions of the two are obviously different in the following perspectives: (1) in Fig. 1(b), the weight gradient in our Binary-L2 term becomes stronger as it approaches 0 by a large margin to the boundaries, which shrinks the margin effectively and drives the weights towards two boundaries $\{-1, +1\}$ friendly. On the contrary, the gradient of regularization term in [25] tends to vanish around 0, and its large gradient at $\pm 1$ is likely to stick the weights at their current binary values. (2) While the motivation in [25] is to apply a regularization on the weights, we propose the Binary-L2 based on the analysis of the discovered phenomenon, which provides a perspective on explaining why this term is effective to improve the performance. A detailed discussion on the potential mechanism behind this phenomenon is given in this paper.

The reminder of this paper is organized as follows. Section 2 presents details about our Binarized-Weight-Networks. Section 3 shows experimental results on several common datasets and the analysis. Concluding remarks will be drawn in Sect. 4.

2 Binarized Weight Networks

In the past, people thought that low-precision networks would suffer from obvious performance drop, especially under the extreme scenario: Binarized DNNs were believed to be highly destructive to the network accuracy [26]. However, recent binarized based networks [21, 22, 24] have exhibited promising results when the weights or activations of DNNs are binarized to $\pm 1$. BinaryConnect [22] algorithm constrains the weights to be only two possible values $\{-1, +1\}$ and use the binarized weights in the forward and backward propagations. To exploit the Stochastic Gradient Descent (SGD) technique to optimize the network, there is a real-valued weight corresponding to each binarized weight. In every step, the real-valued weights are updated according to the gradients of the binarized weights. Courbariaux et al. [22] explain that the constrained function of binarization can be regarded as a new regularization (a binary sampling process) for real-valued weights which is able to improve generalization capability, like Dropout [27] and DropConnect [28] operations.

Motivation: In BinaryConnect [22], there are two types of binarization operation: deterministic (det.) and stochastic (stoch.) binarization. The deterministic binarization operation constrains each real-valued weight to binary values $\pm 1$ using the following sign function:

$$\begin{aligned} w_i^b = \text {sign}(w_i) = {\left\{ \begin{array}{ll} +1 &{} \text {if} ~ w_i \ge 0, \\ -1 &{} \text {if} ~ w_i < 0, \end{array}\right. } \end{aligned}$$

(1)

where $w_i^b$ is the binarized weight and $w_i$ is corresponding real-valued weight. This binarization scheme is easy to be implemented on various devices and effective in practice [22, 24]. On the other hand, the stochastic binarization requires generating random bits that is much more computationally costly and more difficult to be implemented on low-power hardware devices. However, the stochastic binarization method performs better than deterministic scheme in the experiments of BinaryConnect [22]. To explore the reason, we observe the distribution of the real-valued weights after training stage. An interesting phenomenon is observed in deterministic scheme that although a majority of weights converge to the two boundaries $\{-1, +1\}$, there are also quite a few weights scattering along the interval, leaving a large margin to the boundaries. On the contrary, in the stochastic scheme, all weights converge to two boundaries $\{-1, +1\}$ friendly with a very small margin. According to this phenomenon, we guess there may be some relationship between the margin and the performance. Thus we attempt to design a simple approach to eliminate such a margin in deterministic binarization scheme for pursuing better performance of Binarized-Weight-Networks (in Fig. 2).

Fig. 2.

Distribution of real-valued weights after training time.

Method: Through investigations, we find an analogous problem in visual similarity search field. For these algorithms, how to encode high-dimensional image data into short and effective binary codes based on hashing approaches is a crucial problem. Liong et al. [29] insert three constrained conditions into the objective function of DNNs to make the different bits of the learned binary vector $\mathbf h ^b$ possess independent and discriminative properties, one of which aims at minimizing the loss between real-valued feature descriptor and the learned binary vector by the following function:

$$\begin{aligned} \min \dfrac{1}{2} \Vert \mathbf h - \text {sign}(\mathbf h ) \Vert ^2_{2} = \min \dfrac{1}{2} \Vert \mathbf h -\mathbf h ^b \Vert ^2_{2}, \end{aligned}$$

(2)

where h denotes float values of the last fc layer and $\mathbf h ^b$ is binarized values of h. It is similar to our weight constraint term.

As discussed in our motivation, we are trying to get a better performance by reducing the margins between real-valued weights and the boundaries $\{-1, +1\}$, which is a similar task as the one above. Thus, we introduce a weight constraint deterministic binarization scheme. The constraint term formulated as a Binary-L2 regularization (standard norm) can effectively minimize the loss between the learned weights of DNNs and the corresponding binarized weights, which is defined as:

$$\begin{aligned} \begin{aligned} \min \dfrac{1}{2} \Vert \mathbf W - \text {sign}(\mathbf W ) \Vert ^2_{ F}&= \min \dfrac{1}{2} \Vert \mathbf W -\mathbf W ^b \Vert ^2_{ F} \\&= \min \dfrac{1}{2} \Vert |\mathbf W | -\mathbf 1 \Vert ^2_{l2}. \end{aligned} \end{aligned}$$

(3)

Then the optimization objective function is as follows:

$$\begin{aligned} \begin{aligned} \min _\mathbf{W }{} { J}(\mathbf W )&= { L}(\mathbf W ^b, \mathbf X , \mathbf y ) + \dfrac{\lambda }{2} \Vert |\mathbf W | -\mathbf 1 \Vert ^2_{l2}, \\ s.t.~~ \mathbf W ^b&= \text {sign}(\mathbf W ), \end{aligned} \end{aligned}$$

(4)

where L is common loss function (such as SoftmaxLoss or HingeLoss), the second term is our Binary-L2 regularization and $\lambda$ controls the balance between the two terms. X and y represents the training samples and corresponding label vector respectively. W is the weight tensor of the entire DNNs and 1 is the counterpart identity tensor. The first loss function (L) is minimized for high classification accuracy and the Binary-L2 term gives weights awareness to converge to the two edges $\{-1, +1\}$ friendly during parameter update for discovering better accuracy of Binarized-Weight-Networks.

Training Process: The entire Binarized-Weight-Networks can be trained in an end-to-end manner by back propagation (BP) algorithm and SGD optimization or Adam learning rule [30]. Same as BinaryConnect, there is a corresponding real-valued weight for each binary weight for parameter update. In every step, the previous real-valued weights are updated directly by the gradients, which are computed with respect to the binarized weights. Then the binarized weights are obtained using sign function Eq. 1. After training stage, the float weights are discarded for memory saving since the forward propagation only need binarized weights. The whole procedure is illustrated in Algorithm 1, where R denotes Binary-L2 term (the second term in Eq. 4).

Implementation Detail: Since existing deep learning platforms do not support binarized layers and Binary-L2 regularization, we need to implement binarized convolutional layer, binarized fully-connected layer and special Binary-L2 regularization based on Caffe [33] toolbox. For each real-valued weight, the Binary-L2 regularization term is continuously differentiable and its derivative is define as:

$$\begin{aligned} \dfrac{\partial R}{\partial w_i} = \lambda (|w_i|-1)\text {sign}(w_i). \end{aligned}$$

(5)

In addition, we also adopted several useful techniques as used in BinaryConnect algorithm [22] to achieve better Binarized-Weight-Networks. These techniques include Batch Normalization (BN), scaling the weights learning rates ($\gamma$ in Algorithm 1) and clipping operation. The real-valued weights are easily updated in the range [−1, 1] by these operations. The weight learning rates are scaled to enlarge the gradient magnitude with the coefficient $\gamma$ from “Xavier” [34] algorithm, thus we set $\lambda \gamma$ = 0.0001–0.0005 as the ratio of regularization that is similar to L2 weight decay in DNNs. Clipping operation constrains weights to [−1, 1] interval for preventing weights far from the boundary $\{-1, +1\}$ overly.

3 Experiments

We conducted experiments on two widely used datasets (natural image classification task) to evaluate the effectiveness of our proposed binarization scheme with routine convolutional neural networks (CNN). In our experiments, the basic block structure of our binarized CNN contains convolutional or fully-connected layer with binarized weights, BN [31] layer and ReLU [32] activation layer. We use weights initialization of uniform distribution [−1, 1], Binary-L2 regularization and Adam optimization for all Binarized-Weight-Networks, while the “Xavier” [34] initialization and Adam optimization are adopted for the corresponding full precision weight networks. The experimental settings and results are described in following.

3.1 MNIST

The MNIST [35] dataset contains handwritten digit images (0–9) including 60k training images and 10k testing images. The images are of normalized size $28\times 28$. An architecture similar to LeNet5 [35] is used in our experiment, which is also adopted in [36] and shown in Fig. 3 (left).

Fig. 3.

Left: MNIST dataset and Right: CIFAR-10 dataset.

In this picture, the “32@5$\times$5 Conv” denotes that a convolutional layer possess 32 convolutional kernels with size 5$\times$5, “2$\times$2 Max Pooling” presents a max-pooling layer with kernel size 2 and stride 2, “512-FC” is a fully connected layer with 512 output neurons, and L2-SVM indicates the square hinge loss is used as loss function (L in Eq. 4) because L2-SVM has achieved better results than Softmax on several classification dataset [37, 38]. For fairness of the comparison, the same settings are adopted based on MNIST dataset as used in BinaryConnect [22] algorithm. We use the last 10000 samples of training set for validation without retraining on the validation set and do not use any data-augmentation or unsupervised pretraining. Except for our Binary-L2 regularization, no other regularizations such as Dropout or L2 regularization are used. For detailed settings (e.g., batch size, learning rate), please refer to our Caffe [33] configuration files². The whole Binarized-Weight-Networks are trained from scratch and the test accuracy is reported in Table 1.

Table 1 shows that the proposed binarized method improves the accuracy by more than 1.5% compared to the recent ternary or binarized approaches, and is slightly better than corresponding full precision network. Meanwhile, our simple 4-layers binarized weight network also achieve competitive performance compared with some complex full precision networks.

Table 1. Test set accuracy for MNIST of various methods.

3.2 CIFAR-10

The CIFAR-10 [40] dataset consists of $32\times 32$ colour images in 10 classes including 50000 training images and 10000 test images. Generally speaking, the dataset is preprocessed by the global contrast normalization and ZCA whitening as was used in [10, 22, 37, 39]. We adopt the same network architecture (VGG-like in Fig. 3 right) as other binarization methods [22, 23, 36], where the 9-layers architecture is more simple than standard VGG-Net [2] and “$\times$2” denotes the same two layers. To compare with previous state-of-the-art works, we also evaluate our method on this dataset with data augmentation as used in [4, 37]. In the training phase, zero-padding 4 pixels on each side, and a 32$\times$32 crop is randomly sampled from the padded image or its horizontal flip. At test time, we only use the single view of original 32$\times$32 test image.

A comparison of our method with previous binarized or full precision approaches is shown in Table 2, and Fig. 4 exhibits test set error rate curves of several schemes. Experimental results³ show that our binarized weight network not only obtain the new state-of-the-art accuracy compared with other binarized or ternary weight networks, but also surpass most of full precision networks including ResNet-56 structure. It indicates that the extreme low-precision networks do not destroy original network performance. On the contrary, network binarization and Binary-L2 techniques act as regularizer for improving generalization capability of original networks to some extent.

Table 2. Test set accuracy for CIFAR-10 of various methods.

Fig. 4.

Test set accuracy curves on CIFAR-10 dataset.

3.3 Compression Rates

The real compression rates of our binarized weight network are described in Table 3. We pack 32 binary-valued weight units into a single fixed32 ProtoBuffer unit [33], and other parameters such as biases and special parameters of Batch Normalization layers are still stored in float values. Finally, the proposed binarization scheme compress the float-valued similar LeNet5 (2.23 MB) and VGG-9Layer (53.5 MB) to binarized weight models whose storage requirement are 84.7 KB and 1.74 MB, respectively. The compression rates are 26.96$\times$ and 30.75$\times$, respectively.

Table 3. The compression rates (Comp. rate) of Binarized-Weight-Networks on MNIST and CIFAR-10 datasets. ‘N/Y’ denotes without/with data augmentation.

4 Conclusion

In this paper, we present a simple yet effective algorithm for learning binarized weight deep neural network by introducing margin-aware weight constraint. The algorithm results in memory saving 32$\times$ (theoretically) compared with single-float precision models and make it possible to deploy deep neural networks on low-power devices or dedicated Deep Learning hardware. While training by back propagation algorithm and gradient decent optimization, the introduced margin-aware weight constraint brings obvious improvement of the performance with only negligible computational cost. Experimental results show that the proposed algorithm outperforms existing binarization schemes, and achieves better performance than corresponding full precision networks. The proposed binarization scheme is also robust to different network architectures on several classical image classification datasets.