Weighted pooling for image recognition of deep convolutional neural networks

Abstract

There are some traditional pooling methods in convolutional neural network, such as max-pooling, average pooling, stochastic pooling and so on, which determine the results of pooling based on the distribution of each activation in the pooling region. Zeiler and Fergus (Stochastic-pooling for regularization of deep convolutional neural networks, 2013) However, it is difficult for the feature mapping process to select a perfect activation representative of the pooling region, and can lead to the phenomenon of over-fitting. In this paper, the following theoretical basis comes out information theory (Shannon in Bell Syst. Tech. J. 27:379–423, 1948). First, we quantify the information entropy of each pooling region, and then propose an efficient pooling method by comparing the mutual information between activations and the pooling region which they are located in. Moreover, we assign different weights to different activations based on mutual information, and named it weighted-pooling. The main features of the weighted-pooling method are as follows: (1) The information quantity of the pooling region is quantified by information theory for the first time. (2) Also, each activation’s contribution was quantified for the first time and these contributions eliminate the uncertainty of the pooling region which it is located in. (3) For choosing a representative in this pooling region, the weight of each activation obviously superiors to the value of activation. In the experimental part, we respectively use MNIST and CIFAR-10 (Krizhevsky in Learning multiple layers of featurs from tiny images, University of Toronto, 2009; LeCun in The MNIST database, 2012) data sets to compare different pooling methods. The results show that the weighted-pooling method has higher recognition accuracy than other pooling methods and reaches a new state-of-the-art.

Appendix

1.1 Joint entropy

Conditional entropy can be embodied by a fact that the entropy of a pair of stochastic variables is equal to the entropy of one of the stochastic variables plus the conditional entropy of another stochastic variable. $H(X,Y)=H(X)+H(Y|X)$.

Proof

$$\begin{aligned} H(X,Y)= & {} -\sum _{x\in \mathcal {X}}\sum _{y \in \mathcal {Y}} p(x,y)\log (x|y) \nonumber \\= & {} -\sum _{x\in \mathcal {X}}\sum _{y \in \mathcal {Y}} p(x,y)\log p(x)p(y|x)\nonumber \\= & {} -\sum _{x\in \mathcal {X}}\sum _{y \in \mathcal {Y}} p(x,y)\log p(x) -\sum _{x\in \mathcal {X}}\sum _{y \in \mathcal {Y}} p(x,y)\log p(y|x)\nonumber \\= \,& {} \sum _{x\in \mathcal {X}}p(x)\log p(x)-\sum _{x\in \mathcal {X}}\sum _{y \in \mathcal {Y}} p(x,y)\log p(y|x)\nonumber \\= \,& {} H(X)+H(Y|X) \end{aligned}$$

(29)

Equivalently written to:

$$\begin{aligned} \log p(X,Y)=\log p(X)+\log p(Y|X) \end{aligned}$$

(30)

Both sides of the equation take the mathematical expectation, which is the theorem.$\square$

1.2 Mutual information

The mutual information I(X;Y) can be rewritten in the following form.

Proof

$$\begin{aligned} I(X;Y)= & {} \sum _{x,y}p(x,y)\log \frac{p(x,y)}{p(x)p(y)}\nonumber \\= & {} \sum _{x,y}p(x,y)\log \frac{p(x|y)}{p(x)}\nonumber \\= & {} -\sum _{x,y}p(x,y)\log {p(x)}+\sum _{x,y}p(x,y)\log p(x|y)\nonumber \\= & {} -\sum _x p(x) \log p(x) -(-\sum _{x,y}p(x,y)\log p(x|y))\nonumber \\= \,& {} H(X)-H(X|Y) \end{aligned}$$

(31)

$\square$