SRS-DNN: a deep neural network with strengthening response sparsity

Abstract

Inspired by the sparse mechanism of biological neural systems, an approach of strengthening response sparsity for deep learning is presented in this paper. Firstly, an unsupervised sparse pre-training process is implemented and a sparse deep network is begun to take shape. In order to avoid that all the connections of the network will be readjusted backward during the following fine-tuning process, for the loss function of the fine-tuning process, some regularization items which strength the sparse responsiveness are added. More importantly, the unified and concise residual formulae for network updating are deduced, which ensure the backpropagation algorithm to perform successfully. The residual formulae significantly improve the existing sparse fine-tuning methods such as which in sparse autoencoders by Andrew Ng. In this way, the sparse structure obtained in the pre-training can be maintained, and the sparse abstract features of data can be extracted effectively. Numerical experiments show that by this sparsity-strengthened learning method, the sparse deep neural network has the best classification performance among several classical classifiers; meanwhile, the sparse learning abilities and time complexity all are better than traditional deep learning methods.

Appendices

Appendix 1: The deviation of KL divergence for the parameters in Sect. 2.2

$$\begin{aligned} \frac{\partial }{\partial W_{ij}}\sum ^{N_h}_{j=1}KL(\rho \parallel p_j)= & {} \frac{\partial }{\partial W_{ij}}\sum ^{N_h}_{j=1}(\rho \log \frac{\rho }{p_j}+(1-\rho )\log \frac{1-\rho }{1-p_j})\\= & {} \left( -\frac{\rho }{p_j}+\frac{1-\rho }{1-p_j}\right) \frac{\partial p_j}{\partial W_{ij}}\\= & {} \left( -\frac{\rho }{p_j}+\frac{1-\rho }{1-p_j}\right) \frac{1}{N_s}\sum ^{N_s}_{q=1}\sigma ^{(q)}_{j}(1-\sigma ^{(q)}_{j})v^{(q)}_{i}\\= & {} \frac{1}{N_s}\left( -\frac{\rho }{p_j}+\frac{1-\rho }{1-p_j}\right) \sum ^{N_s}_{q=1}\sigma ^{(q)}_{j}(1-\sigma ^{(q)}_{j})v^{(q)}_{i}\\ \frac{\partial }{\partial \beta _{j}}\sum ^{N_h}_{j=1}KL(\rho \parallel p_j)= & {} \frac{\partial }{\partial \beta _{j}}\sum ^{N_h}_{j=1}(\rho \log \frac{\rho }{p_j}+(1-\rho )\log \frac{1-\rho }{1-p_j})\\= & {} \left( -\frac{\rho }{p_j}+\frac{1-\rho }{1-p_j}\right) \frac{\partial p_j}{\partial \beta _{j}}\\= & {} \left( -\frac{\rho }{p_j}+\frac{1-\rho }{1-p_j}\right) \frac{1}{N_s}\sum ^{N_s}_{q=1}\sigma ^{(q)}_{j}(1-\sigma ^{(q)}_{j})\\= & {} \frac{1}{N_s}\left( -\frac{\rho }{p_j}+\frac{1-\rho }{1-p_j}\right) \sum ^{N_s}_{q=1}\sigma ^{(q)}_{j}(1-\sigma ^{(q)}_{j}) \end{aligned}$$

here $\sigma ^{(q)}_{j}=\sigma (\sum ^{N_v}_{i=1}v^{(q)}_{i}W_{ij}+\beta _j)=\frac{1}{1+e^{-\sum ^{N_v}_{i=1}v^{(q)}_{i}W_{ij}-\beta _j}}$.

Appendix 2: The derivation of updating formula for the traditional BP in Sect. 3.1

For the traditional BP, the total error of the network in the backpropagation process, i.e., the loss function is

$$\begin{aligned} J(W)=\frac{1}{2N}\sum ^{N}_{q=1}\sum ^{n_L}_{j=1}(a^{(L)}_{qj}-y_{qj})^2 \end{aligned}$$

where N is the training sample size, $y_{qj}$ is the target output of the j-th neuron in the output layer corresponding to the q-th sample, and $a^{(L)}_{qj}$ is the actual output of it. For simplicity, we first give the parameter updating formula for one sample. Consider $J(W)=\frac{1}{2}\sum ^{n_L}_{j=1}(a^{(L)}_{j}-y_j)^2$ as the error of the network for one sample. Let $\eta _1$ be the learning rate, $W^{(l)}_{ij}$ be the connection weight of the i-th node in the l-th layer and the j-th node in the $(l+1)$-th layer ($1\le i\le n_l+1$, $1\le j\le n_{l+1}$), then we have the following update formula for the network parameters

$$\begin{aligned} W^{(l)}_{ij}= & {} W^{(l)}_{ij}-\eta _1\frac{\partial J(W)}{\partial W^{(l)}_{ij}}=W^{(l)}_{ij}-\eta _1\frac{\partial J(W)}{\partial z^{(l+1)}_{j}}\cdot \frac{\partial z^{(l+1)}_{j}}{\partial W^{(l)}_{ij}} \\= & {} W^{(l)}_{ij}-\eta _1\delta ^{(l+1)}_{j} a^{(l)}_{i} \end{aligned}$$

where $\delta ^{(l+1)}_{j}=\frac{\partial J(W)}{\partial z^{(l+1)}_{j}}$ is the residual of the j-th node in the $(l+1)$-th layer. For the L-th layer, i.e., the output layer, the residual of the j-th node is

$$\begin{aligned} \delta ^{(L)}_{j}= & {} \frac{\partial J(W)}{\partial z^{(L)}_{j}}=\frac{\partial }{\partial z^{(L)}_{j}}\frac{1}{2}\sum ^{n_L}_{k=1}(a^{(L)}_{k}-y_k)^2 \\= & {} \frac{\partial }{\partial z^{(L)}_{j}}\frac{1}{2}\sum ^{n_L}_{k=1}(f(z^{(L)}_{k})-y_k)^2\\= & {} (f(z^{(L)}_{j})-y_j) f^{'}(z^{(L)}_{j}) \end{aligned}$$

Suppose

$$\begin{aligned} \delta ^{(l)}= & {} (\delta ^{(l)}_{1},\delta ^{(l)}_{2},\ldots ,\delta ^{(l)}_{n_l})\\ f^{'}(z^{(l)})= & {} (f^{'}(z^{(l)}_{1}),f^{'}(z^{(l)}_{2}),\ldots ,f^{'}(z^{(l)}_{n_l})) \end{aligned}$$

thus the residual vector of the L-th layer is

$$\begin{aligned} \delta ^{(L)}=(a^{(L)}-y)_{\cdot }*(f^{'}(z^{(L)})) \end{aligned}$$

(20)

where $_{\cdot }*$ is the vector product operator (Hadamard product), which is defined as the product of the corresponding elements for one vector or matrix.

The residual of the j-th node for the $(L-1)$-th layer is and the residual of the j-th node for the $(L-1)$-th layer is

$$\begin{aligned} \delta ^{(L-1)}_{j}= & {} \frac{\partial J(W)}{\partial z^{(L-1)}_{j}}=\frac{\partial }{\partial z^{(L-1)}_{j}}\frac{1}{2}\sum ^{n_L}_{k=1}(a^{(L)}_{k}-y_k)^2 \\= & {} \frac{\partial }{\partial z^{(L-1)}_{j}}\frac{1}{2}\sum ^{n_L}_{k=1}(f(z^{(L)}_{k})-y_k)^2\\= & {} \frac{1}{2}\sum ^{n_L}_{k=1}\frac{\partial }{\partial z^{(L-1)}_{j}}(f(z^{(L)}_{k})-y_k)^2 \\= & {} \sum ^{n_L}_{k=1}(f(z^{(L)}_{k})-y_k)\cdot f^{'}(z^{(L)}_{k})\cdot \frac{\partial z^{(L)}_{k}}{\partial z^{(L-1)}_{j}}\\= & {} \sum ^{n_L}_{k=1}\delta ^{(L)}_{k}\cdot \frac{\partial z^{(L)}_{k}}{\partial z^{(L-1)}_{j}} \\= & {} \sum ^{n_L}_{k=1}\delta ^{(L)}_{k}\cdot \frac{\partial }{\partial z^{(L-1)}_{j}}\sum ^{n_{L-1}}_{s=1}a^{(L-1)}_{s}\cdot W^{(L-1)}_{sk}\\= & {} \sum ^{n_L}_{k=1}W^{(L-1)}_{jk}\delta ^{(L)}_{k} f^{'}(z^{(L-1)}_{j}) \end{aligned}$$

The residual of the j-th node for the l-th layer $(l=L-1,\ldots ,2,1)$ is $\delta ^{(l)}_{j}=(\sum ^{n_{l+1}}_{k=1}W^{(l)}_{jk}\delta ^{(l+1)}_{k})f^{'}(z^{(l)}_{j})$, thus the vector form of the residual for the l-th layer is

$$\begin{aligned} \delta ^{(l)}=((\delta ^{(l+1)})\cdot ({\bar{W}}^{(l)})^{\mathrm{T}})_{\cdot }*f^{'}(z^{(l)}) \end{aligned}$$

(21)

where $\cdot$ is the matrix product, ${\bar{W}}^{(l)}$ is the first $n_l$ rows of $W^{(l)}$. Let

$$\begin{aligned} \Delta W^{(l)}= & {} (a^{(l)})^{\mathrm{T}} \cdot \delta ^{(l+1)} \end{aligned}$$

(22)

in which $\delta ^{(l+1)}$ is defined by (20)–(21) ($l=L-1,\ldots ,2,1$).

For the N samples case, by (22), we have $\Delta {W^{(l)}_q}\_{J}=(a^{(l)})^{\mathrm{T}}_q \cdot \delta ^{(l+1)}_q$ for each sample $q=1,2,\ldots ,N$. Thus, the update formula for the network parameters in a matrix form is

$$\begin{aligned} W^{(l)}= & {} W^{(l)}-\eta _1 \cdot \frac{1}{N} \sum ^{N}_{q=1}\Delta W^{(l)}_q\_{J} \end{aligned}$$

Appendix 3: AR results of SRS-DNN with different parameter sets

The following Table 6 contains different sparse parameter sets of sparsity penalty items for KL and $L_1$ in RBM as well as in BP, i.e., $\lambda _2$, $\lambda _3$, $\tau _1$ and $\tau _2$, and also their corresponding accurate rates of classification. Based on which, in this paper, we select $\lambda _2$, $\lambda _3$, $\tau _1$ and $\tau _2$ to be 0.005, 0.0001, 0.0001 and 0.0002, respectively.

Table 6 AR results of SRS-DNN with different parameter sets