Skip to main content
SpringerLink
Log in

Weight correlation reduction and features normalization: improving the performance for shallow networks

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Although convolutional neural networks (CNNs) show great abilities in image classification, improving their performance is still challenging for shallow networks. The redundancy of the network increases when more convolution kernels are adopted in the network. To alleviate this defect, we propose two methods including Weight Correlation Reduction (WCR) and Features Normalization (FN) to boost the performance of shallow networks. The formal method is designed to eliminate weight redundancy, while the latter is used to increase the sparsity of learned deep features. On benchmarks CIFAR-10 and STL-10, the accuracy rate increased by \(2.29\%\) and \(4.79\%\) for shallow networks, respectively, which indicates the effectiveness of the proposed methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Attwell, D., Laughlin, S.B.: An energy budget for signaling in the grey matter of the brain. J. Cereb. Blood Flow Metab. 21(10), 1133–1145 (2001)

    Article  Google Scholar 

  2. Benesty, J., Chen, J., Huang, Y., Cohen, I.: Pearson correlation coefficient. In: Noise reduction in speech processing, pp. 1–4. Springer (2009)

  3. Chen, H., Sun, K., Tian, Z., Shen, C., Huang, Y., Yan, Y.: BlendMask: Top-down meets bottom-up for instance segmentation. In: Computer Vision and Pattern Recognition, pp. 8573–8581 (2020)

  4. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. Pattern Anal. Mach. Intell. 40(4), 834–848 (2018)

    Article  Google Scholar 

  5. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. arXiv preprint arXiv:1802.02611 (2018)

  6. Chen, W., Huang, H., Peng, S., Zhou, C., Zhang, C.: YOLO-face: a real-time face detector. Vis. Comput. 1–9 (2020)

  7. Chen, Y., Fan, H., Xu, B., Yan, Z., Kalantidis, Y., Rohrbach, M., Yan, S., Feng, J.: Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In: International Conference on Computer Vision, pp. 3435–3444 (2019)

  8. Cheng, Y., Yu, F.X., Feris, R.S., Kumar, S., Choudhary, A., Chang, S.F.: An exploration of parameter redundancy in deep networks with circulant projections. In: International Conference on Computer Vision, pp. 2857–2865 (2015)

  9. Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: the International Conference on Artificial Intelligence and Statistics, JMLR Workshop and Conference Proceedings, pp. 215–223 (2011)

  10. Denil, M., Shakibi, B., Dinh, L., Ranzato, M.A., De Freitas, N., 2013. Predicting parameters in deep learning. arXiv preprint arXiv:1306.0543 (2013)

  11. Ding, S., Sun, Y., An, Y., Jia, W.: Multiple birth support vector machine based on recurrent neural networks. Appl. Intell. 50(7), 2280–2292 (2020)

    Article  Google Scholar 

  12. Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Van Der Smagt, P., Cremers, D., Brox, T.: Flownet: Learning optical flow with convolutional networks. In: Computer Vision and Pattern Recognition, pp. 2758–2766 (2015)

  13. Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., Tian, Q.: Centernet: Keypoint triplets for object detection. In: International Conference on Computer Vision, pp. 6569–6578 (2019)

  14. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2016)

  15. Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: the International Conference on Artificial Intelligence and Statistics, JMLR Workshop and Conference Proceedings, pp. 315–323 (2011)

  16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  17. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

  18. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: Evolution of optical flow estimation with deep networks. In: Computer Vision and Pattern Recognition, pp. 1647–1655 (2017)

  19. Kabbai, L., Abdellaoui, M., Douik, A.: Image classification by combining local and global features. Vis. Comput. 35(5), 679–693 (2019)

    Article  Google Scholar 

  20. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Tech. rep. (2009)

  21. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Neural Information Processing Systems, pp. 1097–1105 (2012)

  22. Lee, H., Battle, A., Raina, R., Ng, A.Y.: Efficient sparse coding algorithms. In: Neural Information Processing Systems, pp. 801–808 (2007)

  23. Lennie, P.: The cost of cortical computation. Curr. Biol. 13(6), 493–497 (2003)

    Article  Google Scholar 

  24. Li, X., Yang, Y., Zhao, Q., Shen, T., Lin, Z., Liu, H.: Spatial pyramid based graph reasoning for semantic segmentation. In: Computer Vision and Pattern Recognition, pp. 8947–8956 (2020)

  25. Liu, B., Wang, M., Foroosh, H., Tappen, M., Pensky, M.: Sparse convolutional neural networks. In: Computer Vision and Pattern Recognition, pp. 806–814 (2015)

  26. Liu, C.T., Wu, Y.H., Lin, Y.S., Chien, S.Y.: A Kernel redundancy removing policy for convolutional neural network. CoRR arXiv:1705.10748 (2017)

  27. Mairal, J., Bach, F., Ponce, J., Sapiro, G. and Zisserman, A.: Supervised dictionary learning. arXiv preprint arXiv:0809.3083 (2008)

  28. Mishkin, D., Matas, J.: All you need is a good init. arXiv preprint arXiv:1511.06422 (2015)

  29. Olshausen, B.A., Field, D.J.: Sparse coding with an overcomplete basis set: A strategy employed by V1? Vis. Res. 37(23), 3311–3325 (1997)

    Article  Google Scholar 

  30. Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018)

  31. Shi, S., Ding, S., Zhang, Z., Jia, W.: Energy-based structural least squares MBSVM for classification. Appl. Intell. 50(3), 681–697 (2020)

    Article  Google Scholar 

  32. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  33. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Computer Vision and Pattern Recognition, pp. 1–9 (2015)

  34. Wang, D., Hu, G., Lyu, C.: Frnet: an end-to-end feature refinement neural network for medical image segmentation. Vis. Comput. 1432–2315 (2020)

  35. Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. arXiv preprint arXiv:1608.03665 (2016)

  36. Wu, B., Liu, Z., Yuan, Z., Sun, G., Wu, C.: Reducing overfitting in deep convolutional neural networks using redundancy regularizer. In: International Conference on Artificial Neural Networks, pp. 49–55 (2017)

  37. Xie, D., Xiong, J., Pu, S.: All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. arXiv preprint arXiv:1703.01827 (2017)

  38. Yu, C., Wang, J., Gao, C., Yu, G., Shen, C., Sang, N.: Context prior for scene segmentation. In: Computer Vision and Pattern Recognition, pp. 12416–12425 (2020)

  39. Yuan, Y., Xie, J., Chen, X., Wang, J.: Segfix: Model-agnostic boundary refinement for segmentation. In: European Conference on Computer Vision, pp. 489–506 (2020)

  40. Zhang, J., Ding, S., Zhang, N., Jia, W.: Adversarial training methods for boltzmann machines. IEEE Access, 8, 4594–4604 (2019)

  41. Zhang, N., Ding, S., Sun, T., Liao, H., Wang, L., Shi, Z.: Multi-view RBM with posterior consistency and domain adaptation. Inf. Sci. 516, 142–157 (2020)

    Article  Google Scholar 

  42. Zhao, C., Ni, B., Zhang, J., Zhao, Q., Zhang, W., Tian, Q.: Variational convolutional neural network pruning. In: Computer Vision and Pattern Recognition, pp. 2780–2789 (2019)

  43. Zhong, X., Gong, O., Huang, W., Li, L., Xia, H.: Squeeze-and-excitation wide residual networks in image classification. In: Conference on Image Processing, pp. 395–399 (2019)

  44. Zhou, X., Wang, D., Krähenbühl, P.: Objects as points. ArXiv Preprint arXiv:1904.07850 (2019)

Download references

Funding

This study was funded by National Natural Science Foundation of China (No. 61502358, 61903164).

Author information

Author notes

  1. C. Song, J. Wu: equal contribution.

Authors and Affiliations

  1. School of Information Science and Engineering, Wuhan University of Science and Technology, Wuhan, China

    Can Song, Jin Wu & Lei Zhu

  2. School of Computer Science and Engineering, Jiangsu University of Science and Technology, Zhenjiang, China

    Xin Zuo

Authors
  1. Can Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xin Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Can Song conceived the main idea, designed the algorithm, performed the experiments, analyzed the data, and wrote the manuscript. Jin Wu and Lei Zhu proofread the manuscript. All the authors discussed the results and commented on the manuscript.

Corresponding author

Correspondence to Lei Zhu.

Ethics declarations

Conflict of interest

The all authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, C., Wu, J., Zhu, L. et al. Weight correlation reduction and features normalization: improving the performance for shallow networks. Vis Comput 38, 2489–2498 (2022). https://doi.org/10.1007/s00371-021-02125-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02125-2

Keywords

Search

Navigation