Skip to main content
SpringerLink
Log in

An efficient way to refine DenseNet

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

DenseNet features dense connections between layers. Such an architecture is elegant but suffers memory-hungry and time-consuming. In this paper, we explore the relation between density of connections and performance of DenseNet (Huang et al., in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017). We find that sometimes even just preserving 25% connections does not harm the performance but get a little promotion. We aim to provide users a trade-off between performance and efficiency. We analyze the relation in two connection-trimming ways. One is preserving connection proportionally as a given rate and the other as a given quantity of connection. We evaluate the performance and efficiency between all the architectures on the competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). Experimental results demonstrate that moderate connection trimming achieves the significant performance for DenseNet, but requires almost less than half of the GPU memories, i.e., 40% fewer parameters and about 40% less time for prediction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

The datasets used in the experiment are from previously reported studies and datasets, which have been cited.

References

  1. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009 (CVPR 2009) pp. 248–255 (2009)

  2. Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 315–323 (2011)

  3. Guo, Y., Ding, G., Han, J.: Robust quantization for general similarity search. IEEE Transactions on Image Processing 27(2), 949–963 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  5. Huang, G., Liu, Z., Weinberger, K.Q., van der Maaten, L.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 1 (2017)

  6. Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. CoRR (2016). arxiv:1603.09382

  7. Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: European Conference on Computer Vision, pp. 646–661. Springer, Berlin (2016)

  8. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift (2015). arXiv preprint arXiv:1502.03167

  9. Krizhevsky, A., Hinton, G.: Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto (2009)

  10. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  11. Lin, M., Chen, Q., Yan, S.: Network in network (2013). arXiv preprint arXiv:1312.4400

  12. Lin, T., Maire, M., Belongie, S.J., Bourdev, L.D., Girshick, R.B., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. CoRR (2014). arxiv:1405.0312

  13. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning, vol. 2011 (2011)

  14. Qi, Y., Zhang, S., Qin, L., Huang, Q., Yao, H., Lim, J., Yang, M.H.: Hedging deep features for visual tracking. IEEE Trans. Pattern Anal. Mach. Intell. PP(99), 1–1 (2018)

    Google Scholar 

  15. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv preprint arXiv:1409.1556

  16. Srivastava, R.K., Greff, K., Schmidhuber, J.: Training very deep networks. In: Advances in Neural Information Processing Systems, pp. 2377–2385 (2015)

  17. Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: International Conference on Machine Learning, pp. 1139–1147 (2013)

  18. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: AAAI, vol. 4 (2017)

  19. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., et al.: Going deeper with convolutions. In: CVPR (2015)

  20. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)

  21. Wu, G., Han, J., Lin, Z., Ding, G., Zhang, B., Ni, Q.: Joint image-text hashing for fast large-scale cross-media retrieval using self-supervised deep learning. In: IEEE Transactions on Industrial Electronics, pp. 1–1 (2018)

  22. Xie, D., Zhang, L., Bai, L.: Deep learning in visual computing and signal processing. Appl. Comput. Intell. Soft Comput. 2017(10), 1–13 (2017)

    Article  Google Scholar 

  23. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987–5995 (2017)

  24. Ye, G., Liu D., Wang, J., Chang S.: Large-scale video hashing via structure learning. In: 2013 IEEE international conference on computer vision, pp 2272–2279 (2013). https://doi.org/10.1109/ICCV.2013.282

  25. Zhang, L., Chai, R., Arefan, Dooman, Sumkin, Jules, S.W.: Deep-learning method for tumor segmentation in breast DCE-MRI. In: Medical Imaging. SPIE (2019)

  26. Zhang, L., Mohamed, A.A., Chai, R., Zheng, B., Wu, S.: Automated deep-learning method for whole-breast segmentation in diffusion-weighted breast MRI. In: Medical Imaging, SPIE (2019)

  27. Zhang, L., Yang, F., Zhang, Y.D., Zhu, Y.J.: Road crack detection using deep convolutional neural network. In: International Conference on Image Processing. IEEE (2016)

  28. Zhao, S., Ding, G., Gao, Y., Zhao, X., Tang, Y., Han, J., Yao, H., Huang, Q.: Discrete probability distribution prediction of image emotions with shared sparse learning. IEEE Trans. Affect. Comput. 1, 1–1 (2018)

    Google Scholar 

  29. Zhao, S., Gao, Y., Ding, G., Chua, T.S.: Real-time multimedia social event detection in microblog. IEEE Trans. Cybern. 99, 1–14 (2017)

    Google Scholar 

  30. Zhao, S., Yao, H., Gao, Y., Ding, G., Chua, T.: Predicting personalized image emotion perceptions in social networks. IEEE Trans. Affect. Comput. 9(4), 526–540 (2018). https://doi.org/10.1109/TAFFC.2016.2628787

    Article  Google Scholar 

  31. Zhao, S., Yao, H., Gao, Y., Ji, R., Ding, G.: Continuous probability distribution prediction of image emotions via multitask shared sparse regression. IEEE Trans. Multimed. 19(3), 632–645 (2017)

    Article  Google Scholar 

  32. Zhao, S., Yao, H., Zhao, S., Jiang, X., Jiang, X.: Multi-modal microblog classification via multi-task learning. Multimed. Tools Appl. 75(15), 8921–8938 (2016)

    Article  Google Scholar 

  33. Zhao, S., Zhao, X., Ding, G., Keutzer, K.: Emotiongan: Unsupervised domain adaptation for learning discrete probability distributions of image emotions. In: 2018 ACM Multimedia Conference on Multimedia Conference, pp. 1319–1327. ACM (2018)

  34. Zhou, Z., Shin, J., Zhang, L., Gurudu, S., Gotway, M., Liang, J.: Fine-tuning convolutional neural networks for biomedical image analysis: actively and incrementally. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7340–7349 (2017)

Download references

Acknowledgements

This work was partly supported by the Science and Technology Funding of China (Nos. 61772158 and 61472103) and the Science and Technology Funding Key Program of China (No. U1711265).

Author information

Authors and Affiliations

  1. Harbin Institute of Technology, Harbin, China

    Xinjie Feng, Hongxun Yao & Shengping Zhang

Authors
  1. Xinjie Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongxun Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shengping Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xinjie Feng.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feng, X., Yao, H. & Zhang, S. An efficient way to refine DenseNet. SIViP 13, 959–965 (2019). https://doi.org/10.1007/s11760-019-01433-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-019-01433-4

Keywords

Search

Navigation