Skip to main content
SpringerLink
Log in

Multiple Regularization and Analysis of Deep Capsule Network

  • Theoretical Advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

With the increase of layers in deep capsule networks, the overfitting problem also becomes more serious. Capsule-based regularization methods are important to solve this problem. However, little attention has been paid to this field. To fill this gap, we propose five regularization methods from the following aspects. In capsules represented by vectors, two methods are proposed to modify the existence and properties of their activation vectors by disturbing the length and orientation of the vectors. In capsules represented by tensors, capsule-based layer normalization is proposed to improve dynamic routing. In the training strategy, a warm restart learning rate with probability is used to improve the efficiency of training. In reconstruction, a novel image decoder provides a better regularization effect by using multiscale information of images. These regularization methods are investigated on CIFAR10, CIFAR100, and SVHN. Experiments show that using these regularization methods can effectively improve the generalization performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  1. Ba J, Kiros J, Hinton GE (2016) Layer normalization. arXiv preprint arXiv:1607.06450

  2. Byerly A, Kalganova T, Dear I (2020) A branching and merging convolutional network with homogeneous filter capsules. arXiv preprint arXiv:2001.09136

  3. Choi J, Seo H, Im S, Kang M (2019) Attention routing between capsules. In: Proceedings of the IEEE/CVF international conference on computer vision workshop, pp 1981–1989. https://doi.org/10.1109/ICCVW.2019.00247

  4. Deliège A, Cioppa A, Droogenbroeck MV (2018) Hitnet: a neural network with capsules embedded in a hit-or-miss layer, extended with hybrid data augmentation and ghost capsules. arXiv preprint arXiv:1806.06519

  5. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  6. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov R (2012) Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580

  7. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the international conference on machine learning 37, pp 448–456

  8. Krizhevsky A (2009) Learning multiple layers of features from tiny images. Technical report. University of Toronto, Toronto

    Google Scholar 

  9. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proceed IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  10. Li S, Ren X, Yang L (2018) Fully capsnet for semantic segmentation. In: Proceedings of the Chinese conference on pattern recognition and computer vision, pp 392–403. https://doi.org/10.1007/978-3-030-03335-4_34

  11. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440. https://doi.org/10.1109/CVPR.2015.7298965

  12. Marchisio A, Bussolino B, Colucci A, Hanif MA, Martina M, Masera G, Shafique M (2019) X-traincaps: accelerated training of capsule nets through lightweight software optimizations. arXiv preprint arXiv:1905.10142

  13. Mukhometzianov R, Carrillo J (2018) Capsnet comparative performance evaluation for image classification. arXiv preprint arXiv:1805.11195

  14. Netzer Y, Wang T, Coates A, Bissacco A, Wu B, Ng AY (2011) Reading digits in natural images with unsupervised feature learning. In: NIPS workshop deep learning unsupervised feature learning

  15. Paik I, Kwak T, Kim I (2019) Capsule networks need an improved routing algorithm. In: Proceedings of the Asian conference on machine learning, vol 101, pp 489–502

  16. Peer D, Stabinger S, Rodriguez-Sanchez A (2019) Limitations of routing-by-agreement based capsule networks. arXiv preprint arXiv:1905.08744

  17. Rajasegaran J, Jayasundara V, Jayasekara S, Jayasekara H, Seneviratne S, Rodrigo R (2019) Deepcaps: going deeper with capsule networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10725–10733. https://doi.org/10.1109/CVPR.2019.01098

  18. Ren H, Lu H (2018) Compositional coding capsule network with k-means routing for text classification. arXiv preprint arXiv:1810.09177

  19. Ren Q, Shang S, He L (2019) Adaptive routing between capsules. arXiv preprint arXiv:1911.08119

  20. Rosario VMd, Borin E, Breternitz M (2019) The multi-lane capsule network. IEEE Signal Process Lett 26(7):1006–1010. https://doi.org/10.1109/LSP.2019.2915661

    Article  Google Scholar 

  21. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. In: Advances in neural information processing systems, vol 30, pp 3856–3866

  22. Sun K, Zhao Y, Jiang B, Cheng T, Xiao B, Liu D, Mu Y, Wang X, Liu W, Wang J (2019) High-resolution representations for labeling pixels and regions. arXiv preprint arXiv:1904.04514

  23. Wu Y, Li J, Wu J, Chang J (2020) Siamese capsule networks with global and local features for text classification. Neurocomputing 390:88–98. https://doi.org/10.1016/j.neucom.2020.01.064

    Article  Google Scholar 

  24. Xi E, Bing S, Jin Y (2017) Capsule network performance on complex data. arXiv preprint arXiv:1712.03480

  25. Xiang C, Zhang L, Tang Y, Zou W, Xu C (2018) Ms-capsnet: a novel multi-scale capsule network. IEEE Signal Process Lett 25(12):1850–1854. https://doi.org/10.1109/LSP.2018.2873892

    Article  Google Scholar 

  26. Zhao Z, Kleinhans A, Sandhu G, Patel I, Unnikrishnan KP (2019) Capsule networks with max-min normalization. arXiv preprint arXiv:1903.09662

Download references

Funding

The work was supported by the National Natural Science Foundation of China under Grant 61472278, and Major project of Tianjin under Grant 18ZXZNGX00150, and the Key Project of Natural Science Foundation of Tianjin University under Grant 2017ZD13, and the Research Project of Tianjin Municipal Education Commission under Grant 2017KJ255.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, 300384, China

    Kun Sun, Haixia Xu, Liming Yuan & Xianbin Wen

  2. Key Laboratory of Computer Vision and System, Ministry of Education, Tianjin, 300384, China

    Kun Sun, Haixia Xu, Liming Yuan & Xianbin Wen

  3. Tianjin Key Laboratory of Intelligence Computing and Novel Software Technology, Tianjin, 300384, China

    Kun Sun, Haixia Xu, Liming Yuan & Xianbin Wen

Authors
  1. Kun Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haixia Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liming Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xianbin Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xianbin Wen.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, K., Xu, H., Yuan, L. et al. Multiple Regularization and Analysis of Deep Capsule Network. Pattern Anal Applic 25, 711–729 (2022). https://doi.org/10.1007/s10044-022-01070-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-022-01070-7

Keywords

Search

Navigation