Skip to main content
SpringerLink
Log in
Book cover

European Conference on Computer Vision

ECCV 2022: Computer Vision – ECCV 2022 Workshops pp 219–235Cite as

Self-attention Capsule Network for Tissue Classification in Case of Challenging Medical Image Statistics

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13803))

Abstract

We propose the first Self-Attention Capsule Network that was designed to deal with unique core challenges of medical imaging, specifically for tissue classification. These challenges are - significant data heterogeneity with statistics variability across imaging domains, insufficient spatial context and local fine-grained details, and limited training data. Moreover, our proposed method solves limitations of the baseline Capsule Networks (CapsNet) such as handling complicated challenging data and limited computational resources. To cope with these challenges, our method is composed of a self-attention module that simplifies the complexity of the input data such that the CapsNet routing mechanism can be efficiently used, while extracting much richer contextual information, compared with CNNs. To demonstrate the strengths of our method, it was extensively evaluated on three diverse medical datasets and three natural benchmarks. The proposed method outperformed other methods we compared with in classification accuracy but also in robustness, within and across different datasets and domains.

B. Wilcox and Y. Gupta—Equal Contributors.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Aminian, M., Khotanlou, H.: Capsnet-based brain tumor segmentation in multimodal MRI images using inhomogeneous voxels in del vector domain. Multimedia Tools Appl. 81(13), 17793–17815 (2022)

    Article  Google Scholar 

  2. Aziz, M.J., et al.: Accurate automatic glioma segmentation in brain MRI images based on capsnet. In: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3882–3885. IEEE (2021)

    Google Scholar 

  3. Bilic, P., et al.: The liver tumor segmentation benchmark (lits). CoRR abs/1901.04056 (2019). http://arxiv.org/abs/1901.04056

  4. Choi, J., Seo, H., Im, S., Kang, M.: Attention routing between capsules. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (2019)

    Google Scholar 

  5. Duarte, K., et al.: Routing with self-attention for multimodal capsule networks (2021)

    Google Scholar 

  6. Elmezain, M., Mahmoud, A., Mosa, D.T., Said, W.: Brain tumor segmentation using deep capsule network and latent-dynamic conditional random fields. J. Imaging 8(7), 190 (2022)

    Article  Google Scholar 

  7. Fan, T., Wang, G., Li, Y., Wang, H.: Ma-net: a multi-scale attention network for liver and tumor segmentation. IEEE Access 8, 179656–179665 (2020)

    Article  Google Scholar 

  8. Fu, H., Wang, H., Yang, J.: Video summarization with a dual attention capsule network. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 446–451 (2021). https://doi.org/10.1109/ICPR48806.2021.9412057

  9. Hahn, T., Pyeon, M., Kim, G.: Self-routing capsule networks. In: NeurIPS (2019)

    Google Scholar 

  10. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  11. Huang, G., Liu, Z., Weinberger, K.Q.: Densely connected convolutional networks. CoRR abs/1608.06993 (2016). http://arxiv.org/abs/1608.06993

  12. Huang, W., Zhou, F.: Da-capsnet: dual attention mechanism capsule network. Sci. Rep. 10(1), 1–13 (2020)

    MathSciNet  Google Scholar 

  13. Jetley, S., Lord, N.A., Lee, N., Torr, P.H.S.: Learn to pay attention. CoRR abs/1804.02391 (2018). http://arxiv.org/abs/1804.02391

  14. Jiménez-Sánchez, A., Albarqouni, S., Mateus, D.: Capsule networks against medical imaging data challenges. In: CVII-STENT/LABELS@MICCAI (2018)

    Google Scholar 

  15. Jo, Y., et al.: Quantitative phase imaging and artificial intelligence: a review. IEEE J. Sel. Top. Quant. Electron. 25, 1–14 (2019)

    Article  Google Scholar 

  16. Kaul, C., Manandhar, S., Pears, N.: Focusnet: an attention-based fully convolutional network for medical image segmentation. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 455–458 (2019). https://doi.org/10.1109/ISBI.2019.8759477

  17. Larochelle, H., Hinton, G.E.: Learning to combine foveal glimpses with a third-order Boltzmann machine. In: Advances in Neural Information Processing Systems, vol. 23. pp. 1243–1251 (2010)

    Google Scholar 

  18. Mazzia, V., Salvetti, F., Chiaberge, M.: Efficient-capsnet: capsule network with self-attention routing. Sci. Rep. 11(1), 1–13 (2021)

    Article  Google Scholar 

  19. Mnih, V., Heess, N., Graves, A., Kavukcuoglu, K.: Recurrent models of visual attention. In: Advances in Neural Information Processing Systems, vol. 3 (2014)

    Google Scholar 

  20. Mobiny, A., Van Nguyen, H.: Fast CapsNet for lung cancer screening. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 741–749. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_82

    Chapter  Google Scholar 

  21. Oktay, O., et al.: Attention u-net: learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 (2018)

  22. Olshausen, B.A., Anderson, C.H., Essen, D.C.V.: A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. J. Neurosci. 13, 4700–4719 (1993)

    Article  Google Scholar 

  23. Paik, I., Kwak, T., Kim, I.: Capsule networks need an improved routing algorithm. In: Lee, W.S., Suzuki, T. (eds.) Proceedings of the Eleventh Asian Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 101, pp. 489–502. PMLR, Nagoya, Japan, 17–19 November 2019

    Google Scholar 

  24. Phaye, S.S.R., Sikka, A., Dhall, A., Bathula, D.: Dense and diverse capsule networks: making the capsules learn better (2018)

    Google Scholar 

  25. Pino, C., Vecchio, G., Fronda, M., Calandri, M., Aldinucci, M., Spampinato, C.: Twinlivernet: predicting TACE treatment outcome from CT scans for hepatocellular carcinoma using deep capsule networks. In: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3039–3043. IEEE (2021)

    Google Scholar 

  26. Qian, W., Jiaxing, Z., Sen, S., Zheng, Z.: Attentional neural network: feature selection using cognitive feedback. In: Advances in Neural Information Processing Systems, vol. 27. pp. 2033–2041 (2014)

    Google Scholar 

  27. Ranjbarzadeh, R., Bagherian Kasgari, A., Jafarzadeh Ghoushchi, S., Anari, S., Naseri, M., Bendechache, M.: Brain tumor segmentation based on deep learning and an attention mechanism using MRI multi-modalities brain images. Sci. Rep. 11(1), 1–17 (2021)

    Article  Google Scholar 

  28. Reichert, D.P., Seriès, P., Storkey, A.J.: A hierarchical generative model of recurrent object-based attention in the visual cortex. In: ICANN (2011)

    Google Scholar 

  29. Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. arXiv preprint arXiv:1710.09829 (2017)

  30. Shang, Y., Xu, N., Jin, Z., Yao, X.: Capsule network based on self-attention mechanism. In: 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP), pp. 1–4. IEEE (2021)

    Google Scholar 

  31. Sinha, A., Dolz, J.: Multi-scale self-guided attention for medical image segmentation. IEEE J. Biomed. Health Inform. 25(1), 121–130 (2021). https://doi.org/10.1109/JBHI.2020.2986926

    Article  Google Scholar 

  32. Survarachakan, S., Johansen, J.S., Pedersen, M.A., Amani, M., Lindseth, F.: Capsule nets for complex medical image segmentation tasks. In: CVCS (2020)

    Google Scholar 

  33. Tang, H., et al.: Spatial context-aware self-attention model for multi-organ segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 939–949 (2021)

    Google Scholar 

  34. Tang, Y., Srivastava, N., Salakhutdinov, R.: Learning generative models with visual attention. arXiv preprint arXiv:1312.6110 (2013)

  35. Tran, M., Ly, L., Hua, B.S., Le, N.: SS-3DCapsNet: self-supervised 3D capsule networks for medical segmentation on less labeled data. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–5. IEEE (2022)

    Google Scholar 

  36. Tsai, Y.H., Srivastava, N., Goh, H., Salakhutdinov, R.: Capsules with inverted dot-product attention routing. CoRR abs/2002.04764 (2020). https://arxiv.org/abs/2002.04764

  37. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. arXiv preprint arXiv:1711.07971 (2017)

  38. Xi, E., Bing, S., Jin, Y.: Capsule network performance on complex data. CoRR abs/1712.03480 (2017)

    Google Scholar 

  39. Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. arXiv preprint arXiv:1805.08318 (2018)

  40. Özbulak, G.: Image colorization by capsule networks (2019)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by grants from the National Cancer Institute, National Institutes of Health, U01CA142555 and 1U01CA190214.

Author information

Authors and Affiliations

  1. Ariel University, Ariel, Israel

    Assaf Hoogi

  2. Stanford University, Stanford, CA, USA

    Brian Wilcox, Yachee Gupta & Daniel Rubin

Authors
  1. Assaf Hoogi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian Wilcox
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yachee Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel Rubin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Assaf Hoogi .

Editor information

Editors and Affiliations

  1. IBM Research - MIT-IBM Watson AI Lab, Massachusetts, USA

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hoogi, A., Wilcox, B., Gupta, Y., Rubin, D. (2023). Self-attention Capsule Network for Tissue Classification in Case of Challenging Medical Image Statistics. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13803. Springer, Cham. https://doi.org/10.1007/978-3-031-25066-8_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25066-8_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25065-1

  • Online ISBN: 978-3-031-25066-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation