Skip to main content
SpringerLink
Log in

Multi-View Coupled Self-Attention Network for Pulmonary Nodules Classification

  • Conference paper
  • First Online:
Computer Vision – ACCV 2022 (ACCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13846))

Included in the following conference series:

  • 356 Accesses

Abstract

Evaluation of the malignant degree of pulmonary nodules plays an important role in early detecting lung cancer. Deep learning-based methods have obtained promising results in this domain with their effectiveness in learning feature representation. Both local and global features are crucial for medical image classification tasks, particularly for 3D medical image data, however, the receptive field of the convolution kernel limits the global feature learning. Although self-attention mechanism can successfully model long-range dependencies by directly flattening the input image to a sequence, which has high computational complexity. Additionally, which unable to model the image local context information across spatial and depth dimensions. To address the above challenges, in this paper, we carefully design a Multi-View Coupled Self-Attention Module (MVCS). Specifically, a novel self-attention module is proposed to model spatial and dimensional correlations sequentially for learning global spatial contexts and further improving the identification accuracy. Compared with vanilla self-attention, which has three-fold advances: 1) uses less memory consumption and computational complexity than the existing self-attention methods; 2) except for exploiting the correlations along the spatial and channel dimension, the dimension correlations are also exploited; 3) the proposed self-attention module can be easily integrated with other frameworks. By adding the proposed module into 3D ResNet, we build a classification network for lung nodules’ malignancy evaluation. The nodule classification network was validated on a public dataset from LIDC-IDRI. Extensive experimental results demonstrate that our proposed model outperforms state-of-the-art approaches. The source code of this work is available at the https://github.com/ahukui/MVCS.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Al-Shabi, M., Lan, B.L., Chan, W.Y., Ng, K.H., Tan, M.: Lung nodule classification using deep local-global networks. Int. J. Comput. Assist. Radiol. Surg. 14(10), 1815–1819 (2019)

    Article  Google Scholar 

  2. Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R.L., Torre, L.A., Jemal, A.: Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: Cancer J. Clinic. 68(6), 394–424 (2018)

    Google Scholar 

  3. Dong, B., Wang, W., Fan, D.P., Li, J., Fu, H., Shao, L.: Polyp-PVT: polyp segmentation with pyramid vision transformers. arXiv preprint arXiv:2108.06932 (2021)

  4. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  5. Du, Y., Yuan, C., Li, B., Zhao, L., Li, Y., Hu, W.: Interaction-aware spatio-temporal pyramid attention networks for action classification. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11220, pp. 388–404. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01270-0_23

    Chapter  Google Scholar 

  6. Fang, W., Han, X.H.: Spatial and channel attention modulated network for medical image segmentation. In: Proceedings of the Asian Conference on Computer Vision (2020)

    Google Scholar 

  7. Guo, X., Guo, X., Lu, Y.: SSAN: separable self-attention network for video representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12618–12627 (2021)

    Google Scholar 

  8. Hussein, S., Cao, K., Song, Q., Bagci, U.: Risk stratification of lung nodules using 3D CNN-based multi-task learning. In: Niethammer, M., et al. (eds.) IPMI 2017. LNCS, vol. 10265, pp. 249–260. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59050-9_20

    Chapter  Google Scholar 

  9. Jiang, H., Gao, F., Xu, X., Huang, F., Zhu, S.: Attentive and ensemble 3D dual path networks for pulmonary nodules classification. Neurocomputing 398, 422–430 (2020)

    Article  Google Scholar 

  10. Jiang, H., Shen, F., Gao, F., Han, W.: Learning efficient, explainable and discriminative representations for pulmonary nodules classification. Pattern Recogn. 113, 107825 (2021)

    Article  Google Scholar 

  11. Kumar, D., Wong, A., Clausi, D.A.: Lung nodule classification using deep features in CT images. In: 2015 12th Conference on Computer and Robot Vision, pp. 133–138. IEEE (2015)

    Google Scholar 

  12. Li, Y., Iwamoto, Y., Lin, L., Chen, Y.W.: Parallel-connected residual channel attention network for remote sensing image super-resolution. In: Proceedings of the Asian Conference on Computer Vision (2020)

    Google Scholar 

  13. Li, Z., Yuan, L., Xu, H., Cheng, R., Wen, X.: Deep multi-instance learning with induced self-attention for medical image classification. In: 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 446–450. IEEE (2020)

    Google Scholar 

  14. Lyu, J., Ling, S.H.: Using multi-level convolutional neural network for classification of lung nodules on CT images. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 686–689. IEEE (2018)

    Google Scholar 

  15. Murugesan, M., Kaliannan, K., Balraj, S., Singaram, K., Kaliannan, T., Albert, J.R.: A hybrid deep learning model for effective segmentation and classification of lung nodules from CT images. J. Intell. Fuzzy Syst. (Preprint), 1–13 (2022)

    Google Scholar 

  16. Shen, S., Han, S.X., Aberle, D.R., Bui, A.A., Hsu, W.: An interpretable deep hierarchical semantic convolutional neural network for lung nodule malignancy classification. Expert Syst. Appl. 128, 84–95 (2019)

    Article  Google Scholar 

  17. Shen, W., et al.: Learning from experts: developing transferable deep features for patient-level lung cancer prediction. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 124–131. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_15

    Chapter  Google Scholar 

  18. Shen, W., Zhou, M., Yang, F., Yang, C., Tian, J.: Multi-scale convolutional neural networks for lung nodule classification. In: Ourselin, S., Alexander, D.C., Westin, C.-F., Cardoso, M.J. (eds.) IPMI 2015. LNCS, vol. 9123, pp. 588–599. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19992-4_46

    Chapter  Google Scholar 

  19. Shen, W., et al.: Multi-crop convolutional neural networks for lung nodule malignancy suspiciousness classification. Pattern Recogn. 61, 663–673 (2017)

    Article  Google Scholar 

  20. Shi, F., et al.: Semi-supervised deep transfer learning for benign-malignant diagnosis of pulmonary nodules in chest CT images. IEEE Trans. Med. Imaging (2021). https://doi.org/10.1109/TMI.2021.3123572

    Article  Google Scholar 

  21. Shi, F., et al.: Semi-supervised deep transfer learning for benign-malignant diagnosis of pulmonary nodules in chest CT images. IEEE Trans. Med. Imaging (2021)

    Google Scholar 

  22. Shi, L., Zhang, Y., Cheng, J., Lu, H.: Decoupled spatial-temporal attention network for skeleton-based action recognition. In: Proceedings of the Asian Conference on Computer Vision (2020)

    Google Scholar 

  23. Wang, W., et al.: Attention-based fine-grained classification of bone marrow cells. In: Proceedings of the Asian Conference on Computer Vision (2020)

    Google Scholar 

  24. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)

    Google Scholar 

  25. Wang, Z., Zhang, J., Zhang, X., Chen, P., Wang, B.: Transformer model for functional near-infrared spectroscopy classification. IEEE J. Biomed. Health Inform. 26(6), 2559–2569 (2022). https://doi.org/10.1109/JBHI.2022.3140531

    Article  Google Scholar 

  26. Xie, Y., et al.: Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest CT. IEEE Trans. Med. Imaging 38(4), 991–1004 (2018)

    Article  Google Scholar 

  27. Xie, Y., Zhang, J., Xia, Y., Fulham, M., Zhang, Y.: Fusing texture, shape and deep model-learned information at decision level for automated classification of lung nodules on chest CT. Inf. Fusion 42, 102–110 (2018)

    Article  Google Scholar 

  28. Xu, X., et al.: MSCS-deepLN: evaluating lung nodule malignancy using multi-scale cost-sensitive neural networks. Med. Image Anal. 65, 101772 (2020)

    Article  Google Scholar 

  29. Yan, X., et al.: Classification of lung nodule malignancy risk on computed tomography images using convolutional neural network: a comparison between 2D and 3D strategies. In: Chen, C.-S., Lu, J., Ma, K.-K. (eds.) ACCV 2016. LNCS, vol. 10118, pp. 91–101. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54526-4_7

    Chapter  Google Scholar 

  30. Zhang, J., Xie, Y., Xia, Y., Shen, C.: Attention residual learning for skin lesion classification. IEEE Trans. Med. Imaging 38(9), 2092–2103 (2019)

    Article  Google Scholar 

  31. Zhang, Y., Liu, H., Hu, Q.: TransFuse: fusing transformers and CNNs for medical image segmentation. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12901, pp. 14–24. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87193-2_2

    Chapter  Google Scholar 

  32. Zhu, Q., Du, B., Yan, P.: Boundary-weighted domain adaptive neural network for prostate MR image segmentation. IEEE Trans. Med. Imaging 39(3), 753–763 (2019)

    Article  Google Scholar 

  33. Zhu, Q., Du, B., Yan, P.: Self-supervised training of graph convolutional networks. arXiv preprint arXiv:2006.02380 (2020)

  34. Zhu, Q., Wang, Y., Du, B., Yan, P.: Oasis: one-pass aligned atlas set for medical image segmentation. Neurocomputing 470, 130–138 (2022)

    Article  Google Scholar 

  35. Zhu, W., Liu, C., Fan, W., Xie, X.: DeepLung: deep 3D dual path nets for automated pulmonary nodule detection and classification. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 673–681. IEEE (2018)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (81872510); Guangdong Provincial People’s Hospital Young Talent Project (GDPPHYTP201902); High-level Hospital Construction Project (DFJH201801); GDPH Scientific Research Funds for Leading Medical Talents and Distinguished Young Scholars in Guangdong Province (No. KJ012019449); Guangdong Basic and Applied Basic Research Foundation (No. 2019B1515130002).

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA

    Qikui Zhu

  2. Department of Gynecology, Renmin Hospital of Wuhan University, Wuhan, China

    Yanqing Wang

  3. School of Medicine, South China University of Technology, Guangzhou, China

    Xiongwen Yang

  4. Guangdong Provincial Key Laboratory of Translational Medicine in Lung Cancer, Guangdong Lung Cancer Institute, Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China

    Xiangpeng Chu, Xiongwen Yang & Wenzhao Zhong

Authors
  1. Qikui Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanqing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiangpeng Chu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiongwen Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wenzhao Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanqing Wang .

Editor information

Editors and Affiliations

  1. University of Wollongong, Wollongong, NSW, Australia

    Lei Wang

  2. University of Bonn, Bonn, Germany

    Juergen Gall

  3. University of Adelaide, Adelaide, SA, Australia

    Tat-Jun Chin

  4. National Institute of Informatics, Tokyo, Japan

    Imari Sato

  5. Johns Hopkins University, Baltimore, MD, USA

    Rama Chellappa

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhu, Q., Wang, Y., Chu, X., Yang, X., Zhong, W. (2023). Multi-View Coupled Self-Attention Network for Pulmonary Nodules Classification. In: Wang, L., Gall, J., Chin, TJ., Sato, I., Chellappa, R. (eds) Computer Vision – ACCV 2022. ACCV 2022. Lecture Notes in Computer Science, vol 13846. Springer, Cham. https://doi.org/10.1007/978-3-031-26351-4_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-26351-4_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-26350-7

  • Online ISBN: 978-3-031-26351-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation