Abstract
Three-dimensional medical images are crucial for the early screening and prognosis of numerous diseases. However, constructing an accurate computer-aided prediction model is challenging when dealing with volumes of different sizes due to numerous slices (native nodes) in a single case and variable-length slice sequence. We propose a Global Semantic-guided Dual-stream Graph model to address this issue. Our approach differs from the existing solution that aligns volumes with varying numbers of slices through downsampling. Instead, we leverage global semantic vectors to guide the grouping of native nodes, construct super-nodes, and build dual-stream graphs by incorporating the sequential association of each volume’s unique slices and the feature association of global semantic vectors. Specifically, we propose a shared global semantic vectors-based grouping method that aligns the number and the semantic distribution of nodes among different volumes without discarding slices. Furthermore, we construct a dual-stream graph module that enables Graph Convolutional Networks (GCN) to make clinical predictions from computer tomography (CT) volumes through the natural sequence association between native nodes and, simultaneously, the latent feature association between semantic vectors. We provide interpretability by visualizing the distribution of native nodes within each group and weakly-supervised slice localization. The results demonstrate that our method outperforms previous work in diagnostic (96.74%, +2.81%) and prognostic accuracy (84.56%, +1.86%) while being more interpretable, making it a promising approach for medical image analysis scenarios with limited fine-grained annotation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Adv. Neural Inf. Process. Syst. 33, 9912–9924 (2020)
Cuturi, M.: Sinkhorn distances: lightspeed computation of optimal transport. In: Advances in Neural Information Processing Systems, vol. 26 (2013)
Dosovitskiy, A., et al.: An image is worth 16 \(\times \) 16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (2021). https://openreview.net/forum?id=YicbFdNTTy
Han, K., Wang, Y., Guo, J., Tang, Y., Wu, E.: Vision GNN: an image is worth graph of nodes. arXiv preprint arXiv:2206.00272 (2022)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Jang, E., Gu, S., Poole, B.: Categorical reparameterization with Gumbel-Softmax. arXiv preprint arXiv:1611.01144 (2016)
Liu, C., Cui, J., Gan, D., Yin, G.: Beyond COVID-19 diagnosis: prognosis with hierarchical graph representation learning. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12907, pp. 283–292. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87234-2_27
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=Bkg6RiCqY7
Maddison, C.J., Mnih, A., Teh, Y.W.: The concrete distribution: a continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 (2016)
Meng, Y., et al.: Bilateral adaptive graph convolutional network on CT based COVID-19 diagnosis with uncertainty-aware consensus-assisted multiple instance learning. Med. Image Anal. 84, 102722 (2023)
Niu, C., Wang, G.: Unsupervised contrastive learning based transformer for lung nodule detection. Phys. Med. Biol. 67(20), 204001 (2022)
Shang, C., Chen, J., Bi, J.: Discrete graph structure learning for forecasting multiple time series. In: International Conference on Learning Representations (2021). https://openreview.net/forum?id=WEHSlH5mOk
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Taleb, A., et al.: 3D self-supervised methods for medical imaging. Adv. Neural Inf. Process. Syst. 33, 18158–18172 (2020)
Tang, Y., et al.: Self-supervised pre-training of Swin transformers for 3D medical image analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20730–20740, June 2022
Vedaldi, A., Asano, Y., Rupprecht, C.: Self-labelling via simultaneous clustering and representation learning (2020)
Wang, X., Han, S., Chen, Y., Gao, D., Vasconcelos, N.: Volumetric attention for 3D medical image segmentation and detection. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 175–184. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32226-7_20
Yeung, P.-H., Namburete, A.I.L., Xie, W.: Sli2Vol: annotate a 3D volume from a single slice with self-supervised learning. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12902, pp. 69–79. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87196-3_7
Yuan, Z., Yan, Y., Sonka, M., Yang, T.: Large-scale robust deep AUC maximization: a new surrogate loss and empirical studies on medical image classification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3040–3049 (2021)
Zhang, K., et al.: Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography. Cell 181(6), 1423–1433 (2020)
Acknowledgments
I would like to thank my wife, Yang Feng, for her support during my doctoral studies.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, S., Guo, X., Zhu, J., Wang, Y. (2023). GSDG: Exploring a Global Semantic-Guided Dual-Stream Graph Model for Automated Volume Differential Diagnosis and Prognosis. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14224. Springer, Cham. https://doi.org/10.1007/978-3-031-43904-9_45
Download citation
DOI: https://doi.org/10.1007/978-3-031-43904-9_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43903-2
Online ISBN: 978-3-031-43904-9
eBook Packages: Computer ScienceComputer Science (R0)
