Abstract
Magnetic resonance imaging (MRI) has become necessary in clinical diagnosis for knee osteoarthritis (OA), while deep neural networks can contribute to the computer-assisted diagnosis. Recent works prove that instead of only using a single-view MR image (e.g., sagittal), integrating multi-view MR images can boost the performance of the deep network. However, existing multi-view networks typically encode each MRI view to a feature vector, fuse the feature vectors of all views, and then derive the final output using a set of shallow computations. Such a global fusion scheme happens at a coarse granularity, which may not effectively localize the often tiny abnormality related to the onset of OA. Therefore, this paper proposes a Local Graph Fusion Network (LGF-Net), which implements graph-based representation of knee MR images and multi-view fusion for OA diagnosis. We first model the multi-view MR images to a unified knee graph. Then, the patches of the same location yet from different views are encoded to one-dimensional features and are exchanged mutually during fusing. The local fusion of the features further propagates following edges by Graph Transformer Network in the LGF-Net, which finally yields the grade of OA. The experimental results show that the proposed framework outperforms state-of-the-art methods, demonstrating the effectiveness of local graph fusion in OA diagnosis.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Alizai, H., et al.: Cartilage lesion score: comparison of a quantitative assessment score with established semiquantitative MR scoring systems. Radiology 271(2), 479–487 (2014)
Astuto, B., et al.: Automatic deep learning-assisted detection and grading of abnormalities in knee MRI studies. Radiol. Artif. Intell. 3(3), e200165 (2021)
Azcona, D., McGuinness, K., Smeaton, A.F.: A comparative study of existing and new deep learning methods for detecting knee injuries using the MRNet dataset. In: 2020 International Conference on Intelligent Data Science Technologies and Applications (IDSTA), pp. 149–155. IEEE (2020)
Belton, N., et al.: Optimising knee injury detection with spatial attention and validating localisation ability. In: Papież, B.W., Yaqub, M., Jiao, J., Namburete, A.I.L., Noble, J.A. (eds.) MIUA 2021. LNCS, vol. 12722, pp. 71–86. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-80432-9_6
Bien, N., et al.: Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of mrnet. PLoS Med. 15(11), e1002699 (2018)
Calivà, F., Namiri, N.K., Dubreuil, M., Pedoia, V., Ozhinsky, E., Majumdar, S.: Studying osteoarthritis with artificial intelligence applied to magnetic resonance imaging. Nat. Rev. Rheumatol. 18, 1–10 (2021)
Chen, S., Ma, K., Zheng, Y.: Med3D: transfer learning for 3D medical image analysis. arXiv preprint arXiv:1904.00625 (2019)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (2020)
Garwood, E.R., Recht, M.P., White, L.M.: Advanced imaging techniques in the knee: benefits and limitations of new rapid acquisition strategies for routine knee MRI. Am. J. Roentgenol. 209(3), 552–560 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Huo, J., et al.: Automatic grading assessments for knee MRI cartilage defects via self-ensembling semi-supervised learning with dual-consistency. Med. Image Anal. 80, 102508 (2022)
Huo, J., et al.: A self-ensembling framework for semi-supervised knee cartilage defects assessment with dual-consistency. In: Rekik, I., Adeli, E., Park, S.H., Valdés Hernández, M.C. (eds.) PRIME 2020. LNCS, vol. 12329, pp. 200–209. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59354-4_19
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18(2), 203–211 (2021)
Jamshidi, A., Pelletier, J.P., Martel-Pelletier, J.: Machine-learning-based patient-specific prediction models for knee osteoarthritis. Nat. Rev. Rheumatol. 15(1), 49–60 (2019)
Liu, F., et al.: Fully automated diagnosis of anterior cruciate ligament tears on knee MR images by using deep learning. Radiol. Artif. Intell. 1(3), 180091 (2019)
Liu, F., et al.: Deep learning approach for evaluating knee MR images: achieving high diagnostic performance for cartilage lesion detection. Radiology 289(1), 160–169 (2018)
Nikolas, W., Jan, L., Carina, M., von Eisenhart-Rothe, R., Rainer, B.: Maintaining the spatial relation to improve deep-learning-assisted diagnosis for magnetic resonace imaging of the knee. Zeitschrift für Orthopädie und Unfallchirurgie 158(S 01), DKOU20-670 (2020)
Peterfy, C., Gold, G., Eckstein, F., Cicuttini, F., Dardzinski, B., Stevens, R.: MRI protocols for whole-organ assessment of the knee in osteoarthritis. Osteoarthr. Cartil. 14, 95–111 (2006)
Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE Trans. Neural Networks 20(1), 61–80 (2008)
Si, L., et al.: Knee cartilage thickness differs alongside ages: a 3-T magnetic resonance research upon 2,481 subjects via deep learning. Front. Med. 7, 1157 (2021)
Suzuki, T., Hosseini, A., Li, J.S., Gill, T.J., IV., Li, G.: In vivo patellar tracking and patellofemoral cartilage contacts during dynamic stair ascending. J. Biomech. 45(14), 2432–2437 (2012)
Tsai, C.H., Kiryati, N., Konen, E., Eshed, I., Mayer, A.: Knee injury detection using MRI with efficiently-layered network (ELNet). In: Medical Imaging with Deep Learning, pp. 784–794. PMLR (2020)
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. In: International Conference on Learning Representations (ICLR), pp. 1–12 (2017)
Zhuang, Z., et al.: Knee cartilage defect assessment by graph representation and surface convolution. arXiv preprint arXiv:2201.04318 (2022)
Acknowledgement
This work was supported by the National Key Research and Development Program of China (2018YFC0116400), National Natural Science Foundation of China (NSFC) grants (62001292) and Interdisciplinary Program of Shanghai Jiao Tong University (YG2019QNA17).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhuang, Z. et al. (2022). Local Graph Fusion of Multi-view MR Images for Knee Osteoarthritis Diagnosis. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13433. Springer, Cham. https://doi.org/10.1007/978-3-031-16437-8_53
Download citation
DOI: https://doi.org/10.1007/978-3-031-16437-8_53
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16436-1
Online ISBN: 978-3-031-16437-8
eBook Packages: Computer ScienceComputer Science (R0)
