Abstract
The joint use of multiple imaging modalities for medical image has been widely studied in recent years. The fusion of information from different modalities has demonstrated the performance improvement for some medical tasks. For nephropathy diagnosis, immunofluorescence (IF) is one of the most widely-used medical image due to its ease of acquisition with low cost, which is also an advanced multi-modality technique. However, the existing methods mainly integrate information from diverse sources by averaging or combining them, failing to exploit multi-modality knowledge in details. In this paper, we observe that the 7 modalities of IF images have different impact on different nephropathy categories. Accordingly, we propose a knowledge distillation framework to transfer knowledge from the trained single-modality teacher networks to a multi-modality student network. On top of this, given a input IF sequence, a recruitment module is developed to dynamically assign weights to teacher models and optimize the performance of student model. By applying on several different architectures, the extensive experimental results verify the effectiveness of our method for nephropathy diagnosis.
This work was supported by NSFC under Grant 62250001, and Alibaba Innovative Reaserch.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cao, Y., Xu, J., Lin, S., Wei, F., Hu, H.: Gcnet: non-local networks meet squeeze-excitation networks and beyond. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (2019)
Gomariz, A., et al.: Modality attention and sampling enables deep learning with heterogeneous marker combinations in fluorescence microscopy. Nature Mach. Intell. 3(9), 799–811 (2021)
Hao, F., Liu, X., Li, M., Han, W.: Accurate kidney pathological image classification method based on deep learning and multi-modal fusion method with application to membranous nephropathy. Life 13(2), 399 (2023)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hinton, G., Vinyals, O., Dean, J., et al.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 2(7) (2015)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Kitamura, S., Takahashi, K., Sang, Y., Fukushima, K., Tsuji, K., Wada, J.: Deep learning could diagnose diabetic nephropathy with renal pathological immunofluorescent images. Diagnostics 10(7), 466 (2020)
Ligabue, G., et al.: Evaluation of the classification accuracy of the kidney biopsy direct immunofluorescence through convolutional neural networks. Clin. J. Am. Soc. Nephrol. 15(10), 1445–1454 (2020)
Liu, Y., Cao, J., Li, B., Hu, W., Maybank, S.: Learning to explore distillability and sparsability: a joint framework for model compression. IEEE Trans. Pattern Anal. Mach. Intell. (2022)
Miles, R., Lopez-Rodriguez, A., Mikolajczyk, K.: Information theoretic representation distillation
Romagnani, P., et al.: Chronic kidney disease. Nat. Rev. Dis. Primers. 3(1), 1–24 (2017)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)
Takahashi, K., Kitamura, S., Fukushima, K., Sang, Y., Tsuji, K., Wada, J.: The resolution of immunofluorescent pathological images affects diagnosis for not only artificial intelligence but also human. J. Nephropathol. 10(3), e26 (2021)
Tan, M., Le, Q.: Efficientnet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)
Wang, Q., Wu, B., Zhu, P.F., Li, P., Zuo, W., Hu, Q.: Eca-net: efficient channel attention for deep convolutional neural networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11531–11539 (2019)
Wang, R., et al.: Ada-ccfnet: classification of multimodal direct immunofluorescence images for membranous nephropathy via adaptive weighted confidence calibration fusion network. Eng. Appl. Artif. Intell. 117, 105637 (2023)
Yang, C., An, Z., Cai, L., Xu, Y.: Mutual contrastive learning for visual representation learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 3045–3053 (2022)
Zhang, L., et al.: Classification of renal biopsy direct immunofluorescence image using multiple attention convolutional neural network. Comput. Methods Programs Biomed. 214, 106532 (2022)
Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018)
Zhou, X., Laszik, Z., Silva, F.: Algorithmic approach to the interpretation of renal biopsy. In: Zhou, X.J., Laszik Z., Nadasdy, T., D’Agati, V.D., Silva, F.G. (eds.) Silva’s Diagnostic Reanal Pathology, pp. pp. 55–57. Cambridge University Press, New York (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Dai, N. et al. (2023). Recruiting the Best Teacher Modality: A Customized Knowledge Distillation Method for if Based Nephropathy Diagnosis. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14224. Springer, Cham. https://doi.org/10.1007/978-3-031-43904-9_51
Download citation
DOI: https://doi.org/10.1007/978-3-031-43904-9_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43903-2
Online ISBN: 978-3-031-43904-9
eBook Packages: Computer ScienceComputer Science (R0)
