Abstract
With the widespread application of deep learning in medical image analysis, its capacity to handle high-dimensional and complex medical images has been widely recognized. However, high-accuracy deep learning models typically demand considerable computational resources and time, while shallow models are generally unable to compete with complex model. To overcome these challenges, this paper introduces a Knowledge Distillation methodology that merges features and soft labels, transfering knowledge encapsulated in the intermediate features and predictions of the teacher model to the student model. The student model imitates the teacher model’s behavior, thereby improving its prediction accuracy. Based on this, we propose the Res-Transformer teacher model bases on the U-Net architecture and the ResU-Net student model incorporates Residuals. The Res-Transformer model employs dual Attention to acquire deep feature maps of the image, and subsequently employs Hierarchical Upsampling to restore the details in these feature maps. The ResU-Net model enhances stability via Residuals and recovers the loss of image information in convolution operations through optimized skip-connection. Finally, we evaluate on multiple disease datasets. The results show that the Res-Transformer achieves accuracy up to 94.3%. By applying knowledge distillation, abundant knowledge from the Res-Transformer is transferring knowledge of the Res-Transformer to the ResU-Net model, improving its accuracy up to 7.1%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
Shen, D., Wu, G., Suk, H.I.: Deep learning in medical image analysis. Ann. Rev. Biomed. Eng. 19, 221–248 (2017)
Yao, L., Mao, C., Luo, Y.: Clinical text classification with rule-based features and knowledge-guided convolutional neural networks. BMC Med. Inf. Decis. Mak. 19(3), 31–39 (2019)
Li, C.Y., Liang, X., Hu, Z., Xing, E.P.: Knowledge-driven encode, retrieve, paraphrase for medical image report generation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 6666–6673 (2019)
Tajbakhsh, N., Jeyaseelan, L., Li, Q., Chiang, J.N., Wu, Z., Ding, X.: Embracing imperfect datasets: a review of deep learning solutions for medical image segmentation. Med. Image Anal. 63, 101693 (2020)
Rajpurkar, P., et al.: Chexnet: radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017)
Wang, D., Khosla, A., Gargeya, R., Irshad, H., Beck, A.H.: Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:1606.05718 (2016)
Pathak, Y., Shukla, P.K., Tiwari, A., Stalin, S., Singh, S.: Deep transfer learning based classification model for covid-19 disease. IRBM 43(2), 87–92 (2022)
Rossini, N., Dassisti, M., Benyounis, K., Olabi, A.G.: Methods of measuring residual stresses in components. Mater. Des. 35, 572–588 (2012)
Vaswani, A., et al.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017)
Siddique, N., Paheding, S., Elkin, C.P., Devabhaktuni, V.: U-net and its variants for medical image segmentation: a review of theory and applications. IEEE Access 9, 82031–82057 (2021)
Du, G., Cao, X., Liang, J., Chen, X., Zhan, Y.: Medical image segmentation based on u-net: a review. J. Imaging Sci. Technol. (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Rahman, T., et al.: Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images. Comput. Biol. Med. 132, 104319 (2021)
Spanhol, F.A., Oliveira, L.S., Petitjean, C., Heutte, L.: A dataset for breast cancer histopathological image classification. IEEE Trans. Biomed. Eng. 63(7), 1455–1462 (2015)
Cruz-Roa, A., et al.: Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks. In: Medical Imaging 2014: Digital Pathology, vol. 9041, p. 904103. SPIE (2014)
Liu, M., et al.: A deep learning method for breast cancer classification in the pathology images. IEEE J. Biomed. Health Inf. 26(10), 5025–5032 (2022)
Fan, Y., Gong, H.: An improved tensor network for image classification in histopathology. In: Pattern Recognition and Computer Vision: 5th Chinese Conference, PRCV 2022, Shenzhen, China, 4–7 November 2022, Proceedings, Part II, pp. 126–137. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-18910-4_11
Li, X., Shen, X., Zhou, Y., Wang, X., Li, T.Q.: Classification of breast cancer histopathological images using interleaved densenet with senet (idsnet). PLoS ONE 15(5), e0232127 (2020)
Acknowledgement
This work was supported by National Natural Science Foundation of China, Regional Science Fund Project, No: 72264037.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liao, Z., Dong, Q., Ge, Y., Liu, W., Chen, H., Song, Y. (2024). Knowledge Distillation of Attention and Residual U-Net: Transfer from Deep to Shallow Models for Medical Image Classification. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14437. Springer, Singapore. https://doi.org/10.1007/978-981-99-8558-6_14
Download citation
DOI: https://doi.org/10.1007/978-981-99-8558-6_14
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8557-9
Online ISBN: 978-981-99-8558-6
eBook Packages: Computer ScienceComputer Science (R0)