Skip to main content
SpringerLink
Log in

Knowledge Distillation of Vision Transformers and Convolutional Networks to Predict Inflammatory Bowel Disease

  • Conference paper
  • First Online:
Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications (CIARP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14469))

Included in the following conference series:

Abstract

Inflammatory bowel disease is a chronic disease of unknown cause that can affect the entire gastrointestinal tract, from the mouth to the anus. It is important for patients with this pathology that a good diagnosis is made as early as possible, so that the inflammation present in the mucosa intestinal is controlled and the most severe symptoms are reduced, thus offering the quality of life to people. Therefore, through this comparative study, we seek to find a way of automating the diagnosis of these patients during the endoscopic examination, reducing the subjectivity that is subject to the observation of a gastroenterologist, using six CNNs: AlexNet, ResNet50, VGG16, ResNet50-MobileNetV2 and Hybrid model. Also, five ViTs were used in this study: ViT-B/32, ViT-S/32, ViT-B/16, ViT-S/16 and R26+S/32. This comparison also consists in applying knowledge distillation to build simpler models, with fewer parameters, based on the learning of the pre-trained architectures on large volumes of data. It is concluded that in the ViTs framework, it is possible to reduce 25x the number of parameters by maintaining good performance and reducing the inference time by 5.32 s. For CNNs the results show that it is possible to reduce 107x the number of parameters, reducing consequently the inference time in 3.84 s.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amorim, J.P., Domingues, I., Abreu, P.H., Santos, J.A.: Interpreting deep learning models for ordinal problems. In: European Symposium on Artificial Neural Networks (ESANN), pp. 373–378 (2018)

    Google Scholar 

  2. Borgli, H., Riegler, M., Thambawita, V., Jha, D., Hicks, S., Halvorsen, P.: The HyperKvasir Dataset. OSF (2019)

    Google Scholar 

  3. Borgli, H., et al.: HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Sci. Data 7, 283 (2020)

    Article  Google Scholar 

  4. Cardoso, J.S., Sousa, R., Domingues, I.: Ordinal data classification using kernel discriminant analysis: A comparison of three approaches. In: 11th International Conference on Machine Learning and Applications, vol. 1, pp. 473–477 (2012)

    Google Scholar 

  5. Chicco, D., Jurman, G.: The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genom. 21, 6 (2020)

    Article  Google Scholar 

  6. Chierici, M., et al.: Automatically detecting Crohn’s disease and Ulcerative Colitis from endoscopic imaging. BMC Med. Inform. Decis. Mak. 22, 300 (2022)

    Article  Google Scholar 

  7. CrohnIPI. https://crohnipi.ls2n.fr/en/crohn-ipi-project/ (Accessed 21 Feb 2023)

  8. Doença inflamatória do intestino | CUF. https://www.cuf.pt/saude-a-z/doenca-inflamatoria-do-intestino (Accessed 2 Nov 2022)

  9. Das, D., Massa, H., Kulkarni, A., Rekatsinas, T.: An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation (2020), arXiv Version: 2

    Google Scholar 

  10. Dosovitskiy, A., et al.: An image is worth 16x16 Words: transformers for image recognition at scale. In: International Conference on Learning Representations (2020)

    Google Scholar 

  11. Flynn, S., Eisenstein, S.: Inflammatory bowel disease presentation and diagnosis. Surg. Clin. North Am. 99(6), 1051–1062 (2019)

    Article  Google Scholar 

  12. Galdran, A., Carneiro, G., Ballester, M.A.G.: Convolutional Nets Versus Vision Transformers for Diabetic Foot Ulcer Classification. Diabetic Foot Ulcers Grand Challenge (2022)

    Google Scholar 

  13. Gamage, C., Wijesinghe, I., Chitraranjan, C., Perera, I.: GI-Net: anomalies classification in gastrointestinal tract through endoscopic imagery with deep learning. In: Moratuwa Engineering Research Conference (MERCon), pp. 66–71. IEEE, Moratuwa, Sri Lanka (Jul 2019)

    Google Scholar 

  14. Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: a survey. Int. J. Comput. Vision 129(6), 1789–1819 (2021)

    Article  Google Scholar 

  15. H. Kassani, S., Hosseinzadeh Kassani, P., Wesolowski, M., Schneider, K., Deters, R.: Classification of histopathological biopsy images using ensemble of deep learning networks. arXiv preprint (2019)

    Google Scholar 

  16. Khan, M.N., Hasan, M.A., Anwar, S.: Improving the robustness of object detection through a multi-camera-based fusion algorithm using fuzzy logic. Front. Artifi. Intell. 4, 638951 (2021)

    Article  Google Scholar 

  17. Li, W., Shao, S., Liu, W., Qiu, Z., Zhu, Z., Huan, W.: What role does data augmentation play in knowledge distillation? In: Computer Vision - ACCV 2022, LNCS. vol. 13842, pp. 507–525. Springer Nature Switzerland (2023). https://doi.org/10.1007/978-3-031-26284-5_31

  18. Maissin, A., et al.: Multi-expert annotation of Crohn’s disease images of the small bowel for automatic detection using a convolutional recurrent attention neural network. Endoscopy Int Open 09, E1136–E1144 (2021)

    Article  Google Scholar 

  19. Marques, F., Duarte, H., Santos, J.A., Domingues, I., Amorim, J.P., Abreu, P.H.: An iterative oversampling approach for ordinal classification. In: 34th ACM/SIGAPP Symposium on Applied Computing, pp. 771–774 (2019)

    Google Scholar 

  20. Maurício, J., Domingues, I.: Deep Neural Networks to distinguish between Crohn’s disease and Ulcerative colitis. In: 11th Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA) (2023)

    Google Scholar 

  21. Maurício, J., Domingues, I.: Interpretability of deep neural networks to diagnose inflammatory bowel disease. In: 29th Edition of the Portuguese Conference on Pattern Recognition (2023) (to appear)

    Google Scholar 

  22. Maurício, J., Domingues, I., Bernardino, J.: Comparing vision transformers and convolutional neural networks for image classification: a literature review. Appli. Sci. 13(9) (2023)

    Google Scholar 

  23. Polat, G., Kani, H.T., Ergenc, I., Alahdab, Y.O., Temizel, A., Atug, O.: Labeled Images for Ulcerative Colitis (LIMUC) Dataset (2022)

    Google Scholar 

  24. Sairenji, T., Collins, K.L., Evans, D.V.: An update on inflammatory bowel disease. Primary Care: Clin. Office Pract. 44, 673–692 (2017)

    Article  Google Scholar 

  25. Seyedian, S.S., Nokhostin, F., Malamir, M.D.: A review of the diagnosis, prevention, and treatment methods of inflammatory bowel disease. J. Med. Life 12, 113–122 (2019)

    Article  Google Scholar 

  26. Shahzadi, I., Tang, T.B., Meriadeau, F., Quyyum, A.: CNN-LSTM: cascaded framework for brain tumour classification. In: IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), pp. 633–637 (2018)

    Google Scholar 

  27. Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., Beyer, L.: How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers. arXiv Version: 2 (2021)

    Google Scholar 

  28. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attentio (2020), arXiv Version: 2

    Google Scholar 

  29. Turan, M., Durmus, F.: UC-NfNet: deep learning-enabled assessment of ulcerative colitis from colonoscopy images. Med. Image Anal. 82, 102587 (2022)

    Article  Google Scholar 

  30. Udristoiu, A.L., et al.: Deep learning algorithm for the confirmation of mucosal healing in crohn’s disease, based on confocal laser endomicroscopy images. J. Gastrointestinal Liver Dis. 30, 59–65 (2021)

    Article  Google Scholar 

  31. Vallée, R., Coutrot, A., Normand, N., Mouchère, H.: Accurate small bowel lesions detection in wireless capsule endoscopy images using deep recurrent attention neural network. In: IEEE 21st Int WS on Multimedia Signal Proc (MMSP) (2019)

    Google Scholar 

  32. Vallée, R., Coutrot, A., Normand, N., Mouchère, H.: Influence of expertise on human and machine visual attention in a medical image classification task. In: European Conference on Visual Perception (2021)

    Google Scholar 

  33. Vallée, R., Maissin, A., Coutrot, A., Mouchère, H., Bourreille, A., Normand, N.: CrohnIPI: an endoscopic image database for the evaluation of automatic Crohn’s disease lesions recognition algorithms. In: Medical Imaging: Biomedical Applications in Molecular, Structural, and Functional Imaging, p. 61. SPIE (2020)

    Google Scholar 

  34. Vankdothu, R., Hameed, M.A., Fatima, H.: A brain tumor identification and classification using deep learning based on CNN-LSTM method. Comput. Electr. Eng. 101, 107960 (2022)

    Article  Google Scholar 

  35. Wang, L., et al.: Development of a convolutional neural network-based colonoscopy image assessment model for differentiating crohn’s disease and ulcerative colitis. Front. Med. 9, 789862 (2022)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Instituto Politécnico de Coimbra, Instituto Superior de Engenharia, Rua Pedro Nunes - Quinta da Nora, 3030-199, Coimbra, Portugal

    José Maurício & Inês Domingues

  2. Centro de Investigação do Instituto Português de Oncologia do Porto (CI-IPOP): Grupo de Física Médica, Radiobiologia e Protecção Radiológica, Coimbra, Portugal

    Inês Domingues

Authors
  1. José Maurício
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Inês Domingues
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to José Maurício .

Editor information

Editors and Affiliations

  1. Polytechnic Institute of Coimbra, Coimbra Institute of Engineering, Coimbra, Portugal

    Verónica Vasconcelos

  2. Polytechnic Institute of Coimbra, Coimbra Institute of Engineering, Coimbra, Portugal

    Inês Domingues

  3. Polytechnic Institute of Coimbra, Coimbra Institute of Engineering, Coimbra, Portugal

    Simão Paredes

Rights and permissions

Reprints and permissions

Copyright information

© 2024 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Maurício, J., Domingues, I. (2024). Knowledge Distillation of Vision Transformers and Convolutional Networks to Predict Inflammatory Bowel Disease. In: Vasconcelos, V., Domingues, I., Paredes, S. (eds) Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications. CIARP 2023. Lecture Notes in Computer Science, vol 14469. Springer, Cham. https://doi.org/10.1007/978-3-031-49018-7_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-49018-7_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-49017-0

  • Online ISBN: 978-3-031-49018-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation