Abstract
Ulcerative colitis (UC) is a chronic inflammatory disease of the large bowel characterised by quisent periodes and relapses. Endoscopic grading of the severity of UC is done by using a widely accepted scoring system known as the “Mayo Endoscopic Scoring” (MES). The MES score is largely based on the recognition of phenotypic features of the mucosal wall, and thus the subjectivity in clinical scoring is unavoidable. An automated grading and characterisation can certainly help to minimise the inter-observer variability and help trainees to get useful insights. For the first time, we a system capable of not only providing an automated MES scoring system, but also of generating a description of visible MES phenotypic mucosal representations in these endoscopic images through captions. Our aim is to combine the visual features together with word sequence embeddings that are learnt jointly through a recurrent neural network to predict such scene descriptions. In this work, we explore various recurrent neural network architectures together with other backbone architectures for visual feature representations. Our experiments on held-out test samples demonstrate high similarity between the reference and the predicted captions.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ali, S.: Where do we stand in AI for endoscopic image analysis? deciphering gaps and future directions. npj Dig. Med. 5(1), 184 (2022)
Bhambhvani, H.P., Zamora, A.: Deep learning enabled classification of mayo endoscopic subscore in patients with ulcerative colitis. Eur. J. Gastroenterol. Hepatol. 33(5), 645–649 (2021)
Borgli, H., et al.: Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Sci. Data 7(1), 283 (2020)
Click, B., Ramos Rivers, C., Koutroubakis, I.E., Babichenko, D., Anderson, A.M., et al.: Demographic and clinical predictors of high healthcare use in patients with inflammatory bowel disease. Inflamm. Bowel Dis. 22(6), 1442–1449 (2016)
Di Ruscio, M., et al.: Role of ulcerative colitis endoscopic index of severity (uceis) versus mayo endoscopic subscore (mes) in predicting patients response to biological therapy and the need for colectomy. Digestion 102(4), 534–545 (2021)
Fan, Y., et al.: A novel deep learning-based computer-aided diagnosis system for predicting inflammatory activity in ulcerative colitis. Gastrointest. Endosc. (2022)
Gong, L., et al.: Automatic captioning of early gastric cancer using magnification endoscopy with narrow-band imaging. Gastrointest. Endosc. 96(6), 929–942 (2022)
Ozawa, T., et al.: Novel computer-assisted diagnosis system for endoscopic disease activity in patients with ulcerative colitis. Gastrointest. Endosc. 89(2), 416–421 (2019)
Pagnini, C., Di Paolo, M.C., Mariani, B.M., Urgesi, R., Pallotta, L., et al.: Mayo endoscopic score and ulcerative colitis endoscopic index are equally effective for endoscopic activity evaluation in ulcerative colitis patients in a real life setting. Gastroenterol. Insights 12(2), 217–224 (2021)
Rabbenou, W., Ullman, T.A.: Risk of colon cancer and recommended surveillance strategies in patients with ulcerative colitis. Gastroenterol. Clin. 49(4), 791–807 (2020)
Schroeder, K.W., Tremaine, W.J., Ilstrup, D.M.: Coated oral 5-aminosalicylic acid therapy for mildly to moderately active ulcerative colitis. New Engl. J. Med. 317(26), 1625–1629 (1987)
Stidham, R.W., Liu, W., Bishu, S., Rice, M.D., Higgins, P.D., et al.: Performance of a deep learning model vs human reviewers in grading endoscopic disease severity of patients with ulcerative colitis. JAMA Netw. Open 2(5), 1–10 (2019)
Sutton, R.T., Zaiane, O.R., Goebel, R., Baumgart, D.C.: Artificial intelligence enabled automated diagnosis and grading of ulcerative colitis endoscopy images. Sci. Rep. 12(1), 1–10 (2022)
Taku, K., Britta, S., Chen, W.S., Ferrante, M., Shen, B., et al.: Ulcerative colitis (primer). Nat. Rev. Dis. Primers 6(1) (2020)
Travis, S.P., Schnell, D., Krzeski, P., Abreu, M.T., Altman, D.G., et al.: Developing an instrument to assess the endoscopic severity of ulcerative colitis: the ulcerative colitis endoscopic index of severity (uceis). Gut 61(4), 535–542 (2012)
Ungaro, R., Mehandru, S., Allen, P., Peyrin-Biroulet, L., Colombel, J.: Colitis ulcerosa. Lancet 389(10080), 1756–1770 (2017)
Vashist, N.M., et al.: Endoscopic scoring indices for evaluation of disease activity in ulcerative colitis. Cochrane Datab. Syst. Rev. (1) (2018)
Xu, Z., Ali, S., East, J., Rittscher, J.: Additive angular margin loss and model scaling network for optimised colitis scoring. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–5. IEEE (2022)
Xu, Z., Ali, S., Gupta, S., Leedham, S., East, J.E., Rittscher, J.: Patch-level instance-group discrimination with pretext-invariant learning for colitis scoring. In: Machine Learning in Medical Imaging, pp. 101–110 (2022)
Acknowledgements
The authors wish to acknowledge the Mexican Council for Science and Technology (CONACYT) for the support in terms of postgraduate scholarships in this project, and the Data Science Hub at Tecnologico de Monterrey for their support on this project. This work has been supported by Azure Sponsorship credits granted by Microsoft’s AI for Good Research Lab through the AI for Health program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Valencia, F.H. et al. (2023). Image Captioning for Automated Grading and Understanding of Ulcerative Colitis. In: Ali, S., van der Sommen, F., van Eijnatten, M., Papież, B.W., Jin, Y., Kolenbrander, I. (eds) Cancer Prevention Through Early Detection. CaPTion 2023. Lecture Notes in Computer Science, vol 14295. Springer, Cham. https://doi.org/10.1007/978-3-031-45350-2_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-45350-2_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45349-6
Online ISBN: 978-3-031-45350-2
eBook Packages: Computer ScienceComputer Science (R0)