Retinal OCT image report generation based on visual and semantic topic attention model

Chao Guo; Weifang Zhu; Ting Wang; Tian Lin; Haoyu Chen; Xinjian Chen

doi:10.1117/12.2611469

4 April 2022 Retinal OCT image report generation based on visual and semantic topic attention model

Chao Guo, Weifang Zhu, Ting Wang, Tian Lin, Haoyu Chen, Xinjian Chen

Proceedings Volume 12032, Medical Imaging 2022: Image Processing; 120322C (2022) https://doi.org/10.1117/12.2611469
Event: SPIE Medical Imaging, 2022, San Diego, California, United States

Conference Poster

Abstract

Optical coherence tomography (OCT) is widely used in the diagnosis of retinal diseases. Reading OCT images and summarizing its insights is a routine, yet nonetheless time-consuming task. Automatic report generation can alleviate this issue. There are two major challenges in this task: (1) An OCT image may contain several fundus abnormalities and it is difficult to detect them all simultaneously. (2) The diagnostic reports are complex, which need to describe multiple lesions. In this paper, we propose a deep learning-based model, named as VSTA model (Visual and Semantic Topic Attention model), which is able to generate report from the input OCT image. Our major contributions include: (1) Semantic attention and visual attention are jointly embedded to the model to generate diagnosis report with complex content. (2) Semantic tags based on image similarity is employed to initialize the semantic attention weights, which increases the prediction accuracy of the model. With the proposed VSTA model, the metric of BLEU-4, CIDEr and ROUGE-L reach 31.16, 264.22 and 52.58, which are better than some existing advanced methods.

Conference Presentation

Citation Download Citation

Chao Guo, Weifang Zhu, Ting Wang, Tian Lin, Haoyu Chen, and Xinjian Chen "Retinal OCT image report generation based on visual and semantic topic attention model", Proc. SPIE 12032, Medical Imaging 2022: Image Processing, 120322C (4 April 2022); https://doi.org/10.1117/12.2611469

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available