Abstract
In this paper, a unified computational framework towards medical image explanation is proposed to promote the ability of computers on understanding and interpreting medical images. Four complementary modules are included, such as the construction of Medical Image-Text Joint Embedding (MITE) based on large-scale medical images and related texts; a Medical Image Semantic Association (MISA) mechanism based on the MITE multimodal knowledge representation; a Hierarchical Medical Image Caption (HMIC) module that is visually understandable to radiologists; and a language-independent medical imaging report generation prototype system by integrating the HMIC and transfer learning method. As an initial study of automatic medical image explanation, preliminary experiments were carried out to verify the feasibility of the proposed framework, including the extraction of large scale medical image-text pairs, semantic concept detection from medical images, and automatic medical imaging reports generation. However, there is still a great challenge to produce medical image interpretations clinically usable, and further research is needed to empower machines explaining medical images like a human being.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Interagency Working Group on Medical Imaging Committee on Science, National Science And Technology Council, Roadmap for medical imaging research and development, 12 (2017)
Ma, Q., Kong, D.: A new variational model for joint restoration and segmentation based on the Mumford-Shah model. J. Vis. Commun. Image Represent. 53, 224–234 (2018)
Wang, X., Peng, Y., Lu, L., et al.: ChestX-ray8.: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: IEEE CVPR 2017, pp. 2097–2106 (2017)
Demnerfushman, D., Kohli, M.D., Rosenman, M.B., et al.: Preparing a collection of radiology examinations for distribution and retrieval. J. Am. Med. Inform. Assoc. Jamia 23(2), 304–310 (2016)
Iii, S.G.A., Mclennan, G., Bidaut, L., et al.: The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans. Med. Phys. 38(2), 9–15 (2011)
Eickhoff, C., Schwall, I., Garca ÃSeco de Herrera, A., Müller, H.: Overview of ImageCLEFcaption 2017 - the image caption prediction and concept extraction tasks to understand biomedical images. In: CLEF 2017 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Dublin, Ireland (2017)
Garca ÃSeco de Herrera, A., Eickhoff, C., Andrearczyk, V., Müller, H.: Overview of the ImageCLEF 2018 caption prediction tasks. In: CLEF 2018 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Avignon, France (2018)
Pelka, O., Friedrich, C.M., Garca ÃSeco de Herrera, A., Müller, H.: Overview of the ImageCLEFmed 2019 concept detection task. In: CLEF 2019 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Lugano, Switzerland (2019). ISSN 1613-0073
Pelka, O., Koitka, S., Rückert, J., Nensa, F., Friedrich, C.M.: Radiology objects in COntext (ROCO): a multimodal image dataset. In: Stoyanov, D., et al. (eds.) LABELS/CVII/STENT -2018. LNCS, vol. 11043, pp. 180–189. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01364-6_20
Wang, X., Zhang, Y., Guo, Z., Li, J.: Identifying concepts from medical images via transfer learning and image retrieval. Math. Biosci. Eng. 16(4), 1978–1991 (2019)
Shin, H., Roberts, K., Lu, L., et al.: Learning to read chest X-rays: recurrent neural cascade model for automated image annotation. In: Computer Vision and Pattern Recognition, pp. 2497–2506 (2016)
Zhou, C., Mao, Y., Wang, X.: Topic-specific image caption generation. In: Sun, M., Wang, X., Chang, B., Xiong, D. (eds.) CCL/NLP-NABD -2017. LNCS (LNAI), vol. 10565, pp. 321–332. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69005-6_27
Krause, J., Johnson, J., Krishna, R., Li, F.: A hierarchical approach for generating descriptive image paragraphs. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Liu, C., Wang, C., Sun, F., et al: Image2Text: a multimodal image captioner. In: ACM multimedia, pp. 746–748 (2016)
You, Q., Jin, H., Wang, Z., et al: Image captioning with semantic attention. In: Computer Vision and Pattern Recognition, pp. 4651–4659 (2016)
Liang, X., Hu, Z., Zhang, H., Gan, C., Xing, E.P.: Recurrent topic-transition GAN for visual paragraph generation. In: The IEEE International Conference on Computer Vision (ICCV) (2017)
Xu, K., Ba, J., Kiros, R., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning, pp. 2048–2057 (2016)
Hasan, S.A., Ling, Y., Liu, J., Sreenivasan, R., Anand, S., Arora, T.R., Datla, V., Lee, K., Qadir, A., Swisher, C., Farri, O.: Attention-based medical caption generation with image modality classification and clinical concept mapping. In: Bellot, P., et al. (eds.) CLEF 2018. LNCS, vol. 11018, pp. 224–230. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98932-7_21
Zhang, Z., Xie, Y., Xing, F., McGough, M., Yang, L.: MDNet: a semantically and visually interpretable medical image diagnosis network, pp. 3549–3557 (2017)
Jing, B., Xie, P., Eric, X.: On the automatic generation of medical imaging reports. In: Conference 2018, CVPR, pp. 2577–2586, Melbourne, Australia (2018)
Wang, X., Peng, Y., Lu, L., et al: TieNet: text-image embedding network for common thorax disease classification and reporting in chest X-rays. In: Conference 2018, CVPR (2018)
Kisilev, P., Walach, E., Barkan, E., et al.: From medical image to automatic medical report generation. IBM J. Res. Dev. 59(2/3), 2:1–2:7 (2015) https://doi.org/10.1147/JRD.2015.2393193
Hsu, W., Glass, J.: Disentangling by partitioning: a representation learning framework for multimodal sensory data, p.1805. arXiv (2018)
Angelov, P.P., Gu, X.: Deep rule based classifier with human-level performance and characteristics. Inf. Sci. 463–464, 196–213 (2018)
Gu, X., Angelov, P.P.: Semi-supervised deep rule-based approach for image classification. Appl. Soft Comput. 68, 53–68 (2018)
The Building Blocks of Interpretability Homepage. https://distill.pub/2018/building-blocks/. Accessed 30 Sep 2019
LUCID Homepage. https://github.com/tensorflow/lucid. Accessed 30 Sep 2019
MetaMap Homepage. https://metamap.nlm.nih.gov/. Accessed 30 Sep 2019
Lux, M., Chatzichristofis, S.A.: Lire: Lucene image retrieval: an extensible Java CBIR library. In: Proceedings of the 16th ACM International Conference on Multimedia. British Columbia, Canada (2008)
Guo, Z., Wang, X., Zhang, Y., Li, J.: ImageSem at ImageCLEFmed caption 2019 task: a two-stage medical concept detection strategy. In: CLEF 2019 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Lugano, Switzerland (2019)
Zhang, Y.: Automatic generation of medical imaging report generation based on deep learning, Peking Union Medical College (2019)
Demner-Fushman, D., Antani, S., Simpson, M., et al.: Design and development of a multimodal biomedical information retrieval system. J. Comput. Sci. Eng. 6(2), 168–177 (2012)
Papineni, K., Roukos, S., Ward, T., et al.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318 (2002)
Denkowski, M., Lavie, A.: Meteor universal: language specific translation evaluation for any target language. In: Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 376–380 (2014)
Lin, C.: Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches out: Proceedings of the ACL-04 Workshop, vol. 8, Barcelona, Spain (2004)
Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015)
Acknowledgments
This study was supported by the Non-profit Central Research Institute Fund of Chinese Academy of Medical Sciences (Grant No. 2018-I2M-AI-016, Grant No. 2017PT63010 and Grant No. 2018PT33024); the National Natural Science Foundation of China (Grant No. 81601573 and Grant No. 61906214).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, X., Zhang, Y., Guo, Z., Li, J. (2019). A Computational Framework Towards Medical Image Explanation. In: Marcos, M., et al. Artificial Intelligence in Medicine: Knowledge Representation and Transparent and Explainable Systems. KR4HC TEAAM 2019 2019. Lecture Notes in Computer Science(), vol 11979. Springer, Cham. https://doi.org/10.1007/978-3-030-37446-4_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-37446-4_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37445-7
Online ISBN: 978-3-030-37446-4
eBook Packages: Computer ScienceComputer Science (R0)