skip to main content
10.1145/3484824.3484918acmotherconferencesArticle/Chapter ViewAbstractPublication PagesdsmlaiConference Proceedingsconference-collections
research-article

Automatic Generation of Chest X-Ray Medical Imaging Reports using LSTM-CNN

Published: 13 January 2022 Publication History

Abstract

Generating medical reports manually is a difficult task, especially in rural areas and in urgent medical cases, where there is an emergency. It can also be error-prone for inexperienced physicians to generate a medical report. There are various deep learning methodologies such as Image captioning, image classification that has been implemented earlier to solve this problem. Generating a medical report automatically is a difficult task, considering the less amount of open-source data available and the paired data which contains medical Images and the report is also limited. One of the challenging tasks is data bias in medical Imaging. A generative encoder-decoder model is suggested to solve this problem in an efficient way. There are various other challenges. First, the medical report itself contains various heterogeneous information such as paragraphs, tags, keywords. Secondly, it is also difficult to identify the abnormal regions in medical images. To solve this problem, a multi-task framework is built, which can perform tag generation and paragraph generation. LSTM (Long Short Term Memory) is built to generate long heterogeneous paragraphs in the medical report. The model working is demonstrated on Chest X-Ray dataset and also on pathology dataset.

References

[1]
Kashyap, Ramgopal, and Vivek Tiwari, 2018. Active contours using global models for medical image segmentation. International Journal of Computational Systems Engineering, 4, no. 2--3. 195--201.
[2]
Rakhlin, A. 2016. Convolutional neural networks for sentence classification." GitHub.
[3]
Choudhary, Meenakshi, Vivek Tiwari, and U. Venkanna 2020. Iris anti-spoofing through score-level fusion of handcrafted and data-driven features. Applied Soft Computing 91: 106206, Elsevier.
[4]
Mikolov, T., Karafiát, M., Burget, L., Černocký, J. and Khudanpur, S., 2010. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association.
[5]
Pawar, K., Jalem, R.S. and Tiwari, V., 2019. Stock market price prediction using LSTM RNN. In Emerging Trends in Expert Applications and Security (pp. 493--503. Springer, Singapore.
[6]
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y., 2014. Generative adversarial nets. Advances in neural information processing systems, 27.
[7]
Sak, H., Senior, A.W. and Beaufays, F., 2014. Long short-term memory recurrent neural network architectures for large scale acoustic modeling.
[8]
Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R. and McDonald, C.J., 2016. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2), pp.304--310.
[9]
Yuan, J., Liao, H., Luo, R. and Luo, J., 2019, October. Automatic radiology report generation based on multi-view image fusion and medical concept enrichment. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 721--729. Springer, Cham.
[10]
Bressem, K.K., Adams, L.C., Erxleben, C., Hamm, B., Niehues, S.M. and Vahldiek, J.L., 2020. Comparing different deep learning architectures for classification of chest radiographs. Scientific reports, 10(1), pp. 1--16.
[11]
Jing, B., Xie, P., Xing, E.P. 2018. On the automatic generation of medical imaging reports. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia. pp. 2577--2586.
[12]
Kisilev, P., Walach, E., Barkan, E., Ophir, B., Alpert, S. and Hashoul, S. Y., 2015. From medical image to automatic medical report generation. IBM Journal of Research and Development, 59(2/3), pp.2--1.
[13]
Shin, H. C., Roberts, K., Lu, L., Demner-Fushman, D., Yao, J. and Summers, R.M., 2016. Learning to read chest x-rays: Recurrent neural cascade model for automated image annotation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2497--2506.
[14]
Zhang, J., Xie, Y., Li, Y., Shen, C. and Xia, Y., 2020. Covid-19 screening on chest x-ray images using deep learning based anomaly detection. arXiv preprint arXiv:2003.12338, 27.
[15]
Yin, C., Qian, B., Wei, J., Li, X., Zhang, X., Li, Y. and Zheng, Q., 2019, November. Automatic generation of medical imaging diagnostic report with hierarchical recurrent neural network. In 2019 IEEE international conference on data mining (ICDM) (pp. 728--737. IEEE.
[16]
He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770--778.
[17]
Choudhary, M., Tiwari, V. and Venkanna, U., 2020. Iris Liveness Detection Using Fusion of Domain-Specific Multiple BSIF and DenseNet Features. IEEE Transactions on Cybernetics.
[18]
Choudhary, M., Tiwari, V. and Uduthalapally, V., 2021. Iris presentation attack detection based on best-k feature selection from YOLO inspired RoI. Neural Computing and Applications, 33(11), pp.5609--5629.
[19]
Mateen, M., Wen, J., Song, S. and Huang, Z., 2019. Fundus image classification using VGG-19 architecture with PCA and SVD. Symmetry, 11(1), p. 1.
[20]
Ba, J., Mnih, V. and Kavukcuoglu, K., 2014. Multiple object recognition with visual attention. arXiv preprint arXiv:1412.7755.
[21]
Faulstich, L.C., Irsig, K., Atalla, M., Varges, S., Bieler, H. and Stede, M., 2011, November. SemScribe: automatic generation of medical reports. In Symposium of the Austrian HCI and Usability Engineering Group (pp. 563--573. Springer, Berlin, Heidelberg.
[22]
Pascanu, R., Gulcehre, C., Cho, K. and Bengio, Y., 2013. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6026.
[23]
Krause, J., Johnson, J., Krishna, R. and Fei-Fei, L., 2017. A hierarchical approach for generating descriptive image paragraphs. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 317--325.
[24]
Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R. and McDonald, C. J., 2016. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2), pp.304--310.
[25]
Internet Pathology Laboratory for Medical Education, Link: https://webpath.med.utah.edu/.
[26]
Papineni, K., Roukos, S., Ward, T. and Zhu, W.J., 2002, July. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (pp. 311--318.
[27]
ROUGE, L.C., 2004, July. A package for automatic evaluation of summaries. In Proceedings of Workshop on Text Summarization of ACL, Spain.
[28]
Xue, Y., Xu, T., Long, L.R., Xue, Z., Antani, S., Thoma, G.R. and Huang, X., 2018, September. Multimodal recurrent model with attention for automated radiology report generation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 457--466. Springer, Cham.

Cited By

View all
  • (2024)InVERGe: Intelligent Visual Encoder for Bridging Modalities in Report Generation2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00208(2028-2038)Online publication date: 17-Jun-2024
  • (2024)Deep Learning Techniques For Improving NearField Synthetic Aperture Radar Imaging2024 IEEE 13th International Conference on Communication Systems and Network Technologies (CSNT)10.1109/CSNT60213.2024.10545795(624-630)Online publication date: 6-Apr-2024
  • (2024)Ensemble of Autoencoders for Anomaly Detection in Biomedical Data: A Narrative ReviewIEEE Access10.1109/ACCESS.2024.336069112(17273-17289)Online publication date: 2024
  • Show More Cited By

Index Terms

  1. Automatic Generation of Chest X-Ray Medical Imaging Reports using LSTM-CNN
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Information & Contributors

          Information

          Published In

          cover image ACM Other conferences
          DSMLAI '21': Proceedings of the International Conference on Data Science, Machine Learning and Artificial Intelligence
          August 2021
          415 pages
          ISBN:9781450387637
          DOI:10.1145/3484824
          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 13 January 2022

          Permissions

          Request permissions for this article.

          Check for updates

          Author Tags

          1. Deep Learning
          2. LSTM
          3. X-Ray
          4. encoder-decoder
          5. medical imaging

          Qualifiers

          • Research-article
          • Research
          • Refereed limited

          Funding Sources

          • DSPM IIIT Naya Raipur (CG), India

          Conference

          DSMLAI '21'

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)60
          • Downloads (Last 6 weeks)4
          Reflects downloads up to 23 Jan 2025

          Other Metrics

          Citations

          Cited By

          View all
          • (2024)InVERGe: Intelligent Visual Encoder for Bridging Modalities in Report Generation2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00208(2028-2038)Online publication date: 17-Jun-2024
          • (2024)Deep Learning Techniques For Improving NearField Synthetic Aperture Radar Imaging2024 IEEE 13th International Conference on Communication Systems and Network Technologies (CSNT)10.1109/CSNT60213.2024.10545795(624-630)Online publication date: 6-Apr-2024
          • (2024)Ensemble of Autoencoders for Anomaly Detection in Biomedical Data: A Narrative ReviewIEEE Access10.1109/ACCESS.2024.336069112(17273-17289)Online publication date: 2024
          • (2023)Hybrid Image Captioning Model2022 OPJU International Technology Conference on Emerging Technologies for Sustainable Development (OTCON)10.1109/OTCON56053.2023.10113957(1-6)Online publication date: 8-Feb-2023
          • (2023)Unsupervised Lung Anomaly Detection from Chest Radiographs for Curative Care using Isolation Forest Algorithm2022 OPJU International Technology Conference on Emerging Technologies for Sustainable Development (OTCON)10.1109/OTCON56053.2023.10113915(1-6)Online publication date: 8-Feb-2023
          • (2023)A Comprehensive Study on the Integration and Impact of Bits and Bytes in the Digital Era2023 IEEE International Conference on ICT in Business Industry & Government (ICTBIG)10.1109/ICTBIG59752.2023.10456337(1-6)Online publication date: 8-Dec-2023
          • (2023)An Analysis of Proposition Abstraction from Domain Text for Conceptual Plan Mining Using Deep Learning Methods2023 IEEE International Conference on ICT in Business Industry & Government (ICTBIG)10.1109/ICTBIG59752.2023.10456155(1-6)Online publication date: 8-Dec-2023
          • (2023)Integrating Wearable Sensor Data and AI for Remote Monitoring and Management of Chronic Respiratory Diseases2023 IEEE International Conference on ICT in Business Industry & Government (ICTBIG)10.1109/ICTBIG59752.2023.10455996(1-6)Online publication date: 8-Dec-2023
          • (2022)US Traffic Sign Recognition by Using Partial OCR and Inbuilt DictionaryICT Infrastructure and Computing10.1007/978-981-19-5331-6_72(713-720)Online publication date: 8-Nov-2022
          • (2021)A Comparative Analysis of Deep Convolution Layered Machine Learning approaches for Detection of Pneumonia from Chest Radiographs2021 IEEE International Conference on Technology, Research, and Innovation for Betterment of Society (TRIBES)10.1109/TRIBES52498.2021.9751653(1-5)Online publication date: 17-Dec-2021

          View Options

          Login options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          Media

          Figures

          Other

          Tables

          Share

          Share

          Share this Publication link

          Share on social media