Abstract
The automatic description of images has made good advances, but limited by the little-sample dataset, that the automatic generation of medical imaging reports is still a challenging problem. Aiming at the problem of training the joint model (CNN-RNN) on little-sample datasets, this paper proposes an improved encoding-decoding mode, in which the encoder uses less parameter in FCN (Fully Convolutional Network) for identifying lesions in mammography, and encoding it into a semantic vector. The decoder uses a LSTM (Long Short-Term Memory network) for solving, thereby reducing sample requirements. In addition, this paper combines multi-label classification (MLC) to assist the semantic coding process and uses post-processing such as the beam search to make the output fit in the natural language description better. Compared to existing models, our improved model on public mammography dataset (INbreast) with real-world data supplement achieved the BLEU score improvements by two points.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wang, W., Ding, Y., Tian, C.: A novel semantic attribute-based feature for image caption generation. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3081–3085. IEEE (2018)
Jing, B., Xie, P., Xing, E.: On the automatic generation of medical imaging reports. arXiv preprint arXiv:1711.08195 (2017)
Johnson J., Karpathy, A., Fei-Fei, L.: DenseCap: fully convolutional localization networks for dense captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4565–4574 (2016)
Krause, J., Johnson, J., Krishna, R., et al.: A hierarchical approach for generating descriptive image paragraphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 317–325 (2017)
Chen, X., Zitnick, C.L.: Learning a recurrent visual representation for image caption generation. arXiv preprint arXiv:1411.5654 (2014)
Vinyals, O., Toshev, A., Bengio, S., et al.: Show and tell: a neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156–3164 (2015)
He, X., Yang, Y., Shi, B., et al.: VD-SAN: visual-densely semantic attention network for image caption generation. Neurocomputing 328, 48–55 (2019)
Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3128–3137 (2015)
Kulkarni, G., Premraj, V., Ordonez, V., et al.: BabyTalk: understanding and generating simple image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2891–2903 (2013)
Feng, Y., Lapata, M.: How many words is a picture worth? Automatic caption generation for news images. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 1239–1249. Association for Computational Linguistics (2010)
Farhadi, A., et al.: Every picture tells a story: generating sentences from images. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 15–29. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_2
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)
Aneja, J., Deshpande, A., Schwing, A.G.: Convolutional image captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5561–5570 (2018)
Liang, X., Hu, Z., Zhang, H., et al.: Recurrent topic-transition GAN for visual paragraph generation. In; Proceedings of the IEEE International Conference on Computer Vision, pp. 3362–3371 (2017)
Shin, H.C., Roberts, K., Lu, L., et al.: Learning to read chest x-rays: recurrent neural cascade model for automated image annotation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2497–2506 (2016)
Kisilev, P., Walach, E., Barkan, E., et al.: From medical image to automatic medical report generation. IBM J. Res. Dev. 59(2/3), 2:1–2:7 (2015)
Graves, A.: Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 (2013)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W., Frangi, F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Papineni, K., Roukos, S., Ward, T., et al.: IBM research report Bleu: a method for automatic evaluation of machine translation. IBM Research Division Technical Report, RC22176 (W0109-022), Yorktown Heights, New York (2001)
Vedantam, R., Lawrence Zitnick, C., Parikh, D.: CIDEr: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015)
Acknowledgment
This work was supported by the Science and Technology Development Foundation of Shanghai (18511102703, 16JC1400802, 16JC1400803), the Special Fund of Shanghai Municipal Commission of Economy and Informatization (RX-RJJC-08-16-0483, 2017-RGZN-01004, XX-XXFZ-02-18-2666, XX-XXFZ-01-18-2604).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Sun, L., Wang, W., Li, J., Lin, J. (2019). Study on Medical Image Report Generation Based on Improved Encoding-Decoding Method. In: Huang, DS., Bevilacqua, V., Premaratne, P. (eds) Intelligent Computing Theories and Application. ICIC 2019. Lecture Notes in Computer Science(), vol 11643. Springer, Cham. https://doi.org/10.1007/978-3-030-26763-6_66
Download citation
DOI: https://doi.org/10.1007/978-3-030-26763-6_66
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-26762-9
Online ISBN: 978-3-030-26763-6
eBook Packages: Computer ScienceComputer Science (R0)