Abstract
Despite recent advances in medical diagnosis domain, many challenges remain in obtaining more accurate conclusions and in presenting semantically and visually interpretable results during the diagnosis process. An interpretable diagnosis process is proposed through the implementation of a deep learning model. This consists of three interrelated models, an image model, an attention model and a conclusion model. The proposed image model extracts the semantic feature using convolutional neural networks (CNNs). The conclusion model, integrated with the semantic attributes attention model, aims to predict the conclusion label by long-short term memory (LSTM), which captures the discriminative relationship between semantic attributes. The network is trained in end-to-end way with different weight of each model. Based upon a cervical intraepithelial neoplasia images, diagnostic report and labels (CINDRAL) dataset, the approach demonstrates significant improvement when comparing the baseline in the conclusion result.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhang, X., Su, H., Yang, L., Zhang, S.: Fine-grained histopathological image analysis via robust segmentation and large-scale retrieval. In: Computer Vision and Pattern Recognition, pp. 5361–5368 (2015)
Chang, H., Zhou, Y., Borowsky, A., Barner, K., Spellman, P., Parvin, B.: Stacked predictive sparse decomposition for classification of histology sections. Int. J. Comput. Vis. 113(1), 3–18 (2015)
Cireşan, D.C., Giusti, A., Gambardella, L.M., Schmidhuber, J.: Mitosis detection in breast cancer histology images with deep neural networks. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8150, pp. 411–418. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40763-5_51
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2014)
Kisilev, P., Walach, E., Hashoul, S., Barkan, E., Ophir, B., Alpert, S.: Semantic description of medical image findings: structured learning approach. In: British Machine Vision Conference, pp. 171.1–171.11 (2015)
Esteva, A., et al.: Corrigendum: dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115–118 (2017)
Chartrand, G., et al.: Deep learning: a primer for radiologists. Radiographics 37(7), 2113–2131 (2017)
Zhang, Z., Chen, P., Sapkota, M., Yang, L.: TandemNet: distilling knowledge from medical images using diagnostic reports as optional semantic references. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 320–328. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_37
Kisilev, P., Walach, E., Barkan, E., Ophir, B.: From medical image to automatic medical report generation. IBM J. Res. Dev. 59(2/3), 2:1–2:7 (2015)
Jing, B., Xie, P., Xing, E.: On the automatic generation of medical imaging reports (2017). arXiv:1711.08195
Surhone, L.M., Tennoe, M.T., Henssonow, S.F.: Long Short Term Memory. Beta Script Publishing (2010)
Zhang, Z., Xie, Y., Xing, F., Mcgough, M., Yang, L.: Mdnet: a semantically and visually interpretable medical image diagnosis network, pp. 3549–3557 (2017)
Wang, Z., Chen, T., Li, G., Xu, R., Lin, L.: Multi-label image recognition by recurrently discovering attentional regions. In: IEEE International Conference on Computer Vision, pp. 464–472 (2017)
Shi, X., Xing, F., Xie, Y., Su, H., Yang, L.: Cell encoding for histopathology image classification. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 30–38. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66185-8_4
Nam, H., Ha, J.W., Kim, J.: Dual attention networks for multimodal reasoning and matching, pp 2156–2164 (2016)
Pedersoli, M., Lucas, T., Schmid, C., Verbeek, J.: Areas of attention for image captioning, pp. 1251–1259 (2017)
Vinyals, O., Fortunato, M., Jaitly, N.: Pointer networks. In: Computer Science (2015)
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: neural image caption generation with visual attention. In: Computer Science, pp. 2048–2057 (2015)
Yu, D., Fu, J., Mei, T., Rui, Y.: Multi-level attention networks for visual question answering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4187–4195 (2017)
Lu, J., Xiong, C., Parikh, D., Socher, R.: Knowing when to look: adaptive attention via a visual sentinel for image captioning, pp. 3242–3250 (2016)
Shin, H.C., Roberts, K., Lu, L., Demnerfushman, D., Yao, J., Summers, R.M.: Learning to read chest x-rays: recurrent neural cascade model for automated image annotation, pp. 2497–2506 (2016)
Wang, X., Peng, Y., Lu, L., Lu, Z., Summers, R.M.: Tienet: text-image embedding network for common thorax disease classification and reporting in chest x-rays (2018). arXiv:1801.04334
Everingham, M., Winn, J.: The pascal visual object classes challenge 2010 development kit contents. In: International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, pp. 117–176 (2011)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Computer Science (2014)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks, pp 1097–1105 (2012)
Huang, G., Liu, Z., Maaten, L.V.D., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR (2017)
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition, pp. 770–778 (2015)
Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., Xu, W.: Cnn-rnn: a unified framework for multi-label image classification, pp. 2285–2294 (2016)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Acknowledgements
This work is supported by the National Key Scientific Instruments and Equipment Development Program of China (2013YQ03065101), the National Natural Science Foundation of China under Grant 61521063 and Grant 61503243.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Ma, K., Wu, K., Cheng, H., Gu, C., Xu, R., Guan, X. (2018). A Pathology Image Diagnosis Network with Visual Interpretability and Structured Diagnostic Report. In: Cheng, L., Leung, A., Ozawa, S. (eds) Neural Information Processing. ICONIP 2018. Lecture Notes in Computer Science(), vol 11306. Springer, Cham. https://doi.org/10.1007/978-3-030-04224-0_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-04224-0_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04223-3
Online ISBN: 978-3-030-04224-0
eBook Packages: Computer ScienceComputer Science (R0)