Abstract
Deep learning models have been widely studied and have achieved expert-level performance in medical imaging tasks such as diagnosis. Recent research also considers integrating data from various sources, for instance, chest X-rays (CXR) radiographs and electronic medical records (EMR), to further improve the performance. However, most existing methods ignore the intrinsic relations among different sources of data, thereby lack interpretability. In this paper, we propose a framework for pulmonary disease diagnosis that combines deep learning and domain-knowledge reasoning. We first formalize the standard medical guidelines into formal-logic rules, and then learn the weights of the rules from medical data, integrating multimodal data for pulmonary disease diagnosis. We verify our method on a real dataset collected from a hospital, and the experimental results show that the proposed method outperforms the previous state-of-the-art multi-modal baselines.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Arevalo, J., Solorio, T., Montes-y Gómez, M., González, F.A.: Gated multimodal units for information fusion. arXiv preprint arXiv:1702.01992 (2017)
Bustos, A., Pertusa, A., Salinas, J.M., de la Iglesia-Vayá, M.: PadChest: a large chest X-ray image dataset with multi-label annotated reports. Med. Image Anal. 66, 101797 (2020)
Çallı, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K.G., Murphy, K.: Deep learning for chest X-ray analysis: a survey. Med. Image Anal. 72, 102125 (2021)
Chapman, W.W., Bridewell, W., Hanbury, P., Cooper, G.F., Buchanan, B.G.: A simple algorithm for identifying negated findings and diseases in discharge summaries. J. Biomed. Inform. 34(5), 301–310 (2001)
Cohen, J.P., et al.: TorchXRayVision: a library of chest X-ray datasets and models (2020). https://github.com/mlmed/torchxrayvision
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015)
Hou, J., Gao, T.: Explainable DCNN based chest X-ray image analysis and classification for COVID-19 pneumonia detection. Sci. Rep. 11(1), 1–15 (2021)
Irvin, J., et al.: CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In: Thirty-Third AAAI Conference on Artificial Intelligence (2019)
Irvin, J., et al.: CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. CORR abs/1901.07031, 1901 (2019)
Jiang, J., Li, X., Zhao, C., Guan, Y., Yu, Q.: Learning and inference in knowledge-based probabilistic model for medical diagnosis. Knowl. Based Syst. 138, 58–68 (2017)
Kiela, D., Bhooshan, S., Firooz, H., Testuggine, D.: Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv:1909.02950 (2019)
Kok, S., et al.: The alchemy system for statistical relational AI. Technical report, Department of Computer Science and Engineering, University of Washington, Seattle, WA (2005)
Li, L., et al.: Real-world data medical knowledge graph: construction and applications. Artif. Intell. Med. 103, 101817 (2020)
Nyga, D., Picklum, M., Beetz, M., et al.: PracMLN - Markov logic networks in Python (2013). https://www.pracmln.org/
Peng, Y., Wang, X., Lu, L., Bagheri, M., Summers, R., Lu, Z.: NegBio: a high-performance tool for negation and uncertainty detection in radiology reports. AMIA Summits Transl. Sci. Proc. 2018, 188 (2018)
Ren, H., et al.: Interpretable pneumonia detection by combining deep learning and explainable models with multisource data. IEEE Access 9, 95872–95883 (2021). https://doi.org/10.1109/ACCESS.2021.3090215
Richardson, M., Domingos, P.: Markov logic networks. Mach. Learn. 62(1), 107–136 (2006)
Shen, Y., et al.: CBN: constructing a clinical Bayesian network based on data from the electronic medical record. J. Biomed. Inform. 88, 1–10 (2018)
Singh, R., et al.: Deep learning in chest radiography: detection of findings and presence of change. PloS One 13(10), e0204155 (2018)
Sirazitdinov, I., Kholiavchenko, M., Kuleev, R., Ibragimov, B.: Data augmentation for chest pathologies classification. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 1216–1219. IEEE (2019)
Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
Wang, G., et al.: A deep-learning pipeline for the diagnosis and discrimination of viral, non-viral and COVID-19 pneumonia from chest X-ray images. Nat. Biomed. Eng. 5(6), 509–521 (2021)
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2097–2106 (2017)
Yin, C., Zhao, R., Qian, B., Lv, X., Zhang, P.: Domain knowledge guided deep learning with electronic health records. In: 2019 IEEE International Conference on Data Mining (ICDM), pp. 738–747. IEEE (2019)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, H. et al. (2022). MMLN: Leveraging Domain Knowledge for Multimodal Diagnosis. In: Bansal, M.S., Cai, Z., Mangul, S. (eds) Bioinformatics Research and Applications. ISBRA 2022. Lecture Notes in Computer Science(), vol 13760. Springer, Cham. https://doi.org/10.1007/978-3-031-23198-8_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-23198-8_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23197-1
Online ISBN: 978-3-031-23198-8
eBook Packages: Computer ScienceComputer Science (R0)