skip to main content
10.1145/3343031.3351033acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Multi-modal Knowledge-aware Hierarchical Attention Network for Explainable Medical Question Answering

Authors Info & Claims
Published:15 October 2019Publication History

Editorial Notes

A corrigendum was issued for this article on October 29, 2019. You can download the corrigendum from the supplemental material section of this citation page.

ABSTRACT

Online healthcare services can offer public ubiquitous access to the medical knowledge, especially with the emergence of medical question answering websites, where patients can get in touch with doctors without going to hospital. Explainability and accuracy are two main concerns for medical question answering. However, existing methods mainly focus on accuracy and cannot provide a good explanation for retrieved medical answers. This paper proposes a novelMulti-Modal Knowledge-aware Hierarchical Attention Network (MKHAN) to effectively exploit multi-modal knowledge graph (MKG) for explainable medical question answering. MKHAN can generate path representation by composing the structural, linguistics, and visual information of entities, and infer the underlying rationale of question-answer interactions by leveraging the sequential dependencies within a path from MKG. Furthermore, a novel hierarchical attention network is proposed to discriminate the salience of paths endowing our model with explainability. We build a large-scale multi-modal medical knowledge graph andtwo real-world medical question answering datasets, the experimental results demonstrate the superior performance on our approachcompared with the state-of-the-art methods.

Skip Supplemental Material Section

Supplemental Material

References

  1. Asma Ben Abacha and Pierre Zweigenbaum. 2015. MEANS: A medical question-answering system combining NLP techniques and semantic Web technologies. Inf. Process. Manage. , Vol. 51 (2015), 570--594.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi-relational Data. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2 (NIPS'13). Curran Associates Inc., USA, 2787--2795.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Edward Choi, Mohammad Taha Bahadori, Elizabeth Searles, Catherine Coffey, Michael Thompson, James Bost, Javier Tejedor-Sojo, and Jimeng Sun. 2016. Multi-layer Representation Learning for Medical Concepts. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16). ACM, New York, NY, USA, 1495--1504.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F. Stewart, and Jimeng Sun. 2017. GRAM: Graph-based Attention Model for Healthcare Representation Learning. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '17). ACM, New York, NY, USA, 787--795.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Wanyun Cui, Yanghua Xiao, Haixun Wang, Yangqiu Song, Seung won Hwang, and Wei Yang Wang. 2017. KBQA: Learning Question Answering over QA Corpora and Knowledge Bases. Proceedings of the VLDB Endowment , Vol. 10 (2017), 565--576.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Jun da Hu, Shengsheng Qian, Quan Fang, and Changsheng Xu. 2018. Attentive Interactive Convolutional Matching for Community Question Answering in Social Multimedia. In ACM International Conference on Multimedia .Google ScholarGoogle Scholar
  7. Hanyin Fang, Fei Wu, Zhou Zhao, Xinyu Duan, Yueting Zhuang, and Martin Ester. 2016. Community-based Question Answering via Heterogeneous Social Network Learning. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI'16). AAAI Press, 122--128.Google ScholarGoogle Scholar
  8. Travis R. Goodwin and Sanda M. Harabagiu. 2016. Medical Question Answering for Clinical Decision Support. (2016), 297--306.Google ScholarGoogle Scholar
  9. Travis R. Goodwin and Sanda M. Harabagiu. 2017. Knowledge Representations and Inference Techniques for Medical Question Answering. ACM Transactions on Intelligent Systems and Technology , Vol. 9 (2017), 14:1--14:26.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Donglin Guo, Min Li, Ying Yu, Yaohang Li, Guihua Duan, Fang-Xiang Wu, and Jianxin Wang. 2018. Disease Inference with Symptom Extraction and Bidirectional Recurrent Neural Network. 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (2018), 864--868.Google ScholarGoogle Scholar
  11. Kalervo J"arvelin and Jaana Kek"al"ainen. 2000. IR Evaluation Methods for Retrieving Highly Relevant Documents. (2000), 41--48.Google ScholarGoogle Scholar
  12. Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge Graph Embedding via Dynamic Mapping Matrix. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)" . Association for Computational Linguistics, Beijing, China, 687--696.Google ScholarGoogle Scholar
  13. Quoc Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. In Proceedings of the 31st International Conference on Machine Learning (Proceedings of Machine Learning Research), , Eric P. Xing and Tony Jebara (Eds.), Vol. 32. PMLR, Bejing, China, 1188--1196.Google ScholarGoogle Scholar
  14. Yaliang Li, Nan Du, Chaochun Liu, Yusheng Xie, Wei Fan, Qi Li, Jing Gao, and Huan Sun. 2017. Reliable Medical Diagnosis from Crowdsourcing: Discover Trustworthy Answers from Non-Experts. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining (WSDM '17). ACM, New York, NY, USA, 253--261.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning Entity and Relation Embeddings for Knowledge Graph Completion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI'15). AAAI Press, 2181--2187.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Siqi Liu, Sidong Liu, Tom Weidong Cai, Hangyu Che, Sonia Pujol, Ron Kikinis, David Dagan Feng, and Michael J. Fulham. 2015. Multimodal Neuroimaging Feature Learning for Multiclass Diagnosis of Alzheimer's Disease. IEEE Transactions on Biomedical Engineering , Vol. 62 (2015), 1132--1140.Google ScholarGoogle ScholarCross RefCross Ref
  17. Fenglong Ma, Yaqing Wang, Houping Xiao, Ye Yuan, Radha Chitta, Jing Zhou, and Jing Gao. 2018a. A General Framework for Diagnosis Prediction via Incorporating Medical Code Descriptions. 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (2018), 1070--1075.Google ScholarGoogle ScholarCross RefCross Ref
  18. Fenglong Ma, Quanzeng You, Houping Xiao, Radha Chitta, Jing Zhou, and Jing Gao. 2018b. KAME: Knowledge-based Attention Model for Diagnosis Prediction in Healthcare. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM '18). ACM, New York, NY, USA, 743--752.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Riccardo Miotto, Li Li, Brian A. Kidd, and Joel Dudley. 2016. Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records. In Scientific reports .Google ScholarGoogle Scholar
  20. Hatem Mousselly Sergieh, Teresa Botschen, Iryna Gurevych, and Stefan Roth. 2018. A Multimodal Translation-Based Approach for Knowledge Graph Representation Learning. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics, New Orleans, Louisiana, 225--234.Google ScholarGoogle Scholar
  21. Phuoc Nguyen, Truyen Tran, Nilmini Wickramasinghe, and Svetha Venkatesh. 2017. Deepr: A Convolutional Net for Medical Records. IEEE Journal of Biomedical and Health Informatics , Vol. 21 (2017), 22--30.Google ScholarGoogle ScholarCross RefCross Ref
  22. Fudong Nian, Bing-Kun Bao, Teng Li, and Changsheng Xu. 2017. Multi-Modal Knowledge Representation Learning via Webly-Supervised Relationships Mining. In Proceedings of the 25th ACM International Conference on Multimedia (MM '17). ACM, New York, NY, USA, 411--419.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. L. Nie, M. Wang , L. Zhang, S. Yan, B. Zhang, and T. Chua. 2015. Disease Inference from Health-Related Questions via Sparse Deep Learning. IEEE Transactions on Knowledge and Data Engineering , Vol. 27, 8 (Aug 2015), 2107--2119.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text Matching As Image Recognition. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI'16). AAAI Press, 2793--2799.Google ScholarGoogle Scholar
  25. Trang Pham, Truyen Tran, Dinh Phung, and Svetha Venkatesh. 2016. DeepCare: A Deep Dynamic Memory Model for'zPredictive Medicine. In Proceedings, Part II, of the 20th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining - Volume 9652 (PAKDD 2016). Springer-Verlag, Berlin, Heidelberg, 30--41.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. S. Qian, T. Zhang , C. Xu, and J. Shao. 2016. Multi-Modal Event Topic Model for Social Event Analysis. IEEE Transactions on Multimedia , Vol. 18, 2 (Feb 2016), 233--246. https://doi.org/10.1109/TMM.2015.2510329Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Ying Shen, Yang Deng, Min Yang, Yaliang Li, Nan Du, Wei Fan, and Kai Lei. 2018. Knowledge-aware Attentive Neural Network for Ranking Question Answer Pairs. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '18). ACM, New York, NY, USA, 901--904.Google ScholarGoogle Scholar
  28. Yikang Shen, Wenge Rong, Zhiwei Sun, Yuanxin Ouyang, and Zhang Xiong. 2015. Question/Answer Matching for CQA System via Combining Lexical and Sequential Information. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI'15). AAAI Press, 275--281.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Benjamin Shickel, Patrick James Tighe, Azra Bihorac, and Parisa Rashidi. 2018. Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis. IEEE Journal of Biomedical and Health Informatics , Vol. 22 (2018), 1589--1604.Google ScholarGoogle ScholarCross RefCross Ref
  30. Heung-Il Suk, Seong-Whan Lee, and Dinggang Shen. 2014. Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. NeuroImage , Vol. 101 (2014), 569--582.Google ScholarGoogle ScholarCross RefCross Ref
  31. Zhu Sun, Jie Yang, Jie Zhang, Alessandro Bozzon, Long-Kai Huang, and Chi Xu. 2018. Recurrent Knowledge Graph Embedding for Effective Recommendation. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys '18). ACM, New York, NY, USA, 297--305.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Yi Tay, Minh C. Phan, Luu Anh Tuan, and Siu Cheung Hui. 2017. Learning to Rank Question Answer Pairs with Holographic Dual LSTM Architecture. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '17). ACM, New York, NY, USA, 695--704.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Cross Temporal Recurrent Networks for Ranking Question Answer Pairs. In Proceedings of the 32rd AAAI Conference on Artificial Intelligence .Google ScholarGoogle Scholar
  34. Truyen Tran, Tu Dinh Nguyen, Dinh Phung, and Svetha Venkatesh. 2015. Learning Vector Representation of Medical Objects via EMR-driven Nonnegative Restricted Boltzmann Machines (eNRBM). J. of Biomedical Informatics , Vol. 54, C (April 2015), 96--105.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In NIPS .Google ScholarGoogle Scholar
  36. Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge Graph Embedding by Translating on Hyperplanes. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence (AAAI'14). AAAI Press, 1112--1119.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Yu Ping Wu, Wei Chung Wu, Can Xu, and Zhoujun Li. 2018. Proceedings of the AAAI Conference on Artificial Intelligence. In AAAI .Google ScholarGoogle Scholar
  38. Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2017. Image-embodied Knowledge Representation Learning. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17. 3140--3146.Google ScholarGoogle ScholarCross RefCross Ref
  39. Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-End Neural Ad-hoc Ranking with Kernel Pooling. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '17). ACM, New York, NY, USA, 55--64.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Xiaodong Zhang, Sujian Li, Lei Sha, and Houfeng Wang. 2017. Attentive Interactive Neural Networks for Answer Selection in Community Question Answering. In Proceedings of the 31th AAAI Conference on Artificial Intelligence .Google ScholarGoogle Scholar

Index Terms

  1. Multi-modal Knowledge-aware Hierarchical Attention Network for Explainable Medical Question Answering

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Conferences
      MM '19: Proceedings of the 27th ACM International Conference on Multimedia
      October 2019
      2794 pages
      ISBN:9781450368896
      DOI:10.1145/3343031

      Copyright © 2019 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 15 October 2019

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

      Acceptance Rates

      MM '19 Paper Acceptance Rate252of936submissions,27%Overall Acceptance Rate995of4,171submissions,24%

      Upcoming Conference

      MM '24
      MM '24: The 32nd ACM International Conference on Multimedia
      October 28 - November 1, 2024
      Melbourne , VIC , Australia

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader