Yingying Zhang
ABSTRACT
Representation learning of medical Knowledge Graph (KG) is an important task and forms the fundamental process for intelligent medical applications such as disease diagnosis and healthcare question answering. Therefore, many embedding models have been proposed to learn vector presentations for entities and relations but they ignore three important properties of medical KG: multi-modal, unbalanced and heterogeneous. Entities in the medical KG can carry unstructured multi-modal content, such as image and text. At the same time, the knowledge graph consists of multiple types of entities and relations, and each entity has various number of neighbors. In this paper, we propose a Multi-modal Multi-Relational Feature Aggregation Network (MMRFAN) for medical knowledge representation learning. To deal with the multi-modal content of the entity, we propose an adversarial feature learning model to map the textual and image information of the entity into the same vector space and learn the multi-modal common representation. To better capture the complex structure and rich semantics, we design a sampling mechanism and aggregate the neighbors with intra and inter-relation attention. We evaluate our model on three knowledge graphs, including FB15k-237, IMDb and Symptoms-in-Chinese with link prediction and node classification tasks. Experimental results show that our approach outperforms state-of-the-art method.
Supplemental Material
- Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi-relational Data. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2 (Lake Tahoe, Nevada) (NIPS'13). Curran Associates Inc., USA, 2787--2795.Google Scholar
Digital Library
- Liwei Cai and William Yang Wang. 2018. KBGAN: Adversarial Learning for Knowledge Graph Embeddings. In NAACL-HLT. Association for Computational Linguistics, 1470--1480.Google Scholar
- Edward Choi, Mohammad Taha Bahadori, Elizabeth Searles, Catherine Coffey, Michael Thompson, James Bost, Javier Tejedor-Sojo, and Jimeng Sun. 2016. Multi-layer Representation Learning for Medical Concepts. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016. ACM, 1495--1504.Google ScholarDigital Library
- Andreea Deac, Yu-Hsiang Huang, Petar Velickovic, Pietro Liò, and Jian Tang. 2019. Drug-Drug Adverse Effect Prediction with Graph Co-Attention. CoRR, Vol. abs/1905.00534 (2019). arxiv: 1905.00534Google Scholar
- Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In NIPS. 3844--3852.Google Scholar
- Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2D Knowledge Graph Embeddings. In AAAI. AAAI Press, 1811--1818.Google Scholar
- Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. 2020. MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding. CoRR, Vol. abs/2002.01680 (2020). arxiv: 2002.01680Google Scholar
- Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francc ois Laviolette, Mario Marchand, and Victor S. Lempitsky. 2016. Domain-Adversarial Training of Neural Networks. J. Mach. Learn. Res., Vol. 17 (2016), 59:1--59:35.Google Scholar
- Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. 2018. Large-Scale Learnable Graph Convolutional Networks. In KDD. ACM, 1416--1424.Google Scholar
- Junyu Gao, Tianzhu Zhang, and Changsheng Xu. 2019. Graph convolutional tracking. In CVPR. 4649--4659.Google Scholar
- Junyu Gao, Tianzhu Zhang, Xiaoshan Yang, and Changsheng Xu. 2017. Deep relative tracking. IEEE TIP, Vol. 26, 4 (2017), 1845--1858.Google Scholar
- Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural Message Passing for Quantum Chemistry. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 1263--1272.Google Scholar
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In NIPS. 2672--2680.Google Scholar
- Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable Feature Learning for Networks. In KDD. ACM, 855--864. https://doi.org/10.1145/2939672.2939754Google Scholar
- Jun Hu, Shengsheng Qian, Quan Fang, and Changsheng Xu. 2019. Hierarchical Graph Semantic Pooling Network for Multi-modal Community Question Answer Matching. In Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019. 1157--1165. https://doi.org/10.1145/3343031.3350966Google ScholarDigital Library
- Xin Huang, Yuxin Peng, and Mingkuan Yuan. 2020. MHTN: Modal-Adversarial Hybrid Transfer Network for Cross-Modal Retrieval. IEEE Trans. Cybern., Vol. 50, 3 (2020), 1047--1059. https://doi.org/10.1109/TCYB.2018.2879846Google Scholar
- Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge Graph Embedding via Dynamic Mapping Matrix. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)". Association for Computational Linguistics, Beijing, China, 687--696.Google Scholar
- Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. ICLR.Google Scholar
- Hongwei Li, Johannes C. Paetzold, Anjany Sekuboyina, Florian Kofler, Jianguo Zhang, Jan S. Kirschke, Benedikt Wiestler, and Bjoern H. Menze. 2019. DiamondGAN: Unified Multi-modal Generative Adversarial Networks for MRI Sequences Synthesis. In MICCAI (Lecture Notes in Computer Science, Vol. 11767). Springer, 795--803.Google Scholar
- Yujia Li, Richard Zemel, Marc Brockschmidt, and Daniel Tarlow. 2016. Gated Graph Sequence Neural Networks. In ICLR. 4453--4462.Google Scholar
- Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning Entity and Relation Embeddings for Knowledge Graph Completion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (Austin, Texas) (AAAI'15). AAAI Press, 2181--2187.Google ScholarDigital Library
- Ye Liu, Hui Li, Alberto Garc'i a-Durá n, Mathias Niepert, Daniel O n oro-Rubio, and David S. Rosenblum. 2019. MMKG: Multi-Modal Knowledge Graphs. CoRR, Vol. abs/1903.05485. arxiv: 1903.05485Google Scholar
- Feng Luo, Xiaoli Wang, Qingfeng Wu, Jiaying Liang, Xueliang Qiu, and Zhifeng Bao. 2020. HQADeepHelper: A Deep Learning System for Healthcare Question Answering. In Companion of The 2020 Web Conference 2020, Taipei, Taiwan, April 20--24, 2020,, Amal El Fallah Seghrouchni, Gita Sukthankar, Tie-Yan Liu, and Maarten van Steen (Eds.). ACM / IW3C2, 194--197. https://doi.org/10.1145/3366424.3383539Google Scholar
- Fenglong Ma, Yaqing Wang, Houping Xiao, Ye Yuan, Radha Chitta, Jing Zhou, and Jing Gao. 2018a. A General Framework for Diagnosis Prediction via Incorporating Medical Code Descriptions. 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (2018), 1070--1075.Google ScholarCross Ref
- Fenglong Ma, Quanzeng You, Houping Xiao, Radha Chitta, Jing Zhou, and Jing Gao. 2018c. KAME: Knowledge-based Attention Model for Diagnosis Prediction in Healthcare. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (Torino, Italy) (CIKM '18). ACM, New York, NY, USA, 743--752.Google ScholarDigital Library
- Tengfei Ma, Cao Xiao, Jiayu Zhou, and Fei Wang. 2018b. Drug Similarity Integration Through Attentive Multi-view Graph Auto-Encoders. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden. ijcai.org, 3477--3483.Google ScholarCross Ref
- Chengsheng Mao, Liang Yao, and Yuan Luo. 2019. MedGCN: Graph Convolutional Networks for Multiple Medical Tasks. CoRR, Vol. abs/1904.00326 (2019). arxiv: 1904.00326 http://arxiv.org/abs/1904.00326Google Scholar
- Mehdi Mirza and Simon Osindero. 2014. Conditional Generative Adversarial Nets. CoRR, Vol. abs/1411.1784 (2014). arxiv: 1411.1784Google Scholar
- Hatem Mousselly Sergieh, Teresa Botschen, Iryna Gurevych, and Stefan Roth. 2018. A Multimodal Translation-Based Approach for Knowledge Graph Representation Learning. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics, New Orleans, Louisiana, 225--234.Google Scholar
- Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A Three-Way Model for Collective Learning on Multi-Relational Data. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011. 809--816.Google ScholarDigital Library
- Shengsheng Qian, Tianzhu Zhang, and Changsheng Xu. 2016a. Multi-modal Multi-view Topic-opinion Mining for Social Event Analysis. In Proceedings of the 2016 ACM Conference on Multimedia Conference, MM 2016, Amsterdam, The Netherlands, October 15-19, 2016. 2--11. https://doi.org/10.1145/2964284.2964294Google ScholarDigital Library
- Shengsheng Qian, Tianzhu Zhang, Changsheng Xu, and Jie Shao. 2016b. Multi-Modal Event Topic Model for Social Event Analysis. IEEE Transactions on Multimedia, Vol. 18, 2 (Feb 2016), 233--246.Google ScholarDigital Library
- Meng Qu, Yoshua Bengio, and Jian Tang. 2019. GMNN: Graph Markov Neural Networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 5241--5250. http://proceedings.mlr.press/v97/qu19a.htmlGoogle Scholar
- Paulo E. Rauber, Alexandre X. Falc a o, and Alexandru C. Telea. 2016. Visualizing Time-Dependent Data Using Dynamic t-SNE. In Eurographics Conference on Visualization, EuroVis 2016, Short Papers, Groningen, The Netherlands, 6-10 June 2016. Eurographics Association, 73--77.Google Scholar
- Tong Ruan, Mengjie Wang, Jian Sun, Ting Wang, Lu Zeng, Yichao Yin, and Ju Gao. 2017. An automatic approach for constructing a knowledge base of symptoms in Chinese. J. Biomedical Semantics, Vol. 8-S, 1 (2017), 71--79.Google ScholarCross Ref
- Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The graph neural network model. IEEE TNN, Vol. 20, 1 (2009), 61--80.Google Scholar
- Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling Relational Data with Graph Convolutional Networks. In ESWC (Lecture Notes in Computer Science, Vol. 10843). Springer, 593--607.Google Scholar
- Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. 2020. GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=S1esMkHYPrGoogle Scholar
- Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.Google Scholar
- Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. 2015. LINE: Large-scale Information Network Embedding. In WWW. ACM, 1067--1077. https://doi.org/10.1145/2736277.2741093Google Scholar
- Yuxing Tang, Youbao Tang, Veit Sandfort, Jing Xiao, and Ronald M. Summers. 2019. TUNA-Net: Task-Oriented UNsupervised Adversarial Network for Disease Recognition in Cross-domain Chest X-rays. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2019 - 22nd International Conference, Shenzhen, China, October 13--17, 2019, Proceedings, Part VI (Lecture Notes in Computer Science, Vol. 11769). Springer, 431--440.Google Scholar
- Thé o Trouillon, Johannes Welbl, Sebastian Riedel, É ric Gaussier, and Guillaume Bouchard. 2016. Complex Embeddings for Simple Link Prediction. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 (JMLR Workshop and Conference Proceedings, Vol. 48). JMLR.org, 2071--2080.Google Scholar
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. 5998--6008.Google Scholar
- Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018a. Graph Attention Networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.Google Scholar
- Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018b. Graph Attention Networks. (2018).Google Scholar
- Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019 a. KGAT: Knowledge Graph Attention Network for Recommendation. In KDD. ACM, 950--958. https://doi.org/10.1145/3292500.3330989Google Scholar
- Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S. Yu. 2019 b. Heterogeneous Graph Attention Network. In WWW. ACM, 2022--2032.Google Scholar
- Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge Graph Embedding by Translating on Hyperplanes. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence (Québec City, Québec, Canada) (AAAI'14). AAAI Press, 1112--1119.Google Scholar
- Man Wu, Shirui Pan, Lan Du, Ivor W. Tsang, Xingquan Zhu, and Bo Du. 2019 b. Long-short Distance Aggregation Networks for Positive Unlabeled Graph Learning. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019. 2157--2160.Google ScholarDigital Library
- Man Wu, Shirui Pan, Chuan Zhou, Xiaojun Chang, and Xingquan Zhu. 2020. Unsupervised Domain Adaptive Graph Convolutional Networks. In WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020. 1457--1467.Google Scholar
- Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. 2019 a. A Comprehensive Survey on Graph Neural Networks. CoRR, Vol. abs/1901.00596 (2019). arxiv: 1901.00596Google Scholar
- Louis-Pascal A. C. Xhonneux, Meng Qu, and Jian Tang. 2019. Continuous Graph Neural Networks. CoRR, Vol. abs/1912.00967 (2019). arxiv: 1912.00967 http://arxiv.org/abs/1912.00967Google Scholar
- Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2017. Image-embodied Knowledge Representation Learning. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17. 3140--3146.Google ScholarCross Ref
- Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.Google Scholar
- Yingying Zhang, Junyu Gao, Xiaoshan Yang, Chang Liu, Yan Li, and Changsheng Xu. 2020. Find Objects and Focus on Highlights: Mining Object Semantics for Video Highlight Detection via Graph Neural Networks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020. 12902--12909.Google ScholarCross Ref
- Yingying Zhang, Shengsheng Qian, Quan Fang, and Changsheng Xu. 2019. Multi-modal Knowledge-aware Hierarchical Attention Network for Explainable Medical Question Answering. In Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019. ACM, 1089--1097.Google ScholarDigital Library
- Ziwei Zhang, Peng Cui, and Wenwu Zhu. 2018. Deep Learning on Graphs: A Survey. CoRR, Vol. abs/1812.04202 (2018).Google Scholar
- Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2018. Graph Neural Networks: A Review of Methods and Applications. CoRR, Vol. abs/1812.08434 (2018).Google Scholar
- Zhaocheng Zhu, Shizhen Xu, Jian Tang, and Meng Qu. 2019. GraphVite: A High-Performance CPU-GPU Hybrid System for Node Embedding. In The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019,, Ling Liu, Ryen W. White, Amin Mantrach, Fabrizio Silvestri, Julian J. McAuley, Ricardo Baeza-Yates, and Leila Zia (Eds.). ACM, 2494--2504.Google ScholarDigital Library
- Marinka Zitnik, Monica Agrawal, and Jure Leskovec. 2018. Modeling polypharmacy side effects with graph convolutional networks. Bioinform., Vol. 34, 13 (2018), i457--i466. https://doi.org/10.1093/bioinformatics/bty294Google ScholarCross Ref
Index Terms
- Multi-modal Multi-relational Feature Aggregation Network for Medical Knowledge Representation Learning
Recommendations
Multi-modal Knowledge-aware Hierarchical Attention Network for Explainable Medical Question Answering
MM '19: Proceedings of the 27th ACM International Conference on MultimediaOnline healthcare services can offer public ubiquitous access to the medical knowledge, especially with the emergence of medical question answering websites, where patients can get in touch with doctors without going to hospital. Explainability and ...
Multi-modal Knowledge Graphs for Recommender Systems
CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge ManagementRecommender systems have shown great potential to solve the information explosion problem and enhance user experience in various online applications. To tackle data sparsity and cold start problems in recommender systems, researchers propose knowledge ...
Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion
Knowledge graphs often suffer from incompleteness, and knowledge graph completion (KGC) aims at inferring the missing triplets through knowledge graph embedding from known factual triplets. However, most existing knowledge graph embedding methods only use ...
Comments