skip to main content
research-article

Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion

Published: 06 February 2023 Publication History

Abstract

Knowledge graphs often suffer from incompleteness, and knowledge graph completion (KGC) aims at inferring the missing triplets through knowledge graph embedding from known factual triplets. However, most existing knowledge graph embedding methods only use the relational information of knowledge graph and treat the entities and relations as IDs with simple embedding layer, ignoring the multi-modal information among triplets, such as text descriptions, images, etc. In this work, we propose a novel network to incorporate different modal information with graph structure information for more precise representation of multi-modal knowledge graph, termed as hyper-node relational graph attention (HRGAT) network. In HRGAT, we use low-rank multi-modal fusion to model the intra-modality and inter-modality dynamics, which transforms the original knowledge graph to a hyper-node graph. Then, relational graph attention (RGAT) network is used, which contains relation-specific attention and entity-relation fusion operation to capture the graph structure information. Finally, we aggregate the updated multi-modal information and graph structure information to generate the final embeddings of knowledge graph to achieve KGC. By exploring multi-modal information and graph structure information, HRGAT embraces faster convergence speed and achieves the state-of-the-art for KGC on the standard datasets. Implementation code is available at https://github.com/broliang/HRGAT.

References

[1]
Bo An, Bo Chen, Xianpei Han, and Le Sun. 2018. Accurate text-enhanced knowledge graph representation learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, Volume 1 (Long Papers). 745–755.
[2]
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007. DBpedia: A nucleus for a web of open data. In The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007 + ASWC 2007. 722–735.
[3]
Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. 2019. Hypernetwork knowledge graph embeddings. In Artificial Neural Networks and Machine Learning - ICANN 2019-28th International Conference on Artificial Neural Networks. 553–565.
[4]
Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. 2019. TuckER: Tensor factorization for knowledge graph completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019. 5184–5193.
[5]
Kurt D. Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008. 1247–1250.
[6]
Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. 2787–2795.
[7]
Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning structured embeddings of knowledge bases. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011.
[8]
Chandramani Chaudhary, Poonam Goyal, Navneet Goyal, and Yi-Ping Phoebe Chen. 2020. Image retrieval for complex queries using knowledge embedding. ACM Trans. Multim. Comput. Commun. Appl. 16, 1 (2020), 13:1–13:23.
[9]
Liyi Chen, Zhi Li, Yijun Wang, Tong Xu, Zhefeng Wang, and Enhong Chen. 2020. MMEA: Entity alignment for multi-modal knowledge graph. In Knowledge Science, Engineering and Management - 13th International Conference, KSEM 2020. 134–147.
[10]
Yao Chen, Jiangang Liu, Zhe Zhang, Shiping Wen, and Wenjun Xiong. 2021. MöbiusE: Knowledge graph embedding on möbius ring. Knowl. Based Syst. 227 (2021), 107181.
[11]
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong, Joshua B. Tenenbaum, and Chuang Gan. 2021. Grounding physical concepts of objects and events through dynamic visual reasoning. In 9th International Conference on Learning Representations, ICLR 2021.
[12]
Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Joshua B. Tenenbaum, Antonio Torralba, and Chuang Gan. 2022. ComPhy: Compositional physical reasoning of objects and events from videos. In 10th International Conference on Learning Representations, ICLR 2022.
[13]
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2D knowledge graph embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI 2018. 1811–1818.
[14]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Volume 1 (Long and Short Papers). 4171–4186.
[15]
Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’14. 601–610.
[16]
Takuma Ebisu and Ryutaro Ichise. 2020. Generalized translation-based embedding of knowledge graph. IEEE Trans. Knowl. Data Eng. 32, 5 (2020), 941–951.
[17]
Alberto García-Durán and Mathias Niepert. 2018. KBlrn: End-to-end learning of knowledge base representations with latent, relational, and numerical features. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018. 372–381.
[18]
Xuqian Huang, Jiuyang Tang, Zhen Tan, Weixin Zeng, Ji Wang, and Xiang Zhao. 2021. Knowledge graph embedding by relational and entity rotation. Knowl. Based Syst. 229 (2021), 107310.
[19]
Dan Jiang, Ronggui Wang, Juan Yang, and Lixia Xue. 2021. Kernel multi-attention neural network for knowledge graph embedding. Knowl. Based Syst. 227 (2021), 107188.
[20]
Thomas N. Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard S. Zemel. 2018. Neural relational inference for interacting systems. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018. 2693–2702.
[21]
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017.
[22]
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI 2015. 2181–2187.
[23]
Hanxiao Liu, Yuexin Wu, and Yiming Yang. 2017. Analogical inference for multi-relational embeddings. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017. 2168–2178.
[24]
Ye Liu, Hui Li, Alberto García-Durán, Mathias Niepert, Daniel Oñoro-Rubio, and David S. Rosenblum. 2019. MMKG: Multi-modal knowledge graphs. In The Semantic Web - 16th International Conference, ESWC 2019. 459–474.
[25]
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Volume 1: Long Papers. 2247–2256.
[26]
Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. 2019. Learning attention-based embeddings for relation prediction in knowledge graphs. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019. 4710–4723.
[27]
Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Q. Phung. 2018. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, Volume 2 (Short Papers). 327–333.
[28]
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011. 809–816.
[29]
Pouya Pezeshkpour, Liyan Chen, and Sameer Singh. 2018. Embedding multimodal relational data for knowledge base completion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, EMNLP 2018. 3208–3218.
[30]
Shengsheng Qian, Jun Hu, Quan Fang, and Changsheng Xu. 2021. Knowledge-aware multi-modal adaptive graph convolutional networks for fake news detection. ACM Trans. Multim. Comput. Commun. Appl. 17, 3 (2021), 98:1–98:23.
[31]
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019. 3980–3990.
[32]
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In The Semantic Web - 15th International Conference, ESWC 2018. 593–607.
[33]
Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. 2019. End-to-end structure-aware convolutional networks for knowledge base completion. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019. 3060–3067.
[34]
Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015.
[35]
Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. YAGO: A core of semantic knowledge. In Proceedings of the 16th International Conference on World Wide Web, WWW 2007. 697–706.
[36]
Rui Sun, Xuezhi Cao, Yan Zhao, Junchen Wan, Kun Zhou, Fuzheng Zhang, Zhongyuan Wang, and Kai Zheng. 2020. Multi-modal knowledge graphs for recommender systems. In CIKM’20: The 29th ACM International Conference on Information and Knowledge Management. 1405–1414.
[37]
Zhiqing Sun, Zhihong Deng, Jianyun Nie, and Jian Tang. 2019. RotatE: Knowledge graph embedding by relational rotation in complex space. In 7th International Conference on Learning Representations, ICLR 2019.
[38]
Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, CVSC 2015. 57–66.
[39]
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceedings of the 33rd International Conference on Machine Learning, ICML 2016. 2071–2080.
[40]
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha P. Talukdar. 2020. Composition-based multi-relational graph convolutional networks. In 8th International Conference on Learning Representations, ICLR 2020.
[41]
Denny Vrandecic and Markus Krötzsch. 2014. Wikidata: A free collaborative knowledgebase. Commun. ACM 57, 10 (2014), 78–85.
[42]
Bo Wu, Sijia Liu, Pin-Yu Chen, Gaoyuan Zhang, and Chuang Gan. 2021. STAR: A benchmark for situated reasoning in real-world videos. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021).
[43]
Han Xiao, Minlie Huang, and Xiaoyan Zhu. 2016. TransG : A generative model for knowledge graph embedding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Volume 1: Long Papers.
[44]
Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2017. Image-embodied knowledge representation learning. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017. 3140–3146.
[45]
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representations, ICLR 2015.
[46]
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. KG-BERT: BERT for knowledge graph completion. CoRR abs/1909.03193 (2019).
[47]
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. 2020. CLEVRER: Collision events for video representation and reasoning. In 8th International Conference on Learning Representations, ICLR 2020.
[48]
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017. 1103–1114.
[49]
Adnan Zeb, Anwar Ul Haqg, Junde Chen, Zhenfeng Lei, and Defu Zhang. 2021. Learning hyperbolic attention-based embeddings for link prediction in knowledge graphs. Knowl. Based Syst. 229 (2021), 107369.
[50]
Yingying Zhang, Quan Fang, Shengsheng Qian, and Changsheng Xu. 2020. Multi-modal multi-relational feature aggregation network for medical knowledge representation learning. In MM’20: The 28th ACM International Conference on Multimedia. 3956–3965.
[51]
Yingying Zhang, Shengsheng Qian, Quan Fang, and Changsheng Xu. 2019. Multi-modal knowledge-aware hierarchical attention network for explainable medical question answering. In Proceedings of the 27th ACM International Conference on Multimedia, MM 2019. 1089–1097.
[52]
Zhaoli Zhang, Zhifei Li, Hai Liu, and Neal N. Xiong. 2022. Multi-scale dynamic convolutional network for knowledge graph embedding. IEEE Trans. Knowl. Data Eng. 34, 5 (2022), 2335–2347.
[53]
Da Zheng, Xiang Song, Chao Ma, Zeyuan Tan, Zihao Ye, Jin Dong, Hao Xiong, Zheng Zhang, and George Karypis. 2020. DGL-KE: Training knowledge graph embeddings at scale. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020. 739–748.
[54]
Zhehui Zhou, Can Wang, Yan Feng, and Defang Chen. 2022. JointE: Jointly utilizing 1D and 2D convolution for knowledge graph embedding. Knowl. Based Syst. 240 (2022), 108100.

Cited By

View all
  • (2024)ShallowBKGC: a BERT-enhanced shallow neural network model for knowledge graph completionPeerJ Computer Science10.7717/peerj-cs.205810(e2058)Online publication date: 15-May-2024
  • (2024)Improvement of Web Semantic and Transformer-Based Knowledge Graph Completion in Low-Dimensional SpacesInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.33691920:1(1-18)Online publication date: 21-Feb-2024
  • (2024)MERGE: A Modal Equilibrium Relational Graph Framework for Multi-Modal Knowledge Graph CompletionSensors10.3390/s2423760524:23(7605)Online publication date: 28-Nov-2024
  • Show More Cited By

Index Terms

  1. Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Multimedia Computing, Communications, and Applications
    ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 2
    March 2023
    540 pages
    ISSN:1551-6857
    EISSN:1551-6865
    DOI:10.1145/3572860
    • Editor:
    • Abdulmotaleb El Saddik
    Issue’s Table of Contents

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 06 February 2023
    Online AM: 30 June 2022
    Accepted: 23 June 2022
    Revised: 25 April 2022
    Received: 26 January 2022
    Published in TOMM Volume 19, Issue 2

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Multi-modal knowledge graph
    2. knowledge graph completion
    3. relational graph attention network
    4. low-rank multi-modal fusion

    Qualifiers

    • Research-article
    • Refereed

    Funding Sources

    • National Natural Science Foundation of China
    • Shenzhen Science and Technology Program
    • Zhejiang Lab’s International Talent Fund for Young Professionals

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)1,107
    • Downloads (Last 6 weeks)44
    Reflects downloads up to 18 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)ShallowBKGC: a BERT-enhanced shallow neural network model for knowledge graph completionPeerJ Computer Science10.7717/peerj-cs.205810(e2058)Online publication date: 15-May-2024
    • (2024)Improvement of Web Semantic and Transformer-Based Knowledge Graph Completion in Low-Dimensional SpacesInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.33691920:1(1-18)Online publication date: 21-Feb-2024
    • (2024)MERGE: A Modal Equilibrium Relational Graph Framework for Multi-Modal Knowledge Graph CompletionSensors10.3390/s2423760524:23(7605)Online publication date: 28-Nov-2024
    • (2024)Link Prediction Based on Feature Mapping and Bi-Directional ConvolutionApplied Sciences10.3390/app1405208914:5(2089)Online publication date: 2-Mar-2024
    • (2024)LAFAProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i8.28744(8957-8965)Online publication date: 20-Feb-2024
    • (2024)Multi-modal Knowledge Graph Completion: A SurveyProceedings of the International Conference on Image Processing, Machine Learning and Pattern Recognition10.1145/3700906.3700925(116-121)Online publication date: 13-Sep-2024
    • (2024)Multimodal Sign Language Knowledge Graph and Representation: Text Video KeyFrames and Motion TrajectoriesProceedings of the 6th ACM International Conference on Multimedia in Asia10.1145/3696409.3700208(1-7)Online publication date: 3-Dec-2024
    • (2024)HKA: A Hierarchical Knowledge Alignment Framework for Multimodal Knowledge Graph CompletionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366428820:8(1-19)Online publication date: 29-Jun-2024
    • (2024)A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-ModalIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.341745146:12(9456-9478)Online publication date: Dec-2024
    • (2024)A Rolling Bearing Fault Diagnosis Method Based on Multimodal Knowledge GraphIEEE Transactions on Industrial Informatics10.1109/TII.2024.343107420:11(13047-13057)Online publication date: Nov-2024
    • Show More Cited By

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    Full Text

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media