Abstract
Knowledge graph completion (KGC) is the task of inferencing missing facts from any given knowledge graphs (KG). Previous KGC methods typically represent knowledge graph entities and relations as trainable continuous embeddings and fuse the embeddings of the entity h (or t) and relation r into hidden representations of query (h, r, ?) (or (?, r, t)) to approximate the missing entities. To achieve this, they either use shallow linear transformations or deep convolutional modules. However, the linear transformations suffer from the expressiveness issue while the deep convolutional modules introduce unnecessary inductive bias, which could potentially degrade the model performance. Thus, we propose a novel Transformer-based Patch Refinement Model (PatReFormer) for KGC. PatReFormer first segments the embedding into a sequence of patches and then employs cross-attention modules to allow bi-directional embedding feature interaction between the entities and relations, leading to a better understanding of the underlying KG. We conduct experiments on four popular KGC benchmarks, WN18RR, FB15k-237, YAGO37 and DB100K. The experimental results show significant performance improvement from existing KGC methods on standard KGC evaluation metrics, e.g., MRR and H@n. Our analysis first verifies the effectiveness of our model design choices in PatReFormer. We then find that PatReFormer can better capture KG information from a large relation embedding dimension. Finally, we demonstrate that the strength of PatReFormer is at complex relation types, compared to other KGC models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: NIPS, pp. 1–9 (2013)
Yang, B., Yih, W.T., He, X., Gao, J., Deng, L.: Embedding entities and relations for learning and inference in knowledge bases. In: ICLR, San Diego, CA, USA (2015)
Trouillon, T., Welbl, J., Riedel, S., Gaussier, È., Bouchard, G.: Complex embeddings for simple link prediction. In: ICML, pp. 2071–2080. PMLR (2016)
Schlichtkrull, M., Kipf, T.N., Bloem, P., van den Berg, R., Titov, I., Welling, M.: Modeling relational data with graph convolutional networks. In: Gangemi, A., et al. (eds.) ESWC 2018. LNCS, vol. 10843, pp. 593–607. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93417-4_38
Shang, C., Tang, Y., Huang, J., Bi, J., He, X., Zhou, B.: End-to-end structure-aware convolutional networks for knowledge base completion. Proc. AAAI 33, 3060–3067 (2019)
Jiang, X., Wang, Q., Wang, B.: Adaptive convolution for multi-relational learning. In: Proceedings of the 2019 NAACL-HIT, pp. 978–987 (2019)
Sun, Z., Deng, Z.-H., Nie, J.-Y., Tang, J.: Rotate: Knowledge Graph Embedding by Relational Rotation In Complex Space, p. 2019. ICLR, New Orleans, LA, USA, May (2019)
Dettmers, T., Minervini, P., Stenetorp, P., Riedel, S.: Convolutional 2D knowledge graph embeddings. In: Proceedings of the AAAI, vol. 32 (2018)
Vashishth, S., Sanyal, S., Nitin, V., Agrawal, N., Talukdar, P.: Interacte: improving convolution-based knowledge graph embeddings by increasing feature interactions. In AAAI 34, 3009–3016 (2020)
Ren, F., et al.: Knowledge graph embedding with atrous convolution and residual learning. In: Proceedings of COLING, Barcelona, Spain (Online), December 8–13, pp. 1532–1543 (2020)
Yao, L., Mao, C., Luo, Y.: KG-BERT: BERT for knowledge graph completion. CoRR, abs/1909.03193 (2019)
Kim, B., Hong, T., Ko, Y., Seo, J.: Multi-task learning for knowledge graph completion with pre-trained language models. In: Proceedings of the 28th COLING 2020, Barcelona, Spain (Online), December 8–13, 2020, pp. 1737–1743 (2020)
Wang, B., Shen, T., Long, G., Zhou, T., Wang, Y., Chang, Y.: Structure-augmented text representation learning for efficient knowledge graph completion. In WWW ’21, Virtual Event / Ljubljana, Slovenia, April 19–23, 2021, pp. 1737–1748 (2021)
Xin Xie, X., et al.: From discrimination to generation: Knowledge graph completion with generative transformer. In: WWW’22, Lyon, France, April 2022, pp. 162–165. ACM (2022)
Nickel, M., Rosasco, L., Poggio, T.: Holographic embeddings of knowledge graphs. In: Proceedings of the AAAI, vol. 30 (2016)
Liu, H., Wu, Y., Yang, Y.: Analogical inference for multi-relational embeddings. In: ICML, pp. 2168–2178. PMLR (2017)
Xu, W., Zheng, S., He, L., Shao, B., Yin, J., Liu, T.Y.: SEEK: segmented embedding of knowledge graphs. In: Proceedings of the 58th ACL 2020, Online, July 5–10, 2020, pp. 3888–3897 (2020)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: 3rd ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings (2015)
Toutanova, K., Chen, D.: Observed versus latent features for knowledge base and text inference. In: Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pp. 57–66 (2015)
Ding, B., Wang, Q., Wang, B., Guo, L.: Improving knowledge graph embedding using simple constraints. In: Proceedings of the 56th ACL 2018, Melbourne, Australia, July 15–20, 2018, Volume 1: Long Papers, pp. 110–121 (2018)
Guo, S., Wang, Q., Wang, L., Wang, B., Guo, L.: Knowledge graph embedding with iterative guidance from soft rules. In: Proceedings of the AAAI, vol. 32 (2018)
Bizer, C., et al.: Dbpedia-a crystallization point for the web of data. J. Web Semantics 7(3), 154–165 (2009)
Mahdisoltani, F., Biega, J., Suchanek, F.M.: Yago3: A knowledge base from multilingual wikipedias. In: 7th CIDR Conference (2014)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4–9, 2017, Long Beach, CA, USA, pp. 5998–6008 (2017)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 NAACL: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2–7, 2019, Volume 1 (Long and Short Papers), pp. 4171–4186 (2019)
Liu, Y., et al.: Roberta: A robustly optimized BERT pretraining approach (2019). CoRR, abs/1907.11692
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21, 140:1–140:67 (2020)
Dosovitskiy, A. et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: 9th ICLR 2021, Virtual Event, Austria, May 3–7 (2021)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on CVPR 2016, Las Vegas, NV, USA, June 27–30, 2016, pp. 770–778. IEEE Computer Society (2016)
Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization (2016). CoRR, abs/1607.06450
Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: NIPS 2019(December), pp. 8–14,: Vancouver, pp. 13–23. BC, Canada (2019)
Radford, A.: Learning transferable visual models from natural language supervision. In: ICML 2021 Jul 1 (pp. 8748–8763). PMLR (2021)
Lin, Y., Liu, Z., Sun, M., Liu, Y., Zhu, X.: Learning entity and relation embeddings for knowledge graph completion. In: Proceedings of AAAI, January 25–30, 2015, Austin, Texas, USA, pp. 2181–2187. AAAI Press (2015)
Wang, Z., Zhang, J., Feng, J., Chen, Z.: Knowledge graph embedding by translating on hyperplanes. In: Proceedings of AAAI, July 27–31, 2014, Québec City, Québec, Canada, pp. 1112–1119. AAAI Press
Dong, X.: Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In: Proceedings of the 20th ACM SIGKDD, pp. 601–610 (2014)
Ravishankar, S., Talukdar, P.P: Revisiting simple neural networks for learning representations of knowledge graphs. In: 6th AKBC@NIPS 2017, Long Beach, California, USA, December 8 (2017)
Parmar, N., et al.: Image transformer. In: ICML, pp. 4055–4064. PMLR (2018)
Acknowledgement
This research is supported by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative and Strategic Capability Research Centres Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, C., Wang, Y., Zhang, Y., Sheng, Q.Z., Lam, KY. (2023). Separate-and-Aggregate: A Transformer-Based Patch Refinement Model for Knowledge Graph Completion. In: Yang, X., et al. Advanced Data Mining and Applications. ADMA 2023. Lecture Notes in Computer Science(), vol 14177. Springer, Cham. https://doi.org/10.1007/978-3-031-46664-9_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-46664-9_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46663-2
Online ISBN: 978-3-031-46664-9
eBook Packages: Computer ScienceComputer Science (R0)