Skip to main content
SpringerLink
Log in

ELECTRA-KG: A Transformer-Knowledge Graph Recommender System

  • Conference paper
  • First Online:
Services Computing – SCC 2022 (SCC 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13738))

Included in the following conference series:

  • 300 Accesses

Abstract

Knowledge graphs (KGs) are becoming popular in recommender systems in recent times because of the wealth of side information they provide. Many researchers rely on KGs to help resolve the issues of cold start, diversity, and explainability in recommendations. However, the existing approaches usually ignore entity descriptions, which are essential in providing content information for entities in KGs. In this work, we propose a contextual language model for KG completion known as ELECTRA-KG (Efficiently Learning an Encoder that Classifies Token Replacements Accurately). We formulate the recommendation task as a KG link prediction task where we have an incomplete knowledge graph and we use state-of-the-art approaches to complete it. We do this by identifying missing facts among entities from our test data. To evaluate and validate our method, we perform a couple of experiments. First, we run experiments to demonstrate how well our model compares to state-of-the-art KG embedding models. Second, we run further experiments with our model on the tag recommendation task and compare our results to existing baselines. Our results show that our model outperforms the existing baselines on the tag recommendation task.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amer, A.A., Abdalla, H.I., Nguyen, L.: Enhancing recommendation systems performance using highly-effective similarity measures. Knowl.-Based Syst. 217, 106842 (2021). https://doi.org/10.1016/j.knosys.2021.106842, https://www.sciencedirect.com/science/article/pii/S0950705121001052

  2. Bordes, A., Glorot, X., Weston, J., Bengio, Y.: A semantic matching energy function for learning with multi-relational data. Mach. Learn. 94(2), 233–259 (2013). https://doi.org/10.1007/s10994-013-5363-6

    Article  MathSciNet  MATH  Google Scholar 

  3. Brown, T.B., et al.: Language models are few-shot learners. CoRR abs/2005.14165 (2020). https://arxiv.org/abs/2005.14165

  4. Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794 (2016)

    Google Scholar 

  5. Clark, K., Luong, M., Le, Q.V., Manning, C.D.: ELECTRA: pre-training text encoders as discriminators rather than generators. CoRR abs/2003.10555 (2020). https://arxiv.org/abs/2003.10555

  6. Dettmers, T., Minervini, P., Stenetorp, P., Riedel, S.: Convolutional 2D knowledge graph embeddings. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI 2018/IAAI 2018/EAAI 2018. AAAI Press (2018)

    Google Scholar 

  7. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis (2019). https://doi.org/10.18653/v1/N19-1423, https://aclanthology.org/N19-1423

  8. Dong, L., Wei, F., Zhou, M., Xu, K.: Question answering over freebase with multi-column convolutional neural networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 260–269. Association for Computational Linguistics (2015). https://doi.org/10.3115/v1/P15-1026

  9. Fletcher, K.K., Liu, X.F.: A collaborative filtering method for personalized preference-based service recommendation. In: 2015 IEEE International Conference on Web Services, pp. 400–407 (2015). https://doi.org/10.1109/ICWS.2015.60

  10. Fletcher, K.: Regularizing matrix factorization with implicit user preference embeddings for web API recommendation. In: 2019 IEEE International Conference on Services Computing (SCC), pp. 1–8. IEEE (2019)

    Google Scholar 

  11. Fletcher, K.K.: A method for dealing with data sparsity and cold-start limitations in service recommendation using personalized preferences. In: 2017 IEEE International Conference on Cognitive Computing (ICCC), pp. 72–79. IEEE (2017)

    Google Scholar 

  12. Fletcher, K.K.: A quality-based web API selection for mashup development using affinity propagation. In: Ferreira, J.E., Spanoudakis, G., Ma, Y., Zhang, L.-J. (eds.) SCC 2018. LNCS, vol. 10969, pp. 153–165. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-94376-3_10

    Chapter  Google Scholar 

  13. Fletcher, K.K.: A quality-aware web API recommender system for mashup development. In: Ferreira, J.E., Musaev, A., Zhang, L.-J. (eds.) SCC 2019. LNCS, vol. 11515, pp. 1–15. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23554-3_1

    Chapter  Google Scholar 

  14. Ji, G., He, S., Xu, L., Liu, K., Zhao, J.: Knowledge graph embedding via dynamic mapping matrix. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 687–696. Association for Computational Linguistics, Beijing (2015). https://doi.org/10.3115/v1/P15-1067, https://aclanthology.org/P15-1067

  15. Kar, S., Maharjan, S., López-Monroy, A.P., Solorio, T.: MPST: a corpus of movie plot synopses with tags. In: et al. (eds.) Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Paris (2018)

    Google Scholar 

  16. Kertkeidkachorn, N., Ichise, R.: T2KG: an end-to-end system for creating knowledge graph from unstructured text. In: AAAI Workshops (2017)

    Google Scholar 

  17. Kibriya, A.M., Frank, E., Pfahringer, B., Holmes, G.: Multinomial naive bayes for text categorization revisited. In: Webb, G.I., Yu, X. (eds.) AI 2004. LNCS (LNAI), vol. 3339, pp. 488–499. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30549-1_43

    Chapter  Google Scholar 

  18. Kwapong, B., Anarfi, R., Fletcher, K.K.: A knowledge graph approach to mashup tag recommendation. In: 2020 IEEE International Conference on Services Computing (SCC), pp. 92–99 (2020). https://doi.org/10.1109/SCC49832.2020.00021

  19. Kwapong, B., Fletcher, K.: A knowledge graph based framework for web API recommendation. In: 2019 IEEE World Congress on Services (SERVICES), vol. 2642–939X, pp. 115–120 (2019). https://doi.org/10.1109/SERVICES.2019.00039

  20. Kwapong, B.A., Anarfi, R., Fletcher, K.K.: Personalized service recommendation based on user dynamic preferences. In: Ferreira, J.E., Musaev, A., Zhang, L.-J. (eds.) SCC 2019. LNCS, vol. 11515, pp. 77–91. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23554-3_6

    Chapter  Google Scholar 

  21. Kwapong, B.A., Anarfi, R., Fletcher, K.K.: Collaborative learning using LSTM-RNN for personalized recommendation. In: Wang, Q., Xia, Y., Seshadri, S., Zhang, L.-J. (eds.) SCC 2020. LNCS, vol. 12409, pp. 35–49. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59592-0_3

    Chapter  Google Scholar 

  22. Lin, Y., Liu, Z., Sun, M., Liu, Y., Zhu, X.: Learning entity and relation embeddings for knowledge graph completion. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29, no. 1 (2015). https://ojs.aaai.org/index.php/AAAI/article/view/9491

  23. Liu, Y., et al.: Roberta: A robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019). http://arxiv.org/abs/1907.11692

  24. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Proceedings of the 26th International Conference on Neural Information Processing Systems, NIPS 2013, vol. 2, pp. 3111–3119. Curran Associates Inc., USA (2013)

    Google Scholar 

  25. Nickel, M., Tresp, V., Kriegel, H.P.: A three-way model for collective learning on multi-relational data. In: Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML 2011, pp. 809–816. Omnipress, Madison (2011)

    Google Scholar 

  26. Nickel, M., Tresp, V., Kriegel, H.P.: Factorizing yago: scalable machine learning for linked data. In: Proceedings of the 21st International Conference on World Wide Web, WWW 2012, pp. 271–280. Association for Computing Machinery, New York (2012). https://doi.org/10.1145/2187836.2187874, https://doi-org.ezproxy.lib.umb.edu/10.1145/2187836.2187874

  27. Palumbo, E., Rizzo, G., Troncy, R.: Entity2rec: learning user-item relatedness from knowledge graphs for top-n item recommendation. In: Proceedings of the Eleventh ACM Conference on Recommender Systems, RecSys 2017, pp. 32–36. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3109859.3109889, https://doi-org.ezproxy.lib.umb.edu/10.1145/3109859.3109889

  28. Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543. Association for Computational Linguistics, Doha (2014). https://doi.org/10.3115/v1/D14-1162, https://aclanthology.org/D14-1162

  29. Rennie, J.D., Shih, L., Teevan, J., Karger, D.R.: Tackling the poor assumptions of naive bayes text classifiers. In: Proceedings of the 20th International Conference on Machine Learning (ICML-2003), pp. 616–623 (2003)

    Google Scholar 

  30. Socher, R., Chen, D., Manning, C.D., Ng, A.Y.: Reasoning with neural tensor networks for knowledge base completion. In: Proceedings of the 26th International Conference on Neural Information Processing Systems, NIPS 2013, vol. 1, pp. 926–934. Curran Associates Inc., Red Hook (2013)

    Google Scholar 

  31. Wang, H., Kulkarni, V., Wang, W.Y.: Dolores: deep contextualized knowledge graph embeddings. ArXiv abs/1811.00147 (2020)

    Google Scholar 

  32. Wang, H., et al.: RippleNet: propagating user preferences on the knowledge graph for recommender systems. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, pp. 417–426. ACM, New York (2018). https://doi.org/10.1145/3269206.3271739

  33. Wang, X., Wu, H., Hsu, C.: Mashup-oriented API recommendation via random walk on knowledge graph. IEEE Access 7, 7651–7662 (2019). https://doi.org/10.1109/ACCESS.2018.2890156

    Article  Google Scholar 

  34. Wang, X., Wang, D., Xu, C., He, X., Cao, Y., Chua, T.: Explainable reasoning over knowledge graphs for recommendation. CoRR abs/1811.04540 (2018). http://arxiv.org/abs/1811.04540

  35. Wang, Z., Zhang, J., Feng, J., Chen, Z.: Knowledge graph embedding by translating on hyperplanes. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 28, no. 1 (2014). https://ojs.aaai.org/index.php/AAAI/article/view/8870

  36. Xu, C., et al.: RC-net: a general framework for incorporating knowledge into word representations. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, pp. 1219–1228. ACM, New York (2014). https://doi.org/10.1145/2661829.2662038

  37. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’ Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019). https://proceedings.neurips.cc/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf

  38. Yao, L., Mao, C., Luo, Y.: KG-BERT: BERT for knowledge graph completion. CoRR abs/1909.03193 (2019). http://arxiv.org/abs/1909.03193

  39. Zhang, F., Yuan, N.J., Lian, D., Xie, X., Ma, W.Y.: Collaborative knowledge base embedding for recommender systems. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2016, pp. 353–362. Association for Computing Machinery, New York (2016). https://doi.org/10.1145/2939672.2939673, https://doi-org.ezproxy.lib.umb.edu/10.1145/2939672.2939673

  40. Zhang, M.L., Zhou, Z.H.: ML-KNN: a lazy learning approach to multi-label learning. Pattern Recogn. 40(7), 2038–2048 (2007)

    Article  MATH  Google Scholar 

  41. Zhang, Y., Ai, Q., Chen, X., Wang, P.: Learning over knowledge-base embeddings for recommendation. ArXiv abs/1803.06540 (2018)

    Google Scholar 

  42. Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., Liu, Q.: ERNIE: enhanced language representation with informative entities. In: ACL (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Massachusetts Boston, Boston, MA, 02125, USA

    Benjamin Kwapong & Kenneth K. Fletcher

  2. Oakland University, Rochester, MI, 48309, USA

    Amartya Sen

Authors
  1. Benjamin Kwapong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amartya Sen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kenneth K. Fletcher
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kenneth K. Fletcher .

Editor information

Editors and Affiliations

  1. Louisiana State University, Baton Rouge, LA, USA

    Wang Qingyang

  2. Kingdee International Software Group Co., Ltd., Shenzhen, China

    Liang-Jie Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kwapong, B., Sen, A., Fletcher, K.K. (2022). ELECTRA-KG: A Transformer-Knowledge Graph Recommender System. In: Qingyang, W., Zhang, LJ. (eds) Services Computing – SCC 2022. SCC 2022. Lecture Notes in Computer Science, vol 13738. Springer, Cham. https://doi.org/10.1007/978-3-031-23515-3_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-23515-3_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-23514-6

  • Online ISBN: 978-3-031-23515-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation