Skip to main content

Advertisement

Springer Nature Link
Log in

DIRS-KG: a KG-enhanced interactive recommender system based on deep reinforcement learning

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Recommender systems play a vital role in discovering contents of interest to users in this information explosion era. However, traditional recommender systems only consider user immediate feedback and tend to recommend similar items according to users’ historical interactions. Moreover, in real-world online applications, they lack sufficient user interaction data. In this paper, we propose a novel interactive recommender system by using deep reinforcement learning, which can take both user immediate rewards and future rewards into account. In order to tackle the effect of interaction data insufficiency on recommendation performance, we leverage the knowledge and relation information among items in external knowledge graphs to enrich the item embedding. We concatenate the user representation and user top-l latest historical interactions as the state and feed into the Bi-LSTM model to capture user dynamic preferences. Extensive experiments on two real-world data sets demonstrate the effectiveness and generality of our proposed recommender system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability and Materials

In this paper, the experimental data set MovieLens 1M can be available at https://grouplens.org/datasets/movielens/1m/.

Notes

  1. https://www.scholat.com/

References

  1. Shu, J., Shen, X., Liu, H., Yi, B., Zhang, Z.: A content-based recommendation algorithm for learning resources. Multimed. Syst. 24(2), 163–173 (2018)

    Article  Google Scholar 

  2. Zhang, X., Liu, H., Chen, X., Zhong, J., Wang, D.: A novel hybrid deep recommendation system to differentiate user’s preference and item’s attractiveness. Inform. Sci. 519, 306–316 (2020)

    Article  Google Scholar 

  3. Zhou, S., Dai, X., Chen, H., Zhang, W., Ren, K., Tang, R., He, X., Yu, Y.: Interactive recommender system via knowledge graph-enhanced reinforcement learning. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval SIGIR 2020, pp. 179–188. Virtual Event, China (2020)

  4. Wang, Q., Zeng, C., Zhou, W., Li, T., Iyengar, S.S., Shwartz, L., Grabarnik, G.Y.: Online interactive collaborative filtering using multi-armed bandit with dependent arms. IEEE Trans. Knowl. Data Eng. 31(8), 1569–1580 (2019)

    Article  Google Scholar 

  5. Zeng, C., Wang, Q., Mokhtari, S., Li, T.: Online context-aware recommendation with time varying multi-armed bandit. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2025–2034, San Francisco (2016)

  6. Wang, X., Wang, Y., Hsu, D., Wang, Y.: Exploration in interactive personalized music recommendation: a reinforcement learning approach. ACM Trans. Multimed. Comput. Commun. Appl. 11(1) (2014)

  7. Xin, X., Karatzoglou, A., Arapakis, I., Jose, J.M.: Self-supervised reinforcement learning for recommender systems. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020, pp. 931–940. Virtual Event, China (2020)

  8. Lei, Y., Li, W.: Interactive recommendation with user-specific deep reinforcement learning. ACM Trans. Knowl. Discov. Data 13(6) (2019)

  9. Zhao, X., Xia, L., Zhang, L., Ding, Z., Yin, D., Tang, J.: Deep reinforcement learning for page-wise recommendations. In: Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, pp. 95–103, Vancouver (2018)

  10. Zheng, G., Zhang, F., Zheng, Z., Xiang, Y., Yuan, N.J., Xie, X., Li, Z.: DRN: a deep reinforcement learning framework for news recommendation. In: Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, pp. 167–176, Lyon (2018)

  11. Huang, L., Fu, M., Li, F., Qu, H., Liu, Y., Chen, W.: A deep reinforcement learning based long-term recommender system. Knowl.-Based Syst. 213, 106706 (2021)

    Article  Google Scholar 

  12. Wang, H., Zhang, F., Wang, J., Zhao, M., Li, W., Xie, X., Guo, M.: Ripplenet: propagating user preferences on the knowledge graph for recommender systems. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, pp. 417–426, Torino (2018)

  13. Wang, X., He, X., Cao, Y., Liu, M., Chua, T.: KGAT: knowledge graph attention network for recommendation. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, pp. 950–958, Anchorage (2019)

  14. Wang, Q., Mao, Z., Wang, B., Guo, L.: Knowledge graph embedding: a survey of approaches and applications. IEEE Trans. Knowl. Data Eng. 29(12), 2724–2743 (2017)

    Article  Google Scholar 

  15. Yao, L., Sheng, Q.Z., Ngu, A.H.H., Yu, J., Segev, A.: Unified collaborative and content-based web service recommendation. IEEE Trans. Serv. Comput. 8(3), 453–466 (2015)

    Article  Google Scholar 

  16. Cui, Z., Xu, X., XUE, F., Cai, X., Cao, Y., Zhang, W., Chen, J.: Personalized recommendation system based on collaborative filtering for iot scenarios. IEEE Trans. Serv. Comput. 13(4), 685–695 (2020)

    Article  Google Scholar 

  17. Nassar, N., Jafar, A., Rahhal, Y.: A novel deep multi-criteria collaborative filtering model for recommendation system. Knowl.-Based Syst. 187, 104811 (2020)

    Article  Google Scholar 

  18. Cai, X., Hu, Z., Chen, J.: A many-objective optimization recommendation algorithm based on knowledge mining. Inform. Sci. 537, 148–161 (2020)

    Article  MathSciNet  Google Scholar 

  19. Lu, Z., Dou, Z., Lian, J., Xie, X., Yang, Q.: Content-based collaborative filtering for news topic recommendation. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence, 2015, pp. 217–223, Austin (2015)

  20. Fu, M., Qu, H., Yi, Z., Lu, L., Liu, Y.: A novel deep learning-based collaborative filtering model for recommendation system. IEEE Trans. Cybern. 49(3), 1084–1096 (2019)

    Article  Google Scholar 

  21. Wan, S., Niu, Z.: A hybrid e-learning recommendation approach based on learners’ influence propagation. IEEE Trans. Knowl. Data Eng. 32(5), 827–840 (2020)

    Article  Google Scholar 

  22. Silva, N., Werneck, H., Silva, T., Pereira, A.C.M., Rocha, L.: Multi-armed bandits in recommendation systems: a survey of the state-of-the-art and future directions. Expert Syst. Appl. 197, 116669 (2022)

    Article  Google Scholar 

  23. Tang, L., Jiang, Y., Li, L., Zeng, C., Li, T.: Personalized recommendation via parameter-free contextual bandits. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 323–332, Santiago (2015)

  24. Zhang, J., Hao, B., Chen, B., Li, C., Chen, H., Sun, J.: Hierarchical reinforcement learning for course recommendation in moocs. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 435–442, Honolulu (2019)

  25. Hu, Y., Da, Q., Zeng, A., Yu, Y., Xu, Y.: Reinforcement learning to rank in e-commerce search engine: formalization, analysis, and application. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, pp. 368–377, London (2018)

  26. Liu, F., Tang, R., Li, X., Zhang, W., Ye, Y., Chen, H., Guo, H., Zhang, Y., He, X.: State representation modeling for deep reinforcement learning based recommendation. Knowl.-Based Syst. 205, 106170 (2020)

    Article  Google Scholar 

  27. Lei, Y., Wang, Z., Li, W., Pei, H., Dai, Q.: Social attentive deep q-networks for recommender systems. IEEE Trans. Knowl. Data Eng. 34 (5), 2443–2457 (2022)

    Article  Google Scholar 

  28. Krishnan, A., Sharma, A., Sundaram, H.: Insights from the long-tail: learning latent representations of online user behavior in the presence of skew and sparsity. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, pp. 297–306, Torino (2018)

  29. Shi, C., Zhang, Z., Luo, P., Yu, P.S., Yue, Y., Wu, B.: Semantic path based personalized recommendation on weighted heterogeneous information networks. In: Proceedings of the 24th ACM International Conference on Information and Knowledge Management, CIKM 2015, pp. 453–462, Melbourne (2015)

  30. Yu, X., Ren, X., Sun, Y., Gu, Q., Sturt, B., Khandelwal, U., Norick, B., Han, J.: Personalized entity recommendation: a heterogeneous information network approach. In: Proceedings of the 7th ACM International Conference on Web Search and Data Mining, WSDM 2014, pp. 283–292, New York (2014)

  31. Zhao, H., Yao, Q., Li, J., Song, Y., Lee, D.L.: Meta-graph based recommendation fusion over heterogeneous information networks. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 635–644, Halifax (2017)

  32. Bordes, A., Usunier, N., García-Durán, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Proceedings of the 27th Annual Conference on Neural Information Processing Systems 2013, pp. 2787–2795, Lake Tahoe (2013)

  33. Ji, G., He, S., Xu, L., Liu, K., Zhao, J.: Knowledge graph embedding via dynamic mapping matrix. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, pp. 687–696, Beijing (2015)

  34. Xiao, H., Huang, M., Zhu, X.: Transg: a generative model for knowledge graph embedding. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin (2016)

  35. Zhang, F., Yuan, N.J., Lian, D., Xie, X., Ma, W.: Collaborative knowledge base embedding for recommender systems. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 353–362, San Francisco (2016)

  36. Huang, J., Zhao, W.X., Dou, H., Wen, J., Chang, E.Y.: Improving sequential recommendation with knowledge-enhanced memory networks. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, pp. 505–514, Ann Arbor (2018)

  37. Zhao, M., Huang, X., Zhu, L., Sang, J., Yu, J.: Knowledge graph-enhanced sampling for conversational recommendation system. IEEE Trans. Knowl. Data Eng.:1–14 (2022)

  38. Li, D., Zhang, N., Dong, M., Chen, H., Ota, K., Tang, Y.: Pm-aio: an effective asynchronous i/o system for persistent memory. IEEE Trans. Emerg. Top. Comput.:1–1 (2021)

  39. Tang, Y., Lin, R., Li, D., Li, Y., Zeng, D.: Fsbrain: an intelligent i/o performance tuning system. J. Syst. Archit. 129, 102623 (2022)

    Article  Google Scholar 

  40. Hasselt, H.V., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. In: Proceedings of the 13th AAAI Conference on Artificial Intelligence, February 12-17, 2016, pp. 2094–2100, Phoenix (2016)

  41. Wang, Z., Schaul, T., Hessel, M., Hasselt, H.V., Lanctot, M., de Freitas, N.: Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33nd International Conference on Machine Learning, ICML 2016. JMLR Workshop and Conference Proceedings, vol. 48, pp. 1995–2003, New York City (2016)

  42. Jalili, M., Ahmadian, S., Izadi, M., Moradi, P., Salehi, M.: Evaluating collaborative filtering recommender algorithms: a survey. IEEE Access 6, 74003–74024 (2018). https://doi.org/10.1109/ACCESS.2018.2883742

    Article  Google Scholar 

  43. Li, L., Chu, W., Langford, J., Schapire, R.E.: A contextual-bandit approach to personalized news article recommendation. In: Proceedings of the 19th International Conference on World Wide Web, WWW 2010, pp. 661–670, Raleigh (2010)

  44. Wang, H., Wu, Q., Wang, H.: Learning hidden features for contextual bandits. In: Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, pp. 1633–1642, Indianapolis (2016)

  45. Dacrema, M.F., Cremonesi, P., Jannach, D.: Are we really making much progress? a worrying analysis of recent neural recommendation approaches. In: Proceedings of the 13th ACM Conference on Recommender Systems. Recsys ’19, pp. 101–109, New York (2019)

  46. Pham, M., Nguyen, H., Dang, L., Nieves, J.A.: Compressive features in offline reinforcement learning for recommender systems. In: 2021 IEEE International Conference on Big Data (Big Data), pp. 5719–5726, Orlando (2021)

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China under Grant U1811263 and Grant 62077045, in part by the Humanity and Social Science Youth Foundation of Ministry of Education of China under Grant 19YJCZH049, in part by the Natural Science Foundation of Guangdong Province of China under Grant 2019A1515011292.

Author information

Authors and Affiliations

  1. School of Computer Science, South China Normal University, Guangzhou, 510631, Guangdong, China

    Ronghua Lin, Chaobo He, Zhengyang Wu & Yong Tang

  2. Pazhou Lab, Guangzhou, 510330, Guangdong, China

    Ronghua Lin, Feiyi Tang, Chaobo He, Chengzhe Yuan & Yong Tang

  3. School of Information Engineering, Guangzhou Panyu Polytechnic, Guangzhou, 511483, Guangdong, China

    Feiyi Tang

  4. School of Electronics and Information, Guangdong Polytechnic Normal University, Guangzhou, 510665, Guangdong, China

    Chengzhe Yuan

Authors
  1. Ronghua Lin
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Feiyi Tang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Chaobo He
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Zhengyang Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Chengzhe Yuan
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Yong Tang
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

R. L. and F. T. wrote the main manuscript text and proposed the method in the manuscript. C. H. and Z. W. conducted the formal analysis and investigation. C. Y. was devoted to data visualization in this manuscript. Y. T. reviewed the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Feiyi Tang.

Ethics declarations

Ethics approval and consent to participate

The authors declare that this paper is original and has not been published elsewhere previously.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Special Issue on Knowledge-Graph-Enabled Methods and Applications for the Future Web Guest Editors: Xin Wang, Jeff Pan, Qingpeng Zhang, Yuan-Fang Li

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, R., Tang, F., He, C. et al. DIRS-KG: a KG-enhanced interactive recommender system based on deep reinforcement learning. World Wide Web 26, 2471–2493 (2023). https://doi.org/10.1007/s11280-022-01135-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-022-01135-x

Keywords