research-article

KERL: A Knowledge-Guided Reinforcement Learning Model for Sequential Recommendation

Authors:

Wayne Xin Zhao,

Jimmy HuangAuthors Info & Claims

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 209 - 218

https://doi.org/10.1145/3397271.3401134

Published: 25 July 2020 Publication History

Abstract

For sequential recommendation, it is essential to capture and predict future or long-term user preference for generating accurate recommendation over time. To improve the predictive capacity, we adopt reinforcement learning (RL) for developing effective sequential recommenders. However, user-item interaction data is likely to be sparse, complicated and time-varying. It is not easy to directly apply RL techniques to improve the performance of sequential recommendation.

Inspired by the availability of knowledge graph (KG), we propose a novel Knowledge-guidEd Reinforcement Learning model (KERL for short) for fusing KG information into a RL framework for sequential recommendation. Specifically, we formalize the sequential recommendation task as a Markov Decision Process (MDP), and make three major technical extensions in this framework, including state representation, reward function and learning algorithm. First, we propose to enhance the state representations with KG information considering both exploitation and exploration. Second, we carefully design a composite reward function that is able to compute both sequence- and knowledge-level rewards. Third, we propose a new algorithm for more effectively learning the proposed model. To our knowledge, it is the first time that knowledge information has been explicitly discussed and utilized in RL-based sequential recommenders, especially for the exploration process. Extensive experiment results on both next-item and next-session recommendation tasks show that our model can significantly outperform the baselines on four real-world datasets.

Supplementary Material

MP4 File (3397271.3401134.mp4)

Presentation video

Download
22.79 MB

References

[1]

Xueying Bai, Jian Guan, and Hongning Wang. 2019. A Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation. In NeurIPS. 10734--10745.

[2]

Antoine Bordes, Nicolas Usunier, Alberto Garc'i a-Durá n, Jason Weston, and Oksana Yakhnenko. 2013a. Translating Embeddings for Modeling Multi-relational Data. In NIPS. 2787--2795.

Digital Library

[3]

Antoine Bordes, Nicolas Usunier, Alberto García-Durá n, Jason Weston, and Oksana Yakhnenko. 2013b. Translating Embeddings for Modeling Multi-relational Data. In NIPS. 2787--2795.

[4]

Kyunghyun Cho, Bart van Merrienboer, cC aglar Gü lcc ehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In EMNLP. 1724--1734.

[5]

Tim Donkers, Benedikt Loepp, and Jü rgen Ziegler. 2017. Sequential User-based Recurrent Neural Network Recommendations. In RecSys. 152--160.

[6]

Yue Feng, Jun Xu, Yanyan Lan, Jiafeng Guo, Wei Zeng, and Xueqi Cheng. 2018. From Greedy Selection to Exploratory Decision-Making: Diverse Ranking with Policy-Value Networks. In SIGIR. 125--134.

[7]

Gaole He, Junyi Li, Wayne Xin Zhao, Peiju Liu, and Ji-Rong Wen. 2020. Mining Implicit Entity Preference from User-Item Interaction Data for Knowledge Graph Completion via Adversarial Learning. In WWW. 740--751.

[8]

Ruining He and Julian J. McAuley. 2016. Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering. In WWW. 507--517.

[9]

Balazs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016a. Session-based Recommendations with Recurrent Neural Networks. In ICLR.

[10]

Balázs Hidasi, Massimo Quadrana, Alexandros Karatzoglou, and Domonkos Tikk. 2016b. Parallel Recurrent Neural Network Architectures for Feature-rich Session-based Recommendations. In RecSys. 241--248.

[11]

Jin Huang, Zhaochun Ren, Wayne Xin Zhao, Gaole He, Ji-Rong Wen, and Daxiang Dong. 2019. Taxonomy-Aware Multi-Hop Reasoning Networks for Sequential Recommendation. In WSDM. 573--581.

[12]

Jin Huang, Wayne Xin Zhao, Hong-Jian Dou, Ji-Rong Wen, and Edward Y. Chang. 2018. Improving Sequential Recommendation with Knowledge-Enhanced Memory Networks. In SIGIR. 505--514.

[13]

Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P. Xing. 2019. Rethinking Knowledge Graph Propagation for Zero-Shot Learning. In CVPR. 11487--11496.

[14]

Wang-Cheng Kang and Julian J. McAuley. 2018. Self-Attentive Sequential Recommendation. In ICDM. 197--206.

[15]

Yehuda Koren, Robert M. Bell, and Chris Volinsky. 2009. Matrix Factorization Techniques for Recommender Systems. IEEE Computer, Vol. 42, 8 (2009), 30--37.

Digital Library

[16]

Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017. Neural Attentive Session-based Recommendation. In CIKM. 1419--1428.

[17]

Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning Entity and Relation Embeddings for Knowledge Graph Completion. In AAAI. 2181--2187.

[18]

Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2016. A Review of Relational Machine Learning for Knowledge Graphs. Proc. IEEE, Vol. 104, 1 (2016), 11--33.

[19]

Rajiv Pasricha and Julian McAuley. 2018. Translation-based factorization machines for sequential recommendation. In RecSys. 63--71.

[20]

Massimo Quadrana, Alexandros Karatzoglou, Balázs Hidasi, and Paolo Cremonesi. 2017. Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks. In RecSys. 130--137.

[21]

Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence Level Training with Recurrent Neural Networks. In ICLR.

[22]

Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010. Factorizing Personalized Markov Chains for Next-basket Recommendation. In WWW. 811--820.

[23]

Markus Schedl. 2016. The LFM-1b Dataset for Music Retrieval and Recommendation. In ICMR. 103--110.

[24]

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, and George van den Driessche et.al. 2016. Mastering the game of Go with deep neural networks and tree search. Nature, Vol. 529, 7587 (2016), 484--489.

[25]

Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement learning - an introduction. MIT Press, Cambridge, MA, 2018.

Digital Library

[26]

Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo. 2018. RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems. In CIKM. 417--426.

[27]

Pengfei Wang, Jiafeng Guo, Yanyan Lan, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2015.

[28]

Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019. KGAT: Knowledge Graph Attention Network for Recommendation. In KDD. 950--958.

Digital Library

[29]

Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander J. Smola, and How Jing. 2017. Recurrent Recommender Networks. In WSDM. 495--503.

[30]

Yikun Xian, Zuohui Fu, S. Muthukrishnan, Gerard de Melo, and Yongfeng Zhang. 2019. Reinforcement Knowledge Graph Reasoning for Explainable Recommendation. In SIGIR. 285--294.

[31]

Feng Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2016. A Dynamic Recurrent Model for Next Basket Recommendation. In SIGIR. 729--732.

[32]

Qingheng Zhang, Zequn Sun, Wei Hu, Muhao Chen, Lingbing Guo, and Yuzhong Qu. 2019. Multi-view Knowledge Graph Embedding for Entity Alignment. In IJCAI. 5429--5435.

[33]

Wayne Xin Zhao, Gaole He, Kunlin Yang, Hongjian Dou, Jin Huang, Siqi Ouyang, and Ji-Rong Wen. 2019. KB4Rec: A Data Set for Linking Knowledge Bases with Recommender Systems. Data Intell., Vol. 1, 2 (2019), 121--136.

[34]

Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. 2018. Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning. In KDD. 1040--1048.

[35]

Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin. 2019 b. Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems. In KDD. 2810--2818.

[36]

Lixin Zou, Long Xia, Pan Du, Zhuo Zhang, Ting Bai, Weidong Liu, Jian-Yun Nie, and Dawei Yin. 2020. Pseudo Dyna-Q: A Reinforcement Learning Framework for Interactive Recommendation. In WSDM. 816--824.

[37]

Shihao Zou, Zhonghua Li, Mohammad Akbari, Jun Wang, and Peng Zhang. 2019 a. MarlRank: Multi-agent Reinforced Learning to Rank. CIKM. 2073--2076.

Cited By

Wang JKaratzoglou AArapakis IJose JNejdl WAuer SKarras OCha MMoens MNajork M(2025)Large Language Model driven Policy Exploration for Recommender SystemsProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining10.1145/3701551.3703496(107-116)Online publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1145/3701551.3703496
Shi CYan SZhang SWang HLin K(2025)Knowledge-Guided Semantically Consistent Contrastive Learning for sequential recommendationNeural Networks10.1016/j.neunet.2025.107191185(107191)Online publication date: May-2025
https://doi.org/10.1016/j.neunet.2025.107191
Wang JWu LLiu QYang Y(2025)An efficient continuous control perspective for reinforcement-learning-based sequential recommendationKnowledge-Based Systems10.1016/j.knosys.2025.113133312(113133)Online publication date: Mar-2025
https://doi.org/10.1016/j.knosys.2025.113133
Show More Cited By

Index Terms

KERL: A Knowledge-Guided Reinforcement Learning Model for Sequential Recommendation
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems
    2. Users and interactive retrieval
      1. Personalization

Recommendations

Self-Supervised Reinforcement Learning for Recommender Systems
SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

In session-based or sequential recommendation, it is important to consider a number of factors like long-term user engagement, multiple types of user-item interactions such as clicks, purchases etc. The current state-of-the-art supervised approaches fail ...
Teach and Explore: A Multiplex Information-guided Effective and Efficient Reinforcement Learning for Sequential Recommendation
Casting sequential recommendation (SR) as a reinforcement learning (RL) problem is promising and some RL-based methods have been proposed for SR. However, these models are sub-optimal due to the following limitations: (a) they fail to leverage the ...
CDARL: a contrastive discriminator-augmented reinforcement learning framework for sequential recommendations
Abstract
Sequential recommendations play a crucial role in many real-world applications. Due to the sequential nature, reinforcement learning has been employed to iteratively produce recommendations based on an observed stream of user behavior. In this ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2020

2548 pages

ISBN:9781450380164

DOI:10.1145/3397271

General Chairs:
Jimmy Huang
York University, Canada
,
Yi Chang
Jilin University, China
,
Xueqi Cheng
Chinese Academy of Sciences, China
,
Program Chairs:
Jaap Kamps
University of Amsterdam, Netherlands
,
Vanessa Murdock
Amazon, U.S.A.
,
Ji-Rong Wen
Renmin University of China, China
,
Yiqun Liu
Tsinghua University, China

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

SIGIR '20

Sponsor:

SIGIR

SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval

July 25 - 30, 2020

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

96
Total Citations
View Citations
2,622
Total Downloads

Downloads (Last 12 months)235
Downloads (Last 6 weeks)17

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang JKaratzoglou AArapakis IJose JNejdl WAuer SKarras OCha MMoens MNajork M(2025)Large Language Model driven Policy Exploration for Recommender SystemsProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining10.1145/3701551.3703496(107-116)Online publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1145/3701551.3703496
Shi CYan SZhang SWang HLin K(2025)Knowledge-Guided Semantically Consistent Contrastive Learning for sequential recommendationNeural Networks10.1016/j.neunet.2025.107191185(107191)Online publication date: May-2025
https://doi.org/10.1016/j.neunet.2025.107191
Wang JWu LLiu QYang Y(2025)An efficient continuous control perspective for reinforcement-learning-based sequential recommendationKnowledge-Based Systems10.1016/j.knosys.2025.113133312(113133)Online publication date: Mar-2025
https://doi.org/10.1016/j.knosys.2025.113133
Zheng SWang SLi KLi XSun F(2025)When Feature Encoder Meets Diffusion Model for Sequential RecommendationsInformation Sciences10.1016/j.ins.2025.121903(121903)Online publication date: Jan-2025
https://doi.org/10.1016/j.ins.2025.121903
Luo TLiu YPan S(2024)Collaborative Sequential Recommendations via Multi-view GNN-transformersACM Transactions on Information Systems10.1145/364943642:6(1-27)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1145/3649436
Ye SLu J(2024)Robust Recommender Systems with Rating Flip NoiseACM Transactions on Intelligent Systems and Technology10.1145/364128516:1(1-19)Online publication date: 26-Dec-2024
https://dl.acm.org/doi/10.1145/3641285
Preuett L(2024)Learning Personalized Health Recommendations via Offline Reinforcement LearningProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688021(1355-1357)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688021
Yan SShi CWang HChen LJiang LGuo RLin K(2024)Teach and Explore: A Multiplex Information-guided Effective and Efficient Reinforcement Learning for Sequential RecommendationACM Transactions on Information Systems10.1145/363000342:5(1-26)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3630003
Wang JKaratzoglou AArapakis IJose JHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Reinforcement Learning-based Recommender Systems with Large Language Models for State Reward and Action ModelingProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657767(375-385)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657767
Hu HGuo WLiu XLiu YTang RZhang RKan MAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)User Behavior Enriched Temporal Knowledge Graphs for Sequential RecommendationProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635762(266-275)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635762
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten