skip to main content
10.1145/3477495.3531844acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

Learning to Infer User Implicit Preference in Conversational Recommendation

Published: 07 July 2022 Publication History

Abstract

Conversational recommender systems (CRS) enable traditional recommender systems to interact with users by asking questions about attributes and recommending items. The attribute-level and item-level feedback of users can be utilized to estimate users' preferences. However, existing works do not fully exploit the advantage of explicit item feedback --- they only use the item feedback in rather implicit ways such as updating the latent user and item representation. Since CRS has multiple chances to interact with users, leveraging the context in the conversation may help infer users' implicit feedback (e.g., some specific attributes) when recommendations get rejected. To address the limitations of existing methods, we propose a new CRS framework called Conversational Recommender with Implicit Feedback (CRIF). CRIF formulates the conversational recommendation scheme as a four-phase process consisting of offline representation learning, tracking, decision, and inference. In the inference module, by fully utilizing the relation between users' attribute-level and item-level feedback, our method can explicitly deduce users' implicit preferences. Therefore, CRIF is able to achieve more accurate user preference estimation. Besides, in the decision module, to better utilize the attribute-level and item-level feedback, we adopt inverse reinforcement learning to learn a flexible decision strategy that selects the suitable action at each conversation turn. Through extensive experiments on four benchmark CRS datasets, we validate the effectiveness of our approach, which significantly outperforms the state-of-the-art CRS methods.

Supplementary Material

MP4 File (SIGIR22-fp1743.mp4)
This is the presentation video of our paper.

References

[1]
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. 2009. A survey of robot learning from demonstration. Robotics and autonomous systems, Vol. 57 (2009), 469--483.
[2]
Saurabh Arora and Prashant Doshi. 2021. A survey of inverse reinforcement learning: Challenges, methods and progress. AI, Vol. 297 (2021), 103500.
[3]
Keping Bi, Qingyao Ai, Yongfeng Zhang, and W Bruce Croft. 2019. Conversational product search based on negative feedback. In CIKM. 359--368.
[4]
Senthilkumar Chandramohan, Matthieu Geist, Fabrice Lefevre, and Olivier Pietquin. 2011. User simulation in dialogue systems using inverse reinforcement learning. In Interspeech 2011 . 1025--1028.
[5]
Haokun Chen, Xinyi Dai, Han Cai, Weinan Zhang, Xuejian Wang, Ruiming Tang, Yuzhou Zhang, and Yong Yu. 2019 a. Large-scale interactive recommendation with tree-structured policy gradient. In AAAI, Vol. 33. 3312--3320.
[6]
Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019 b. Towards Knowledge-Based Recommender Dialog System. In EMNLP-IJCNLP. 1803--1813.
[7]
Konstantina Christakopoulou, Alex Beutel, Rui Li, Sagar Jain, and Ed H Chi. 2018. Q&R: A Two-Stage Approach toward Interactive Recommendation. In SIGKDD . 139--148.
[8]
Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. 2016. Towards conversational recommender systems. In SIGKDD. 815--824.
[9]
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In NeurIPS . 4299--4307.
[10]
Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. 2021. Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning. In SIGIR. 1431--1441.
[11]
Justin Fu, Katie Luo, and Sergey Levine. 2018. Learning Robust Rewards with Adverserial Inverse Reinforcement Learning. In ICLR .
[12]
Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2021. Advances and challenges in conversational recommender systems: A survey. arXiv preprint arXiv:2101.09459 (2021).
[13]
Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. NIPS, Vol. 29 (2016), 4565--4573.
[14]
Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. 2021. A survey on conversational recommender systems. CSUR, Vol. 54 (2021), 1--36.
[15]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR .
[16]
Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR .
[17]
Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2020 a. Conversational recommendation: Formulation, methods, and evaluation. In SIGIR . 2425--2428.
[18]
Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. 2020 b. Estimation--Action--Reflection: Towards Deep Interaction Between Conversational and Recommender Systems. In WSDM. 304--312.
[19]
Wenqiang Lei, Gangyi Zhang, Xiangnan He, Yisong Miao, Xiang Wang, Liang Chen, and Tat-Seng Chua. 2020 c. Interactive path reasoning on graph for conversational recommendation. In SIGKDD . 2073--2083.
[20]
Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards Deep Conversational Recommendations. In NeurIPS. 9748--9758.
[21]
Shijun Li, Wenqiang Lei, Qingyun Wu, Xiangnan He, Peng Jiang, and Tat-Seng Chua. 2021. Seamlessly unifying attributes and items: Conversational recommendation for cold-start users. TOIS, Vol. 39 (2021), 1--29.
[22]
Lizi Liao, Yunshan Ma, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2018. Knowledge-aware Multimodal Dialogue Systems. In ACM MM. 801--809.
[23]
Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu. 2020. Towards Conversational Recommendation over Multi-Type Dialogs. In ACL . 1036--1049.
[24]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nat. (2015), 529--533.
[25]
Andrew Y. Ng and Stuart J. Russell. 2000. Algorithms for Inverse Reinforcement Learning. In ICML. 663--670.
[26]
Xuhui Ren, Hongzhi Yin, Tong Chen, Hao Wang, Zi Huang, and Kai Zheng. 2021. Learning to ask appropriate questions in conversational recommendation. In SIGIR . 808--817.
[27]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In UAI. 452--461.
[28]
Yueming Sun and Yi Zhang. 2018. Conversational Recommender System. In SIGIR. 235--244.
[29]
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. 1999. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In NeurIPS . 1057--1063.
[30]
Kerui Xu, Jingxuan Yang, Jun Xu, Sheng Gao, Jun Guo, and Ji-Rong Wen. 2021. Adapting User Preference to Online Feedback in Multi-round Conversational Recommendation. In WSDM. 364--372.
[31]
Tong Yu, Yilin Shen, and Hongxia Jin. 2019. An Visual Dialog Augmented Interactive Recommender System. In SIGKDD. 157--165.
[32]
Xiaoying Zhang, Hong Xie, Hang Li, and John CS Lui. 2020. Conversational contextual bandit: Algorithm and application. In WWW. 662--672.
[33]
Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W Bruce Croft. 2018. Towards conversational search and recommendation: System ask, user respond. In CIKM . 177--186.
[34]
Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. 2018. Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning. In SIGKDD . 1040--1048.
[35]
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018. DRN: A Deep Reinforcement Learning Framework for News Recommendation. In WWW . 167--176.
[36]
Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020 a. Improving conversational recommender systems via knowledge graph based semantic fusion. In SIGKDD. 1006--1014.
[37]
Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji-Rong Wen. 2020 b. Towards Topic-Guided Conversational Recommender System. In COLING. 4128--4139.
[38]
Jie Zou, Yifan Chen, and Evangelos Kanoulas. 2020. Towards question-based recommender systems. In SIGIR. 881--890.

Cited By

View all
  • (2025)Conversational Recommendations With User Entity Focus and Multi-Granularity Latent Variable EnhancementIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.352328337:3(1126-1139)Online publication date: Mar-2025
  • (2024)Knowledge-Enhanced Conversational Recommendation via Transformer-Based Sequential ModelingACM Transactions on Information Systems10.1145/367737642:6(1-27)Online publication date: 18-Oct-2024
  • (2024)TUT4CRS: Time-aware User-preference Tracking for Conversational Recommendation SystemProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681259(5856-5864)Online publication date: 28-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval
July 2022
3569 pages
ISBN:9781450387323
DOI:10.1145/3477495
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. conversational recommendation
  2. inverse reinforcement learning
  3. user preference inference

Qualifiers

  • Research-article

Conference

SIGIR '22
Sponsor:

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)192
  • Downloads (Last 6 weeks)10
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Conversational Recommendations With User Entity Focus and Multi-Granularity Latent Variable EnhancementIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.352328337:3(1126-1139)Online publication date: Mar-2025
  • (2024)Knowledge-Enhanced Conversational Recommendation via Transformer-Based Sequential ModelingACM Transactions on Information Systems10.1145/367737642:6(1-27)Online publication date: 18-Oct-2024
  • (2024)TUT4CRS: Time-aware User-preference Tracking for Conversational Recommendation SystemProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681259(5856-5864)Online publication date: 28-Oct-2024
  • (2024)FairCRS: Towards User-oriented Fairness in Conversational Recommendation SystemsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688150(126-136)Online publication date: 8-Oct-2024
  • (2024)An Empirical Analysis on Multi-turn Conversational Recommender SystemsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657893(841-851)Online publication date: 10-Jul-2024
  • (2024)A Survey on Reinforcement Learning for Recommender SystemsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.328016135:10(13164-13184)Online publication date: Oct-2024
  • (2024)Improving conversational recommender systems via multi-preference modelling and knowledge-enhancedKnowledge-Based Systems10.1016/j.knosys.2023.111361286:COnline publication date: 17-Apr-2024
  • (2024)Graph-based dynamic attribute clipping for conversational recommendationDiscover Computing10.1007/s10791-024-09437-627:1Online publication date: 10-May-2024
  • (2024)Towards Multi-subsession Conversational RecommendationAdvances in Knowledge Discovery and Data Mining10.1007/978-981-97-2262-4_15(182-194)Online publication date: 7-May-2024
  • (2024)Enhancing Adaptive E-Learning with Generative AI: Expanding the Horizon Beyond Recommendation SystemsProceedings of Third International Conference on Computing and Communication Networks10.1007/978-981-97-0892-5_59(755-767)Online publication date: 21-Jul-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media