REDRL: A review-enhanced Deep Reinforcement Learning model for interactive recommendation

Highlights

• Mining information in reviews and interaction data via the pretrained model.

• Modeling long-term dynamic preferences of users accurately and discriminately.

• Filtering irrelevant items and getting candidate items dynamically from a new anger.

• Better interactive recommendation based on deep reinforcement learning.

Abstract

Recent advances in interactive recommender systems (IRS) have received wide attention due to its flexible recommendation strategy and optimization for users’ long-term utility. Considering this interaction paradigm of IRS, researchers have made some attempts to incorporate reinforcement learning (RL) models into IRS, because of the excellent ability of RL in long-term optimizing and decision making. However, data sparsity is an intractable problem most IRS urgently need to address. Although a small amount of work has exploited reviews to address data sparsity, they ignored the varying importance of items for modeling the user. In addition, most existing RL-based approaches suffer from decision-making difficulties when the action space becomes large. To solve above problems, in this work, we present a Review-enhanced Deep Reinforcement Learning model (REDRL) for interactive recommendation. Specifically, we utilize text reviews, combined with a pretrained review representation model to acquire item review-enhanced embedding representations. Then we formalize the recommendation problem as a Markov Decision Process (MDP), and exploit deep reinforcement learning (DRL) to model the interactive recommendation. Notably, we introduce a multi-head self-attention technique to capture distinct importance of various items in the sequence behavior, which is overlooked by existing work when modeling the user preference. In this way, we can model long-term dynamic preferences of users accurately and discriminately for comprehensive interactive recommendation. Moreover, we subtly combine the semantic structure information in the user–item bipartite graph with meta-paths in heterogeneous information networks (HIN), to filter some irrelevant items and obtain candidate items dynamically. By this means, the size of the discrete action space is effectively reduced from a new anger. The experimental results based on three benchmark datasets demonstrate the efficiency of our method with significant improvement over state-of-the-art.

Introduction

In the age with explosive growth of information, recommender systems (RS) are crucial in solving the issue of information overload. RS help users make decisions by displaying items (goods, services, news, information, etc.) that meet their needs as much as possible, according to their purchasing behaviors and preferences. Recently, interactive recommender systems (IRS) have received wide attention from the research community. Different from traditional RS, where the recommendation process is considered as a stationary one and follows a constant strategy, IRS sequentially recommend items to users and meanwhile update their recommendation policy based on users’ feedback. This interaction paradigm presented by IRS has been extensively used in some popular systems, such as Spotify, TikTok, and YouTube.

One way to implement IRS is multi-armed bandit (MAB) methods (Li et al., 2010, Wang et al., 2017, Zeng et al., 2016), and they focus on how to achieve the trade-off between exploration and exploitation. However, these methods assume that the user preference stays the same during the recommendation process, which goes against the dynamic nature of IRS. Recently, there have been some attempts to incorporate reinforcement learning (RL) models into IRS (Lei and Li, 2019, Lei et al., 2020, Zhou et al., 2020, Zou et al., 2020), due to the excellent ability of RL in long-term optimizing and decision making. However, most existing RL-based approaches suffer from decision-making difficulties when the action space becomes large, because the time complexity of decision making is linearly related to the scale of the discrete action space.

For more efficient decision-making, on the basis of Deep Deterministic Policy Gradient (DDPG), Wang, Guo, Li, Pan, and Li (2020) designed an efficient approach for constructing an action candidate set by dividing users into some clusters and using the idea of collaborative filtering. However, such a method introduced the burden of clustering users and the quality of the candidate set relied on the accuracy of the clustering.

Simultaneously, most existing RL-based approaches only consider user–item interaction rating data to model item characteristics and user preferences. However, the reliance on the single rating information can easily make the model fall into the dilemma of the sparsity problem. It is well recognized that textual reviews contain large amount of abundant semantic knowledge, which can help mitigate the data sparsity problem. And as a supplementary information, they are easily accessible from many e-commerce and review websites. Thanks to the progress made by deep learning (DL) in natural language processing (NLP), there have been some successful works to leverage reviews to enhance recommendation performance (Wu, Quan, Li, and Ji, 2018, Wu et al., 2019, Zheng et al., 2017). But in the scenario of interactive recommendation, most existing RL-based approaches either ignore the reviews information or have difficulty in incorporating with reviews. Although Wang et al. (2020) put forward an Actor–Critic architecture framework to alleviate the data sparsity problem by leveraging textual information, they failed to model user preference accurately and discriminately, because they ignored the varying importance of items for modeling the user. Furthermore, when modeling the user preference, most RL-based methods consider the output (the last hidden state) of the recurrent neural network (RNN) as the representation of user state. This approach ignores the importance of the contribution of different items to the overall modeling in the sequence behavior.

Therefore, to solve the above problems, in this work, we present a Review-enhanced Deep Reinforcement Learning model (REDRL) for interactive recommendation.

Specifically, to reduce the impact from the sparsity problem, we utilize text reviews, combined with a pretrained review representation model to acquire item review-enhanced embedding representations. Then we formalize the recommendation problem as a Markov Decision Process (MDP), and exploit deep reinforcement learning (DRL) to model the interactive recommendation. To solve the problem that existing RL-based models ignore the distinct importance of various items in sequential behaviors when modeling user preference, we introduce a multi-head self-attention technique. In this way, we can model long-term dynamic preferences of users accurately and discriminately for comprehensive interactive recommendation.

Moreover, to solve the problem of large action spaces that would be faced when applying reinforcement learning to the recommendation domain, we combine the semantic structure information in the user–item bipartite graph with meta-paths in heterogeneous information networks (HIN), to filter some irrelevant items and obtain candidate items dynamically from a fresh perspective. The main contributions of our work are summarized below:

• We propose a novel Review-enhanced Deep Reinforcement Learning model for interactive recommendation, which comprehensively considers the rating information and review information to mitigate the problem of data sparsity. Moreover, by introducing the multi-head attention technique, it can characterize long-term dynamic preferences of users accurately and discriminately.

• We combine the semantic structure information in the user–item bipartite graph with meta-paths in heterogeneous information networks, to filter some irrelevant items and acquire candidate items dynamically from a fresh perspective. These potential candidate items not only reduce the scale of discrete action space, but also increase the effectiveness of sampling for policy optimization.

• We conduct experiments on three benchmark datasets. Experimental results demonstrate that our proposed model REDRL is superior to several state-of-the-art alternatives.