Staying or Leaving: A Knowledge-Enhanced User Simulator for Reinforcement Learning Based Short Video Recommendation

Abstract

Reinforcement learning has been widely used  in recommender systems in order to optimize users’ long-term utilities. An accurate and explainable user simulator is crucial for reinforcement learning based recommendation, as an online interactive environment is often unavailable. On short video platforms, it is very important to keep users on the platform as long as possible in each session. Thus, session-based user utilities depend on two factors: how much users like every single video (video preference) and the number of videos watched (video views) in each session. To this end, the simulator should simultaneously model the user’s degree of liking for each video and video views. However, most previous studies on the short video recommendation only paid attention to the former. In this work, we propose KESWA, a Knowledge-Enhanced Session-Wide Attention method for short video user simulation. KESWA fuses information foraging theory with a deep learning model for both video preference and video views modeling, providing an explainable prediction for users’ staying and leaving behavior. Comparative experiments demonstrate that KESWA provides a better simulation of video views compared with existing models. Meanwhile, reinforcement learning agents can achieve higher session-based user utilities trained by KESWA than by other user simulators.