Changhua Pei
Fei Sun
Peng Jiang
Yongfeng Zhang
ABSTRACT
Existing recommendation algorithms mostly focus on optimizing traditional recommendation measures, such as the accuracy of rating prediction in terms of RMSE or the quality of top-k recommendation lists in terms of precision, recall, MAP, etc. However, an important expectation for commercial recommendation systems is to improve the final revenue/profit of the system. Traditional recommendation targets such as rating prediction and top-k recommendation are not directly related to this goal.
In this work, we blend the fundamental concepts in online advertising and micro-economics into personalized recommendation for profit maximization. Specifically, we propose value-aware recommendation based on reinforcement learning, which directly optimizes the economic value of candidate items to generate the recommendation list. In particular, we generalize the basic concept of click conversion rate (CVR) in computational advertising into the conversation rate of an arbitrary user action (XVR) in E-commerce, where the user actions can be clicking, adding to cart, adding to wishlist, etc. In this way, each type of user action is mapped to its monetized economic value. Economic values of different user actions are further integrated as the reward of a ranking list, and reinforcement learning is used to optimize the recommendation list for the maximum total value. Experimental results in both offline benchmarks and online commercial systems verified the improved performance of our framework, in terms of both traditional top-k ranking tasks and the economic profits of the system.
- Ashish Agarwal, Kartik Hosanagar, and Michael D Smith. 2011. Location, location, location: An analysis of profitability of position in online advertising markets. Journal of marketing research 48, 6 (2011), 1057-1073.Google Scholar
Cross Ref
- Andrei Broder, Marcus Fontoura, Vanja Josifovski, and Lance Riedel. 2007. A semantic approach to contextual advertising. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 559-566. Google ScholarDigital Library
- Robin Burke, Gediminas Adomavicius, Ido Guy, Jan Krasnodebski, Luiz Pizzato, Yi Zhang, and Himan Abdollahpouri. 2017. VAMS 2017: Workshop on Value-Aware and Multistakeholder Recommendation. In Proceedings of the Eleventh ACM Conference on Recommender Systems. ACM, 378-379. Google ScholarDigital Library
- Shi-Yong Chen, Yang Yu, Qing Da, Jun Tan, Hai-Kuan Huang, and Hai-Hong Tang. 2018. Stabilizing reinforcement learning in dynamic environment with application to online recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM. Google ScholarDigital Library
- Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. 2010. Performance of recommender algorithms on top-n recommendation tasks. In Proceedings of the fourth ACM conference on Recommender systems. ACM, 39-46. Google ScholarDigital Library
- Kushal Dave, Vasudeva Varma, 2014. Computational advertising: Techniques for targeting relevant ads. Foundations and Trends® in Information Retrieval 8, 4-5(2014), 263-418. Google ScholarDigital Library
- Gediminas Adomavicius Dietmar Jannach. 2017. Price and Profit Awareness in Recommender Systems. arXiv preprint arXiv:1801.00209(2017).Google Scholar
- Michael D Ekstrand, John T Riedl, Joseph A Konstan, 2011. Collaborative filtering recommender systems. Foundations and Trends® in Human-Computer Interaction 4, 2(2011), 81-173. Google ScholarDigital Library
- Jun Feng, Heng Li, Minlie Huang, Shichen Liu, Wenwu Ou, Zhirong Wang, and Xiaoyan Zhu. 2018. Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent Reinforcement Learning. In WWW. 1939-1948. Google ScholarDigital Library
- Yingqiang Ge, Shuyuan Xu, Shuchang Liu, Shijie Geng, Zuohui Fu, and Yongfeng Zhang. 2019. Maximizing Marginal Utility per Dollar for Economic Recommendation. In WWW. Google ScholarDigital Library
- Anindya Ghose and Sha Yang. 2009. An empirical analysis of search engine advertising: Sponsored search in electronic markets. Management science 55, 10 (2009), 1605-1622. Google ScholarDigital Library
- Avi Goldfarb and Catherine Tucker. 2011. Online display advertising: Targeting and obtrusiveness. Marketing Science 30, 3 (2011), 389-404. Google ScholarDigital Library
- Asela Gunawardana and Guy Shani. 2009. A survey of accuracy evaluation metrics of recommendation tasks. Journal of Machine Learning Research 10, Dec (2009), 2935-2962. Google ScholarDigital Library
- Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 173-182. Google ScholarDigital Library
- Jonathan L Herlocker, Joseph A Konstan, Al Borchers, and John Riedl. 2017. An algorithmic framework for performing collaborative filtering. In ACM SIGIR Forum, Vol. 51. ACM, 227-234. Google ScholarDigital Library
- Yujing Hu, Qing Da, Anxiang Zeng, Yang Yu, and Yinghui Xu. 2018. Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM. Google ScholarDigital Library
- Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender systems. Computer8(2009), 30-37. Google ScholarDigital Library
- Daniel D Lee and H Sebastian Seung. 2001. Algorithms for non-negative matrix factorization. In Advances in neural information processing systems. 556-562.Google Scholar
- Kuang-chih Lee, Burkay Orten, Ali Dasdan, and Wentong Li. 2012. Estimating conversion rate in display advertising from past erformance data. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 768-776. Google ScholarDigital Library
- Tie-Yan Liu. 2009. Learning to Rank for Information Retrieval. Found. Trends Inf. Retr. 3, 3 (March 2009), 225-331. Google ScholarDigital Library
- Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018. Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. ACM, 1137-1140. Google ScholarDigital Library
- Andriy Mnih and Ruslan R Salakhutdinov. 2008. Probabilistic matrix factorization. In Advances in neural information processing systems.Google Scholar
- Michael J Pazzani and Daniel Billsus. 2007. Content-based recommendation systems. In The adaptive web. Springer, 325-341. Google ScholarDigital Library
- Steffen Rendle and Christoph Freudenthaler. 2014. Improving pairwise learning for item recommendation from implicit feedback. In Proceedings of the 7th ACM international conference on Web search and data mining. ACM, 273-282. Google ScholarDigital Library
- Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864(2017).Google Scholar
- Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web. ACM, 285-295. Google ScholarDigital Library
- Guy Shani, David Heckerman, and Ronen I Brafman. 2005. An MDP-based recommender system. Journal of Machine Learning Research 6, Sep (2005), 1265-1295. Google ScholarDigital Library
- SIGKDD. 2015. Life-stage Prediction for Product Recommendation in E-commerce. ACM, 1879-1888. Google ScholarDigital Library
- Richard S Sutton, Andrew G Barto, 1998. Reinforcement learning: An introduction. MIT press. Google ScholarDigital Library
- Nima Taghipour and Ahmad Kardan. 2008. A hybrid web recommender system based on q-learning. In Proceedings of the 2008 ACM symposium on Applied computing. ACM, 1164-1168. Google ScholarDigital Library
- Hao Wang, Naiyan Wang, and Dit-Yan Yeung. 2015. Collaborative deep learning for recommender systems. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1235-1244. Google ScholarDigital Library
- Qiaolin Xia, Peng Jiang, Fei Sun, Yi Zhang, Xiaobo Wang, and Zhifang Sui. 2018. Modeling Consumer Buying Decision for Recommendation Based on Multi-Task Deep Learning. In CIKM. ACM, 1703-1706. Google ScholarDigital Library
- Shuai Zhang, Lina Yao, and Aixin Sun. 2017. Deep learning based recommender system: A survey and new perspectives. arXiv preprint arXiv:1707.07435(2017).Google Scholar
- Yongfeng Zhang, Qi Zhao, Yi Zhang, Daniel Friedman, Min Zhang, Yiqun Liu, and Shaoping Ma. 2016. Economic recommendation with surplus maximization. In WWW. Google ScholarDigital Library
- Qi Zhao, Yongfeng Zhang, Yi Zhang, and Daniel Friedman. 2017. Multi-product utility maximization for economic recommendation. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. ACM, 435-443. Google ScholarDigital Library
- Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2018. Deep Reinforcement Learning for Page-wise Recommendations. In Proceedings of the 12th ACM Conference on Recommender Systems. ACM. Google ScholarDigital Library
- Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. 2018. Recommendation with Negative Feedback via Pairwise Deep Reinforcement Learning. Proceedings of the 24th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2018). Google ScholarDigital Library
- Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Dawei Yin, Yihong Zhao, and Jiliang Tang. 2017. Deep Reinforcement Learning for List-wise Recommendations. arXiv preprint arXiv:1801.00209(2017).Google Scholar
- Yong Zheng. 2017. Multi-Stakeholder Recommendation: Applications and Challenges. arXiv preprint arXiv:1707.08913(2017).Google Scholar
- Han Zhu, Junqi Jin, Chang Tan, Fei Pan, Yifan Zeng, Han Li, and Kun Gai. {n. d.}. Optimized cost per click in taobao display advertising.Google Scholar
Recommendations
User-Specific Feature-Based Similarity Models for Top-n Recommendation of New Items
Survey Paper, Regular Papers and Special Section on Participatory Sensing and Crowd IntelligenceRecommending new items for suitable users is an important yet challenging problem due to the lack of preference history for the new items. Noncollaborative user modeling techniques that rely on the item features can be used to recommend new items. ...
Hybrid Trust-Aware Model for Personalized Top-N Recommendation
CODS '17: Proceedings of the 4th ACM IKDD Conferences on Data SciencesDue to the large quantity and diversity of content being easily available to users, recommender systems (RS) have become an integral part of nearly every online system. They allow users to resolve the information overload problem by proactively ...
Context-aware reinforcement learning for course recommendation
AbstractOnline course recommendation is an extremely relevant ingredient for the efficiency of e-learning. The current recommendation methods cannot guarantee the effectiveness and accuracy of course recommendation, especially when a user has enrolled in ...
Highlights- We present a context-aware reinforcement learning model for course recommendation.
- A novel interaction improves the effectiveness of course recommendation.
- We propose a contextual policy gradient method with approximation for RRL.
Comments