ABSTRACT
Click-through rate (CTR) prediction aims to recall the advertisements that users are interested in and to lead users to click, which is of critical importance for a variety of online advertising systems. In practice, CTR prediction is generally formulated as a conventional binary classification problem, where the clicked advertisements are positive samples and the others are negative samples. However, directly treating unclicked advertisements as negative samples would suffer from the severe label noise issue, since there exist many reasons why users are interested in a few advertisements but do not click. To address such serious issue, we propose a reinforcement learning based noise filtering approach, dubbed RLNF, which employs a noise filter to select effective negative samples. In RLNF, such selected, effective negative samples can be used to enhance the CTR prediction model, and meanwhile the effectiveness of the noise filter can be enhanced through reinforcement learning using the performance of CTR prediction model as reward. Actually, by alternating the enhancements of the noise filter and the CTR prediction model, the performance of both the noise filter and the CTR prediction model is improved. In our experiments, we equip 7 state-of-the-art CTR prediction models with RLNF. Extensive experiments on a public dataset and an industrial dataset present that RLNF significantly improves the performance of all these 7 CTR prediction models, which indicates both the effectiveness and the generality of RLNF.
Supplemental Material
- Ting Bai, Lixin Zou, Wayne Xin Zhao, Pan Du, Weidong Liu, Jian-Yun Nie, and Ji-Rong Wen. 2019. CTRec: A Long-Short Demands Evolution Model for Continuous-Time Recommendation. In Proceedings of SIGIR 2019. 675--684.Google ScholarDigital Library
- Fidel Cacheda, Nicola Barbieri, and Roi Blanco. 2017. Click Through Rate Prediction for Local Search Results. In Proceedings of WSDM 2017. 171--180.Google ScholarDigital Library
- Patrick P. K. Chan, Xian Hu, Lili Zhao, Daniel S. Yeung, Dapeng Liu, and Lei Xiao. 2018. Convolutional Neural Networks based Click-Through Rate Prediction with Multiple Feature Sequences. In Proceedings of IJCAI 2018. 2007--2013.Google ScholarCross Ref
- Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016. Wide & Deep Learning for Recommender Systems. In Proceedings of DLRS@RecSys 2016. 7--10.Google ScholarDigital Library
- Haibin Cheng, Eren Manavoglu, Ying Cui, Ruofei Zhang, and Jianchang Mao. 2012. Dynamic ad layout revenue optimization for display advertising. In Proceedings of ADKDD 2012. 9:1--9:9.Google ScholarDigital Library
- Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of RecSys 2016. 191--198.Google ScholarDigital Library
- Wei Deng, Junwei Pan, Tian Zhou, Deguang Kong, Aaron Flores, and Guang Lin. 2021. DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving. In Proceedings of WSDM 2021. 922--930.Google ScholarDigital Library
- Yue Deng, Yilin Shen, and Hongxia Jin. 2017. Disguise Adversarial Networks for Click-through Rate Prediction. In Proceedings of IJCAI 2017. 1589--1595.Google ScholarCross Ref
- Bora Edizel, Amin Mantrach, and Xiao Bai. 2017. Deep Character-Level Click-Through Rate Prediction for Sponsored Search. In Proceedings of SIGIR 2017. 305--314.Google ScholarDigital Library
- Yufei Feng, Fuyu Lv, Weichen Shen, Menghan Wang, Fei Sun, Yu Zhu, and Keping Yang. 2019. Deep Session Interest Network for Click-Through Rate Prediction. In Proceedings of IJCAI 2019. 2301--2307.Google ScholarCross Ref
- Hongchang Gao, Deguang Kong, Miao Lu, Xiao Bai, and Jian Yang. 2018. Attention Convolutional Neural Network for Advertiser-level Click-through Rate Forecasting. In Proceedings of WWW 2018. 1855--1864.Google ScholarDigital Library
- Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In Proceedings of IJCAI 2017. 1725--1731.Google ScholarCross Ref
- Wei Guo, Ruiming Tang, Huifeng Guo, Jianhua Han, Wen Yang, and Yuzhou Zhang. 2019. Order-aware Embedding Neural Network for CTR Prediction. In Proceedings of SIGIR 2019. 1121--1124.Google ScholarDigital Library
- Zai Huang, Zhen Pan, Qi Liu, Bai Long, Haiping Ma, and Enhong Chen. 2017. An Ad CTR Prediction Method Based on Feature Learning of Deep and Shallow Layers. In Proceedings of CIKM 2017. 2119--2122.Google ScholarDigital Library
- Kalervo J"a rvelin and Jaana Kek"a l"a inen. 2000. IR evaluation methods for retrieving highly relevant documents. In Proceedings of SIGIR 2000. 41--48.Google Scholar
- Sofia Ira Ktena, Alykhan Tejani, Lucas Theis, Pranay Kumar Myana, Deepak Dilipkumar, Ferenc Huszá r, Steven Yoo, and Wenzhe Shi. 2019. Addressing delayed feedback for continuous training with neural networks in CTR prediction. In Proceedings of RecSys 2019. 187--195.Google ScholarDigital Library
- Feng Li, Zhenrui Chen, Pengjie Wang, Yi Ren, Di Zhang, and Xiaoyu Zhu. 2019 a. Graph Intention Network for Click-through Rate Prediction in Sponsored Search. In Proceedings of SIGIR 2019. 961--964.Google ScholarDigital Library
- Zeyu Li, Wei Cheng, Yang Chen, Haifeng Chen, and Wei Wang. 2020. Interpretable Click-Through Rate Prediction through Hierarchical Attention. In Proceedings of WSDM 2020. 313--321.Google ScholarDigital Library
- Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang, and Liang Wang. 2019 b. Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction. In Proceedings of CIKM 2019. 539--548.Google ScholarDigital Library
- Bin Liu, Niannan Xue, Huifeng Guo, Ruiming Tang, Stefanos Zafeiriou, Xiuqiang He, and Zhenguo Li. 2020 a. AutoGroup: Automatic Feature Grouping for Modelling Explicit High-Order Feature Interactions in CTR Prediction. In Proceedings of SIGIR 2020. 199--208.Google Scholar
- Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xiuqiang He, Zhenguo Li, and Yong Yu. 2020 b. AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction. In Proceedings of KDD 2020. 2636--2645.Google ScholarDigital Library
- Weiwen Liu, Ruiming Tang, Jiajin Li, Jinkai Yu, Huifeng Guo, Xiuqiang He, and Shengyu Zhang. 2018. Field-aware probabilistic embedding neural network for CTR prediction. In Proceedings of RecSys 2018. 412--416.Google ScholarDigital Library
- Wantong Lu, Yantao Yu, Yongzhe Chang, Zhen Wang, Chenhui Li, and Bo Yuan. 2020. A Dual Input-aware Factorization Machine for CTR Prediction. In Proceedings of IJCAI 2020. 3139--3145.Google ScholarCross Ref
- Chuan Luo, Pu Zhao, Chen Chen, Bo Qiao, Chao Du, Hongyu Zhang, Wei Wu, Shaowei Cai, Bing He, Saravanakumar Rajmohan, and Qingwei Lin. 2021. PULNS: Positive-Unlabeled Learning with Effective Negative Sample Selector. In Proceedings of AAAI 2021 .Google ScholarCross Ref
- Zequn Lyu, Yu Dong, Chengfu Huo, and Weijun Ren. 2020. Deep Match to Rank Model for Personalized Click-Through Rate Prediction. In Proceedings of AAAI 2020. 156--163.Google ScholarCross Ref
- Wentao Ouyang, Xiuwu Zhang, Shukui Ren, Chao Qi, Zhaojie Liu, and Yanlong Du. 2019. Representation Learning-Assisted Click-Through Rate Prediction. In Proceedings of IJCAI 2019. 4561--4567.Google ScholarCross Ref
- Jiarui Qin, Weinan Zhang, Xin Wu, Jiarui Jin, Yuchen Fang, and Yong Yu. 2020. User Behavior Retrieval for Click-Through Rate Prediction. In Proceedings of SIGIR 2020. 2347--2356.Google ScholarDigital Library
- Steffen Rendle. 2010. Factorization Machines. In Proceedings of ICDM 2010. 995--1000.Google Scholar
- Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting clicks: estimating the click-through rate for new ads. In Proceedings of WWW 2007. 521--530.Google ScholarDigital Library
- Shu-Ting Shi, Wenhao Zheng, Jun Tang, Qing-Guo Chen, Yao Hu, Jianke Zhu, and Ming Li. 2020. Deep Time-Stream Framework for Click-through Rate Prediction by Tracking Interest Evolution. In Proceedings of AAAI 2020. 5726--5733.Google ScholarCross Ref
- Qingquan Song, Dehua Cheng, Hanning Zhou, Jiyan Yang, Yuandong Tian, and Xia Hu. 2020. Towards Automated Neural Interaction Discovery for Click-Through Rate Prediction. In Proceedings of KDD 2020. 945--955.Google ScholarDigital Library
- Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. 1999. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Proceedings of NIPS 1999. 1057--1063.Google Scholar
- Zhulin Tao, Xiang Wang, Xiangnan He, Xianglin Huang, and Tat-Seng Chua. 2020. HoAFM: A High-order Attentive Factorization Machine for CTR Prediction. Information Processing and Management, Vol. 57, 6 (2020), 102076.Google ScholarCross Ref
- Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & Cross Network for Ad Click Predictions. In Proceedings of ADKDD 2017. 12:1--12:7.Google ScholarDigital Library
- Yikai Wang, Liang Zhang, Quanyu Dai, Fuchun Sun, Bo Zhang, Yang He, Weipeng Yan, and Yongjun Bao. 2019. Regularized Adversarial Sampling and Deep Time-aware Attention for Click-Through Rate Prediction. In Proceedings of CIKM 2019. 349--358.Google ScholarDigital Library
- Shu Wu, Feng Yu, Xueli Yu, Qiang Liu, Liang Wang, Tieniu Tan, Jie Shao, and Fan Huang. 2020. TFNet: Multi-Semantic Feature Interaction for CTR Prediction. In Proceedings of SIGIR 2020. 1885--1888.Google ScholarDigital Library
- Weinan Xu, Hengxu He, Minshi Tan, Yunming Li, Jun Lang, and Dongbai Guo. 2020 a. Deep Interest with Hierarchical Attention Network for Click-Through Rate Prediction. In Proceedings of SIGIR 2020. 1905--1908.Google ScholarDigital Library
- Weinan Xu, Hengxu He, Minshi Tan, Yunming Li, Jun Lang, and Dongbai Guo. 2020 b. Deep Interest with Hierarchical Attention Network for Click-Through Rate Prediction. In Proceedings of SIGIR 2020. 1905--1908.Google ScholarDigital Library
- Niannan Xue, Bin Liu, Huifeng Guo, Ruiming Tang, Fengwei Zhou, Stefanos Zafeiriou, Yuzhou Zhang, Jun Wang, and Zhenguo Li. 2020. AutoHash: Learning Higher-order Feature Interactions for Deep CTR Prediction. IEEE Transactions on Knowledge and Data Engineering (2020).Google Scholar
- Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Deep Interest Evolution Network for Click-Through Rate Prediction. In Proceedings of AAAI 2019. 5941--5948.Google ScholarDigital Library
- Guorui Zhou, Xiaoqiang Zhu, Chengru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep Interest Network for Click-Through Rate Prediction. In Proceedings of KDD 2018. 1059--1068.Google ScholarDigital Library
- Jieming Zhu, Jinyang Liu, Weiqi Li, Jincai Lai, Xiuqiang He, Liang Chen, and Zibin Zheng. 2020. Ensembled CTR Prediction via Knowledge Distillation. In Proceedings of CIKM 2020. 2941--2958.Google ScholarDigital Library
Index Terms
- RLNF: Reinforcement Learning based Noise Filtering for Click-Through Rate Prediction
Recommendations
An ensemble learning framework for click-through rate prediction based on a reinforcement learning algorithm with parameterized actions
AbstractClick-through rate (CTR) prediction is essential for targeted advertising systems. Although there have been many studies on CTR prediction and forming some representative models, building a CTR prediction model that performs well under various ...
Ad Click-Through Rate Prediction: A Survey
Database Systems for Advanced Applications. DASFAA 2021 International WorkshopsAbstractAd click-through rate prediction (CTR), as an essential task of charging advertisers in the field of E-commerce, provides users with appropriate advertisements according to user interests to increase users’ click-through rate based on user clicks. ...
Improving click-through rate prediction accuracy in online advertising by transfer learning
WI '17: Proceedings of the International Conference on Web IntelligenceAs the main revenue source of Internet companies, online advertising is always a significant topic, where click-through rate (CTR) prediction plays a central role. In online advertising systems, there are often many advertisement products. Due to the ...
Comments