skip to main content
10.1145/3640457.3688152acmconferencesArticle/Chapter ViewAbstractPublication PagesrecsysConference Proceedingsconference-collections
research-article

Ranking-Aware Unbiased Post-Click Conversion Rate Estimation via AUC Optimization on Entire Exposure Space

Published: 08 October 2024 Publication History

Abstract

Estimating the post-click conversion rate (CVR) accurately in ranking systems is crucial in industrial applications. However, this task is often challenged by data sparsity and selection bias, which hinder accurate ranking. Previous approaches to address these challenges have typically focused on either modeling CVR across the entire exposure space which includes all exposure events, or providing unbiased CVR estimation separately. However, the lack of integration between these objectives has limited the overall performance of CVR estimation. Therefore, there is a pressing need for a method that can simultaneously provide unbiased CVR estimates across the entire exposure space. To achieve it, we formulate the CVR estimation task as an Area Under the Curve (AUC) optimization problem and propose the Entire-space Weighted AUC (EWAUC) framework. EWAUC utilizes sample reweighting techniques to handle selection bias and employs pairwise AUC risk, which incorporates more information from limited clicked data, to handle data sparsity. In order to model CVR across the entire exposure space unbiasedly, EWAUC treats the exposure data as both conversion data and non-conversion data to calculate the loss. The properties of AUC risk guarantee the unbiased nature of the entire space modeling. We provide comprehensive theoretical analysis to validate the unbiased nature of our approach. Additionally, extensive experiments conducted on real-world datasets demonstrate that our approach outperforms state-of-the-art methods in terms of ranking performance for the CVR estimation task.

References

[1]
Shivani Agarwal, Thore Graepel, Ralf Herbrich, Sariel Har-Peled, and Dan Roth. 2005. Generalization bounds for the area under the ROC curve. JMLR 6 (2005), 393–425.
[2]
Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, 2021. Big self-supervised models advance medical image classification. In ICCV. IEEE, Montreal, 3478–3488.
[3]
Zhongxin Bai, Xiao-Lei Zhang, and Jingdong Chen. 2020. Partial AUC optimization based deep speaker embeddings with class-center learning for text-independent speaker verification. In ICASSP. IEEE, IEEE, Barcelona, 6819–6823.
[4]
Elias Bareinboim, Jin Tian, and Judea Pearl. 2014. Recovering from selection bias in causal and statistical inference. In AAAI. AAAI Press, Québec City, 2410–2416.
[5]
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016. Wide & Deep Learning for Recommender Systems. In RecSys, Alexandros Karatzoglou, Balázs Hidasi, Domonkos Tikk, Oren Sar Shalom, Haggai Roitman, Bracha Shapira, and Lior Rokach (Eds.). ACM, Boston, 7–10. https://doi.org/10.1145/2988450.2988454
[6]
Stéphan Clémençon, Gábor Lugosi, and Nicolas Vayatis. 2008. Ranking and empirical minimization of U-statistics. The Annals of Statistics 36, 2 (2008), 844–874.
[7]
Corinna Cortes and Mehryar Mohri. 2003. AUC optimization vs. Error rate minimization. In NIPS. MIT Press, Vancouver, 313–320.
[8]
David S Evans. 2009. The online advertising industry: Economics, evolution, and privacy. Journal of economic perspectives 23, 3 (2009), 37–60.
[9]
Asghar Feizi. 2020. Hierarchical detection of abnormal behaviors in video surveillance through modeling normal behaviors based on AUC maximization. Soft Computing 24, 14 (2020), 10401–10413.
[10]
Yoav Freund, Raj Iyer, Robert E Schapire, and Yoram Singer. 2003. An efficient boosting algorithm for combining preferences. JMLR 4, Nov (2003), 933–969.
[11]
Wei Gao, Lu Wang, Rong Jin, Shenghuo Zhu, and Zhi-Hua Zhou. 2016. One-pass AUC optimization. Artif. Intell. 236 (2016), 1–29.
[12]
Wei Gao and Zhi-Hua Zhou. 2015. On the Consistency of AUC Pairwise Optimization. In IJCAI. AAAI Press, Buenos Aires, 939–945.
[13]
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In IJCAI, Carles Sierra (Ed.). ijcai.org, Melbourne, 1725–1731. https://doi.org/10.24963/IJCAI.2017/239
[14]
James A Hanley and Barbara J McNeil. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 1 (1982), 29–36.
[15]
Thorsten Joachims. 2005. A support vector method for multivariate performance measures. In ICML. ACM, Bonn, 377–384.
[16]
Diederik P Kingma and Jimmy Ba. 2015. Adam: a method for stochastic optimization. In ICLR. Curran Associates, San Diega, 15 pages.
[17]
Cheng Li, Yue Lu, Qiaozhu Mei, Dong Wang, and Sandeep Pandey. 2015. Click-through prediction for advertising in twitter timeline. In SIGKDD. ACM, Sydney, 1959–1968.
[18]
Mingrui Liu, Zhuoning Yuan, Yiming Ying, and Tianbao Yang. 2020. Stochastic AUC maximization with deep neural networks. In ICLR. OpenReview.net, Addis Ababa, 10 pages.
[19]
Jie Lu, Dianshuang Wu, Mingsong Mao, Wei Wang, and Guangquan Zhang. 2015. Recommender system application developments: a survey. Decision support systems 74 (2015), 12–32.
[20]
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In SIGKDD. ACM, London, 1930–1939.
[21]
Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018. Entire space multi-task model: An effective approach for estimating post-click conversion rate. In SIGIR. ACM, Ann Arbor Michigan, 1137–1140.
[22]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: an imperative style, high-performance deep learning library. In NeurIPS. Curran Associates, Vancouver, 8024–8035.
[23]
pengcheng Li, Runze Li, Qing Da, An-Xiang Zeng, and Lijun Zhang. 2020. Improving Multi-Scenario Learning to Rank in E-commerce by Exploiting Task Relationships in the Label Space. In CIKM. ACM, Ireland, 2605–2612.
[24]
Zhen Qin, Yicheng Cheng, Zhe Zhao, Zhe Chen, Donald Metzler, and Jingzheng Qin. 2020. Multitask mixture of sequential experts for user activity streams. In SIGKDD. ACM, California, 3083–3091.
[25]
Xiang-Rong Sheng, Jingyue Gao, Yueyao Cheng, Siran Yang, Shuguang Han, Hongbo Deng, Yuning Jiang, Jian Xu, and Bo Zheng. 2023. Joint optimization of ranking and calibration with contextualized hybrid model. In SIGKDD. ACM, Long Beach, CA, 4813–4822.
[26]
Kent A Spackman. 1989. Signal detection theory: Valuable tools for evaluating inductive learning. In ML. Morgan Kaufmann, New York, 160–163.
[27]
Harald Steck. 2010. Training and testing of recommender systems on data missing not at random. In SIGKDD. ACM, Washington, 713–722.
[28]
Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020. Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In RecSys. ACM, Brazil, 269–278.
[29]
Nicolas Usunier, Massih-Reza Amini, and Patrick Gallinari. 2005. A data-dependent generalisation error bound for the AUC. In ICML workshops. ACM, Bonn, 9 pages.
[30]
Nicolas Usunier, Massih R Amini, and Patrick Gallinari. 2005. Generalization error bounds for classifiers trained with interdependent data. In NIPS. Curran Associates, Vancouver, 313–320.
[31]
Vladimir Vapnik. 1991. Principles of risk minimization for learning theory. In NIPS, Vol. 4. Morgan Kaufmann, Denver, 831–838.
[32]
Hao Wang, Tai-Wei Chang, Tianqiao Liu, Jianmin Huang, Zhichao Chen, Chao Yu, Ruopeng Li, and Wei Chu. 2022. ESCM2: entire space counterfactual multi-task model for post-click conversion rate estimation. In SIGIR. ACM, Madrid, 363–372.
[33]
Jizhe Wang, Pipei Huang, Huan Zhao, Zhibo Zhang, Binqiang Zhao, and Dik Lun Lee. 2018. Billion-scale commodity embedding for e-commerce recommendation in alibaba. In SIGKDD. ACM, London, 839–848.
[34]
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & Cross Network for Ad Click Predictions. In ADKDD. ACM, Halifax, 12:1–12:7. https://doi.org/10.1145/3124749.3124754
[35]
Yifan Wang, Peijie Sun, Min Zhang, Qinglin Jia, Jingjie Li, and Shaoping Ma. 2023. Unbiased Delayed Feedback Label Correction for Conversion Rate Prediction. In SIGKDD. ACM, Long Beach CA USA, 2456–2466.
[36]
Penghui Wei, Hongjian Dou, Shaoguo Liu, Rongjun Tang, Li Liu, Liang Wang, and Bo Zheng. 2023. Fedads: A benchmark for privacy-preserving cvr estimation with vertical federated learning. In SIGIR. ACM, Taipei, 3037–3046.
[37]
Dongbo Xi, Zhen Chen, Peng Yan, Yinger Zhang, Yongchun Zhu, Fuzhen Zhuang, and Yu Chen. 2021. Modeling the sequential dependence among audience multi-step conversions with multi-task learning in targeted display advertising. In SIGKDD. ACM, Singapore, 3745–3755.
[38]
Zheng Xie and Ming Li. 2018. Cutting the software building efforts in continuous integration by semi-supervised online AUC optimization. In IJCAI. ijcai.org, Stockholm, 2875–2881.
[39]
Zheng Xie and Ming Li. 2018. Semi-supervised AUC optimization without guessing labels of unlabeled data. In AAAI. AAAI Press, New Orleans, 4310–4317.
[40]
Zheng Xie, Yu Liu, Hao-Yuan He, Ming Li, and Zhi-Hua Zhou. 2024. Weakly Supervised AUC Optimization: A Unified Partial AUC Approach. TPAMI (2024), 1–16.
[41]
Zheng Xie, Yu Liu, and Ming Li. 2024. AUC Optimization from Multiple Unlabeled Datasets. In AAAI, Vol. 38. AAAI Press, Vancouver, 16058–16066.
[42]
Yuriko Yamaguchi, Mimpei Morishita, Youichi Inagaki, Reyn Nakamoto, Jianwei Zhang, Junichi Aoi, and Shinsuke Nakajima. 2016. Web advertising recommender system based on estimating users’ latent interests. In iiWAS. ACM, Singapore, 42–49.
[43]
Yan Yan, Zitao Liu, Meng Zhao, Wentao Guo, Weipeng P Yan, and Yongjun Bao. 2019. A practical deep online ranking system in e-commerce recommendation. In ECML/PKDD, Vol. 11053. Springer, Springer, Dublin, 186–201.
[44]
Tianbao Yang and Yiming Ying. 2022. AUC maximization in the era of big data and AI: A survey. ACM computing surveys 55, 8 (2022), 1–37.
[45]
Zhiyong Yang, Qianqian Xu, Shilong Bao, Xiaochun Cao, and Qingming Huang. 2021. Learning with multiclass AUC: theory and algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 11 (2021), 7747–7763.
[46]
Zhiyong Yang, Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao, and Qingming Huang. 2021. When all we need is a piece of the pie: a generic framework for optimizing two-way partial AUC. In ICML. PMLR, Virtual Event, 11820–11829.
[47]
Yao Yao, Qihang Lin, and Tianbao Yang. 2022. Large-scale optimization of partial AUC in a range of false positive rates. In NeurIPS. Curran Associates, New Orleans, 31239–31253.
[48]
Yiming Ying, Longyin Wen, and Siwei Lyu. 2016. Stochastic online AUC maximization. In NIPS. Curran Associates, Inc., Barcelona, 451–459.
[49]
Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. 2021. Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification. In ICCV. IEEE, Montreal, 3020–3029.
[50]
Dandan Zhang, Haotian Wu, Guanqi Zeng, Yao Yang, Weijiang Qiu, Yujie Chen, and Haoyuan Hu. 2022. CTnoCVR: A novelty auxiliary task making the lower-CTR-higher-CVR upper. In SIGIR. ACM, Madrid, 2272–2276.
[51]
Wenhao Zhang, Wentian Bao, Xiao-Yang Liu, Keping Yang, Quan Lin, Hong Wen, and Ramin Ramezani. 2020. Large-scale causal approaches to debiasing post-click conversion rate estimation with multi-task learning. In WWW. ACM, Taipei, 2775–2781.
[52]
Yu Zhang and Qiang Yang. 2021. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering 34, 12 (2021), 5586–5609.
[53]
Dixian Zhu, Gang Li, Bokun Wang, Xiaodong Wu, and Tianbao Yang. 2022. When AUC meets DRO: optimizing partial AUC for deep learning with non-convex convergence guarantee. In ICML. PMLR, Baltimore, 27548–27573.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
RecSys '24: Proceedings of the 18th ACM Conference on Recommender Systems
October 2024
1438 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 October 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. AUC Optimization
  2. Data Sparsity
  3. Post-click Conversion Rate Estimation
  4. Recommender System
  5. Selection Bias

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • NSFC
  • Major Program (JD) of Hubei Province
  • NSFC

Conference

Acceptance Rates

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 402
    Total Downloads
  • Downloads (Last 12 months)402
  • Downloads (Last 6 weeks)31
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media