short-paper

Posterior Probability Matters: Doubly-Adaptive Calibration for Neural Predictions in Online Advertising

Authors:

Bo ZhengAuthors Info & Claims

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 2645 - 2649

https://doi.org/10.1145/3477495.3531911

Published: 07 July 2022 Publication History

Abstract

Predicting user response probabilities is vital for ad ranking and bidding. We hope that predictive models can produce accurate probabilistic predictions that reflect true likelihoods. Calibration techniques aims to post-process model predictions to posterior probabilities. Field-level calibration -- which performs calibration w.r.t. to a specific field value -- is fine-grained and more practical. In this paper we propose a doubly-adaptive approach AdaCalib. It learns an isotonic function family to calibrate model predictions with the guidance of posterior statistics, and field-adaptive mechanisms are designed to ensure that the posterior is appropriate for the field value to be calibrated. Experiments verify that AdaCalib achieves significant improvement on calibration performance. It has been deployed online and beats previous approach.

References

[1]

Alexey Borisov, Julia Kiseleva, Ilya Markov, and Maarten de Rijke. 2018. Calibration: A simple way to improve click models. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management . 1503--1506.

Digital Library

[2]

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et almbox. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7--10.

Digital Library

[3]

Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for YouTube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191--198.

Digital Library

[4]

Chao Deng, Hao Wang, Qing Tan, Jian Xu, and Kun Gai. 2020. Calibrating user response predictions in online advertising. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases . 208--223.

[5]

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning . 1321--1330.

[6]

Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web. 507--517.

Digital Library

[7]

Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et almbox. 2014. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the eighth international workshop on data mining for online advertising . 1--9.

Digital Library

[8]

Siguang Huang, Yunli Wang, Lili Mou, Huayue Zhang, Han Zhu, Chuan Yu, and Bo Zheng. 2022. MBCT: Tree-Based Feature-Aware Binning for Individual Uncertainty Calibration. In Proceedings of The Web Conference 2022 .

Digital Library

[9]

Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical Reparameterization with Gumbel-Softmax. In International Conference on Learning Representations .

[10]

Meelis Kull, Telmo Silva Filho, and Peter Flach. 2017. Beta calibration: A well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statistics . 623--631.

[11]

Ananya Kumar, Percy S Liang, and Tengyu Ma. 2019. Verified uncertainty calibration. Advances in Neural Information Processing Systems, Vol. 32 (2019).

[12]

Wonbin Kweon, SeongKu Kang, and Hwanjo Yu. 2022. Obtaining Calibrated Probabilities with Personalized Ranking Models. In Proceedings of the 36th AAAI Conference on Artificial Intelligence .

[13]

Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018. Entire space multi-task model: An effective approach for estimating post-click conversion rate. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval . 1137--1140.

Digital Library

[14]

H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et almbox. 2013. Ad click prediction: A view from the trenches. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining . 1222--1230.

Digital Library

[15]

Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using Bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence .

Digital Library

[16]

Feiyang Pan, Xiang Ao, Pingzhong Tang, Min Lu, Dapeng Liu, Lei Xiao, and Qing He. 2020. Field-aware calibration: A simple and empirically strong method for reliable probabilistic predictions. In Proceedings of The Web Conference 2020 . 729--739.

Digital Library

[17]

John Platt et almbox. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, Vol. 10, 3 (1999), 61--74.

[18]

Penghui Wei, Weimin Zhang, Zixuan Xu, Shaoguo Liu, Kuang-chih Lee, and Bo Zheng. 2021. AutoHERI: Automated Hierarchical Representation Integration for Post-Click Conversion Rate Estimation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management . 3528--3532.

Digital Library

[19]

Zixuan Xu, Penghui Wei, Weimin Zhang, Shaoguo Liu, Liang Wang, and Bo Zheng. 2022. UKD: Debiasing Conversion Rate Estimation via Uncertainty-regularized Knowledge Distillation. In Proceedings of The Web Conference 2022 .

Digital Library

[20]

Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning. 609--616.

Digital Library

[21]

Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. 694--699.

Digital Library

[22]

Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining . 1059--1068.

Digital Library

Cited By

Yang SYang HZou ZXu LYuan SZeng YBaeza-Yates RBonchi F(2024)Deep Ensemble Shape Calibration: Multi-Field Post-hoc Calibration in Online AdvertisingProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671529(6117-6126)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671529
Zhao YWu CJia QZhu HYan JZong LZhang LDong ZZhang MSerra ESpezzano F(2024)Confidence-Aware Multi-Field Model CalibrationProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680043(5111-5118)Online publication date: 21-Oct-2024
https://doi.org/10.1145/3627673.3680043
Guo YAo XLiu QHe QFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Leveraging Post-Click User Behaviors for Calibrated Conversion Rate Prediction Under Delayed Feedback in Online AdvertisingProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615161(3918-3922)Online publication date: 21-Oct-2023
https://doi.org/10.1145/3583780.3615161
Show More Cited By

Index Terms

Posterior Probability Matters: Doubly-Adaptive Calibration for Neural Predictions in Online Advertising
1. Information systems
  1. Information systems applications
    1. Computational advertising

Recommendations

Reliable Posterior Probability Estimation for Streaming Face Recognition
CVPRW '14: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops

Increasing access to large, non-stationary face datasets and corresponding demands to process, analyze and learn from this data. This has lead to a new class of online/ incremental face recognition problems. While it is advantageous to build large scale ...
Obtaining calibrated probability using ROC Binning

Obtaining calibrated probability, or actual occurrence, is crucial in many real problems because it effectively supports the decision-making process with good assessment of cost and effect. Estimating calibrated probability is a more significant issue ...
Your Hometown Matters: Popularity-Difference Bias in Online Reputation Platforms
We study a new source of bias in online review platforms that originates from the popularity difference between the traveling reviewer’s hometown and destination.
Through analyzing a large set of restaurant reviews from a major online reputation platform, we find empirical evidence that popularity difference affects both the assigned rating and the text-encoded sentiment of a review. When reviewers travel to a less ...
We study a new source of bias in online review platforms that originates from the popularity difference between the traveling reviewer’s hometown and destination (popularity-difference bias). In particular, we model popularity-difference bias as a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2022

3569 pages

ISBN:9781450387323

DOI:10.1145/3477495

General Chairs:
Enrique Amigo
UNED
,
Pablo Castells
UAM and Amazon
,
Julio Gonzalo
UNED
,
Program Chairs:
Ben Carterette
Spotify
,
J. Shane Culpepper
RMIT University
,
Gabriella Kazai
Waseda University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Short-paper

Conference

SIGIR '22

Sponsor:

SIGIR

SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2022

Madrid, Spain

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
125
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)1

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang SYang HZou ZXu LYuan SZeng YBaeza-Yates RBonchi F(2024)Deep Ensemble Shape Calibration: Multi-Field Post-hoc Calibration in Online AdvertisingProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671529(6117-6126)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671529
Zhao YWu CJia QZhu HYan JZong LZhang LDong ZZhang MSerra ESpezzano F(2024)Confidence-Aware Multi-Field Model CalibrationProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680043(5111-5118)Online publication date: 21-Oct-2024
https://doi.org/10.1145/3627673.3680043
Guo YAo XLiu QHe QFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Leveraging Post-Click User Behaviors for Calibrated Conversion Rate Prediction Under Delayed Feedback in Online AdvertisingProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615161(3918-3922)Online publication date: 21-Oct-2023
https://doi.org/10.1145/3583780.3615161
Wei PDou HLiu STang RLiu LWang LZheng BChen HDuh WHuang HKato MMothe JPoblete B(2023)FedAds: A Benchmark for Privacy-Preserving CVR Estimation with Vertical Federated LearningProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591909(3037-3046)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591909

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten