Joel Barajas
ABSTRACT
Online advertising has historically been approached as an ad-to-user matching problem within sophisticated optimization algorithms. As the research and ad-tech industries have progressed, advertisers have increasingly emphasized the causal effect estimation of their ads (incrementality) using controlled experiments (A/B testing). With low lift effects and sparse conversion, the development of incrementality testing platforms at scale suggests tremendous engineering challenges in measurement precision. Similarly, the correct interpretation of results addressing a business goal requires significant data science and experimentation research expertise. We propose a practical tutorial in the incrementality testing landscape, including: The business need; Literature solutions and industry practices; Designs in the development of testing platforms; The testing cycle, case studies, and recommendations. We provide first-hand lessons based on the development of such a platform in a major combined DSP and ad network, and after running several tests for up to two months each over recent years.
- Alberto Abadie, Alexis Diamond, and Jens Hainmueller. 2010. Synthetic control methods for comparative case studies: Estimating the effect of California's tobacco control program. Journal of the American statistical Association 105, 490 (2010), 493--505.Google Scholar
Cross Ref
- Joel Barajas, Ram Akella, Marius Holtan, and Aaron Flores. 2016. Experimental designs and estimation for online display advertising attribution in marketplaces. Marketing Science 35, 3 (2016), 465--483. Google ScholarDigital Library
- Joel Barajas and Narayan Bhamidipati. 2021. Incrementality Testing in Programmatic Advertising: Enhanced Precision with Double-Blind Designs. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW '21). Association for Computing Machinery, New York, NY, USA, 2818--2827. https://doi.org/10.1145/3442381.3450106 Google ScholarDigital Library
- Joel Barajas, Narayan Bhamidipati, and James G. Shanahan. 2021. Online Advertising Incrementality Testing And Experimentation: Industry Practical Lessons. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (Virtual Event, Singapore) (KDD '21). Association for Computing Machinery, New York, NY, USA, 4027--4028. https://doi.org/10.1145/3447548.3470819 Google ScholarDigital Library
- Joel Barajas, Tom Zidar, and Mert Bay. 2020. Advertising Incrementality Measurement using Controlled Geo-Experiments: The Universal App Campaign Case Study. (2020).Google Scholar
- Thomas Blake, Chris Nosko, and Steven Tadelis. 2015. Consumer heterogeneity and paid search effectiveness: A large-scale field experiment. Econometrica 83, 1 (2015), 155--174.Google ScholarCross Ref
- David Chan, Rong Ge, Ori Gershony, Tim Hesterberg, and Diane Lambert. 2010. Evaluating Online Ad Campaigns in a Pipeline: Causal Models at Scale. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Washington, DC, USA) (KDD '10). ACM, New York, NY, USA, 7--16. https://doi.org/10.1145/1835804.1835809 Google ScholarDigital Library
- Ayman Farahat and James Shanahan. 2013. Econometric analysis and digital marketing: how to measure the effectiveness of an ad. In Proceedings of the sixth ACM international conference on Web search and data mining. 785--785. Google ScholarDigital Library
- C. E. Frangakis and D. B. Rubin. 2002. Principal stratification in causal inference. Biometrics 58, 1 (2002), 21--29. https://doi.org/10.1111/j.0006--341X.2002.00021.xGoogle ScholarCross Ref
- Brett R Gordon, Florian Zettelmeyer, Neha Bhargava, and Dan Chapsky. 2019. A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science 38, 2 (2019), 193--225.Google ScholarDigital Library
- Guido W. Imbens and Donald B. Rubin. 1997. Bayesian Inference for Causal Effects in Randomized Experiments with Noncompliance. The Annals of Statistics 25, 1 (1997), 305--327. http://www.jstor.org/stable/2242722Google ScholarCross Ref
- Garrett A. Johnson, Randall A. Lewis, and Elmar I. Nubbemeyer. 2017. Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness. Journal of Marketing Research 54, 6 (2017), 867--884. https://doi.org/10.1509/jmr.15.0297 arXiv:https://doi.org/10.1509/jmr.15.0297Google ScholarCross Ref
- Pavel Kireyev, Koen Pauwels, and Sunil Gupta. 2016. Do display ads influence search? Attribution and dynamics in online advertising. International Journal of Research in Marketing 33, 3 (2016), 475--490.Google ScholarCross Ref
- Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.Google Scholar
- Randall A. Lewis and Justin M. Rao. 2015. The Unfavorable Economics of Measuring the Returns to Advertising *. The Quarterly Journal of Economics 130, 4 (2015), 1941--1973.Google ScholarCross Ref
- Randall A. Lewis, Justin M. Rao, and David H. Reiley. 2011. Here, There, and Everywhere: Correlated Online Behaviors Can Lead to Overestimates of the Effects of Advertising. In Proceedings of the 20th International Conference on World Wide Web (Hyderabad, India) (WWW '11). ACM, New York, NY, USA, 157--166. https://doi.org/10.1145/1963405.1963431 Google ScholarDigital Library
- Hongshuang (Alice) Li and P.K. Kannan. 2014. Attributing Conversions in a Multichannel Online Marketing Environment: An Empirical Model and a Field Experiment. Journal of Marketing Research 51, 1 (2014), 40--56. https://doi.org/10.1509/jmr.13.0050 arXiv:https://doi.org/10.1509/jmr.13.0050Google ScholarCross Ref
- Tesary Lin and Sanjog Misra. 2020. The Identity Fragmentation Bias. arXiv:2008.12849 [econ.EM]Google Scholar
- Donald B Rubin. 2005. Causal Inference Using Potential Outcomes. J. Amer. Statist. Assoc. 100, 469 (2005), 322--331. https://doi.org/10.1198/016214504000001880 arXiv:https://doi.org/10.1198/016214504000001880Google ScholarCross Ref
Index Terms
- Online Advertising Incrementality Testing: Practical Lessons And Emerging Challenges
Recommendations
Online Advertising Incrementality Testing And Experimentation: Industry Practical Lessons
KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data MiningOnline advertising has historically been approached as user targeting and ad-to-user matching problems within sophisticated optimization algorithms. As the research area and ad tech industry have progressed over the last couple of decades, advertisers ...
Online Advertising Incrementality Testing: Practical Lessons, Paid Search and Emerging Challenges
Advances in Information RetrievalAbstractOnline advertising has historically been approached as an ad-to-user matching problem within sophisticated optimization algorithms. As the research and ad tech industries have progressed, advertisers have increasingly emphasized the causal effect ...
An empirical analysis of paid placement in online advertising
ICEC '07: Proceedings of the ninth international conference on Electronic commerceThe Internet has brought about a fundamental change in the way consumers obtain and process information. It is well documented now that the online retailing revolution has established a new distribution channel that represents a fundamental paradigm ...
Comments