Abstract
Online advertising has historically been approached as an ad-to-user matching problem within sophisticated optimization algorithms. As the research and ad tech industries have progressed, advertisers have increasingly emphasized the causal effect estimation of their ads (incrementality) using controlled experiments (A/B testing). With low lift effects and sparse conversion, the development of incrementality testing platforms at scale suggests tremendous engineering challenges in measurement precision. Similarly, the correct interpretation of results addressing a business goal requires significant data science and experimentation research expertise.
We propose a practical tutorial in the incrementality testing landscape, including:
– The business need
– Literature solutions and industry practices
– Designs in the development of testing platforms
– The testing cycle, case studies, and recommendations
– Paid search effectiveness in the marketplace
– Emerging privacy challenges for incrementality testing and research solutions
We provide first-hand lessons based on the development of such a platform in a major combined DSP and ad network, and after running several tests for up to two months each over recent years. With increasing privacy constraints, we survey literature and current practices. These practices include private set union and differential privacy for conversion modeling, and geo-testing combined with synthetic control techniques.
J. Barajas—Work done while the author was employed at Yahoo.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abadie, A., Diamond, A., Hainmueller, J.: Synthetic control methods for comparative case studies: estimating the effect of California’s tobacco control program. J. Am. Stat. Assoc. 105(490), 493–505 (2010)
Barajas, J., Akella, R., Holtan, M., Flores, A.: Experimental designs and estimation for online display advertising attribution in marketplaces. Mark. Sci. 35(3), 465–483 (2016)
Barajas, J., Bhamidipati, N.: Incrementality testing in programmatic advertising: enhanced precision with double-blind designs. In: Proceedings of the Web Conference 2021, pp. 2818–2827. WWW 2021, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3442381.3450106
Barajas, J., Bhamidipati, N., Shanahan, J.G.: Online advertising incrementality testing and experimentation: industry practical lessons. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 4027–4028. KDD 2021, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3447548.3470819
Barajas, J., Bhamidipati, N., Shanahan, J.G.: Online advertising incrementality testing: practical lessons and emerging challenges. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 4838–4841. CIKM 2021, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3459637.3482031
Barajas, J., Zidar, T., Bay, M.: Advertising incrementality measurement using controlled geo-experiments: the universal app campaign case study (2020)
Blake, T., Nosko, C., Tadelis, S.: Consumer heterogeneity and paid search effectiveness: a large-scale field experiment. Econometrica 83(1), 155–174 (2015)
Chan, D., Ge, R., Gershony, O., Hesterberg, T., Lambert, D.: Evaluating online ad campaigns in a pipeline: causal models at scale. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 7–16. KDD 2010, ACM, New York, NY, USA (2010). https://doi.org/10.1145/1835804.1835809,http://doi.acm.org/10.1145/1835804.1835809
Dai, D., Luca, M.: Effectiveness of paid search advertising: Experimental evidence. Technical report, Harvard Business School (October 2016). workin Paper No. 17–025
Farahat, A., Shanahan, J.: Econometric analysis and digital marketing: how to measure the effectiveness of an ad. In: Proceedings of the sixth ACM International Conference on Web Search and Data Mining, pp. 785–785 (2013)
Frangakis, C., Rubin, D.: Principal stratification in causal inference. Biometrics 58(1), 21–29 (2002). https://doi.org/10.1111/j.0006-341X.2002.00021.x
Gordon, B.R., Zettelmeyer, F., Bhargava, N., Chapsky, D.: A comparison of approaches to advertising measurement: evidence from big field experiments at Facebook. Mark. Sci. 38(2), 193–225 (2019)
Imbens, G.W., Rubin, D.B.: Bayesian inference for causal effects in randomized experiments with noncompliance. Ann. Stat. 25(1), 305–327 (1997). http://www.jstor.org/stable/2242722
Johnson, G.A., Lewis, R.A., Nubbemeyer, E.I.: Ghost ads: improving the economics of measuring online ad effectiveness. J. Mark. Res. 54(6), 867–884 (2017). https://doi.org/10.1509/jmr.15.0297
Kireyev, P., Pauwels, K., Gupta, S.: Do display ads influence search? Attribution and dynamics in online advertising. Int. J. Res. Mark. 33(3), 475–490 (2016)
Kohavi, R., Tang, D., Xu, Y.: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, Cambridge (2020)
Lewis, R.A., Rao, J.M.: The unfavorable economics of measuring the returns to advertising *. Q. J. Econ. 130(4), 1941–1973 (2015)
Lewis, R.A., Rao, J.M., Reiley, D.H.: Here, there, and everywhere: correlated online behaviors can lead to overestimates of the effects of advertising. In: Proceedings of the 20th International Conference on World Wide Web, pp. 157–166. WWW 2011, ACM, New York, NY, USA (2011). https://doi.org/10.1145/1963405.1963431,http://doi.acm.org/10.1145/1963405.1963431
Li, H.A., Kannan, P.: Attributing conversions in a multichannel online marketing environment: an empirical model and a field experiment. J. Mark. Res. 51(1), 40–56 (2014)
Lin, T., Misra, S.: The identity fragmentation bias (2020)
Moshary, S.: Sponsored search in equilibrium: evidence from two experiments. Available at SSRN 3903602 (2021)
Rubin, D.B.: Causal inference using potential outcomes. J. Am. Stat. Assoc. 100(469), 322–331 (2005). https://doi.org/10.1198/016214504000001880
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Barajas, J., Bhamidipati, N., Shanahan, J.G. (2022). Online Advertising Incrementality Testing: Practical Lessons, Paid Search and Emerging Challenges. In: Hagen, M., et al. Advances in Information Retrieval. ECIR 2022. Lecture Notes in Computer Science, vol 13186. Springer, Cham. https://doi.org/10.1007/978-3-030-99739-7_72
Download citation
DOI: https://doi.org/10.1007/978-3-030-99739-7_72
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-99738-0
Online ISBN: 978-3-030-99739-7
eBook Packages: Computer ScienceComputer Science (R0)