skip to main content
10.1145/3583780.3615021acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

Quantifying the Effectiveness of Advertising: A Bootstrap Proportion Test for Brand Lift Testing

Published: 21 October 2023 Publication History

Abstract

Brand Lift test is a widely deployed statistical tool for measuring the effectiveness of online advertisements on brand perception such as ad recall, brand familiarity and favorability. By formulating the problem of interest into a two-sample test on the binomial proportions from the control group (p_0) and the treatment group (p_1), Brand Lift test evaluates ads impact based on the statistical significance of test results. Traditional approaches construct the test statistics based on the absolute difference between the two observed proportions, a.k.a, absolute lift. In this work, we propose a new bootstrap test based on the percentage difference between the two observed proportions, i.e., relative lift. We provide rigorous theoretical guarantees on the asymptotic validity of the proposed relative-lift-based test. Our numerical studies suggest that the relative-lift-based test requires less stringent conditions than the absolute-lift-based test for controlling the type-I error rate. Interestingly, we also prove that the relative-lift-based test is more powerful than the absolute-lift-based test when the alternative is positive (i.e., p1 - p0 > 0), but less powerful when the alternative is negative (i.e., p1 - p0 < 0). The empirical performance of the proposed test is demonstrated by extensive simulation studies, an application to a publicly available A/B testing dataset from advertising, and real datasets collected from the Brand Lift Testing platform at LinkedIn.

References

[1]
Alan Agresti and Brian Caffo. 2000. Simple and effective confidence intervals for proportions and differences of proportions result from adding two successes and two failures. The American Statistician 54, 4 (2000), 280--288.
[2]
Eytan Bakshy, Dean Eckles, and Michael S Bernstein. 2014. Designing and deploying online field experiments. In Proceedings of the 23rd International Conference on World Wide Web. 283--292.
[3]
GA Barnard. 1947. Significance tests for 2× 2 tables. Biometrika 34, 1/2 (1947), 123--138.
[4]
George Alfred Barnard. 1945. A new test for 2× 2 tables. Nature 156, 3954 (1945), 177.
[5]
Iavor Bojinov and Somit Gupta. 2022. Online experimentation: Benefits, operational and methodological challenges, and scaling guide. Harvard Data Science Review 4, 3 (2022).
[6]
Lawrence D Brown, T Tony Cai, and Anirban DasGupta. 2001. Interval estimation for a binomial proportion. Statist. Sci. 16, 2 (2001), 101--133.
[7]
David Chan, Rong Ge, Ori Gershony, Tim Hesterberg, and Diane Lambert. 2010. Evaluating online ad campaigns in a pipeline: causal models at scale. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 7--16.
[8]
Hokwon Cho. 2013. Approximate confidence limits for the ratio of two binomial variates with unequal sample sizes. Communications for Statistical Applications and Methods 20, 5 (2013), 347--356.
[9]
William J Conover. 1974. Some reasons for not using the Yates continuity correction on 2× 2 contingency tables. J. Amer. Statist. Assoc. 69, 346 (1974), 374--376.
[10]
Alex Deng, Ulf Knoblich, and Jiannan Lu. 2018. Applying the Delta method in metric analytics: A practical guide with novel ideas. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 233--242.
[11]
Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. 2013. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining. 123--132.
[12]
Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap. CRC press.
[13]
Jianqing Fan, Yuan Liao, and Jiawei Yao. 2015. Power enhancement in high-dimensional cross-sectional tests. Econometrica 83, 4 (2015), 1497--1541.
[14]
Rachel Fan, Tim Hesterberg, Ying Liu, and Lu Zhang. 2018. Methods for Measuring Brand Lift of Online Ads. (2018).
[15]
Ronald A Fisher. 1935. The logic of inductive inference. Journal of the Royal Statistical Society 98, 1 (1935), 39--82.
[16]
Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. 2013. Statistical methods for rates and proportions. John Wiley & Sons.
[17]
BK Ghosh. 1979. A comparison of some approximate confidence intervals for the binomial parameter. J. Amer. Statist. Assoc. 74, 368 (1979), 894--900.
[18]
Brett R Gordon, Florian Zettelmeyer, Neha Bhargava, and Dan Chapsky. 2019. A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science 38, 2 (2019), 193--225.
[19]
Garrett A Johnson, Randall A Lewis, and Elmar I Nubbemeyer. 2017. Ghost ads: Improving the economics of measuring online ad effectiveness. Journal of Marketing Research 54, 6 (2017), 867--884.
[20]
Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. 2013. Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1168--1176.
[21]
Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press.
[22]
Randall A Lewis and Justin M Rao. 2015. The unfavorable economics of measuring the returns to advertising. The Quarterly Journal of Economics 130, 4 (2015), 1941--1973.
[23]
Min Liu, Jialiang Mao, and Kang Kang. 2021. Trustworthy and Powerful Online Marketplace Experimentation with Budget-split Design. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 3319--3329.
[24]
Karl Pearson. 1900. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 50, 302 (1900), 157--175.
[25]
Diane Tang, Ashish Agarwal, Deirdre O'Brien, and Mike Meyer. 2010. Overlapping experiment infrastructure: More, better, faster experimentation. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 17--26.
[26]
Keith D Tocher. 1950. Extension of the Neyman-Pearson theory of tests to discontinuous variates. Biometrika 37, 1/2 (1950), 130--144.
[27]
Stein Emil Vollset. 1993. Confidence intervals for a binomial proportion. Statistics in Medicine 12, 9 (1993), 809--824.
[28]
Ya Xu, Nanyu Chen, Addrian Fernandez, Omar Sinno, and Anmol Bhasin. 2015. From infrastructure to culture: A/B testing challenges in large scale social networks. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2227--2236.
[29]
Frank Yates. 1934. Contingency tables involving small numbers and the X2 test. Supplement to the Journal of the Royal Statistical Society 1, 2 (1934), 217--235.
[30]
Xiufan Yu, Danning Li, Lingzhou Xue, and Runze Li. 2022. Power-enhanced simultaneous test of high-dimensional mean vectors and covariance matrices with application to gene-set testing. J. Amer. Statist. Assoc. in press (2022), 1--14.
[31]
Xiufan Yu, Jiawei Yao, and Lingzhou Xue. 2023. Power enhancement for testing multi-factor asset pricing models via Fisher's method. Journal of Econometrics in press (2023), 1--22.

Cited By

View all

Index Terms

  1. Quantifying the Effectiveness of Advertising: A Bootstrap Proportion Test for Brand Lift Testing

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management
    October 2023
    5508 pages
    ISBN:9798400701245
    DOI:10.1145/3583780
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 21 October 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. biased test
    2. binomial distribution
    3. brand advertising
    4. power enhancement
    5. relative lift

    Qualifiers

    • Research-article

    Conference

    CIKM '23
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

    Upcoming Conference

    CIKM '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 87
      Total Downloads
    • Downloads (Last 12 months)46
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 08 Mar 2025

    Other Metrics

    Citations

    Cited By

    View all

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media