skip to main content
10.1145/3530019.3534983acmotherconferencesArticle/Chapter ViewAbstractPublication PageseaseConference Proceedingsconference-collections
extended-abstract

When experimentation starts as a solution to raise ROI: The pitfall of not having the right scope for experimentation ROI

Published: 13 June 2022 Publication History

Abstract

Experimentation in organizations often starts with the marketing department hiring an agency to raise the ROI of their marketing efforts. Scaling experimentation in the whole organization leads to a larger quantity of experiments, preventing changes without testing, thus making better decisions, and reducing risk. However, we have learned that broadly scaling experimentation has negative effects on the marketing departments KPI of more revenue, which is preventing experimentation from scaling up. The main objective of this paper is to make organizations aware of this Catch-22 when starting experimentation with the scope of raising ROI. We want to convince marketing departments, their agencies, and their vendors that if experimentation starts at marketing, then they need to create more awareness for the full power of experimentation by presenting all the results including the more common inconclusive and negative outcomes of experiments. Our learnings suggest that this awareness must lead to budget and resources for a stand-alone experimentation Center of Excellence (CoE) outside of the marketing department, that different (not ROI focused) KPIs are needed when scaling up experimentation, and that specific organizational conditions must be met to maintain a future-proof set-up.

References

[1]
Online Dialogue study of vendor and agency websites published case studies (2022). Retrieved from https://anali.st/vendoragencycases
[2]
Stefan H. Thomke (2020): Experimentation Works, Harvard Business Review Press
[3]
A. Fabijan, P. Dmitriev, H. H. Olsson and J. Bosch, "The Evolution of Continuous Experimentation in Software Product Development: From Data to a Data-Driven Organization at Scale," 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), 2017, pp. 770-780.
[4]
Khoja, T.H. (2010): Highlights on Evidence Based Medicine. Executive Board of the Health. Ministers'Council, Sixth Edition, L.D. No. 1430/1040
[5]
Kohavi, Crook, Longbotham (2009): Online Experimentation at Microsoft. Retrieved from https://exp-platform.com/Documents/ExP_DMCaseStudies.pdf
[6]
Bryan, Kerin, Ian Herbert (2011): “The Centre of Excellence”, E&T Magazine, pp. 1-9, 06.10.2011
[7]
A. Fabijan, B. Arai, P. Dmitriev and L. Vermeer, "It takes a Flywheel to Fly: Kickstarting and Growing the A/B testing Momentum at Scale," 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2021, pp. 109-118.
[8]
D. Tang, R. Kohavi, Y. Xu (2020): Trustworthy Online Controlled Experiments, Cambridge University Press
[9]
Raphael Lopez Kaufman, Jegar Pitchforth, Lukas Vermeer (2017): Democratizing online controlled experiments at Booking.com. Retrieved from: https://doi.org/10.48550/arXiv.1710.08217

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
EASE '22: Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering
June 2022
466 pages
ISBN:9781450396134
DOI:10.1145/3530019
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2022

Check for updates

Author Tags

  1. Experimentation
  2. Experimentation Center of Excellence
  3. Experimentation Program
  4. Scaling Experimentation

Qualifiers

  • Extended-abstract
  • Research
  • Refereed limited

Conference

EASE 2022

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 56
    Total Downloads
  • Downloads (Last 12 months)5
  • Downloads (Last 6 weeks)1
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media