short-paper

Sign-Aware Periodicity Metrics of User Engagement for Online Search Quality Evaluation

Author:

SIGIR '15: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 779 - 782

https://doi.org/10.1145/2766462.2767814

Published: 09 August 2015 Publication History

Get Access

Abstract

Modern Internet companies improve evaluation criteria of their data-driven decision-making that is based on online controlled experiments (also known as A/B tests). The amplitude metrics of user engagement are known to be well sensitive to service changes, but they could not be used to determine, whether the treatment effect is positive or negative. We propose to overcome this sign-agnostic issue by paying attention to the phase of the corresponding DFT sine wave. We refine the amplitude metrics of the first frequency by the phase ones and formalize our intuition in several novel overall evaluation criteria. These criteria are then verified over A/B experiments on real users of Yandex. We find that our approach holds the sensitivity level of the amplitudes and makes their changes sign-aware w.r.t. the treatment effect.

References

[1]

T. Crook, B. Frasca, R. Kohavi, and R. Longbotham. Seven pitfalls to avoid when running controlled experiments on the web. In KDD'2009, pages 1105--1114. ACM, 2009.

Digital Library

Google Scholar

[2]

A. Deng, Y. Xu, R. Kohavi, and T. Walker. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In WSDM'2013, pages 123--132. ACM, 2013.

Digital Library

Google Scholar

[3]

A. Drutsa, G. Gusev, and P. Serdyukov. Engagement periodicity in search engine usage: Analysis and its application to search quality evaluation. In WSDM'2015, pages 27--36, 2015.

Digital Library

Google Scholar

[4]

A. Drutsa, G. Gusev, and P. Serdyukov. Future user engagement prediction and its application to improve the sensitivity of online experiments. In WWW'2015, 2015.

Digital Library

Google Scholar

[5]

G. Dupret and M. Lalmas. Absence time and user engagement: evaluating ranking functions. In WSDM'2013, pages 173--182, 2013.

Digital Library

Google Scholar

[6]

R. Kohavi, A. Deng, B. Frasca, R. Longbotham, T. Walker, and Y. Xu. Trustworthy online controlled experiments: Five puzzling outcomes explained. In KDD'2012, pages 786--794. ACM, 2012.

Digital Library

Google Scholar

[7]

R. Kohavi, A. Deng, R. Longbotham, and Y. Xu. Seven rules of thumb for web site experimenters. In KDD'2014, 2014.

Digital Library

Google Scholar

[8]

R. Kohavi, R. Longbotham, D. Sommerfield, and R. M. Henne. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, 18(1):140--181, 2009.

Digital Library

Google Scholar

[9]

Y. Song, X. Shi, and X. Fu. Evaluating and predicting user engagement change with degraded search relevance. In WWW'2013, pages 1213--1224, 2013.

Digital Library

Google Scholar

[10]

D. Tang, A. Agarwal, D. O'Brien, and M. Meyer. Overlapping experiment infrastructure: More, better, faster experimentation. In KDD'2010, pages 17--26. ACM, 2010.

Digital Library

Google Scholar

Cited By

View all

Drutsa AGusev GKharitonov EKulemyakin DSerdyukov PYashkov IPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Effective Online Evaluation for Web SearchProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331378(1399-1400)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331378
Budylin RDrutsa AKatsev ITsoy VChang YZhai CLiu YMaarek Y(2018)Consistent Transformation of Ratio Metrics for Efficient Online Controlled ExperimentsProceedings of the Eleventh ACM International Conference on Web Search and Data Mining10.1145/3159652.3159699(55-63)Online publication date: 2-Feb-2018
https://dl.acm.org/doi/10.1145/3159652.3159699
Drutsa AGusev GSerdyukov PBarrett RCummings RAgichtein EGabrilovich E(2017)Using the Delay in a Treatment Effect to Improve Sensitivity and Preserve Directionality of Engagement Metrics in A/B ExperimentsProceedings of the 26th International Conference on World Wide Web10.1145/3038912.3052664(1301-1310)Online publication date: 3-Apr-2017
https://dl.acm.org/doi/10.1145/3038912.3052664
Show More Cited By

Index Terms

Sign-Aware Periodicity Metrics of User Engagement for Online Search Quality Evaluation
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
      1. Cognitive science
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Practical Aspects of Sensitivity in Online Experimentation with User Engagement Metrics
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

Online controlled experiments, e.g., A/B testing, is the state-of-the-art approach used by modern Internet companies to improve their services based on data-driven decisions. The most challenging problem is to define an appropriate online metric of user ...
Periodicity in User Engagement with a Search Engine and Its Application to Online Controlled Experiments

Nowadays, billions of people use the Web in connection with their daily needs. A significant part of these needs are constituted by search tasks that are usually addressed by search engines. Thus, daily search needs result in regular user engagement ...
Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments
WWW '15: Proceedings of the 24th International Conference on World Wide Web

Modern Internet companies improve their services by means of data-driven decisions that are based on online controlled experiments (also known as A/B tests). To run more online controlled experiments and to get statistically significant results faster ...

Comments

Information & Contributors

Information

Published In

SIGIR '15: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval

August 2015

1198 pages

ISBN:9781450336215

DOI:10.1145/2766462

General Chair:
Ricardo Baeza-Yates
Yahoo Labs, USA
,
Program Chairs:
Mounia Lalmas
Yahoo Labs, UK
,
Alistair Moffat
University of Melbourne, Australia
,
Berthier Ribeiro-Neto
Google, Brazil, and UFMG, Brazil

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 August 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

SIGIR '15

Sponsor:

SIGIR

SIGIR '15: The 38th International ACM SIGIR conference on research and development in Information Retrieval

August 9 - 13, 2015

Santiago, Chile

Acceptance Rates

SIGIR '15 Paper Acceptance Rate 70 of 351 submissions, 20%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
176
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Drutsa AGusev GKharitonov EKulemyakin DSerdyukov PYashkov IPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Effective Online Evaluation for Web SearchProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331378(1399-1400)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331378
Budylin RDrutsa AKatsev ITsoy VChang YZhai CLiu YMaarek Y(2018)Consistent Transformation of Ratio Metrics for Efficient Online Controlled ExperimentsProceedings of the Eleventh ACM International Conference on Web Search and Data Mining10.1145/3159652.3159699(55-63)Online publication date: 2-Feb-2018
https://dl.acm.org/doi/10.1145/3159652.3159699
Drutsa AGusev GSerdyukov PBarrett RCummings RAgichtein EGabrilovich E(2017)Using the Delay in a Treatment Effect to Improve Sensitivity and Preserve Directionality of Engagement Metrics in A/B ExperimentsProceedings of the 26th International Conference on World Wide Web10.1145/3038912.3052664(1301-1310)Online publication date: 3-Apr-2017
https://dl.acm.org/doi/10.1145/3038912.3052664
Drutsa AGusev GSerdyukov P(2017)Periodicity in User Engagement with a Search Engine and Its Application to Online Controlled ExperimentsACM Transactions on the Web10.1145/285682211:2(1-35)Online publication date: 14-Apr-2017
https://dl.acm.org/doi/10.1145/2856822
Poyarkov ADrutsa AKhalyavin AGusev GSerdyukov PKrishnapuram BShah MSmola AAggarwal CShen DRastogi R(2016)Boosted Decision Tree Regression Adjustment for Variance Reduction in Online Controlled ExperimentsProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining10.1145/2939672.2939688(235-244)Online publication date: 13-Aug-2016
https://dl.acm.org/doi/10.1145/2939672.2939688
Nikolaev KDrutsa AGladkikh EUlianov AGusev GSerdyukov PCao LZhang CJoachims TWebb GMargineantu DWilliams G(2015)Extreme States Distribution Decomposition Method for Search Engine Online EvaluationProceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining10.1145/2783258.2783415(845-854)Online publication date: 10-Aug-2015
https://dl.acm.org/doi/10.1145/2783258.2783415

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Practical Aspects of Sensitivity in Online Experimentation with User Engagement Metrics

Periodicity in User Engagement with a Search Engine and Its Application to Online Controlled Experiments

Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations