Pavel Procházka
ABSTRACT
Blending of search results from several vertical sources became standard among web search engines. Similar scenarios appear in computational advertising, news recommendation, and other interactive systems. As such environments give only partial feedback, the evaluation of new policies conventionally requires expensive online A/B tests. Counterfactual approach is a promising alternative, nevertheless, it requires specific conditions for a valid off-policy evaluation. We release a large-scale, real-world vertical-blending dataset gathered bySeznam.cz web search engine. The dataset contains logged partial feedback with the corresponding propensity and is thus suited for counterfactual evaluation. We provide basic checks for validity and evaluate several learning methods.
- Jaime Arguello. 2017. Aggregated Search. Found. Trends Inf. Retr. 10, 5 (March 2017), 365--502. Google Scholar
Digital Library
- Jie at al. 2013. A Unified Search Federation System Based on Online User Feedback. In Proceedings of the 19th ACM SIGKDD (KDD '13). ACM, New York, NY, USA, 1195--1203. Google ScholarDigital Library
- Vorobev at al. 2015. Gathering Additional Feedback on Search Results by Multi-Armed Bandits with Respect to Production Ranking. In Proceedings of the 24th International Conference on WWW (WWW '15). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 1177--1187. Google ScholarDigital Library
- Miroslav Dudík, John Langford, and Lihong Li. 2011. Doubly Robust Policy Evaluation and Learning. In Proceedings of the 28th ICML (ICML'11). Omnipress,USA, 1097--1104. Google ScholarDigital Library
- Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham,and Simon Dollé. 2018. Offline A/B Testing for Recommender Systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (WSDM '18). ACM, New York, NY, USA, 198--206. Google ScholarDigital Library
- Katja Hofmann, Shimon Whiteson, and Maarten de Rijke. 2011. Contextual Bandits for Information Retrieval. In NIPS 2011: Workshop on Bayesian Optimization,Experimental Design and Bandits: Theory and Applications, Vol. 12.Google Scholar
- Thorsten Joachims and Adith Swaminathan. 2016. SIGIR Tutorial on Counterfactual Evaluation and Learning for Search, Recommendation and Ad Placement. In Proceedings of the 39th International ACM SIGIR. ACM, 1199--1201. Google ScholarDigital Library
- Thorsten Joachims, Adith Swaminathan, and Maarten de Rijke. 2018. Deep Learning with Logged Bandit Feedback. In International Conference on Learning Representations(iclr 2018 ed.).Google Scholar
- Oren Kurland and J. Shane Culpepper. 2018. Fusion in Information Retrieval:SIGIR 2018 Half-Day Tutorial. In The 41st International ACM SIGIR (SIGIR '18). ACM, New York, NY, USA, 1383--1386. Google ScholarDigital Library
- John Langford, Lihong Li, and Alexander Strehl. 2007. Vowpal wabbit opensource project. (2007). https://github.com/VowpalWabbit/vowpalwabbit/wikiGoogle Scholar
- Damien Lefortier, Adith Swaminathan, Xiaotao Gu, Thorsten Joachims, and Maarten de Rijke. 2016. Large-scale Validation of Counterfactual Learning Methods: A Test-Bed. CoRRabs/1612.00367 (2016). arXiv:1612.00367Google Scholar
- Lihong Li, Shunbao Chen, Jim Kleban, and Ankur Gupta. 2015. Counterfactual estimation and optimization of click metrics in search engines: A case study. In Proceedings of the 24th International Conference on WWW. ACM, 929--934. Google ScholarDigital Library
- Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on WWW. ACM, 661--670. Google ScholarDigital Library
- James McInerney, Benjamin Lacker, Samantha Hansen, Karl Higley, Hugues Bouchard, Alois Gruson, and Rishabh Mehrotra. 2018. Explore, Exploit, and Ex-plain: Personalizing Explainable Recommendations with Bandits. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys '18). ACM, New York, NY, USA, 31--39. Google ScholarDigital Library
- Paul R. Rosenbaum and Donald B. Rubin. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika 70, 1 (04 1983), 41--55.Google Scholar
- Daniel J. Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. 2018. A Tutorial on Thompson Sampling. Foundations and Trends in Machine Learning 11, 1 (2018), 1--96. Google ScholarDigital Library
- Marc Sloan and Jun Wang. 2012. Dynamical Information Retrieval Modelling: A Portfolio-armed Bandit Machine Approach. In Proceedings of the 21st International Conference on WWW (WWW '12 Companion). ACM, New York, NY, USA, 603--604. Google ScholarDigital Library
- Adith Swaminathan and Thorsten Joachims. 2015. The self-normalized estimator for counterfactual learning. In Advances in Neural Information Processing Systems.3231--3239. Google ScholarDigital Library
- Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miroslav Dudík,John Langford, Damien Jose, and Imed Zitouni. 2016. Off-policy evaluation for slate recommendation. CoRRabs/1605.04812 (2016). arXiv:1605.04812 http://arxiv.org/abs/1605.04812Google Scholar
Index Terms
- Vertical Search Blending: A Real-world Counterfactual Dataset
Recommendations
Incorporating vertical results into search click models
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrievalIn modern search engines, an increasing number of search result pages (SERPs) are federated from multiple specialized search engines (called verticals, such as Image or Video). As an effective approach to interpret users' click-through behavior as ...
Focused ranking in a vertical search engine
SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrievalSince the debut of PageRank and HITS, hyperlink-induced Web document ranking has come a long way. The Web has become increasingly vast and topically diverse. Such vastness has led many into the area of topic-sensitive ranking and its variants. We ...
Comments