Abstract
Query Auto Completion is the task of suggesting queries to the users of a search engine while they are typing a query in the search box. Over the recent years there has been a renewed interest in research on improving the quality of this task. The published improvements were assessed by using offline evaluation techniques and metrics. In this paper, we provide a comparison of online and offline assessments for Query Auto Completion. We show that there is a large potential for significant bias if the raw data used in an online experiment is re-used for offline experiments afterwards to evaluate new methods.
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
- 2.
An average of 2.06 documents were clicked per query. Queries without document clicks were not recorded.
- 3.
Note that in our online experiments, the average query length on which the users clicked as a completion is 11 characters.
- 4.
- 5.
References
Bar-Yossef, Z., Kraus, N.: Context-sensitive query auto-completion. In: Proceedings of WWW (2011)
Cai, F., de Rijke, M.: Selectively personalizing query auto-completion. In: Proceedings of SIGIR (2016)
Cai, F., Liang, S., de Rijke, M.: Time-sensitive personalized query auto completion. In: Proceedings of CIKM (2014)
Chapelle, O., Joachims, T., Radlinski, F., Yue, Y.: Large-scale validation and analysis of interleaved search evaluation. ACM Trans. Inf. Syst. 30, 1–41 (2012)
Di Santo, G., McCreadie, R., Macdonald, C., Ounis, I.: Comparing approaches for query autocompletion. In: Proceedings of SIGIR (2015)
Jiang, J.-Y., Ke, Y.-Y., Chien, P.-Y., Cheng, P.-J.: Learning user reformulation behavior for query auto-completion. In: Proceedings of SIGIR (2014)
Joachims, T.: Evaluating retrieval performance using clickthrough data. Text Mining, pp. 79–96 (2003)
Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Mining Knowl. Discov. 18, 140–181 (2009)
Radlinski, F., Kurup, M., Joachims, T.: How does clickthrough data reflect retrieval quality? In: Proceedings of CIKM (2008)
Shokouhi M.: Learning to personalize query auto-completion. In: Proceedings of SIGIR (2013)
Shokouhi, M., Radinsky K.: Time-sensitive query auto-completion. In: Proceedings of SIGIR (2012)
Acknowledgment
This research was partially supported by the EU Project KConnect (Grant No.: 644753) and the Austrian FWF Project ADmIRE (Project No.: P25905-N23).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Bampoulidis, A., Palotti, J., Lupu, M., Brassey, J., Hanbury, A. (2017). Does Online Evaluation Correspond to Offline Evaluation in Query Auto Completion?. In: Jose, J., et al. Advances in Information Retrieval. ECIR 2017. Lecture Notes in Computer Science(), vol 10193. Springer, Cham. https://doi.org/10.1007/978-3-319-56608-5_70
Download citation
DOI: https://doi.org/10.1007/978-3-319-56608-5_70
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-56607-8
Online ISBN: 978-3-319-56608-5
eBook Packages: Computer ScienceComputer Science (R0)