Abstract
Search engines use replication and distribution of large indices across many query servers to achieve efficient retrieval. Under high query load, queries can be scheduled to replicas that are expected to be idle soonest, facilitated by the use of predicted query response times. However, the overhead of making response time predictions can hinder the usefulness of query scheduling under low query load. In this paper, we propose a hybrid scheduling approach that combines the scheduling methods appropriate for both low and high load conditions, and can adapt in response to changing conditions. We deploy a simulation framework, which is prepared with actual and predicted response times for real Web search queries for one full day. Our experiments using different numbers of shards and replicas of the 50 million document ClueWeb09 corpus show that hybrid scheduling can reduce the average waiting times of one day of queries by 68% under high load conditions and by 7% under low load conditions w.r.t. traditional scheduling methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Dean, J.: Challenges in building large-scale information retrieval systems: invited talk. In: Proceedings of WSDM 2009 (2009)
Cacheda, F., Carneiro, V., Plachouras, V., Ounis, I.: Performance analysis of distributed information retrieval architectures using an improved network simulation model. Inf. Process. Manage. 43(1), 204–224 (2007)
Macdonald, C., Tonellotto, N., Ounis, I.: Learning to predict response times for online query scheduling. In: Proceedings of SIGIR 2012, pp. 621–630 (2012)
Freire, A., Macdonald, C., Tonellotto, N., Ounis, I., Cacheda, F.: Scheduling queries across replicas. In: Proceedings of SIGIR 2012, pp. 1139–1140 (2012)
Silvestri, F.: Mining query logs: Turning search usage data into knowledge. Foundations and Trends in Information Retrieval 4(1-2), 1–174 (2010)
Moffat, A., Webber, W., Zobel, J., Baeza-Yates, R.: A pipelined architecture for distributed text query evaluation. Inf. Retr. 10, 205–231 (2007)
Broder, A.Z., Carmel, D., Herscovici, M., Soffer, A., Zien, J.: Efficient query evaluation using a two-level retrieval process. In: Proceedings of CIKM 2003, pp. 426–434 (2003)
Brutlag, J.D., Hutchinson, H., Stone, M.: User preference and search engine latency. In: JSM Proceedings: Quality and Productivity Research Section (2008)
Lu, J., Callan, J.: Content-based retrieval in hybrid peer-to-peer networks. In: Proceedings of CIKM 2003, pp. 199–206 (2003)
Craswell, N., Jones, R., Dupret, G., Viegas, E. (eds.): Proceedings of the Web Search Click Data Workshop at WSDM 2009 (2009)
Macdonald, C., McCreadie, R., Santos, R., Ounis, I.: From puppy to maturity: Experiences in developing Terrier. In: Proc. of the OSIR at SIGIR 2012 (2012)
Moffat, A., Zobel, J.: Self-indexing inverted files for fast text retrieval. Transactions on Information Systems 14(4), 349–379 (1996)
Amati, G., Ambrosi, E., Bianchi, M., Gaibisso, C., Gambosi, G.: FUB, IASI-CNR and University of Tor Vergata at TREC 2007 blog track. In: Proceedings of TREC 2007 (2007)
Tonellotto, N., Macdonald, C., Ounis, I.: Query efficiency prediction for dynamic pruning. In: Proceedings of LSDS-IR at CIKM 2011 (2011)
Friedman, J.H.: Greedy function approximation: A gradient boosting machine. Annals of Statistics 29, 1189–1232 (2000)
Ganjisaffar, Y., Caruana, R., Lopes, C.: Bagging gradient-boosted trees for high precision, low variance ranking models. In: Proc. of SIGIR 2011, pp. 85–94 (2011)
Simmons, B., McCloskey, A., Lutfiyya, H.: Dynamic provisioning of resources in data centers. In: Proceedings of ICAS 2007, pp. 40–46 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Freire, A., Macdonald, C., Tonellotto, N., Ounis, I., Cacheda, F. (2013). Hybrid Query Scheduling for a Replicated Search Engine. In: Serdyukov, P., et al. Advances in Information Retrieval. ECIR 2013. Lecture Notes in Computer Science, vol 7814. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-36973-5_37
Download citation
DOI: https://doi.org/10.1007/978-3-642-36973-5_37
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-36972-8
Online ISBN: 978-3-642-36973-5
eBook Packages: Computer ScienceComputer Science (R0)