Skip to main content
SpringerLink
Log in

Context-Aware Query Term Difficulty Estimation for Performance Prediction

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14611))

Included in the following conference series:

  • 254 Accesses

Abstract

Research has already found that many retrieval methods are sensitive to the choice and order of terms that appear in a query, which can significantly impact retrieval effectiveness. We capitalize on this finding in order to predict the performance of a query. More specifically, we propose to learn query term difficulty weights specifically within the context of each query, which could then be used as indicators of whether each query term has the likelihood of making the query more effective or not. We show how such difficulty weights can be learnt through the finetuning of a language model. In addition, we propose an approach to integrate the learnt weights into a cross-encoder architecture to predict query performance. We show that our proposed approach shows a consistently strong performance prediction on the MSMARCO collection and its associated widely used Trec Deep Learning tracks query sets. Our findings demonstrate that our method is able to show consistently strong performance prediction over different query sets (MSMARCO Dev, TREC DL’19, ’20, Hard) and a range of evaluation metrics (Kendall, Spearman, sMARE).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Arabzadeh, N., Bigdeli, A., Hamidi Rad, R., Bagheri, E.: Quantifying ranker coverage of different query subspaces. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2298–2302 (2023)

    Google Scholar 

  2. Arabzadeh, N., Bigdeli, A., Seyedsalehi, S., Zihayat, M., Bagheri, E.: Matches made in heaven: toolkit and large-scale datasets for supervised query reformulation. In: Proceedings of the 30th ACM International Conference on Information and Knowledge Management, pp. 4417–4425 (2021)

    Google Scholar 

  3. Arabzadeh, N., Bigdeli, A., Zihayat, M., Bagheri, E.: Query performance prediction through retrieval coherency. In: Hiemstra, D., et al. (eds.) ECIR 2021. LNCS, vol. 12657, pp. 193–200. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72240-1_15

  4. Arabzadeh, N., Hamidi Rad, R., Khodabakhsh, M., Bagheri, E.: Noisy perturbations for estimating query difficulty in dense retrievers. In: CIKM (2023)

    Google Scholar 

  5. Arabzadeh, N., Khodabakhsh, M., Bagheri, E.: Bert-qpp: contextualized pre-trained transformers for query performance prediction. In: CIKM (2021)

    Google Scholar 

  6. Arabzadeh, N., Mitra, B., Bagheri, E.: MS marco chameleons: challenging the MS marco leaderboard with extremely obstinate queries. In: Proceedings of the 30th ACM International Conference on Information and Knowledge Management, pp. 4426–4435 (2021)

    Google Scholar 

  7. Arabzadeh, N., Yan, X., Clarke, C.L.A.: Predicting efficiency/effectiveness trade-offs for dense vs. sparse retrieval strategy selection. arXiv preprint arXiv:2109.10739 (2021)

  8. Arabzadeh, N., Zarrinkalam, F., Jovanovic, J., Al-Obeidat, F., Bagheri, E.: Neural embedding-based specificity metrics for pre-retrieval query performance prediction. Inf. Process. Manag. 57(4), 102248 (2020)

    Google Scholar 

  9. Arabzadeh, N., Zarrinkalam, F., Jovanovic, J., Bagheri, E.: Neural embedding-based metrics for pre-retrieval query performance prediction. In: Jose, J.M., et al. (eds.) ECIR 2020. LNCS, vol. 12036, pp. 78–85. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45442-5_10

  10. Arabzadeh, N., Zarrinkalam, F., Jovanovic, J., Bagheri, E.: Geometric estimation of specificity within embedding spaces. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 2109–2112 (2019)

    Google Scholar 

  11. Carmel, D., Yom-Tov, E.: Estimating the query difficulty for information retrieval. Synth. Lect. Inf. Concept. Retriev. Serv. 2(1), 1–89 (2010)

    Google Scholar 

  12. Carmel, D., Yom-Tov, E., Darlow, A., Pelleg, D.: What makes a query difficult? In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 390–397 (2006)

    Google Scholar 

  13. Craswell, N., Mitra, B., Yilmaz, E., Campos, D.: Overview of the TREC 2020 deep learning track. arXiv preprint arXiv:2102.07662 (2021)

  14. Craswell, N., Mitra, B., Yilmaz, E., Campos, D., Voorhees, E.M.: Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820 (2020)

  15. Dai, Z., Callan, J.: Context-aware sentence/passage term importance estimation for first stage retrieval. arXiv preprint arXiv:1910.10687 (2019)

  16. Dai, Z., Callan, J.: Context-aware term weighting for first stage passage retrieval. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1533–1536 (2020)

    Google Scholar 

  17. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  18. Faggioli, G., Zendel, O., Culpepper, J.S., Ferro, N., Scholer, F.: Smare: a new paradigm to evaluate and understand query performance prediction methods. Inf. Retriev. J. 25(2), 94–122 (2022)

    Google Scholar 

  19. Hauff, C.: Predicting the effectiveness of queries and retrieval systems. In: SIGIR Forum, vol. 44, p. 88 (2010)

    Google Scholar 

  20. Hauff, C., Hiemstra, D., de Jong, F.: A survey of pre-retrieval query performance predictors. In: Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM 2008, Napa Valley, California, 26–30 October 2008, pp. 1419–1420 (2008). https://doi.org/10.1145/1458082.1458311

  21. He, B., Ounis, I.: Inferring query performance using pre-retrieval predictors. In: Apostolico, A., Melucci, M. (eds.) String Processing and Information Retrieval. LNCS, vol. 3246, pp. 43–54. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30213-1_5

  22. He, B., Ounis, I.: Query performance prediction. Inf. Syst. 31(7), 585–594 (2006)

    Article  Google Scholar 

  23. Khodabakhsh, M., Bagheri, E.: Semantics-enabled query performance prediction for ad hoc table retrieval. Inf. Process. Manag. 58(1), 102399 (2021)

    Google Scholar 

  24. Khodabakhsh, M., Bagheri, E.: Learning to rank and predict: multi-task learning for ad hoc retrieval and query performance prediction. Inf. Sci. 639, 119015 (2023)

    Google Scholar 

  25. Kwok, K.L.: A new method of weighting query terms for ad-hoc retrieval. In: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1996), 18–22 August 1996, Zurich (Special Issue of the SIGIR Forum), pp. 187–195 (1996). https://doi.org/10.1145/243199.243266

  26. Mackie, I., Dalton, J., Yates, A.: How deep is your learning: the dl-hard annotated deep learning dataset. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (2021)

    Google Scholar 

  27. Meng, C., Arabzadeh, N., Aliannejadi, M., de Rijke, M.: Query performance prediction: from ad-hoc to conversational search. arXiv preprint arXiv:2305.10923 (2023)

  28. Nguyen, T., et al.: MS marco: a human generated machine reading comprehension dataset. In: CoCo@ NIPS (2016)

    Google Scholar 

  29. Nogueira, R., Cho, K.: Passage re-ranking with bert. arXiv preprint arXiv:1901.04085 (2019)

  30. Nogueira, R., Lin, J., Epistemic, A.: From doc2query to doctttttquery. Online preprint 6, 2 (2019)

    Google Scholar 

  31. Nogueira, R., Yang, W., Lin, J., Cho, K.: Document expansion by query prediction. arXiv preprint arXiv:1904.08375 (2019)

  32. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(140), 1–67 (2020). http://jmlr.org/papers/v21/20-074.html

  33. Raiber, F., Kurland, O.: Query-performance prediction: setting the expectations straight. In: Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 13–22 (2014)

    Google Scholar 

  34. Reimers, N., Gurevych, I.: Sentence-bert: sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)

  35. Roy, D., Ganguly, D., Mitra, M., Jones, G.J.F.: Estimating Gaussian mixture models in the local neighbourhood of embedded word vectors for query performance prediction. Inf. Process. Manag. 56(3), 1026–1045 (2019). https://doi.org/10.1016/j.ipm.2018.10.009

  36. Salamat, S., Arabzadeh, N., Seyedsalehi, S., Bigdeli, A., Zihayat, M., Bagheri, E.: Neural disentanglement of query difficulty and semantics. In: CIKM, pp. 4264–4268 (2023)

    Google Scholar 

  37. Tamannaee, M., Fani, H., Zarrinkalam, F., Samouh, J., Paydar, S., Bagheri, E.: Reque: a configurable workflow and dataset collection for query refinement. In: Proceedings of the 29th ACM International Conference on Information and Knowledge Management, pp. 3165–3172 (2020)

    Google Scholar 

  38. Yang, P., Fang, H., Lin, J.: Anserini: enabling the use of lucene for information retrieval research. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1253–1256 (2017)

    Google Scholar 

  39. Yom-Tov, E., Fine, S., Carmel, D., Darlow, A.: Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 512–519 (2005)

    Google Scholar 

  40. Zhao, Y., Scholer, F., Tsegay, Y.: Effective pre-retrieval query performance prediction using similarity and variability evidence. In: Advances in Information Retrieval, 30th European Conference on IR Research, ECIR 2008, Glasgow, 30 March–3 April 2008. Proceedings. pp. 52–64 (2008). https://doi.org/10.1007/978-3-540-78646-7_8

  41. Zhou, Y., Croft, W.B.: Query performance prediction in web search environments. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 543–550 (2007)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Toronto Metropolitan University, Toronto, ON, Canada

    Abbas Saleminezhad, Soosan Beheshti & Ebrahim Bagheri

  2. University of Waterloo, Toronto, ON, Canada

    Negar Arabzadeh

Authors
  1. Abbas Saleminezhad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Negar Arabzadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Soosan Beheshti
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ebrahim Bagheri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abbas Saleminezhad .

Editor information

Editors and Affiliations

  1. Georgetown University, Washington, WA, USA

    Nazli Goharian

  2. University of Pisa, PISA, Pisa, Italy

    Nicola Tonellotto

  3. King's College London, London, UK

    Yulan He

  4. University College London, London, UK

    Aldo Lipani

  5. University of Glasgow, Glasgow, UK

    Graham McDonald

  6. University of Glasgow, Glasgow, UK

    Craig Macdonald

  7. University of Glasgow, Glasgow, UK

    Iadh Ounis

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Saleminezhad, A., Arabzadeh, N., Beheshti, S., Bagheri, E. (2024). Context-Aware Query Term Difficulty Estimation for Performance Prediction. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14611. Springer, Cham. https://doi.org/10.1007/978-3-031-56066-8_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56066-8_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56065-1

  • Online ISBN: 978-3-031-56066-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation