Skip to main content
SpringerLink
Log in

Contrasting Neural Click Models and Pointwise IPS Rankers

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13980))

Included in the following conference series:

  • 1484 Accesses

Abstract

Inverse-propensity scoring and neural click models are two popular methods for learning rankers from user clicks that are affected by position bias. Despite their prevalence, the two methodologies are rarely directly compared on equal footing. In this work, we focus on the pointwise learning setting to compare the theoretical differences of both approaches and present a thorough empirical comparison on the prevalent semi-synthetic evaluation setup in unbiased learning-to-rank. We show theoretically that neural click models, similarly to IPS rankers, optimize for the true document relevance when the position bias is known. However, our work also finds small but significant empirical differences between both approaches indicating that neural click models might be affected by position bias when learning from shared, sometimes conflicting, features instead of treating each document separately.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    LightGBM Version 3.3.2, using 100 trees, 31 leafs, and learning rate 0.1.

  2. 2.

    \(\text {optimizer} \in \{Adam, Adagrad, SGD\}\).

  3. 3.

    \(\text {learning rate} \in \{0.1,0.05,0.01,0.005,0.001,0.0005,0.0001\}\).

References

  1. Agarwal, A., Takatsu, K., Zaitsev, I., Joachims, T.: A general framework for counterfactual learning-to-rank. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2019)

    Google Scholar 

  2. Agarwal, A., Zaitsev, I., Wang, X., Li, C., Najork, M., Joachims, T.: Estimating position bias without intrusive interventions. In: International Conference on Web Search and Data Mining (WSDM) (2019)

    Google Scholar 

  3. Ai, Q., Bi, K., Luo, C., Guo, J., Croft, W.B.: Unbiased learning to rank with unbiased propensity estimation. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2018)

    Google Scholar 

  4. Ai, Q., Yang, T., Wang, H., Mao, J.: Unbiased learning to rank: Online or offline? ACM Trans. Inform. Syst. (TOIS) 39(2), 1–29 (2021)

    Google Scholar 

  5. Bekker, J., Robberechts, P., Davis, J.: Beyond the selected completely at random assumption for learning from positive and unlabeled data. In: Machine Learning and Knowledge Discovery in Databases: European Conference (ECML PKDD) (2019)

    Google Scholar 

  6. Bonferroni, C.: Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R. Istituto Superiore di Scienze Economiche e Commericiali di Firenze 8, 3–62 (1936)

    Google Scholar 

  7. Borisov, A., Markov, I., de Rijke, M., Serdyukov, P.: A neural click model for web search. In: The World Wide Web Conference (WWW) (2016)

    Google Scholar 

  8. Burges, C.J.: From ranknet to lambdarank to lambdamart: An overview. Tech. Rep. MSR-TR-2010-82, Microsoft (2010)

    Google Scholar 

  9. Chapelle, O., Chang, Y.: Yahoo! learning to rank challenge overview. J. Mach. Learn. Res. (JMLR) 14, 1–24 (2011)

    Google Scholar 

  10. Chapelle, O., Metlzer, D., Zhang, Y., Grinspan, P.: Expected reciprocal rank for graded relevance. In: International Conference on Information and Knowledge Management (CIKM) (2009)

    Google Scholar 

  11. Chapelle, O., Zhang, Y.: A dynamic bayesian network click model for web search ranking. In: The World Wide Web Conference (WWW) (2009)

    Google Scholar 

  12. Chen, J., Mao, J., Liu, Y., Zhang, M., Ma, S.: A context-aware click model for web search. In: International Conference on Web Search and Data Mining (WSDM) (2020)

    Google Scholar 

  13. Chu, W., Li, S., Chen, C., Xu, L., Cui, H., Liu, K.: A general framework for debiasing in ctr prediction (2021). https://doi.org/10.48550/arXiv.2112.02767

  14. Chuklin, A., Markov, I., de Rijke, M.: Click Models for Web Search. Morgan & Claypool (2015), ISBN 9781627056489. https://doi.org/10.2200/S00654ED1V01Y201507ICR043

  15. Covington, P., Adams, J., Sargin, E.: Deep neural networks for youtube recommendations. In: ACM Conference on Recommender Systems (RecSys) (2016)

    Google Scholar 

  16. Craswell, N., Zoeter, O., Taylor, M., Ramsey, B.: An experimental comparison of click position-bias models. In: International Conference on Web Search and Data Mining (WSDM) (2008)

    Google Scholar 

  17. Dato, D., et al.: Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Trans. Inform. Syst. (TOIS) 35(2), 1–31 (2016)

    Google Scholar 

  18. Diaz, F., White, R., Buscher, G., Liebling, D.: Robust models of mouse movement on dynamic web search results pages. In: International Conference on Information and Knowledge Management (CIKM) (2013)

    Google Scholar 

  19. Dupret, G.E., Piwowarski, B.: A user browsing model to predict search engine click data from past observations. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2008)

    Google Scholar 

  20. Gomez-Uribe, C.A., Hunt, N.: The netflix recommender system: Algorithms, business value, and innovation. ACM Trans. Manage. Inform. Syst. (TMIS) 6(4), 1–19 (2016)

    Google Scholar 

  21. Guo, F., et al.: Click chain model in web search. In: The World Wide Web Conference (WWW) (2009)

    Google Scholar 

  22. Guo, H., Yu, J., Liu, Q., Tang, R., Zhang, Y.: Pal: A position-bias aware learning framework for ctr prediction in live recommender systems. In: ACM Conference on Recommender Systems (RecSys) (2019)

    Google Scholar 

  23. Haldar, M., et al.: Improving deep learning for airbnb search. In: International Conference on Knowledge Discovery and Data Mining (SIGKDD) (2020)

    Google Scholar 

  24. Hofmann, K., Schuth, A., Whiteson, S., de Rijke, M.: Reusing historical interaction data for faster online learning to rank for ir. In: International Conference on Web Search and Data Mining (WSDM) (2013)

    Google Scholar 

  25. Hu, Z., Wang, Y., Peng, Q., Li, H.: Unbiased lambdamart: An unbiased pairwise learning-to-rank algorithm. In: The World Wide Web Conference (WWW) (2019)

    Google Scholar 

  26. Jagerman, R., Oosterhuis, H., de Rijke, M.: To model or to intervene: A comparison of counterfactual and online learning to rank from user interactions. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2019)

    Google Scholar 

  27. Joachims, T., Granka, L., Pan, B., Hembrooke, H., Gay, G.: Accurately interpreting clickthrough data as implicit feedback. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2005)

    Google Scholar 

  28. Joachims, T., Swaminathan, A., Schnabel, T.: Unbiased learning-to-rank with biased feedback. In: International Conference on Web Search and Data Mining (WSDM) (2017)

    Google Scholar 

  29. Ke, G., et al.: Lightgbm: A highly efficient gradient boosting decision tree. In: International Conference on Neural Information Processing Systems (NIPS) (2017)

    Google Scholar 

  30. Lin, J., et al.: A graph-enhanced click model for web search. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2021)

    Google Scholar 

  31. Liu, T.Y., et al.: Learning to rank for information retrieval. Found. Trends Inf. Retr. 3, 225–331 (2009)

    Article  Google Scholar 

  32. Oosterhuis, H.: Reaching the end of unbiasedness: Uncovering implicit limitations of click-based learning to rank. In: International Conference on the Theory of Information Retrieval (ICTIR) (2022)

    Google Scholar 

  33. Oosterhuis, H., de Rijke, M.: Differentiable unbiased online learning to rank. In: International Conference on Information and Knowledge Management (CIKM) (2018)

    Google Scholar 

  34. Oosterhuis, H., de Rijke, M.: Policy-aware unbiased learning to rank for top-k rankings. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2020)

    Google Scholar 

  35. Oosterhuis, H., de Rijke, M.: Unifying online and counterfactual learning to rank: A novel counterfactual estimator that effectively utilizes online interventions. In: International Conference on Web Search and Data Mining (WSDM) (2021)

    Google Scholar 

  36. Ovaisi, Z., Ahsan, R., Zhang, Y., Vasilaky, K., Zheleva, E.: Correcting for selection bias in learning-to-rank systems. In: The Web Conference (2020)

    Google Scholar 

  37. Qin, T., Liu, T.: Introducing letor 4.0 datasets (2013), 10.48550/arXiv. 1306.2597

    Google Scholar 

  38. Qin, Z., et al.: Are neural rankers still outperformed by gradient boosted decision trees? In: International Conference on Learning Representations (ICLR) (2021)

    Google Scholar 

  39. Richardson, M., Dominowska, E., Ragno, R.: Predicting clicks: Estimating the click-through rate for new ads. In: The World Wide Web Conference (WWW) (2007)

    Google Scholar 

  40. Saito, Y., Yaginuma, S., Nishino, Y., Sakata, H., Nakata, K.: Unbiased recommender learning from missing-not-at-random implicit feedback. In: International Conference on Web Search and Data Mining (WSDM) (2020)

    Google Scholar 

  41. Sanderson, M., et al.: Test collection based evaluation of information retrieval. Found. Trends Inf. Retr. 4, 247–375 (2010)

    Article  MATH  Google Scholar 

  42. Sorokina, D., Cantu-Paz, E.: Amazon search: The joy of ranking products. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2016)

    Google Scholar 

  43. Student: The probable error of a mean. Biometrika pp. 1–25 (1908)

    Google Scholar 

  44. Swaminathan, A., Joachims, T.: The self-normalized estimator for counterfactual learning. In: International Conference on Neural Information Processing Systems (NIPS) (2015)

    Google Scholar 

  45. Vardasbi, A., Oosterhuis, H., de Rijke, M.: When inverse propensity scoring does not work: Affine corrections for unbiased learning to rank. In: International Conference on Information and Knowledge Management (CIKM) (2020)

    Google Scholar 

  46. Vardasbi, A., de Rijke, M., Markov, I.: Cascade model-based propensity estimation for counterfactual learning to rank. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2020)

    Google Scholar 

  47. Vardasbi, A., de Rijke, M., Markov, I.: Mixture-Based Correction for Position and Trust Bias in Counterfactual Learning to Rank (2021)

    Google Scholar 

  48. Wang, C., Liu, Y., Wang, M., Zhou, K., Nie, J.y., Ma, S.: Incorporating non-sequential behavior into click models. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2015)

    Google Scholar 

  49. Wang, X., Bendersky, M., Metzler, D., Najork, M.: Learning to rank with selection bias in personal search. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2016)

    Google Scholar 

  50. Wang, X., Golbandi, N., Bendersky, M., Metzler, D., Najork, M.: Position bias estimation for unbiased learning to rank in personal search. In: International Conference on Web Search and Data Mining (WSDM) (2018)

    Google Scholar 

  51. Wang, Y.X., Agarwal, A., Dudík, M.: Optimal and adaptive off-policy evaluation in contextual bandits. In: International Conference on Machine Learning (ICML) (2017)

    Google Scholar 

  52. Xie, X., et al.: Investigating examination behavior of image search users. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2017)

    Google Scholar 

  53. Xie, X., Mao, J., de Rijke, M., Zhang, R., Zhang, M., Ma, S.: Constructing an interaction behavior model for web image search. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2018)

    Google Scholar 

  54. Yan, L., Qin, Z., Zhuang, H., Wang, X., Bendersky, M., Najork, M.: Revisiting two-tower models for unbiased learning to rank. In: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (2022)

    Google Scholar 

  55. Zhao, Z., et al.: Recommending what video to watch next: A multitask ranking system. In: ACM Conference on Recommender Systems (RecSys) (2019)

    Google Scholar 

  56. Zhuang, H., et al.: Cross-positional attention for debiasing clicks. In: The Web Conference (2021)

    Google Scholar 

Download references

Acknowledgements

We thank our reviewers for their time and valuable feedback. For insightful discussions and their comments, we thank Shashank Gupta, Romain Deffayet, Kathrin Parchatka, and Harrie Oosterhuis.

This research was supported by the Mercury Machine Learning Lab, a collaboration between TU Delft, the University of Amsterdam, and Booking.com. Maarten de Rijke was supported by the Hybrid Intelligence Center, a 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, https://hybrid-intelligence-centre.nl.

All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.

Author information

Authors and Affiliations

  1. University of Amsterdam, Amsterdam, The Netherlands

    Philipp Hager & Maarten de Rijke

  2. Booking.com, Amsterdam, The Netherlands

    Onno Zoeter

Authors
  1. Philipp Hager
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maarten de Rijke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Onno Zoeter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Philipp Hager .

Editor information

Editors and Affiliations

  1. University of Amsterdam, Amsterdam, The Netherlands

    Jaap Kamps

  2. Université Grenoble-Alpes, Saint-Martin-d’Hères, France

    Lorraine Goeuriot

  3. Università della Svizzera Italiana, Lugano, Switzerland

    Fabio Crestani

  4. University of Copenhagen, Copenhagen, Denmark

    Maria Maistro

  5. University of Tsukuba, Ibaraki, Japan

    Hideo Joho

  6. Dublin City University, Dublin, Ireland

    Brian Davis

  7. Dublin City University, Dublin, Ireland

    Cathal Gurrin

  8. Universität Regensburg, Regensburg, Germany

    Udo Kruschwitz

  9. Dublin City University, Dublin, Ireland

    Annalina Caputo

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hager, P., de Rijke, M., Zoeter, O. (2023). Contrasting Neural Click Models and Pointwise IPS Rankers. In: Kamps, J., et al. Advances in Information Retrieval. ECIR 2023. Lecture Notes in Computer Science, vol 13980. Springer, Cham. https://doi.org/10.1007/978-3-031-28244-7_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-28244-7_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-28243-0

  • Online ISBN: 978-3-031-28244-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation