skip to main content
10.1145/3477495.3532056acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

Risk-Sensitive Deep Neural Learning to Rank

Published: 07 July 2022 Publication History

Abstract

Learning to Rank (L2R) is the core task of many Information Retrieval systems. Recently, a great effort has been put on exploring Deep Neural Networks (DNNs) for L2R, with significant results. However, risk-sensitiveness, an important and recent advance in the L2R arena, that reduces variability and increases trust, has not been incorporated into Deep Neural L2R yet. Risk-sensitive measures are important to assess the risk of an IR system to perform worse than a set of baseline IR systems for several queries. However, the risk-sensitive measures described in the literature have a non-smooth behavior, making them difficult, if not impossible, to be optimized by DNNs. In this work we solve this difficult problem by proposing a family of new loss functions -- \riskloss\ -- that support a smooth risk-sensitive optimization. \riskloss\ introduces two important contributions: (i) the substitution of the traditional NDCG or MAP metrics in risk-sensitive measures with smooth loss functions that evaluate the correlation between the predicted and the true relevance order of documents for a given query and (ii) the use of distinct versions of the same DNN architecture as baselines by means of a multi-dropout technique during the smooth risk-sensitive optimization, avoiding the inconvenience of assessing multiple IR systems as part of DNN training. We empirically demonstrate significant achievements of the proposed \riskloss\ functions when used with recent DNN methods in the context of well-known web-search datasets such as WEB10K, YAHOO, and MQ2007. Our solutions reach improvements of 8% in effectiveness (NDCG) while improving in around 5% the risk-sensitiveness (\grisk\ measure) when applied together with a state-of-the-art Self-Attention DNN-L2R architecture. Furthermore, \riskloss\ is capable of reducing by 28% the losses over the best evaluated baselines and significantly improving over the risk-sensitive state-of-the-art non-DNN method (by up to 13.3%) while keeping (or even increasing) overall effectiveness. All these results ultimately establish a new level for the state-of-the-art on risk-sensitiveness and DNN-L2R research.

References

[1]
Qingyao Ai, Keping Bi, Jiafeng Guo, and W. Bruce Croft. 2018. Learning a Deep Listwise Context Model for Ranking Refinement. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR '18). Association for Computing Machinery, New York, NY, USA, 135--144.
[2]
Irwan Bello, Sayali Kulkarni, Sagar Jain, Craig Boutilier, Ed Huai-hsin Chi, Elad Eban, Xiyang Luo, Alan Mackey, and Ofer Meshi. 2018. Seq2Slate: Reranking and Slate Optimization with RNNs. CoRR abs/1810.02019 (2018), 5 pages. arXiv:1810.02019
[3]
Rodger Benham, Ben Carterette, J. Shane Culpepper, and Alistair Moffat. 2020. Bayesian Inferential Risk Evaluation On Multiple IR Systems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR '20). Association for Computing Machinery, New York, NY, USA, 339--348.
[4]
Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. 2020. Fast Differentiable Sorting and Ranking. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13--18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119). PMLR, 950--959.
[5]
Sebastian Bruch. 2021. An Alternative Cross Entropy Loss for Learning-to- Rank. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW '21). Association for Computing Machinery, New York, NY, USA, 118--126.
[6]
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to Rank: From Pairwise Approach to Listwise Approach. In Proceedings of the 24th International Conference on Machine Learning (Corvalis, Oregon, USA) (ICML '07). Association for Computing Machinery, New York, NY, USA, 129--136.
[7]
Pingjun Chen, Linlin Gao, Xiaoshuang Shi, Kyle Allen, and Lin Yang. 2019. Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss. Computerized Medical Imaging and Graphics 75 (2019), 84--92.
[8]
B. Taner Dinçer, Craig Macdonald, and Iadh Ounis. 2014. Hypothesis Testing for the Risk-Sensitive Evaluation of Retrieval Systems. In Proceedings of the 37th International ACM SIGIR (Gold Coast, Queensland, Australia) (SIGIR '14). Association for Computing Machinery, New York, NY, USA, 23--32.
[9]
B. Taner Dinçer, Craig Macdonald, and Iadh Ounis. 2016. Risk-Sensitive Evaluation and Learning to Rank Using Multiple Baselines. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (Pisa, Italy) (SIGIR '16). Association for Computing Machinery, New York, NY, USA, 483--492.
[10]
Hiroshi Inoue. 2019. Multi-Sample Dropout for Accelerated Training and Better Generalization. CoRR abs/1905.09788 (2019). arXiv:1905.09788 http://arxiv.org/ abs/1905.09788
[11]
Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the user experience of recommender systems. Journal of User Modeling and User-Adapted Interaction 22 (2012), 441--504.
[12]
Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the user experience of recommender systems. Journal of User Modeling and User-Adapted Interaction 22 (2012), 441--504.
[13]
Ling Li and Hsuan-Tien Lin. 2006. Ordinal Regression by Extended Binary Classification. In Proceedings of the 19th International Conference on Neural Information Processing Systems (Canada) (NIPS'06). MIT Press, Cambridge, MA, USA, 865--872.
[14]
Tie-Yan Liu. 2009. Learning to Rank for Information Retrieval. Foundations and Trends® in Information Retrieval 3, 3 (2009), 225--331.
[15]
Tie-Yan Liu. 2011. Learning To Rank For Information Retrieval. Springer, New York, USA. 1--304 pages.
[16]
Jarana Manotumruksa, Craig Macdonald, and Iadh Ounis. 2019. A Contextual Recurrent Collaborative Filtering framework for modelling sequences of venue checkins. Information Processing and Management 56 (2019), 102092.
[17]
Josiane Mothe and Md Zia Ullah. 2021. Defining an Optimal Configuration Set for Selective Search Strategy - A Risk-Sensitive Approach. In Proceedings of the 30th ACM International Conference on Information amp; Knowledge Management (CIKM '21). Association for Computing Machinery, New York, NY, USA, 1335--1345.
[18]
Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Jingfang Xu, and Xueqi Cheng. 2017. DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 257--266.
[19]
Przemyslaw Pobrotyn, Tomasz Bartczak, Mikolaj Synowiec, Radoslaw Bialobrzeski, and Jaroslaw Bojar. 2020. Context-Aware Learning to Rank with Self- Attention. CoRR abs/2005.10084 (2020). arXiv:2005.10084
[20]
Taof Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 Datasets. CoRR abs/1306.2597 (2013). arXiv:1306.2597
[21]
Tao Qin, Tie-Yan Liu, and Hang Li. 2010. A General Approximation Framework for Direct Optimization of Information Retrieval Measures. Inf. Retr. 13, 4 (Aug. 2010), 375--397.
[22]
Tao Qin, Xu-Dong Zhang, Ming-Feng Tsai, De-Sheng Wang, Tie-Yan Liu, and Hang Li. 2008. Query-level loss functions for information retrieval. Information Processing & Management 44, 2 (2008), 838--855. Evaluating Exploratory Search Systems Digital Libraries in the Context of Users' Broader Activities.
[23]
Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2021. Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 16 pages.
[24]
Daniel Xavier Sousa, Sérgio Canuto, Marcos André Gonçalves, Thierson Couto Rosa, and Wellington Santos Martins. 2019. Risk-Sensitive Learning to Rank with Evolutionary Multi-Objective Feature Selection. ACM Trans. Inf. Syst. 37, 2, Article 24 (Feb. 2019), 34 pages.
[25]
Daniel Xavier De Sousa, Sérgio Daniel Canuto, Thierson Couto Rosa, Wellington Santos Martins, and Marcos André Gonçalves. 2016. Incorporating Risk- Sensitiveness into Feature Selection for Learning to Rank. In CIKM'2016. ACM, New York, NY, USA, 257--266.
[26]
C. Spearman. 1987. The Proof and Measurement of Association between Two Things. The American Journal of Psychology 100, 3/4 (1987), 441--471.
[27]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15, 56 (2014), 1929--1958.
[28]
Shuo Sun and Kevin Duh. 2020. Modeling Document Interactions for Learning to Rank with Regularized Self-Attention. arXiv:2005.03932 [cs.IR]
[29]
Wei Ting Pang, Prabha Rajagopal, Mengjia Wang, Shuxiang Zhang, and Sri Devi Ravana. 2019. Exploring topic difficulty in information retrieval systems evaluation. In 1st International Conference Computer Science and Engineering, IC2SE 2019 (Journal of Physics: Conference Series, 1), Robbi Rahim, Robby Dharma, Billy Hendrik, and Abulwafa Muhammad (Eds.). IOP Publishing, United Kingdom.
[30]
Lidan Wang, Paul N. Bennett, and Kevyn Collins-Thompson. 2012. Robust Ranking Models via Risk-Sensitive Optimization. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval (Portland, Oregon, USA) (SIGIR '12). Association for Computing Machinery, New York, NY, USA, 761--770.
[31]
XuanhuiWang, Cheng Li, Nadav Golbandi, Michael Bendersky, and Marc Najork. 2018. The LambdaLoss Framework for Ranking Metric Optimization. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (Torino, Italy) (CIKM '18). Association for Computing Machinery, New York, NY, USA, 1313--1322.
[32]
Peng Zhang, Linxue Hao, Dawei Song, Jun Wang, Yuexian Hou, and Bin Hu. 2014. Generalized Bias-Variance Evaluation of TREC Participated Systems. In CIKM'2014. ACM, New York, NY, USA, 1911--1914.
[33]
Zhaohui Zheng, Hongyuan Zha, Tong Zhang, Olivier Chapelle, Keke Chen, and Gordon Sun. 2007. A General Boosting Method and Its Application to Learning Ranking Functions for Web Search. In Proceedings of the 20th International Conference on Neural Information Processing Systems (Vancouver, British Columbia, Canada) (NIPS'07). Curran Associates Inc., Red Hook, NY, USA, 1697--1704.
[34]
Xiaofeng Zhu and Diego Klabjan. 2020. Listwise learning to rank by exploring unique ratings. CoRR abs/2001.01828 (2020), 798--806.

Index Terms

  1. Risk-Sensitive Deep Neural Learning to Rank

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval
    July 2022
    3569 pages
    ISBN:9781450387323
    DOI:10.1145/3477495
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 July 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. deep neural network
    2. loss function
    3. risk-sensitiveness

    Qualifiers

    • Research-article

    Funding Sources

    • CAPES
    • CNPq
    • FAPEG
    • FAPEMIG
    • IFG

    Conference

    SIGIR '22
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 311
      Total Downloads
    • Downloads (Last 12 months)29
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 28 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media