skip to main content
10.1145/3594315.3594351acmotherconferencesArticle/Chapter ViewAbstractPublication PagesiccaiConference Proceedingsconference-collections
research-article

An Empirical Perspective on Learning-to-rank

Published: 02 August 2023 Publication History

Abstract

Learning-to-rank has been widely studied and applied in document retrieval. Typically, existing learning-to-rank methods treat ranking as an independent matching process among different queries. Hence, their ranking functions remain unchanged once the learning process is accomplished. However, these methods ignore the fact that previous ranking performance could help fine-tune the ranking function. In a contrast, this paper proposes an empirical perspective-based learning-to-rank model (EPRank) by formulating ranking among different queries as a sequential ranking process where previous ranking feedback could help refine the ranking function for future ranking. Specifically, EPRank constructs a clip feedback mapping where the ranking feedback is mapped to distinct feature embeddings to roughly represent ranking performance for efficient learning. Then, EPRank explicitly incorporates previous queries’ clip feedback embeddings upon earlier ranking performance into a recurrent neural network to iteratively refine the ranking function. Experiments demonstrate the effectiveness of EPRank compared to previous learning-to-rank methods in terms of Normalized Discounted Cumulative Gain (NDCG) and Expected Reciprocal Rank (ERR).

References

[1]
Qingyao Ai, Keping Bi, Jiafeng Guo, and W Bruce Croft. 2018. Learning a deep listwise context model for ranking refinement. In The 41st international ACM SIGIR conference on research & development in information retrieval. 135–144.
[2]
Qingyao Ai, Xuanhui Wang, Sebastian Bruch, Nadav Golbandi, Michael Bendersky, and Marc Najork. 2019. Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. ACM, 85–92.
[3]
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. 2019. On the convergence rate of training recurrent neural networks. Advances in neural information processing systems 32 (2019).
[4]
Irwan Bello, Sayali Kulkarni, Sagar Jain, Craig Boutilier, Ed Chi, Elad Eban, Xiyang Luo, Alan Mackey, and Ofer Meshi. 2018. Seq2slate: Re-ranking and slate optimization with rnns. arXiv preprint arXiv:1810.02019 (2018).
[5]
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning. ACM, 89–96.
[6]
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning. ACM, 129–136.
[7]
BH ChandraShekar and G Shoba. 2009. Classification of documents using kohonen’s self-organizing map. International Journal of Computer Theory and Engineering 1, 5 (2009), 610.
[8]
Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. 2009. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM conference on Information and knowledge management. ACM, 621–630.
[9]
Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).
[10]
Wei Chu and Zoubin Ghahramani. 2005. Gaussian processes for ordinal regression. Journal of machine learning research 6, Jul (2005), 1019–1041.
[11]
Wei Chu and S Sathiya Keerthi. 2005. New approaches to support vector ordinal regression. In Proceedings of the 22nd international conference on Machine learning. 145–152.
[12]
Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W Bruce Croft. 2017. Neural ranking models with weak supervision. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 65–74.
[13]
Yue Feng, Jun Xu, Yanyan Lan, Jiafeng Guo, Wei Zeng, and Xueqi Cheng. 2018. From Greedy Selection to Exploratory Decision-Making: Diverse Ranking with Policy-Value Networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. ACM, 125–134.
[14]
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM, 55–64.
[15]
M Hosseinia, K Badie, and A Moeini. 2012. Aspect-Oriented Document Clustering for Facilitating Retrieval Process. International Journal of Computer Theory and Engineering 4, 5 (2012), 707.
[16]
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Conference on information and knowledge management. 2333–2338.
[17]
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS) 20, 4 (2002), 422–446.
[18]
Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation (1972).
[19]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[20]
Fan Li, Hong Qu, Mingsheng Fu, Liyan Zhang, Fan Zhang, Wenyu Chen, Ruixuan Sun, and Haixian Zhang. 2022. Neural Reranking-Based Collaborative Filtering by Leveraging Listwise Relative Ranking Information. IEEE Transactions on Systems, Man, and Cybernetics: Systems (2022).
[21]
Pengfei Li and Kezhi Mao. 2019. Knowledge-oriented convolutional neural network for causal relation extraction from natural language texts. Expert Systems with Applications 115 (2019), 512–523.
[22]
Pengfei Li, Peixiang Zhong, Kezhi Mao, Dongzhe Wang, Xuefeng Yang, Yunfeng Liu, Jianxiong Yin, and Simon See. 2021. Act: an attentive convolutional transformer for efficient text classification. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 13261–13269.
[23]
Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou. 2021. A survey of convolutional neural networks: analysis, applications, and prospects. IEEE transactions on neural networks and learning systems (2021).
[24]
Tie-Yan Liu 2009. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval 3, 3 (2009), 225–331.
[25]
Tao Qin, Tie-Yan Liu, and Hang Li. 2010. A general approximation framework for direct optimization of information retrieval measures. Information retrieval 13, 4 (2010), 375–397.
[26]
Tao Qin, Tie-Yan Liu, Jun Xu, and Hang Li. 2010. LETOR: A benchmark collection for research on learning to rank for information retrieval. Information Retrieval 13, 4 (2010), 346–374.
[27]
Leonardo Rigutini, Tiziano Papini, Marco Maggini, and Franco Scarselli. 2011. SortNet: Learning to rank by a neural preference function. IEEE transactions on neural networks 22, 9 (2011), 1368–1380.
[28]
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, 1995. Okapi at TREC-3. Nist Special Publication Sp 109 (1995), 109.
[29]
Aditi Sharma, Nishtha Adhao, and Anju Mishra. 2013. A survey: Static and dynamic ranking. International Journal of Computer Applications 70, 14 (2013), 7–12.
[30]
Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. 2017. Irgan: A minimax game for unifying generative and discriminative information retrieval models. In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval. 515–524.
[31]
Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. 2008. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning. ACM, 1192–1199.
[32]
Long Xia, Jun Xu, Yanyan Lan, Jiafeng Guo, Wei Zeng, and Xueqi Cheng. 2017. Adapting Markov decision process for search result diversification. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 535–544.
[33]
Hanlin Yin, X Rong Li, and Jian Lan. 2017. Pairwise comparison based ranking vector approach to estimation performance ranking. IEEE Transactions on Systems, Man, and Cybernetics: Systems 48, 6 (2017), 942–953.
[34]
Hai-Tao Yu, Adam Jatowt, Hideo Joho, Joemon M Jose, Xiao Yang, and Long Chen. 2019. WassRank: Listwise Document Ranking Using Optimal Transport Theory. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. ACM, 24–32.

Index Terms

  1. An Empirical Perspective on Learning-to-rank

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ICCAI '23: Proceedings of the 2023 9th International Conference on Computing and Artificial Intelligence
    March 2023
    824 pages
    ISBN:9781450399029
    DOI:10.1145/3594315
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 August 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Learning-to-rank
    2. document retrieval
    3. ranking feedback
    4. sequential ranking process

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    ICCAI 2023

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 45
      Total Downloads
    • Downloads (Last 12 months)22
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 10 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media