skip to main content
10.1145/3331184.3331347acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
short-paper
Open access

Revisiting Approximate Metric Optimization in the Age of Deep Neural Networks

Published: 18 July 2019 Publication History

Abstract

Learning-to-Rank is a branch of supervised machine learning that seeks to produce an ordering of a list of items such that the utility of the ranked list is maximized. Unlike most machine learning techniques, however, the objective cannot be directly optimized using gradient descent methods as it is either discontinuous or flat everywhere. As such, learning-to-rank methods often optimize a loss function that either is loosely related to or upper-bounds a ranking utility instead. A notable exception is the approximation framework originally proposed by Qin et al. that facilitates a more direct approach to ranking metric optimization. We revisit that framework almost a decade later in light of recent advances in neural networks and demonstrate its superiority empirically. Through this study, we hope to show that the ideas from that work are more relevant than ever and can lay the foundation of learning-to-rank research in the age of deep neural networks.

References

[1]
Mart'in Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et almbox. 2016. Tensorflow: a system for large-scale machine learning. In Proc. of the 12th USENIX Symposium on Operating Systems Design and Implementation. 265--283.
[2]
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proc. of the 22nd International Conference on Machine Learning. 89--96.
[3]
Christopher J.C. Burges. 2010. From RankNet to LambdaRank to LambdaMART: An Overview. Technical Report Technical Report MSR-TR-2010--82. Microsoft Research.
[4]
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: from pairwise approach to listwise approach. In Proc. of the 24th International Conference on Machine Learning. 129--136.
[5]
Olivier Chapelle and Yi Chang. 2011. Yahoo! learning to rank challenge overview. In Proc. of the Learning to Rank Challenge. 1--24.
[6]
Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of Statistics, Vol. 29, 5 (2001), 1189--1232.
[7]
Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proc. of the 32nd International Conference on Machine Learning (ICML). 448--456.
[8]
Kalervo J"arvelin and Jaana Kekalainen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, Vol. 20, 4 (2002), 422--446.
[9]
Thorsten Joachims. 2006. Training linear SVMs in linear time. In Proc. of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 217--226.
[10]
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Advances in Neural Information Processing Systems 30. 3146--3154.
[11]
Donald A Metzler, W Bruce Croft, and Andrew Mccallum. 2005. Direct maximization of rank-based metrics for information retrieval. CIIR report 429. University of Massachusetts.
[12]
Rama Kumar Pasumarthi, Sebastian Bruch, Xuanhui Wang, Cheng Li, Michael Bendersky, Marc Najork, Jan Pfeifer, Nadav Golbandi, Rohan Anil, and Stephan Wolf. 2019. TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. (to appear).
[13]
Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 Datasets. (2013). arxiv: 1306.2597
[14]
Tao Qin, Tie-Yan Liu, and Hang Li. 2010. A general approximation framework for direct optimization of information retrieval measures. Information Retrieval, Vol. 13, 4 (2010), 375--397.
[15]
Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka. 2008. SoftRank: Optimizing Non-smooth Rank Metrics. In Proc. of the 1st International Conference on Web Search and Data Mining. 77--86.
[16]
Xuanhui Wang, Cheng Li, Nadav Golbandi, Michael Bendersky, and Marc Najork. 2018. The LambdaLoss Framework for Ranking Metric Optimization. In Proc. of the 27th ACM International Conference on Information and Knowledge Management. 1313--1322.
[17]
Qiang Wu, Christopher JC Burges, Krysta M Svore, and Jianfeng Gao. 2010. Adapting boosting for information retrieval measures. Information Retrieval, Vol. 13, 3 (2010), 254--270.
[18]
Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. 2008. Listwise approach to learning to rank: theory and algorithm. In Proc. of the 25th International Conference on Machine Learning. 1192--1199.
[19]
Jun Xu and Hang Li. 2007. AdaRank: A Boosting Algorithm for Information Retrieval. In Proc. of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 391--398.

Cited By

View all
  • (2024)JDRec: Practical Actor-Critic Framework for Online Combinatorial Recommender SystemProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3663244(2612-2614)Online publication date: 6-May-2024
  • (2024)GS2PProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/936(8433-8438)Online publication date: 3-Aug-2024
  • (2024)A Learning-to-Rank Formulation of Clustering-Based Approximate Nearest Neighbor SearchProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657931(2261-2265)Online publication date: 10-Jul-2024
  • Show More Cited By

Index Terms

  1. Revisiting Approximate Metric Optimization in the Age of Deep Neural Networks

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval
    July 2019
    1512 pages
    ISBN:9781450361729
    DOI:10.1145/3331184
    Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 18 July 2019

    Check for updates

    Author Tags

    1. deep neural networks for IR
    2. direct ranking metric optimization
    3. learning to rank

    Qualifiers

    • Short-paper

    Conference

    SIGIR '19
    Sponsor:

    Acceptance Rates

    SIGIR'19 Paper Acceptance Rate 84 of 426 submissions, 20%;
    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)423
    • Downloads (Last 6 weeks)34
    Reflects downloads up to 19 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)JDRec: Practical Actor-Critic Framework for Online Combinatorial Recommender SystemProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3663244(2612-2614)Online publication date: 6-May-2024
    • (2024)GS2PProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/936(8433-8438)Online publication date: 3-Aug-2024
    • (2024)A Learning-to-Rank Formulation of Clustering-Based Approximate Nearest Neighbor SearchProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657931(2261-2265)Online publication date: 10-Jul-2024
    • (2024)Optimizing Learning-to-Rank Models for Ex-Post Fair RelevanceProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657751(1525-1534)Online publication date: 10-Jul-2024
    • (2024)Hazards Prioritization With Cognitive Attention Maps for Supporting Driving Decision-MakingIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2024.341367525:11(16221-16234)Online publication date: Nov-2024
    • (2024)PlayerRank: Leveraging Learning-to-Rank AI for Player Positioning in CricketIEEE Access10.1109/ACCESS.2024.349552812(177504-177519)Online publication date: 2024
    • (2024)Listwise learning to rank method combining approximate NDCG ranking indicator with Conditional Generative Adversarial NetworksPattern Recognition Letters10.1016/j.patrec.2024.01.015179(31-37)Online publication date: Mar-2024
    • (2024)Integrating listwise ranking into pairwise-based image-text retrievalKnowledge-Based Systems10.1016/j.knosys.2024.111431287(111431)Online publication date: Mar-2024
    • (2024)GS2P: a generative pre-trained learning to rank model with over-parameterization for web-scale searchMachine Learning10.1007/s10994-023-06469-9113:8(5331-5349)Online publication date: 5-Jan-2024
    • (2024)Learning bivariate scoring functions for rankingDiscover Computing10.1007/s10791-024-09444-727:1Online publication date: 27-Sep-2024
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media