Ke Zhou
ABSTRACT
Designing effective ranking functions is a core problem for information retrieval and Web search since the ranking functions directly impact the relevance of the search results. The problem has been the focus of much of the research at the intersection of Web search and machine learning, and learning ranking functions from preference data in particular has recently attracted much interest. The objective of this paper is to empirically examine several objective functions that can be used for learning ranking functions from preference data. Specifically, we investigate the roles of ties in the learning process. By ties, we mean preference judgments that two documents have equal degree of relevance with respect to a query. This type of data has largely been ignored or not properly modeled in the past. In this paper, we analyze the properties of ties and develop novel learning frameworks which combine ties and preference data using statistical paired comparison models to improve the performance of learned ranking functions. The resulting optimization problems explicitly incorporating ties and preference data are solved using gradient boosting methods. Experimental studies are conducted using three publicly available data sets which demonstrate the effectiveness of the proposed new methods.
- R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison Wesley, May 1999. Google Scholar
Digital Library
- R. Bradley and M.Terry. The rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39, 1952.Google Scholar
- C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine learning, 2005. Google ScholarDigital Library
- C. J. Burges, R. Ragno, and Q. V. Le. Learning to rank with nonsmooth cost functions. In Advances in Neural Information Processing Systems 19. 2007.Google Scholar
- Y. Cao, J. Xu, T.-Y. Liu, H. Li, Y. Huang, and H.-W. Hon. Adapting ranking svm to document retrieval. In Proceedings of the 29th ACM SIGIR, 2006. Google ScholarDigital Library
- Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, 2007. Google ScholarDigital Library
- C. Cortes, M. Mohri, and A. Rastogi. Magnitude-preserving ranking algorithms. In Proceedings of the 24th ICML, 2007. Google ScholarDigital Library
- H. A. David. The Method of Paired Comparisons. Oxford University Press, 2nd edition, 1988.Google Scholar
- Y. Freund, R. D. Iyer, R. E. Schapire, and Y. Singer. An efficient boosting algorithm for combining preferences. In Proceedings of the Fifteenth International Conference on Machine Learning, 1998. Google ScholarDigital Library
- J. Friedman. Greedy function approximation: a gradient boosting machine. The Annals of Statistics, 29, 2001.Google Scholar
- J. Friedman. Stochastic gradient boosting. Computational Statistics and Data Analysis, 38, 2002. Google ScholarDigital Library
- G. Fung, R. Rosales, and B. Krishnapuram. Learning rankings via convex hull separation. In Advances in Neural Information Processing Systems 18. 2006.Google Scholar
- T. Hastie, R. Tibshirani, and J. H. Friedman. The Elements of Statistical Learning. Springer, 2001.Google ScholarCross Ref
- R. Herbrich, T. Graepel, and K. Obermayer. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers, Cambridge, MA, 2000. MIT Press.Google Scholar
- K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transaction of Information Systems, 20(4), 2002. Google ScholarDigital Library
- T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of ACM SIGKDD, 2002. Google ScholarDigital Library
- T. Joachims. A support vector method for multivariate performance measures. In Proceedings of the 22nd international conference on Machine learning, 05. Google ScholarDigital Library
- T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th ACM SIGIR conference, 2005. Google ScholarDigital Library
- J. Lafferty and C. Zhai. Document language models, query models, and risk minimization for information retrieval. In Proceedings of the 24th ACM SIGIR, 2001. Google ScholarDigital Library
- T. Y. Liu, J. Xu, T. Qin, W. Xiong, and H. Li. Letor: Benchmark dataset for research on learning to rank for information retrieval. In Proceedings of the Learning to Rank workshop in the 30th ACM SIGIR, 2007.Google Scholar
- F. Radlinski and T. Joachims. Active exploration for learning rankings from clickthrough data. In Proceedings of the 13th ACM SIGKDD, 2007. Google ScholarDigital Library
- P. Rao and L. Kupper. Ties in paired-comparison experiments: A generalization of the bradley-terry model,. Journal of the American Statistical Association, 62, March 1967.Google Scholar
- S. Robertson and D. A. Hull. The TREC-9 filtering track final report. Proceedings of the 9th Text REtrieval Conference (TREC-9), 2001.Google Scholar
- G. Salton, A. Wong, and C. S. Yang. A vector space model for automatic indexing. Commun. ACM, 18(11), 1975. Google ScholarDigital Library
- F. Song and W. B. Croft. A general language model for information retrieval. In Proceedings of CIKM, 1999. Google ScholarDigital Library
- L. Thurstone. A law of comparative judgement. Psychological Review, 34, 1927.Google Scholar
- M.-F. Tsai, T.-Y. Liu, T. Qin, H.-H. Chen, and W.-Y. Ma. Frank: a ranking method with fidelity loss. In Proceedings of ACM SIGIR, pages 383--390, 2007. Google ScholarDigital Library
- J. Xu and H. Li. Adarank: a boosting algorithm for information retrieval. In Proceedings of the 30th ACM SIGIR, 2007. Google ScholarDigital Library
- Y. Yue, T. Finley, F. Radlinski, and T. Joachims. A support vector method for optimizing average precision. In Proceedings of ACM SIGIR, 2007. Google ScholarDigital Library
- Z. Zheng, K. Chen, G. Sun, and H. Zha. A regression framework for learning ranking functions using relative relevance judgments. In Proceedings of the 30th ACM SIGIR conference, 2007. Google ScholarDigital Library
- Z. Zheng, H. Zha, T. Zhang, O. Chapelle, K. Chen, and G. Sun. A general boosting method and its application to learning ranking functions for web search. In Advances in Neural Information Processing Systems. 2007.Google Scholar
Index Terms
- Learning to rank with ties
Recommendations
A regression framework for learning ranking functions using relative relevance judgments
SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrievalEffective ranking functions are an essential part of commercial search engines. We focus on developing a regression framework for learning ranking functions for improving relevance of search engines serving diverse streams of user queries. We explore ...
Smoothing DCG for learning to rank: a novel approach using smoothed hinge functions
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge managementDiscounted cumulative gain (DCG) is widely used for evaluating ranking functions. It is therefore natural to learn a ranking function that directly optimizes DCG. However, DCG is non-smooth, rendering gradient-based optimization algorithms inapplicable. ...
An adaptive learning to rank algorithm: Learning automata approach
The recent years have witnessed the birth and explosive growth of the web. It is obvious that the exponential growth of the web has made it into a huge interconnected source of information wherein finding a document without a searching tool is ...
Comments