Abstract
A general model is proposed for studying ranking problems. We investigate learning methods based on empirical minimization of the natural estimates of the ranking risk. The empirical estimates are of the form of a U-statistic. Inequalities from the theory of U-statistics and U-processes are used to obtain performance bounds for the empirical risk minimizers. Convex risk minimization methods are also studied to give a theoretical framework for ranking algorithms based on boosting and support vector machines. Just like in binary classification, fast rates of convergence are achieved under certain noise assumption. General sufficient conditions are proposed in several special cases that guarantee fast rates of convergence.
This research was supported in part by Spanish Ministry of Science and Technology and FEDER, grant BMF2003-03324, and by the PASCAL Network of Excellence under EC grant no. 506778.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Agarwal, S., Graepel, T., Herbrich, R., Roth, D.: A large deviation bound for the area under the ROC curve. In: Proceedings of the 18th Annual Conference on Neural Information Processing Systems, Vancouver, Canada (2004)
Agarwal, S., Har-Peled, S., Roth, D.: A uniform convergence bound for the area under the ROC curve. In: Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics, Barbados (2005)
Arcones, M.A., Giné, E.: U-processes indexed by Vapnik-Chervonenkis classes of functions with applications to asymptotics and bootstrap of U-statistics with estimated parameters. Stochastic Processes and their Applications 52, 17–38 (1994)
Bartlett, P.L., Jordan, M.I., McAuliffe, J.D.: Convexity, classification, and risk bounds. Technical Report 638, Department of Statistics, U.C. Berkeley (2003)
Bartlett, P.L., Mendelson, S.: Empirical minimization. Technical Report, Department of Statistics, U.C. Berkeley (2003)
Blanchard, G., Lugosi, G., Vayatis, N.: On the rate of convergence of regularized boosting classifiers. Journal of Machine Learning Research 4, 861–894 (2003)
Bousquet, O., Boucheron, S., Lugosi, G.: Theory of classification: a survey of recent advances. ESAIM: Probability and Statistics, (2004) (to appear)
Breiman, L.: Population theory for boosting ensembles. Annals of Statistics 32, 1–11 (2004)
de la Peña, V., Giné, E.: Decoupling: from dependence to independence. Springer, Heidelberg (1999)
Devroye, L., Györfi, L., Lugosi, G.: A Probabilistic Theory of Pattern Recognition. Springer, New York (1996)
Freund, Y., Iyer, R., Schapire, R.E., Singer, Y.: An Efficient Boosting Algorithm for Combining Preferences. Journal of Machine Learning Research 4, 933–969 (2003)
Haussler, D.: Sphere packing numbers for subsets of the boolean n-cube with bounded Vapnik-Chervonenkis dimension. Journal of Combinatorial Theory, Series A 69, 217–232 (1995)
Herbrich, R., Graepel, T., Obermayer, K.: Large margin rank boundaries for ordinal regression. In: Smola, A., et al. (eds.) Advances in Large Margin Classifiers, pp. 115–132. MIT Press, Cambridge (2000)
Hoeffding, W.: Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association 58, 13–30 (1963)
Jiang, W.: Process consistency for Adaboost (with discussion). Annals of Statistics 32, 13–29 (2004)
Koltchinskii, V., Panchenko, D.: Empirical margin distribution and bounding the generalization error of combined classifiers. Annals of Statistics 30, 1–50 (2002)
Lugosi, G.: Pattern classification and learning theory. In: Györfi, L. (ed.) Principles of Nonparametric Learning, pp. 1–56. Springer, Heidelberg (2002)
Lugosi, G., Vayatis, N.: On the Bayes-risk consistency of boosting methods (with discussion). Annals of Statistics 32, 30–55 (2004)
Massart, P., Nédélec, E.: Risk bounds for statistical learning, Preprint, Université Paris XI (2003)
McDiarmid, C.: On the method of bounded differences. In: Surveys in Combinatorics 1989, pp. 148–188. Cambridge University Press, Cambridge (1989)
Serfling, R.J.: Approximation theorems of mathematical statistics. John Wiley & Sons, Chichester (1980)
Tsybakov, A.: Optimal aggregation of classifiers in statistical learning. Annals of Statistics 32, 135–166 (2004)
Zhang, T.: Statistical behavior and consistency of classification methods based on convex risk minimization (with discussion). Annals of Statistics 32, 56–85 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Clémençon, S., Lugosi, G., Vayatis, N. (2005). Ranking and Scoring Using Empirical Risk Minimization. In: Auer, P., Meir, R. (eds) Learning Theory. COLT 2005. Lecture Notes in Computer Science(), vol 3559. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11503415_1
Download citation
DOI: https://doi.org/10.1007/11503415_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-26556-6
Online ISBN: 978-3-540-31892-7
eBook Packages: Computer ScienceComputer Science (R0)