Skip to main content
Springer Nature Link
Log in

Large-Margin Thresholded Ensembles for Ordinal Regression: Theory and Practice

  • Conference paper
Algorithmic Learning Theory (ALT 2006)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 4264))

Included in the following conference series:

  • 865 Accesses

Abstract

We propose a thresholded ensemble model for ordinal regression problems. The model consists of a weighted ensemble of confidence functions and an ordered vector of thresholds. We derive novel large-margin bounds of common error functions, such as the classification error and the absolute error. In addition to some existing algorithms, we also study two novel boosting approaches for constructing thresholded ensembles. Both our approaches not only are simpler than existing algorithms, but also have a stronger connection to the large-margin bounds. In addition, they have comparable performance to SVM-based algorithms, but enjoy the benefit of faster training. Experimental results on benchmark datasets demonstrate the usefulness of our boosting approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Herbrich, R., Graepel, T., Obermayer, K.: Large margin rank boundaries for ordinal regression. In: Advances in Large Margin Classifiers, pp. 115–132. MIT Press, Cambridge (2000)

    Google Scholar 

  2. Shashua, A., Levin, A.: Ranking with large margin principle: Two approaches. In: Advances in Neural Information Processing Systems 15, pp. 961–968. MIT Press, Cambridge (2003)

    Google Scholar 

  3. Chu, W., Keerthi, S.S.: New approaches to support vector ordinal regression. In: Proceedings of ICML 2005, pp. 145–152. Omnipress (2005)

    Google Scholar 

  4. Crammer, K., Singer, Y.: Online ranking by projecting. Neural Computation 17, 145–175 (2005)

    Article  MATH  Google Scholar 

  5. Li, L., Lin, H.T.: Ordinal regression by extended binary classification. Under review (2007)

    Google Scholar 

  6. Rudin, C., Cortes, C., Mohri, M., Schapire, R.E.: Margin-based ranking meets boosting in the middle. In: Auer, P., Meir, R. (eds.) COLT 2005. LNCS (LNAI), vol. 3559, pp. 63–78. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  7. Freund, Y., Iyer, R., Shapire, R.E., Singer, Y.: An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research 4, 933–969 (2003)

    Article  Google Scholar 

  8. Meir, R., Rätsch, G.: An introduction to boosting and leveraging. In: Mendelson, S., Smola, A.J. (eds.) Advanced Lectures on Machine Learning. LNCS, vol. 2600, pp. 118–183. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  9. Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Machine Learning: ICML 1996, pp. 148–156. Morgan Kaufmann, San Francisco (1996)

    Google Scholar 

  10. Vapnik, V.N.: The Nature of Statistical Learning Theory. Springer, Heidelberg (1995)

    MATH  Google Scholar 

  11. Schapire, R.E., Freund, Y., Bartlett, P., Lee, W.S.: Boosting the margin: A new explanation for the effectiveness of voting methods. The Annals of Statistics 26, 1651–1686 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  12. Lin, H.-T., Li, L.: Infinite ensemble learning with support vector machines. In: Gama, J., Camacho, R., Brazdil, P.B., Jorge, A.M., Torgo, L. (eds.) ECML 2005. LNCS, vol. 3720, pp. 242–254. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  13. Mason, L., Baxter, J., Bartlett, P., Frean, M.: Functional gradient techniques for combining hypotheses. In: Advances in Large Margin Classifiers, pp. 221–246. MIT Press, Cambridge (2000)

    Google Scholar 

  14. Robertson, T., Wright, F.T., Dykstra, R.L.: Order Restricted Statistical Inference. John Wiley & Sons, Chichester (1988)

    MATH  Google Scholar 

  15. Rätsch, G., Mika, S., Schölkopf, B., Müller, K.R.: Constructing boosting algorithms from SVMs: An application to one-class classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 1184–1199 (2002)

    Article  Google Scholar 

  16. Li, L.: Perceptron learning with random coordinate descent. Technical Report CaltechCSTR:2005.006, California Institute of Technology (2005)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Learning Systems Group, California Institute of Technology, USA

    Hsuan-Tien Lin & Ling Li

Authors
  1. Hsuan-Tien Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ling Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Departament de Llenguatges i Sistemes Informàtics Laboratori d’Algorísmica Relacional, Complexitat i Aprenentatge, Universitat Politècnica de Catalunya, Barcelona,  

    José L. Balcázar

  2. Google, 1600 Amphitheatre Parkway, 94043, Mountain View, CA, USA

    Philip M. Long

  3. Department of Computer Science and Department of Mathematics, National University of Singapore, 117543, Singapore, Republic of Singapore

    Frank Stephan

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Lin, HT., Li, L. (2006). Large-Margin Thresholded Ensembles for Ordinal Regression: Theory and Practice. In: Balcázar, J.L., Long, P.M., Stephan, F. (eds) Algorithmic Learning Theory. ALT 2006. Lecture Notes in Computer Science(), vol 4264. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11894841_26

Download citation

  • DOI: https://doi.org/10.1007/11894841_26

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-46649-9

  • Online ISBN: 978-3-540-46650-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics