Abstract
We consider geometric conditions on a labeled data set which guarantee that boosting algorithms work well when linear classifiers are used as weak learners. We start by providing conditions on the error of the weak learner which guarantee that the empirical error of the composite classifier is small. We then focus on conditions required in order to insure that the linear weak learner itself achieves an error which is smaller than 1/2 - γ, where the advantage parameter ? is strictly positive and independent of the sample size. Such a condition guarantees that the generalization error of the boosted classifier decays to its minimal value at a rate of \( 1/\sqrt m \) , where m is the sample size.The required conditions, which are based solely on geometric concepts, can be easily verified for any data set in time O( m2), and may serve as an indication for the effectiveness of linear classifiers as weak learners for a particular data set.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
R. Alexander. Geometric methods in the study of irregularities of distribution. Combinatorica, 10(2):115–136, 1990.
R. Alexander.Principles of a new method in the study of irregularities of distribution. Invent. Math., 103:279–296, 1991.
L. Devroye and Györfi Nonparametric Density Estimation: The L1 View. John Wiley, New York, 1985.
Y. Freund. Boosting a weak learning algorithm by majority. Information and Computation, 121:256–285, 1995.
Y. Freund and R.E. Schapire. Experiments with a new boosting algorithm. In Proceeding of the Thirteenth International Conference on Machine Learning, pages 148–156, 1996.
Y. Freund and R.E. Schapire. Game theory, on-line prediction and boosting. In Proceedings of the Ninth Annual Conference on Computational Learning Theory, 1996, pages 325–332, 1996.
J. Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression: a statistical view of boosting. The Annals of Statistics, 38(2):337–374, 2000.
M. Goldman, L. Håstad, and A. Razborov. Majority gates vs. general weighted threshold gates. Jour. of Comput. Complexity, 1(4):277–300, 1992.
V. Koltchinskii, D. Panchenko, and F. Lozano. Some new bounds on the generlization error of combined classifiers. In T. Dietterich, editor, Advances in Neural Information Processing Systems 14, Boston, 2001. MIT Press.
S. Mannor and R. Meir. On the existence of weak learners and applications to boosting. Machine Learning, 2001.T o appear.
R.E. Schapire, Y. Freund, P. Bartlett, and W.S. Lee. Boosting the margin: a new explanation for the effectiveness of voting methods. The Annals of Statistics, 26(5):1651–1686, 1998.
R.E. Schapire and Y. Singer. Improved boosting algorithms using confidence-rated predictions. Machine Learning, 37(3):297–336, 1999.
J. Shaw-Taylor, P.L. Bartlett, R.C. Williamson, and M Anthony. Structural risk minimization over data-dependent hierarchies. IEEE Trans. Inf. Theory, 44(5):1926–1940, September 1998.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mannor, S., Meir, R. (2001). Geometric Bounds for Generalization in Boosting. In: Helmbold, D., Williamson, B. (eds) Computational Learning Theory. COLT 2001. Lecture Notes in Computer Science(), vol 2111. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44581-1_30
Download citation
DOI: https://doi.org/10.1007/3-540-44581-1_30
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-42343-0
Online ISBN: 978-3-540-44581-4
eBook Packages: Springer Book Archive