Abstract
We present a reduction from cost-sensitive classification to binary classification based on (a modification of) error correcting output codes. The reduction satisfies the property that ε regret for binary classification implies l 2-regret of at most 2ε for cost estimation. This has several implications:
-
1
Any regret-minimizing online algorithm for 0/1 loss is (via the reduction) a regret-minimizing online cost-sensitive algorithm. In particular, this means that online learning can be made to work for arbitrary (i.e., totally unstructured) loss functions.
-
2
The output of the reduction can be thresholded so that ε regret for binary classification implies at most 4\({\sqrt{\epsilon Z}}\) regret for cost-sensitive classification where Z is the expected sum of costs.
-
3
For multiclass problems, ε binary regret translates into l 2-regret of at most 4ε in the estimation of class probabilities. For classification, this implies at most 4\({\sqrt{\epsilon}}\) multiclass regret.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Allwein, E., Schapire, R., Singer, Y.: Reducing multiclass to binary: A unifying approach for margin classifiers. Journal of Machine Learning Research 1, 113–141 (2000)
Beygelzimer, A., Dani, V., Hayes, T., Langford, J.: Reductions between classification tasks. Electronic Colloquium on Computational Complexity, TR04-077 (2004)
Dietterich, T., Bakiri, G.: Solving multiclass learning problems via error-correcting output codes. Journal of Artificial Intelligence Research 2, 263–286 (1995)
Freund, Y., Schapire, R.: A decision-theoretic generalization of online learning and an application to boosting. Journal of Computer and System Sciences 55(1), 119–139 (1997)
Guruswami, V., Sahai, A.: Multiclass learning, boosting, and error-correcting codes. In: Proceedings of the 12th Annual Conference on Computational Learning Theory (COLT), pp. 145–155 (1999)
Kalai, A., Servedio, R.: Boosting in the Presence of Noise. In: Proceedings of the 35th Annual ACM Symposium on the Theory of Computing (STOC), pp. 195–205 (2003)
Langford, J., Zadrozny, B.: Estimating class membership probabilities using classifier learners. In: Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics (2005)
Lee, Y., Lin, Y., Wahba, G.: Multicategory Support Vector Machines: Theory and Application to the Classification of Microarray Data and Satelite Radiance Data. Journal of the American Statistical Association 99, 465, 67–81 (2004)
Littlestone, N., Warmuth, M.: The Weighted Majority Algorithm. In: Foundations of Computer Science (1992)
Platt, J.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Smola, A., Bartlett, P., Schölkopf, B., Schuurmans, D. (eds.) Advances in Large Margin Classifiers, pp. 61–74 (1999)
Blake, C., Merz, C.: UCI Repository of Machine Learning Databases University of California, Irvine, http://www.ics.uci.edu/~mlearn/MLRepository.html
Valiant, L.: Learning disjunctions of conjunctions. In: Proceedings of the 9th IJCAI, pp. 560–566 (1985)
Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Heidelberg (1995)
Vapnik, V., Chervonenkis, A.: On the uniform convergence of relative frequencies of event to their probabilities. Theory of Probability and its Applications 16(2), 264–280 (1971)
Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools with Java implementations. Morgan Kaufmann, San Francisco (2000), http://www.cs.waikato.ac.nz/ml/weka/
Zadrozny, B., Langford, J., Abe, N.: Cost-Sensitive Learning by Cost-Proportionate Example Weighting. In: Proceedings of the 3rd IEEE International Conference on Data Mining (ICDM), pp. 435–442 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Langford, J., Beygelzimer, A. (2005). Sensitive Error Correcting Output Codes. In: Auer, P., Meir, R. (eds) Learning Theory. COLT 2005. Lecture Notes in Computer Science(), vol 3559. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11503415_11
Download citation
DOI: https://doi.org/10.1007/11503415_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-26556-6
Online ISBN: 978-3-540-31892-7
eBook Packages: Computer ScienceComputer Science (R0)