research-article

Multi-class cost-sensitive boosting with p-norm loss functions

Authors:

Aurélie C. Lozano,

Naoki AbeAuthors Info & Claims

KDD '08: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 506 - 514

https://doi.org/10.1145/1401890.1401953

Published: 24 August 2008 Publication History

Abstract

We propose a family of novel cost-sensitive boosting methods for multi-class classification by applying the theory of gradient boosting to p-norm based cost functionals. We establish theoretical guarantees including proof of convergence and convergence rates for the proposed methods. Our theoretical treatment provides interpretations for some of the existing algorithms in terms of the proposed family, including a generalization of the costing algorithm, DSE and GBSE-t, and the Average Cost method. We also experimentally evaluate the performance of our new algorithms against existing methods of cost sensitive boosting, including AdaCost, CSB2, and AdaBoost.M2 with cost-sensitive weight initialization. We show that our proposed scheme generally achieves superior results in terms of cost minimization and, with the use of higher order p-norm loss in certain cases, consistently outperforms the comparison methods, thus establishing its empirical advantage.

References

[1]

N. Abe, B. Zadrozny, and J. Langford. An iterative method for multi-class cost-sensitive learning. In KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 3--11, New York, NY, USA, 2004. ACM.

Digital Library

[2]

C. L. Blake and C. J. Merz. UCI repository of machine learning databases. Department of Information and Computer Sciences, University of California, Irvine, 1998. http://www.ics.uci.edu/~mlearn/MLRepository.html.

[3]

P. Domingos. MetaCost: A general method for making classifiers cost sensitive. In Proceedings of the Fifth International Conference on Knowledge Discovery and Data Mining, pages 155--164. ACM Press, 1999.

Digital Library

[4]

Charles Elkan. Magical thinking in data mining: Lessons from coil challenge 2000. In Proceedings of the Seventh International Conference on Knowledge Discovery and Data Mining, pages 426--431. ACM Press, 2001.

Digital Library

[5]

W. Fan, S. J. Stolfo, J. Zhang, and P. K. Chan. AdaCost: Misclassification cost sensitive boosting. In Proceedings of the Sixteenth International Conference on Machine Learning, pages 97--105, 1999.

Digital Library

[6]

Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119--139, 1997.

Digital Library

[7]

U. Knoll, G. Nakhaeizadeh, and B. Tausend. Cost-sensitive pruning of decision trees. In Proceedings of the Eight European Conference on Machine Learning, pages 383--386, 1994.

Digital Library

[8]

D. Margineantu. Methods for Cost-Sensitive Learning. PhD thesis, Department of Computer Science, Oregon State University, Corvallis, 2001.

Digital Library

[9]

L. Mason, J. Baxter, P. Barlett, and M. Frean. Boosting algorithms as gradient descent. In Advances in Neural Information Processing systems 12, pages 512--158, 2000.

[10]

L. Mason, J. Baxter, P. Bartlett, and M. Frean. Boosting algorithms as gradient descent in function space. Technical report, RSISE, Australian National University, 1999. http://wwwsyseng.anu.edu.au/~jon/papers/doom2.ps.gz.

[11]

J. Quinlan. C4.5: Programs for Machine Learning. San Mateo, CA: Morgan Kaufmann, 1993.

Digital Library

[12]

R. E. Schapire and Y. Singer. Improved boosting using confidence-rated predictions. Machine Learning, 37(3):297--336, 1999.

Digital Library

[13]

K. M. Ting. A comparative study of cost-sensitive boosting algorithms. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 983--990, 2000.

Digital Library

[14]

I. H. Witten and E. Frank. Data Mining: Practical machine learning tools with Java implementations. Morgan Kaufmann, 2000.

Digital Library

[15]

B. Zadrozny and C. Elkan. Learning and making decisions when costs and probabilities are both unknown. In Proceedings of the Seventh International Conference on Knowledge Discovery and Data Mining, pages 204--213. ACM Press, 2001.

Digital Library

[16]

B. Zadrozny, J. Langford, and N. Abe. Cost-sensitive learning by cost-proportionate example weighting. In Proceedings of the Third IEEE International Conference on Data Mining, pages 435--442, 2003.

Digital Library

[17]

T. Zhang. Sequential greedy approximation for certain convex optimization problems. volume IT-49, pages 682--691, 2003.

Digital Library

[18]

Z. H. Zhou and X. Y. Liu. On multi-class cost sensitive learning. In Proceedings of the 21st National Conference on Artificial Intelligence, pages 567--572, 2006.

Digital Library

Cited By

Ramos-López DMaldonado A(2021)Cost-Sensitive Variable Selection for Multi-Class Imbalanced Datasets Using Bayesian NetworksMathematics10.3390/math90201569:2(156)Online publication date: 13-Jan-2021
https://doi.org/10.3390/math9020156
Buenaposada JBaumela L(2020)Improving multi-class Boosting-based object detectionIntegrated Computer-Aided Engineering10.3233/ICA-200636(1-16)Online publication date: 3-Jul-2020
https://doi.org/10.3233/ICA-200636
Hao SLu JZhao PZhang CHoi SMiao C(2018)Second-Order Online Active Learning and Its ApplicationsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2017.277809730:7(1338-1351)Online publication date: 1-Jul-2018
https://doi.org/10.1109/TKDE.2017.2778097
Show More Cited By

Index Terms

Multi-class cost-sensitive boosting with p-norm loss functions
1. Computing methodologies
  1. Machine learning

Recommendations

An iterative method for multi-class cost-sensitive learning
KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining

Cost-sensitive learning addresses the issue of classification in the presence of varying costs associated with different types of misclassification. In this paper, we present a method for solving multi-class cost-sensitive learning problems using any ...
Cost-Sensitive Boosting

A novel framework is proposed for the design of cost-sensitive boosting algorithms. The framework is based on the identification of two necessary conditions for optimal cost-sensitive learning that 1) expected losses must be minimized by optimal cost-...
Semi-supervised boosting for multi-class classification
ECMLPKDD'08: Proceedings of the 2008th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume Part II

Most semi-supervised learning algorithms have been designed for binary classification, and are extended to multi-class classification by approaches such as one-against-the-rest. The main shortcoming of these approaches is that they are unable to exploit ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '08: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining

August 2008

1116 pages

ISBN:9781605581934

DOI:10.1145/1401890

General Chair:
Ying Li
Microsoft adCenter Labs
,
Program Chairs:
Bing Liu
University of Illinois at Chicago
,
Sunita Sarawagi
Indian Institute of Technology, Bombay

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD08

Sponsor:

KDD08: The 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 24 - 27, 2008

Nevada, Las Vegas, USA

Acceptance Rates

KDD '08 Paper Acceptance Rate 118 of 593 submissions, 20%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
734
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)1

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ramos-López DMaldonado A(2021)Cost-Sensitive Variable Selection for Multi-Class Imbalanced Datasets Using Bayesian NetworksMathematics10.3390/math90201569:2(156)Online publication date: 13-Jan-2021
https://doi.org/10.3390/math9020156
Buenaposada JBaumela L(2020)Improving multi-class Boosting-based object detectionIntegrated Computer-Aided Engineering10.3233/ICA-200636(1-16)Online publication date: 3-Jul-2020
https://doi.org/10.3233/ICA-200636
Hao SLu JZhao PZhang CHoi SMiao C(2018)Second-Order Online Active Learning and Its ApplicationsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2017.277809730:7(1338-1351)Online publication date: 1-Jul-2018
https://doi.org/10.1109/TKDE.2017.2778097
Shirui Pan Jia Wu Xingquan Zhu (2015)CogBoost: Boosting for Fast Cost-Sensitive Graph ClassificationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2015.239111527:11(2933-2946)Online publication date: 1-Nov-2015
https://dl.acm.org/doi/10.1109/TKDE.2015.2391115
Zhao PZhuang FWu MLi XHoi S(2015)Cost-Sensitive Online Classification with Adaptive Regularization and Its ApplicationsProceedings of the 2015 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM.2015.51(649-658)Online publication date: 14-Nov-2015
https://dl.acm.org/doi/10.1109/ICDM.2015.51
Wang JZhao PHoi S(2014)Cost-Sensitive Online ClassificationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2013.15726:10(2425-2438)Online publication date: Oct-2014
https://doi.org/10.1109/TKDE.2013.157
Sheng VGu BFang WWu J(2014)Cost-sensitive learning for defect escalationKnowledge-Based Systems10.1016/j.knosys.2014.04.03366:1(146-155)Online publication date: 1-Aug-2014
https://dl.acm.org/doi/10.1016/j.knosys.2014.04.033
Lomax SVadera S(2013)A survey of cost-sensitive decision tree induction algorithmsACM Computing Surveys10.1145/2431211.243121545:2(1-35)Online publication date: 12-Mar-2013
https://dl.acm.org/doi/10.1145/2431211.2431215
Scott C(2012)Calibrated asymmetric surrogate lossesElectronic Journal of Statistics10.1214/12-EJS6996:noneOnline publication date: 1-Jan-2012
https://doi.org/10.1214/12-EJS699
Wang JZhao PHoi S(2012)Cost-Sensitive Online ClassificationProceedings of the 2012 IEEE 12th International Conference on Data Mining10.1109/ICDM.2012.116(1140-1145)Online publication date: 10-Dec-2012
https://dl.acm.org/doi/10.1109/ICDM.2012.116
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten