Article

An iterative method for multi-class cost-sensitive learning

Authors:

Bianca Zadrozny,

John LangfordAuthors Info & Claims

KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 3 - 11

https://doi.org/10.1145/1014052.1014056

Published: 22 August 2004 Publication History

Abstract

Cost-sensitive learning addresses the issue of classification in the presence of varying costs associated with different types of misclassification. In this paper, we present a method for solving multi-class cost-sensitive learning problems using any binary classification algorithm. This algorithm is derived using hree key ideas: 1) iterative weighting; 2) expanding data space; and 3) gradient boosting with stochastic ensembles. We establish some theoretical guarantees concerning the performance of this method. In particular, we show that a certain variant possesses the boosting property, given a form of weak learning assumption on the component binary classifier. We also empirically evaluate the performance of the proposed method using benchmark data sets and verify that our method generally achieves better results than representative methods for cost-sensitive learning, in terms of predictive performance (cost minimization) and, in many cases, computational efficiency.

References

[1]

E. L. Allwein, R. E. Schapire, and Y. Singer. Reducing multiclass to binary: A unifying approach for margin classifiers. Journal of Machine Learning Research, 1:113--141, 2000.

Digital Library

[2]

S. D. Bay. UCI KDD archive. Department of Information and Computer Sciences, University of California, Irvine, 2000. http://kdd.ics.uci.edu/.

[3]

C. L. Blake and C. J. Merz. UCI repository of machine learning databases. Department of Information and Computer Sciences, University of California, Irvine, 1998. http://www.ics.uci.edu/~mlearn/MLRepository.html.

[4]

J. Bradford, C. Kunz, R. Kohavi, C. Brunk, and C. Brodley. Pruning decision trees with misclassification costs. In Proceedings of the European Conference on Machine Learning, pages 131--136, 1998.

Digital Library

[5]

L. Breiman. Bagging predictors. Machine Learning, 24(2):123--140, 1996.

[6]

L. Breiman, J. H. Friedman, R. A. Olsen, and C. J. Stone. Classification and Regression Trees. Wadsworth International Group, 1984.

[7]

P. Chan and S. Stolfo. Toward scalable learning with non-uniform class and cost distributions. In Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining, pages 164--168, 1998.

Digital Library

[8]

P. Domingos. MetaCost: A general method for making classifiers cost sensitive. In Proceedings of the Fifth International Conference on Knowledge Discovery and Data Mining, pages 155--164. ACM Press, 1999.

Digital Library

[9]

C. Drummond and R. C. Holte. C4.5, class imbalance, and cost-sensitivity: Why under-sampling beats over-sampling. In Workshop Notes, Workshop on Cost-Sensitive Learning, International Conference on Machine Learning, June 2000.

[10]

C. Elkan. Magical thinking in data mining: Lessons from coil challenge 2000. In Proceedings of the Seventh International Conference on Knowledge Discovery and Data Mining, pages 426--431. ACM Press, 2001.

Digital Library

[11]

W. Fan, S. J. Stolfo, J. Zhang, and P. K. Chan. AdaCost: Misclassification cost-sensitive boosting. In Proceedings of the Sixteenth International Conference on Machine Learning, pages 97--105, 1999.

Digital Library

[12]

Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119--139, 1997.

Digital Library

[13]

G. Fumera and F. Roli. Cost-sensitive learning in support vector machines. In VIII Convegno Associazione Italiana per L'Intelligenza Artificiale, 2002.

[14]

P. Geibel and F. Wysotzki. Perceptron based learning with example dependent and noisy costs. In Proceedings of the Twentieth International Conference on Machine Learning, 2003.

[15]

U. Knoll, G. Nakhaeizadeh, and B. Tausend. Cost-sensitive pruning of decision trees. In Proceedings of the Eight European Conference on Machine Learning, pages 383--386, 1994.

Digital Library

[16]

D. Margineantu. Methods for Cost-Sensitive Learning. PhD thesis, Department of Computer Science, Oregon State University, Corvallis, 2001.

Digital Library

[17]

L. Mason, J. Baxter, P. Barlett, and M. Frean. Boosting algorithms as gradient descent. In Advances in Neural Information Processing systems 12, pages 512--158, 2000.

[18]

J. Quinlan. C4.5: Programs for Machine Learning. San Mateo, CA: Morgan Kaufmann, 1993.

Digital Library

[19]

B. Zadrozny and C. Elkan. Learning and making decisions when costs and probabilities are both unknown. In Proceedings of the Seventh International Conference on Knowledge Discovery and Data Mining, pages 204--213. ACM Press, 2001.

Digital Library

[20]

B. Zadrozny, J. Langford, and N. Abe. Cost-sensitive learning by cost-proportionate example weighting. In Proceedings of the Third IEEE International Conference on Data Mining, pages 435--442, 2003.

Digital Library

Cited By

Delgado RFernández-Peláez FPallarés NDiaz-Brito VIzquierdo EOriol ISimonetti ATebé CVidela SCarratalà J(2024)Predictive risk models for COVID-19 patients using the multi-thresholding meta-algorithmScientific Reports10.1038/s41598-024-77386-714:1Online publication date: 18-Nov-2024
https://doi.org/10.1038/s41598-024-77386-7
Kumar P(2024)Adversarial attacks and defenses for large language models (LLMs): methods, frameworks & challengesInternational Journal of Multimedia Information Retrieval10.1007/s13735-024-00334-813:3Online publication date: 25-Jun-2024
https://doi.org/10.1007/s13735-024-00334-8
Zhang YTang LHuang YMa Y(2024)Smart data augmentation: One equation is all you needStatistical Analysis and Data Mining: The ASA Data Science Journal10.1002/sam.1167217:2Online publication date: 27-Mar-2024
https://doi.org/10.1002/sam.11672
Show More Cited By

Recommendations

MetaCost: a general method for making classifiers cost-sensitive
KDD '99: Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining
Editorial: special issue on learning from imbalanced data sets
Special issue on learning from imbalanced datasets
Does cost-sensitive learning beat sampling for classifying rare classes?
UBDM '05: Proceedings of the 1st international workshop on Utility-based data mining

A highly-skewed class distribution usually causes the learned classifier to predict the majority class much more often than the minority class. This is a consequence of the fact that most classifiers are designed to maximize accuracy. In many instances, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining

August 2004

874 pages

ISBN:1581138881

DOI:10.1145/1014052

General Chairs:
Won Kim
Cyber Database Solutions
,
Ronny Kohavi
Amazon.com
,
Program Chairs:
Johannes Gehrke
Cornell University
,
William DuMouchel
AT&T Labs Research

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 August 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

KDD04

Sponsor:

KDD04: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 22 - 25, 2004

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

104
Total Citations
View Citations
1,456
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)3

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Delgado RFernández-Peláez FPallarés NDiaz-Brito VIzquierdo EOriol ISimonetti ATebé CVidela SCarratalà J(2024)Predictive risk models for COVID-19 patients using the multi-thresholding meta-algorithmScientific Reports10.1038/s41598-024-77386-714:1Online publication date: 18-Nov-2024
https://doi.org/10.1038/s41598-024-77386-7
Kumar P(2024)Adversarial attacks and defenses for large language models (LLMs): methods, frameworks & challengesInternational Journal of Multimedia Information Retrieval10.1007/s13735-024-00334-813:3Online publication date: 25-Jun-2024
https://doi.org/10.1007/s13735-024-00334-8
Zhang YTang LHuang YMa Y(2024)Smart data augmentation: One equation is all you needStatistical Analysis and Data Mining: The ASA Data Science Journal10.1002/sam.1167217:2Online publication date: 27-Mar-2024
https://doi.org/10.1002/sam.11672
Li ZChen JLaber ELiu FBaumgartner R(2023)Optimal Treatment Regimes: A Review and Empirical ComparisonInternational Statistical Review10.1111/insr.1253691:3(427-463)Online publication date: 22-Feb-2023
https://doi.org/10.1111/insr.12536
Shen HChen SWang RWang X(2023)Adversarial Learning With Cost-Sensitive ClassesIEEE Transactions on Cybernetics10.1109/TCYB.2022.314638853:8(4855-4866)Online publication date: Aug-2023
https://doi.org/10.1109/TCYB.2022.3146388
Yadav AShekhar SVidyarthi APrakash RGowri R(2023)Hyper-Parameter Tuning with Grid and Randomized Search Techniques for Predictive Models of Hotel Booking2023 International Conference on Electrical, Electronics, Communication and Computers (ELEXCOM)10.1109/ELEXCOM58812.2023.10370718(1-5)Online publication date: 26-Aug-2023
https://doi.org/10.1109/ELEXCOM58812.2023.10370718
Kumaravel AVijayan T(2023)Comparing cost sensitive classifiers by the false-positive to false- negative ratio in diagnostic studiesExpert Systems with Applications10.1016/j.eswa.2023.120303227(120303)Online publication date: Oct-2023
https://doi.org/10.1016/j.eswa.2023.120303
Duan YLiu XJatowt AYu HLynden SKim KMatono A(2022)Long-Tailed Graph Representation Learning via Dual Cost-Sensitive Graph Convolutional NetworkRemote Sensing10.3390/rs1414329514:14(3295)Online publication date: 8-Jul-2022
https://doi.org/10.3390/rs14143295
Duan YLiu XJatowt AYu HLynden SKim KMatono A(2022)Dual Cost-sensitive Graph Convolutional Network2022 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN55064.2022.9892598(1-8)Online publication date: 18-Jul-2022
https://doi.org/10.1109/IJCNN55064.2022.9892598
Garg ASani DAnand S(2022)Learning Hierarchy Aware Features for Reducing Mistake SeverityComputer Vision – ECCV 202210.1007/978-3-031-20053-3_15(252-267)Online publication date: 6-Nov-2022
https://doi.org/10.1007/978-3-031-20053-3_15
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten