Multi-label thresholding for cost-sensitive classification
Section snippets
Introduction and motivation
Classification is one of the most prominent learning tasks in machine learning, in which the task is to classify a new instance into one or more potential classes. Traditional classification allows only a single label (binary or multi-class) whereas multi-label classification considers more than one label for each instance simultaneously [1], [2], [3]. Usually, it is inadequate to classify each instance under just one single label, because several labels could describe its content concurrently.
Background and related work
Cost-sensitive learning is “a type of learning in data mining that takes misclassification costs and possibly other types of cost into consideration” [8]. In cost-sensitive learning, the key idea is to treat different misclassification costs differently to achieve higher classification accuracy. Costs are not necessarily monetary but can, for example, be a waste of time or illness. Many real-world applications such as medical decision making, fraud detection, target marketing and object
Preliminaries
In this section, we define key concepts and notation used throughout the paper. Let be the instance space and be the label space. The number of labels is denoted q – in a single-label classification (binary or multi-class) , while in a multi-label classification , because the label space is extended to multiple vectors enabling multiple labels per instance. Label sets are subsets of .
is any dataset with m instances, where is an instance and
Evaluating multi-label classifiers
As mentioned in the introduction, in multi-label learning, a prediction is obtained for each (instance, label) pair which is typically real-valued score. A thresholding strategy will be used to convert the predicted scores to actual labels. Multi-label classification evaluation measures are divided into two main categories: 1) instance-based methods that compute the average differences of the actual and the predicted labels over all instances; and 2) label-based methods that break down the
Threshold choice methods for multi-label problems
In this section, we present different approaches to perform the final labelling of a multi-label dataset given a set of scores for the potential labels. Let us assume we have trained a multi-label model with a given training dataset . Then, we run the classifier on a test set and we obtain a confidence score matrix that indicates for each test instance the predicted scores. The final task is to decide the best labels for the test instances. We distinguish between the number of thresholds
Experimental evaluation
13 multi-label datasets have been used in our experiments to clarify the differences among different thresholding techniques. We used the train/test splits that are provided in Meka1 and Mulan2 repositories [43]. Neither repository provide train/test splits for three of the datasets, namely, Cal500, Language log and Slashdot. Instead, we used the random splits (75% train, 25% test) that are made available by the KDIS research group.3
Concluding remarks
There is a great deal of literature on multi-label learning. To the best of our knowledge, none of these studies have introduced changes in misclassification costs across contexts. In this paper, we explore multi-label threshold choice methods: fixed, rate-driven, optimal, RCut and MCut. In addition, we introduce two novel thresholding methods for multi-label classification: score-driven and global optimal. Score-driven threshold can be adjusted globally (per dataset) or locally (per label).
We
CRediT authorship contribution statement
Reem Alotaibi: Conceptualization, Formal analysis, Investigation, Visualization, Methodology, Software, Writing - original draft. Peter Flach: Conceptualization, Formal analysis, Investigation, Supervision, Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgement
This project is funded by Deanship of Scientific Research (DSR), King Abdulaziz University, Saudi Arabia, Jeddah under Grant No. (J-159-612-1440).
Dr. Reem Alotaibi received her PhD in computer science from University of Bristol, Bristol, U.K., in 2017. SShe he She is an assistant professor at the Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia. Her research interests include Artificial Intelligence, Machine earning, Data mining and Crowd management. Dr. Alotaibi’s research has been funded by several sources in Saudi Arabia including Deputyship for Research & Innovation, Ministry of
References (45)
Stacked generalization
Neural Networks
(1992)- G. Tsoumakas, I. Katakis, I. Vlahavas, Mining multi-label data, in: Data Mining and Knowledge Discovery Handbook, 2010,...
- et al.
A review on multi-label learning algorithms
IEEE Trans. Knowl. Data Eng.
(2014) - E. Gibaja, S. Ventura, A tutorial on multilabel learning, ACM Comput. Surveys 47 (3) (2015) 52:1–52:38. ISSN 0360-0300,...
- R. Al-Otaibi, M. Kull, P. Flach, Declaratively capturing local label correlations with multi-label trees, in: G.A....
- E.K. Yapp, X. Li, W.F. Lu, P.S. Tan, Comparison of base classifiers for multi-label learning, Neurocomputing. ISSN...
- R. Al-Otaibi, P.A. Flach, M. Kull, Multi-label classification: a comparative study on threshold selection methods,...
- A. Rivolli, A. de Carvalho, The utiml package: multi-label classification in R, R J. 10 (2019) 24....
- C.X. Ling, V.S. Sheng, Cost-Sensitive Learning and the Class Imbalance Problem, 2008, Springer, pp. 869–875. ISBN...
- C. Elkan, The foundations of cost-sensitive learning, in: Proceedings of the Seventeenth International Joint Conference...
Cost-sensitive Learning and the Class Imbalanced Problem
Cited by (13)
Contrastively enforcing distinctiveness for multi-label image classification
2023, NeurocomputingJoint optimization of scoring and thresholding models for online multi-label classification
2023, Pattern RecognitionA contradiction solving method for complex product conceptual design based on deep learning and technological evolution patterns
2023, Advanced Engineering InformaticsCS-ResNet: Cost-sensitive residual convolutional neural network for PCB cosmetic defect detection
2021, Expert Systems with ApplicationsCitation Excerpt :Different from cost-sensitive learning, threshold-moving adjusted the output threshold toward majority classes such that samples with higher costs became harder to be misclassified (Zhu et al., 2018). Some researches have shown that threshold-moving is as effective as resampling in addressing the class imbalance and cost-sensitive problems (Alotaibi & Flach, 2021). The flowchart of our proposed PCB cosmetic defect detection system is shown in Fig. 3.
From Scores to Predictions in Multi-Label Classification: Neural Thresholding Strategies
2023, Applied Sciences (Switzerland)
Dr. Reem Alotaibi received her PhD in computer science from University of Bristol, Bristol, U.K., in 2017. SShe he She is an assistant professor at the Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia. Her research interests include Artificial Intelligence, Machine earning, Data mining and Crowd management. Dr. Alotaibi’s research has been funded by several sources in Saudi Arabia including Deputyship for Research & Innovation, Ministry of Education, King Abdulaziz City for Science and Technology (KACST) and Deanship of Scientifc Research (DSR), King Abdulaziz University.
Peter Flach received the PhD degree in Computer Science from Tilburg University, the Netherlands in 1995. He is a Professor of artificial intelligence at the University of Bristol. His research interests include mining highly structured data and the evaluation and improvement of machine learning models. Flach has been Editor-in-Chief of the Machine Learning journal since 2010, and is the author of Machine Learning: The Art and Science of Algorithms That Make Sense of Data (Cambridge University Press, 2012).