Comparing classifiers when the misallocation costs are uncertain

doi:10.1016/S0031-3203(98)00154-X

Pattern Recognition

Volume 32, Issue 7, July 1999, Pages 1139-1147

https://doi.org/10.1016/S0031-3203(98)00154-X Get rights and content

Abstract

Receiver Operating Characteristic (ROC) curves are popular ways of summarising the performance of two class classification rules. In fact, however, they are extremely inconvenient. If the relative severity of the two different kinds of misclassification is known, then an awkward projection operation is required to deduce the overall loss. At the other extreme, when the relative severity is unknown, the area under an ROC curve is often used as an index of performance. However, this essentially assumes that nothing whatsoever is known about the relative severity – a situation which is very rare in real problems. We present an alternative plot which is more revealing than an ROC plot and we describe a comparative index which allows one to take advantage of anything that may be known about the relative severity of the two kinds of misclassification.

References (0)

Cited by (139)

Predicting trips to health care facilities: A binary logit and receiver operating characteristics (ROC) approach
2024, Research in Transportation Economics
This paper discusses an overarching framework that integrates latent variable models and prediction success with receiver operator characteristics (ROC) curves. In three illustrative examples that focus on trip purpose, health market indicators, and resources, the paper employs binary logit models and ROC methodology to identify factors that best discriminate individuals' trips to out-patient health care facilities. Data for the examples are a three-year longitudinal survey of persons 45 years and older in China. The study contributes to the sparse empirical economics literature using ROC methodologies and more broadly to the transportation and health literatures. The ROC applications provide new insights on health care trips, finding that out-patient trips for treatment, paying out-of-pocket costs, and lack of monetary resources are salient discriminators in one's trip choice decisions.
Incorporating financial development indicators into early warning systems
2023, Journal of Economic Asymmetries
Citation Excerpt :
A related criticism is that the ROC approach weights all types of errors equally, while in applications to predicting financial crises the type I and type II errors may not have the same importance. When misclassification costs are unequal, summarising over all possible threshold values is flawed (Adams & Hand, 1999). The usefulness represents the gain using an EWS in comparison with the naïve approach in a decision making process.
We set up an early warning system for financial crises based on the Random Forrest approach. We use a novel set of predictors that comprises financial development indicators in addition to conventional imbalances measures. The evaluation of the model is conducted using a three-step procedure (i.e. training, validation and testing sub-samples). The results indicate that combining financial imbalances and financial development indicators helps to improve the out-of-sample accuracy of the early warning system.
Deep learning features in facial identification and the likelihood ratio bound
2023, Forensic Science International
In recent years, the score-based likelihood ratio (SLR) method for facial comparison has attracted considerable research attention. This method relies on the match scores that are calculated from the features obtained from facial recognition systems, deep learning based in particular. However, this concept has not been completely understood. Therefore, this study is aimed at investigating deep learning facial features, and the SLR levels of their match scores. We propose a new interpretation that the deep learning feature is a class characteristic. Based on a large-scale data set experiment, we present evidence that the log SLR value of deep learning features can reach 8 in some data sets. The study results imply that the SLR of deep learning features is a useful method for facial identification, especially when the suspected image is obtained via a CCTV camera.
Case study: Lymph node malignancy classification for head and neck cancer radiation therapy
2023, Machine Learning and Artificial Intelligence in Radiation Oncology: A Guide for Clinicians
The accurate identification of malignant lymph nodes (LNs) plays a pivotal role in the success of radiation therapy for treating head and neck cancer. In this chapter, we present a case study detailing the progression of a hybrid model, combining radiomics and convolutional neural networks, aiming at achieving accurate identification of malignant lymph nodes. Initially, our model was designed to classify LNs into three classes: malignant, suspicious, benign. This initial development phase leveraged patient data from the INFIELD trial, where nodal status was determined through physicians’ evaluation. Subsequently, we fine-tuned the model to classify LNs into two classes: benign and malignant. This fine-tuning involved a dataset comprising patients who had undergone surgical resection, with nodal status confirmed by pathological examination. The fine-tuned model was then utilized to identify malignant LN in a prospective phase II clinical trial (INRT-AIR). Notably, this trial featured a novel treatment approach, as it delivered therapeutic dose to malignant LNs only, in contrast to conventional radiation therapy for head and neck cancer, which treats the whole neck. Finally, we discussed additional considerations regarding our model uncertainty quantification, as well as its interpretability and explainability. These facets are crucial for facilitating the translation of our model into clinical practice.
Use and Misuse of Receiver Operating Characteristic Analysis in Eyewitness Identification
2021, Journal of Applied Research in Memory and Cognition
On the class overlap problem in imbalanced data classification
2021, Knowledge-Based Systems
Class imbalance is an active research area in the machine learning community. However, existing and recent literature showed that class overlap had a higher negative impact on the performance of learning algorithms. This paper provides detailed critical discussion and objective evaluation of class overlap in the context of imbalanced data and its impact on classification accuracy. First, we present a thorough experimental comparison of class overlap and class imbalance. Unlike previous work, our experiment was carried out on the full scale of class overlap and an extreme range of class imbalance degrees. Second, we provide an in-depth critical technical review of existing approaches to handle imbalanced datasets. Existing solutions from selective literature are critically reviewed and categorised as class distribution-based and class overlap-based methods. Emerging techniques and the latest development in this area are also discussed in detail. Experimental results in this paper are consistent with existing literature and show clearly that the performance of the learning algorithm deteriorates across varying degrees of class overlap whereas class imbalance does not always have an effect. The review emphasises the need for further research towards handling class overlap in imbalanced datasets to effectively improve learning algorithms’ performance.

View all citing articles on Scopus

View full text