MD-ELM: Originally Mislabeled Samples Detection using OP-ELM Model

Abstract

This paper proposes a methodology for identifying data samples that are likely to be mislabeled in a c-class classification problem (dataset). The methodology relies on an assumption that the generalization error of a model learned from the data decreases if a label of some mislabeled sample is changed to its correct class. A general classification model used in the paper is OP-ELM; it also provides a fast way to estimate the generalization error by PRESS Leave-One-Out. It is tested on two toy datasets, as well as on real life datasets for one of which expert knowledge about the identified potential mislabels has been sought.

Introduction

This work focuses on finding data samples with incorrect labels in a given dataset. Such samples create “label noise”, which is generally considered more harmful than feature noise [1]. The work is motivated by studies on a financial dataset [2] where each sample corresponds to a company labeled as either healthy or bankrupt. In this dataset, samples with incorrect labels are important both by themselves (i.e. as companies eligible for a loan but mislabeled by “bankrupt”), and for the whole dataset to allow building more precise machine learning models with a limited amount of data (because each sample is expensive and slow to obtain). There are other areas where the detection of particular mislabeled samples is important, like medical applications [3].

There exist multiple sources of label noise. First, noise can be generated by simple mistakes in data gathering and processing, like people mistyping or sensor malfunction [4], [1]. For real datasets, such noise is estimated to be roughly 5% not including other factors [5]. Second, experts who label the data can make mistakes. This happens especially in cases where labeling quality is traded for lower labeling price, for instance with crowdsourcing [6] like Amazon Mechanical Turk [7] framework. Third, labeling criterion may be vague, then different experts will produce different labels. For example, in EEG segmentation exact beginnings and ends of signals are not formally defined, and different doctors give slightly different signal boundaries [8]. At last, the existing information may be insufficient for reliable labeling of data [4].

Recent methods of machine learning in the presence of mislabeled data can be aggregated in three categories [9]. Data cleansing (or filtering) methods [4] pre-process dataset and fix incorrect labels or remove such samples [10]. The resulting clean dataset is used with general machine learning methods. Noise-robust methods [11], [12] like k-nearest neighbors [13] are tuned to perform well despite the presence of label noise. It is even possible to achieve same theoretical performance with label noise as without one, although in simple cases [14]. Noise-tolerant methods include label noise in their model; an extensive overview of such methods is presented in [9], Section 7. A good survey is given by Frénay in his PhD thesis [15].

The idea of detecting mislabeled samples is to utilize their effect of increasing the model complexity [4], [16]. In a Single Layer Feed-forward Neural network (SLFN) model, a more complex model requires more hidden neurons to learn [24], or equally a more complex model will result in a lower accuracy with the same amount of hidden neurons. Correcting an incorrect sample label will decrease a model error of a fixed SLFN. Note a difference between mislabeled samples and outliers—an outlier is not a typical sample of any class, so changing its label will not lead to a decrease in model error; although outlier detector methods are applicable for dealing with mislabeled data [17].

One existing work on mislabel detection topic is written by Gamberger and Lavrac [3] before millennium for a medical domain. They performed a medical evaluation of particular detected mislabeled samples (corresponding to patients), and found them really mislabeled. No more recent works focused on detecting mislabeled samples were found in literature review.

In [18], the MD-ELM methodology is proposed for the problem of binary classification. This paper changes the method to give efficient performance and stopping criteria, reduces chances of missing a mislabeled sample and improves overall performance with multiple SLFN models, extends the methodology to multiple classes, optimizes computational time. Also the impact of method parameters is analyzed, and their values validated on toy datasets.

Section 2 describes the proposed methodology, including an ELM model, fast LOO error estimation, and the final MD-ELM algorithm. Experimental results, Section 3, present the method performance with artificially added mislabeled samples in binary and multiclass cases. Several real life data sets are then used, in which the method attempts to identify potential original mislabels.