Active learning through label error statistical methods

Highlights

• We define two label error statistics functions and build clustering-based practical statistical models to guide block splitting.

• We propose a center-and-edge instance selection strategy to choose critical instances.

• We design an algorithm called active learning through label error statistical methods (ALSE).

• Results of significance test verify the superiority of ALSE to state-of-the-art algorithms.

Abstract

Clustering-based active learning splits data into a number of blocks and queries the labels of the most critical instances. An active learner must decide how to choose these critical instances and how to split the blocks. In this paper, we present theoretical and practical statistical methods for analyzing the relationship between the label error and the neighbor radius, and design new split and selection strategies to handle these two issues. First, we define statistical functions for the label error based on a single instance and instance pairs. Second, we build practical statistical models, calculate empirical label errors, and guide the block splitting process. Third, using these practical models, we develop a center-and-edge instance selection strategy for choosing critical instances. Fourth, we design a new algorithm called active learning through label error statistical methods (ALSE). Learning experiments were performed with 20 datasets from various domains. The results of significance tests verify the effectiveness of ALSE and its superiority over state-of-the-art active learning algorithms.

Introduction

Active learning [1], [2] is a subfield of machine learning in which the algorithm is able to interactively query an oracle to obtain the desired data. The objective is to train an accurate prediction model with the minimum cost by labeling the most informative instances [3]. As obtaining class labels are expensive and time consuming, it is reasonable to select instances whose labels will shrink the version space as fast as possible [4], [5], [6]. The most popular approach is to query the most informative instances, such as query-by-committee [5], uncertainty sampling [6], and optimal experimental design [7]. The main weakness of these approaches is that they are unable to exploit the abundance of unlabeled data and are prone to sample bias [8].

Clustering-based active learning [8], [9], [10] explores the clustering structure of unlabeled data and designs appropriate clustering strategies. One approach is to pre-cluster the data and query the cluster centroids to form the initial training set [11]. This is a form of “warm start” active learning, instead of beginning with a random sample. Another approach is to iteratively cluster the most informative instances and query the most representative among them [9], [12]. During the active learning process, the clustering is adjusted using the coarse-to-fine strategy. For example, Dasgupta et al. [9] constructed a hierarchical clustering tree and adopted a pruning strategy to iteratively refine the clustering. However, the clusters may not actually correspond to the hidden class labels. Moreover, it is difficult to determine the block granularity that is suitable for active learning. To explore a more appropriate cluster structure that is as consistent as possible with the class labels, we should consider the following questions: (1) How can we determine the relationship between the cluster distribution and the label errors? (2) How can we determine the most appropriate diameter for the cluster block? and (3) How can we develop a suitable critical instance selection strategy?

In this paper, we present theoretical and practical statistical methods for analyzing the relationship between the label error and the neighbor radius, and design new split and selection strategies to handle these two issues. There are four main contributions of this study. First, we propose a theoretical method for analyzing the relationship between the statistical label error and the neighbor radius. The closer any two instances are, the less likely they are to have different labels. With this intrinsic feature of the data distribution, we define two new label error statistical functions based on single instances and instance pairs. These functions are used to quantitatively calculate label inconsistencies according to the neighborhood radius.

Second, we propose a practical statistical method for obtaining the empirical label error of the cluster block. With any clustering algorithm, this method can guide the splitting of the blocks. We use the clustering by fast search and find of density peaks (CFDP) algorithm to cluster the training datasets. For each cluster block, we use statistical functions for single instances and instance pairs to calculate the label error. Finally, we use curve fitting techniques to obtain the corresponding empirical label error curves. In fact, there are some existing works about statistical active learning. For example, Gaussian statistical active learning [13], [14] explores unlabeled data spatial connectivity to build an instance subset, from which critical instances are selected. In contrast, we take advantages of clustering algorithms to form blocks, and explore the label distribution in each block to guide the splitting process.

Third, we propose a new center-and-edge instance selection strategy for choosing the critical instances. This strategy takes into account the representativeness and information of each instance. Edge instances are used to reduce the uncertainty of the model, and central instances are used to represent the overall features of all unlabeled data. The strategy is based on the principle of maximum error, thus guaranteeing the accuracy of classification.

Fourth, we design a new algorithm called active learning through label error statistical model (ALSE). Fig. 1 illustrates the ALSE process using a running example. Part 1 is the input, which contains two types of datasets: Iris ($DB<1.2$) and Sonar ($DB>1.2$). Part 2 is the theoretical and practical label error statistical methods. Theoretical label error statistical methods present the single-instance label error statistical function $e_s\left({\lambda }_s\right)$ and instance-pair statistical function $e_p\left({\lambda }_p\right)$. Practical label error statistical models provide two empirical label error functions $\phi \left({\lambda }_s\right)$ and $\phi \left({\lambda }_p\right)$ obtained using statistical method. Part 3 is an example of iterative query, split and prediction using the Iris dataset. Three different sizes of cluster sub-blocks are obtained by clustering. The diameters of the three clusters are ${\lambda }^{\prime }$, ${\lambda }^{\prime \prime }$ and ${\lambda }^{\prime \prime \prime }$, respectively. For block 1, $\phi \left({\lambda }^{\prime }\right)<\epsilon$, we select representative instances 5, 23, 41. Since $l\left(5\right)=l\left(23\right)=l\left(41\right)=1$, block 1 is pure and we will predict all remaining instances. For block 2, $\phi \left({\lambda }^{\prime \prime }\right)<\epsilon$, while judging that block 2 is impure, we need to split the block. For block 3, $\phi \left({\lambda }^{\prime \prime \prime }\right)>\epsilon$, we will split the block directly. In this way, the ALSE algorithm iteratively queries, splits, and predicts until all instances obtain the label. Part 4 is the output.

Experiments are undertaken on 20 datasets with 150–1,025,010 instances, 2–16,833 attributes, and 2–31 classes. These datasets include continuous, discrete, and mixed attribute types. We compare the ALSE algorithm with popular classifiers and state-of-the-art active learning algorithms. The Friedman test and Nemenyi post-hoc test are used to verify the significance of the differences between ALSE and the other algorithms. The results show that ALSE outperforms all of these competing algorithms in terms of classification accuracy.

The remainder of this paper is organized as follows. In Section 2, we briefly review active learning and cluster-based active learning. Section 3 presents the theoretical label error statistical methods. Section 4 illustrates the construction of practical label error statistical models. Section 5 introduces the pseudocode of the ALSE algorithm and analyzes its two key issues. Section 6 discusses the experimental results and Section 7 presents our conclusions.