Learning performance of LapSVM based on Markov subsampling

Abstract

It has become common to collect massive datasets in modern applications. The massive and highly noise contaminated data pose serious challenges to conventional semi-supervised learning methods. To tackle such challenges from the large-quantity-low-quality situation, we propose a distribution-free Markov subsampling strategy based on Laplacian support vector machine (LapSVM) to achieve robust and effective estimation. The core idea is to construct an informative subset which allows us to conservatively correct a rough initial estimate towards the true classifier. Specifically, the proposed subsampling strategy selects samples with small losses via a probabilistic procedure, constructing a subset which stands a good chance of excluding the noise data and providing a safe improvement over the rough initial estimate. Theoretically, we show that the obtained classifier is statistically consistent and can achieve fast learning rate under mild conditions. The promising performance is also supported by simulation studies and real data examples.

Introduction

Classification is one of the fundamental tasks in statistics and machine learning; it aims to identify the category of a new observation based on the knowledge from a training dataset containing observations with known category membership (labels). In modern scientific research, it is increasingly frequent that analysts need to deal with a huge amount of training data, where people may only label the category membership for a small partial of observations. For example, customs need to flag the illegitimate shipments from hundreds of thousands of declarations in a day; due to the limited recourses, they may only check and label a few hundred of them. Apparently, simply deleting all the unlabeled data from the training set may lead to a significant loss of data information. How to construct a good classifier utilizing both labeled and unlabeled data has been an important research problem.

In the literature, several strategies have been proposed for learning with unlabeled data. In particular, as a benchmark semi-supervised learning method, Laplacian Support Vector Machine (LapSVM) has attracted considerable attention in the recent years, see, for example, [1], [2], [3], [4]. It incorporates the manifold information of unlabeled data into classification using a similarity measure between observations. LapSVM has been demonstrated effective and efficient in the applications when the size of training set is moderate. When the amount of training data is huge, LapSVM becomes less effective or even infeasible, as solving the associated regularization problem is numerically costly. Moreover, in many applications, the big training set often comes with a complex structure and is highly noise-contaminated, as the data are not collected from a designed experiment. When the noise level is high, incorporating all the unlabeled data into the training process might deteriorate the predictive performance of the classifier due to overfitting.

To tackle the challenges from such “large-quantity-low-quality” situation, one natural idea is to first select an informative subset from the original data [5], [6], [7], and then train a classifier based on the subset. For the computational convenience and the effectiveness of training, such an informative subset should include only a moderate number of observations and can capture the essential classification information contained in the original data. Apparently, finding such a good subset is not straightforward, especially in the case of our problem setup, where the majority of observations are unlabeled and their impacts on finding the classifier are unknown beforehand.

In this paper, we propose a distribution-free Markov subsampling scheme to implement the idea of subset learning discussed above. Suppose that a training set contains $n_1$ labeled observations and $n_2$ unlabeled observations, with the total number of observations $n=n_1+n_2$ being huge and $n_1\ll n_2$. The proposed scheme first finds a pilot function $f_0$ based on all the labeled observations and a small number of randomly selected unlabeled observations. The function $f_0$ then is used to impute the category membership for any unlabeled observation with a plausible class label. Starting from a random observation in the original data, the scheme then recursively evaluates the ratio of classification risk between the current observation and the candidate observations, and accepts/rejects the candidate observations in probabilities proportional to the contribution of them towards finding a better classifier. The procedure stops when m observations are selected after a short burning period. With the proposed scheme, a set of uniformly ergodic Markov chain (u.e.M.c) observations are then generated. We will show that so obtained m observations constitute an manageable informative subset ${\mathcal{D}}_M$, which is a refined representative of the original big training set and can be more efficiently used to train an ideal classifier.

Compared with many other Markov sampling methods in the literature (see, e.g., [8], [9], [10]), the proposed scheme does not require any prior information on distribution of data. Thus, it is particularly suited for learning from complex data. Under mild conditions, we will show that the LapSVM based on the selected ${\mathcal{D}}_M$ is consistent in statistics, with an optimal ${\mathcal{O}}_p\left(m^{-1}\right)$ convergence rate to the oracle classifier. The promising performance of the proposed method is supported by both simulations and real data applications.

The rest of the paper is organized as follows. The problem is set up in Section 2. Section 3 states the proposed Markov subsampling scheme. Section 4 presents the generalization analysis results on the LapSVM with u.e.M.c observations. Section 5 then demonstrates the experimental evaluation results for the proposed Markov subsampling strategy. Finally, Section 6 summarizes the paper with some useful remarks.