A hybrid generative/discriminative method for semi-supervised classification

Abstract

Training methods for machine learning are often characterized as being generative or discriminative. We present a new co-training style algorithm which employs a generative classifier (Naive Bayes) and a discriminative classifier (Support Vector Machine) as base classifiers, to take advantage of both methods. Furthermore, we introduce a pair of weight parameters to balance the impact of labeled and pseudo-labeled data, and define a hybrid objective function to tune their values during co-training. The final prediction is given by the combination of base classifiers, and we define a pseudo-validation set to regulate their weight. Additionally, we present a strategy of pseudo-labeled data selecting to deal with the class imbalance problem. Experimental results on six datasets show that our method performs much better in practice, especially when the amount of labeled data is small.

Introduction

Most machine learning algorithms rely on the availability of enough labeled data, however, labeled data are often expensive to get in many application domains, while unlabeled data are readily available in abundance. As a consequence, semi-supervised learning (SSL), which attempts to learn from both labeled and unlabeled data, has attracted much attention in recent years.

A variety of SSL algorithms have been proposed for either generative or discriminative classifiers. In a probabilistic framework, generative classifiers model the joint probability, p(x, y), of the input x and the label y, and make predictions by using Bayes rules to calculate p(y∣x). In contrast, discriminative classifiers directly model the posterior class probabilities p(y∣x), and usually achieve better performance than generative ones. However, when the amount of labeled data is small, generative classifiers can provide better accuracy even when their models are not a very good fit to the data [1], [2], [3]. Additionally, discriminative classifiers cannot naturally incorporate unlabeled data in SSL, and the reason is that they do not model p(x) as generative ones do.

To leverage the power of both generative method and discriminative method, several hybrid methods have been proposed recent years. Most of these methods define a probabilistic hybrid objective function, in which the parameter distributions of both the generative part and the discriminative part should belong to a same family, e.g. PCP [2], or even be the same one, e.g. MCL [4]. As a result, some prominent classifiers, such as the Support Vector Machine (SVM), can hardly be incorporated into these hybrid methods.

Co-training is a popular SSL algorithm which employs two Naive Bayes (NB) as base classifiers to retrain each other iteratively [5], [6]. Due to the prominent performance of SVM [7], [8], several researchers recently replace the NBs with SVMs in co-training style algorithms [8], [9], [10]. However, the SVM is very sensitive to noise, which are inevitable among pseudo-labeled data of co-training style algorithms. Additionally, SVM performs worse when classes are strongly overlapping.

We present a new co-training style algorithm, which is named Co-NB-SVM, to flexibly combine a generative classifier (NB) and a discriminative classifier (SVM). Unlike conventional co-training algorithms, our method does not require two sufficient and redundant views (i.e., attribute sets). In order to balance the impact of labeled and pseudo-labeled data, we introduce a pair of weight parameters for all pseudo-labeled data, and this method avoids the plague of local maxima (minima) in probabilistic co-training style algorithms [6], [8], furthermore, we define a hybrid objective function to estimate the optimum values of weight parameters during co-training.

The final prediction is given by combining two classifiers, with a parameter μ to control their weights. In addition, we define a pseudo-validation set to tune the value of μ for a better balance between the generative component and discriminative component.

The class imbalance problem appears in many real-world tasks, and it hurts the performance of standard learning methods. In this paper, we also present different strategies to select pseudo-labeled data for NB and SVM respectively, to deal with the class imbalance problem. Finally, we demonstrate our framework on six text categorization problems addressed in [11].