Exploiting probabilistic topic models to improve text categorization under class imbalance

Abstract

In text categorization, it is quite often that the numbers of documents in different categories are different, i.e., the class distribution is imbalanced. We propose a unique approach to improve text categorization under class imbalance by exploiting the semantic context in text documents. Specifically, we generate new samples of rare classes (categories with relatively small amount of training data) by using global semantic information of classes represented by probabilistic topic models. In this way, the numbers of samples in different categories can become more balanced and the performance of text categorization can be improved using this transformed data set. Indeed, the proposed method is different from traditional re-sampling methods, which try to balance the number of documents in different classes by re-sampling the documents in rare classes. Such re-sampling methods can cause overfitting. Another benefit of our approach is the effective handling of noisy samples. Since all the new samples are generated by topic models, the impact of noisy samples is dramatically reduced. Finally, as demonstrated by the experimental results, the proposed methods can achieve better performance under class imbalance and is more tolerant to noisy samples.

Introduction

Text categorization/classification (Sebastiani, 2002) is a common technique for automatically organizing documents into predefined categories. While existing text categorization techniques have shown promising results in many application scenarios (e.g. Sebastiani, 2002, Ko and Seo, 2009, Paradis and Nie, 2007), text categorization techniques for handling data sets with imbalanced class distributions remain a challenging research issue.

In the case of class imbalance, the classifiers tend to ignore rare classes in favor of larger classes due to the size effect. In fact Yang and Liu (1999) compared the robustness and classification performances of several text categorization methods such as Support Vector Machine (SVM), Naive Bayes classifier, and K-Nearest Neighbor (KNN) classifier on data sets with various class distributions, and the experimental results show that all these classifiers achieve relatively low performance for rare classes. A promising direction to handle the class imbalance problem is applying re-sampling techniques. Specifically, over-sampling techniques (Japkowicz, 2000) can be used to increase the number of data instances in rare classes and under-sampling techniques can be used to reduce the number of data instances in large classes (Japkowicz & Stephen, 2002). The ultimate goal is to adjust the sizes of classes to a relatively balanced level.

Although both over sampling and under sampling can alleviate the class imbalance problem, there are some side effects. For instance, replicating samples by over sampling could result in overfitting. Also, some useful samples in large classes may be missing due to under sampling. This, in turn, will hinder the classification performance. Therefore, many modified re-sampling methods were developed to overcome the disadvantages described above. For example, the overfitting problem in random over-sampling methods can be avoided to a certain extent by bringing in random Gaussian noise to samples in rare classes or synthetically generating samples for rare classes (Chawla, Bowyer, Hall, & Kegelmeyer, 2002). Also, a better performance than random under-sampling methods can be achieved by only eliminating samples that are further away from class boundaries (Batista, Prati, & Monard, 2004).

As a matter of fact, most re-sampling techniques were developed for general types of data rather than for text. Our approach intends to exploit the semantic characteristics uniquely existing in text documents to improve text categorization under class imbalance. New samples of rare classes are generated by using global semantic information of classes represented by probabilistic topic models instead of replicating samples in rare classes. Probabilistic topic models (Blei et al., 2003, Griffiths and Steyvers, 2004) are effective tools to capture the semantic topics (more details Section 2.1). By exploiting probabilistic topic models of rare classes to generate new samples, overfitting is less likely to occur. In addition, if the original training samples are replaced by new samples generated by probabilistic topic models, the training data can be smoothed and the impact on the classification performance by noisy samples will also be alleviated.

In this paper, we propose two re-sampling methods based on probabilistic topic models: DECOM (data re-sampling with probabilistic topic models) and DECODER (data re-sampling with probabilistic topic models after smoothing). DECOM deals with class imbalance by generating new samples of rare classes using probabilistic topic models. In addition, for data sets with a high proportion of noisy samples, DECODER first smoothes data by regenerating all samples in data sets and then generates more samples for rare classes by probabilistic topic models. Experimental results on various real-world data sets show that both DECOM and DECODER can achieve better classification performances on rare classes. Also, DECODER is more robust in handling very noisy data.

Overview: The remainder of this paper is organized as follows. Section 2 introduces probabilistic topic models as well as two re-sampling methods based on probabilistic topic models. In Section 3, we present experimental results on a number of real-world document data sets. Section 4 describes some related work. Finally, we draw conclusions in Section 5.