Text classification is the most fundamental and essential task in natural language processing. The last decade has seen a surge of research in this area due to the unprecedented success of deep learning. Numerous methods, datasets, and evaluation metrics have been proposed in the literature, raising the need for a comprehensive and updated survey. This paper fills the gap by reviewing the state-of-the-art approaches from 1961 to 2021, focusing on models from traditional models to deep learning. We create a taxonomy for text classification according to the text involved and the models used for feature extraction and classification. We then discuss each of these categories in detail, dealing with both the technical developments and benchmark datasets that support tests of predictions. A comprehensive comparison between different techniques, as well as identifying the pros and cons of various evaluation metrics are also provided in this survey. Finally, we conclude by summarizing key implications, future research directions, and the challenges facing the research area.

1 Introduction

Text classification – the procedure of designating pre-defined labels for text – is an essential and significant task in many Natural Language Processing (NLP) applications, such as sentiment analysis [1, 2], topic labeling [3, 4], question answering [5, 6] and dialog act classification [7]. In the era of information explosion, it is time-consuming and challenging to process and classify large amounts of text data manually. Besides, the accuracy of manual text classification can be easily influenced by human factors, such as fatigue and expertise. It is desirable to use machine learning methods to automate the text classification procedure to yield more reliable and less subjective results. Moreover, this can also help enhance information retrieval efficiency and alleviate the problem of information overload by locating the required information.

Figure 1 illustrates a flowchart of the procedures involved in the text classification, under the light of traditional and deep analysis. Text data is different from numerical, image, or signal data. It requires NLP techniques to be processed carefully. The first important step is to preprocess text data for the model. Traditional models usually need to obtain good sample features by artificial methods and then classify them with classic machine learning algorithms. Therefore, the effectiveness of the method is largely restricted by feature extraction. However, different from traditional models, deep learning integrates feature engineering into the model fitting process by learning a set of nonlinear transformations that serve to map features directly to outputs.

Fig. 1.

From the 1960s until the 2010s, traditional text classification models dominated. Traditional methods mean statistics-based models, such as Naïve Bayes (NB) [8], K-Nearest Neighbor (KNN) [9], and Support Vector Machine (SVM) [10]. Comparing with the earlier rule-based methods, this method has obvious advantages in accuracy and stability. However, these approaches still need to do feature engineering, which is time-consuming and costly. Besides, they usually disregard the natural sequential structure or contextual information in textual data, making it challenging to learn the semantic information of the words. Since the 2010s, text classification has gradually changed from traditional models to deep learning models. Compared with the methods based on traditional, deep learning methods avoid designing rules and features by humans and automatically provide semantically meaningful representations for text mining. Therefore, most of the text classification research works are based on Deep Neural Networks (DNNs) [11], which are data-driven approaches with high computational complexity. Few works focus on traditional models to settle the limitations of computation and data.

1.1 Major Differences and Contributions

There have been several works reviewing text classification and its subproblems recently. Two of them are reviews of text classification. Kowsari et al. [12] surveyed different text feature extraction, dimensionality reduction methods, basic model structure for text classification, and evaluation methods. Minaee et al. [13] reviewed recent deep learning based text classification methods, benchmark datasets, and evaluation metrics. Unlike existing text classification reviews, we conclude existing models from traditional models to deep learning with works of recent years. Traditional models emphasize the feature extraction and classifier design. Once the text has well-designed characteristics, it can be quickly converged by training the classifier. DNNs can perform feature extraction automatically and learn well without domain knowledge. We then give the datasets and evaluation metrics for single-label and multi-label tasks and summarize future research challenges from data, models, and performance perspectives. Moreover, we summarize various information in three tables, including the necessary information of classic deep learning models, primary information of main datasets, and a general benchmark of state-of-the-art methods under different applications. In summary, this study’s main contributions are as follows:

•

We introduce the process and development of text classification and present comprehensive analysis and research on primary models – from traditional to deep learning models – according to their model structures. We summarize the necessary information of deep learning models in terms of basic model structures in Table 1, including publishing years, methods, venues, applications, evaluation metrics, datasets, and code links.

Table 1.

•

We introduce the present datasets and give the formulation of main evaluation metrics with the comparison of metrics, including single-label and multi-label text classification tasks. We summarize the necessary information of primary datasets in Table 2, including the number of categories, average sentence length, the size of each dataset, related papers, and data addresses.

Table 2.

Datasets	#C	#L	#N	Language	Related Papers	Sources	Applications
MR	2	20	10,662	English	[5, 17, 101, 155]	[226]	SA
SST-1	5	18	11,855	English	[17, 64] [1, 2][112]	[227]	SA
SST-2	2	19	9,613	English	[6, 17, 64] [19, 79]	[228]	SA
MPQA	2	3	10,606	English	[17, 60, 120]	[229]	SA
IMDB	2	294	50,000	English	[69][108] [6] [79] [85]	[230]	SA
Yelp.P	2	153	598,000	English	[93, 98]	[231]	SA
Yelp.F	5	155	700,000	English	[93, 98, 108]	[231]	SA
Amz.P	2	91	4,000,000	English	[93, 122]	[232]	SA
Amz.F	5	93	3,650,000	English	[93, 108, 122]	[232]	SA
Twitter	3	19	11,209	English	[5][100]	[233]	SA
NLP & CC 2013	2	-	115,606	Multi-language	[110]	[111]	SA
20NG	20	221	18,846	English	[72, 75, 106, 155, 157]	[34]	NC
AG News	4	45/7	127,600	English	[98, 100] [101, 138]	[234]	NC
R8	8	66	7,674	English	[155, 157] [159]	[235]	NC
R52	52	70	9,100	English	[155, 157] [159]	[235]	NC
Sogou	6	578	510,000	Chinese	[93]	[236]	NC
Newsgroup	20	-	18,846	English	[237]	[237]	NC
DBPedia	14	55	630,000	English	[85, 93, 98, 136]	[238]	TL
Ohsumed	23	136	7,400	English	[155, 157, 159]	[239]	TL
YahooA	10	112	1,460,000	English	[93, 108]	[93]	TL
EUR-Lex	3,956	1,239	19,314	English	[96] [134, 161] [134]	[240]	TL
Amazon670K	670	244	643,474	English	[103, 122]	[241]	TL
Google news	152	6	11,109	English	[3, 242, 243]	[242]	TL
TweetSet 2011-2012	89	-	2,472	English	[242, 243]	[242]	TL
TweetSet 2011-2015	269	8	30,322	English	[3, 4]	[4]	TL
Bing	4	20	34,871	English	[100]	[244]	TL
Fudan	20	2981	18,655	Chinese	[72]	[245]	TL
SQuAD	-	5,000	5,570	English	[118, 118, 140, 146]	[246]	QA
TREC-QA	-	1,162	68	English	[197]	[247]	QA
TREC	6	10	5,952	English	[5, 6, 17] [100]	[248]	QA
WikiQA	-	873	243	English	[197, 249]	[249]	QA
Subj	2	23	10,000	English	[17, 79, 101]	[250]	QA
CR	2	19	3,775	English	[17, 101]	[251]	QA
Reuters	90	168	10,788	English	[101, 166]	[252]	ML
Reuters10	10	168	9,979	English	[253]	[254]	ML
RCV1	103	240	807,595	English	[75, 103, 134, 153]	[255]	ML
RCV1-V2	103	124	804,414	English	[116, 166]	[256]	ML
AAPD	54	163	55,840	English	[116, 166]	[117]	ML

Table 2. Summary Statistics for the Datasets

C: Number of target classes. L: Average sentence length. N: Dataset size.

•

We summarize classification accuracy scores of models given in their articles, on benchmark datasets in Table 4 and conclude the survey by discussing the main challenges facing the text classification and key implications stemming from this study.

1.2 Organization of the Survey

The rest of the survey is organized as follows. Section 2 summarizes the existing models related to text classification, including traditional and deep learning models, including a summary table. Section 3 introduces the primary datasets with a summary table and evaluation metrics on single-label and multi-label tasks. We then give quantitative results of the leading models in classic text classification datasets in Section 4. Finally, we summarize the main challenges for deep learning text classification in Section 5 before concluding the article in Section 6.

2 Text Classification Methods

Text classification is referred to as extracting features from raw text data and predicting the categories of text data based on such features. Numerous models have been proposed in the past few decades for text classification. For traditional models, NB [8] is the first model used for the text classification task. Whereafter, generic classification models are proposed, such as KNN [9], SVM [10], and Random Forest (RF) [14], which are called classifiers, and are widely used for text classification. Recently, the eXtreme Gradient Boosting (XGBoost) [15] and the Light Gradient Boosting Machine (LightGBM) [16] have arguably the potential to provide excellent performance. For deep learning models, TextCNN [17] has the highest number of references in these models, wherein a Convolutional Neural Network (CNN) [18] model has been introduced to solve the text classification problem for the first time. While not specifically designed for handling text classification tasks, the Bidirectional Encoder Representation from Transformers (BERT) [19] has been widely employed when designing text classification models, considering its effectiveness on numerous text classification datasets.

2.1 Traditional Models

Traditional models accelerate text classification with improved accuracy and make the application scope of traditional expand. The first thing is to preprocess the raw input text for training traditional models, which generally consists of word segmentation, data cleaning, and statistics. Then, text representation aims to express preprocessed text in a form that is much easier for computers and minimizes information loss, such as Bag-Of-Words (BOW) [20], N-gram [21], Term Frequency-Inverse Document Frequency (TF-IDF) [22], word2vec [23], and Global Vectors for word representation (GloVe) [24]. BOW means that all the words in the corpus are formed into a mapping array. According to the mapping array, a sentence can be represented as a vector. The \(i\)-th element in the vector represents the frequency of the \(i\)-th word in the mapping array of the sentence. The vector is the BOW of the sentence. At the core of the BOW is representing each text with a dictionary-sized vector. The individual value of the vector denotes the word frequency corresponding to its inherent position in the text. Compared to BOW, N-gram considers the information of adjacent words and builds a dictionary by considering the adjacent words. It is used to calculate the probability model of a sentence. The probability of a sentence is expressed as the joint probability of each word in the sentence. The probability of a sentence can be calculated by predicting the probability of the \(N\)-th word, given the sequence of the \((N-1)\)-th words. To simplify the calculation, the N-gram model adopts the Markov hypothesis [21]. A word appears only concerning the words that preceded it. Therefore, the N-gram model performs a sliding window with size N. By counting and recording the occurrence frequency of all fragments, the probability of a sentence can be calculated using the frequency of relevant fragments in the record. TF-IDF [22] uses the word frequency and inverses the document frequency to model the text. TF is the word frequency of a word in a specific article, and IDF is the reciprocal of the proportion of the articles containing this word to the total number of articles in the corpus. TF-IDF is the multiplication of the two. TF-IDF assesses the importance of a word to one document in a set of files or a corpus. The importance of a word increases proportionally with the number of times it appears in a document. However, it decreases inversely with its frequency in the corpus as a whole. The word2vec [23] employs local context information to obtain word vectors, as shown in Figure 2. Word vector refers to a fixed-length real value vector specified as the word vector for any word in the corpus. The word2vec uses two essential models: CBOW and Skip-gram. The former is to predict the current word on the premise that the context of the current word is known. The latter is to predict the context when the current word is known. The GloVe [24] – with both the local context and global statistical features – trains on the nonzero elements in a word-word co-occurrence matrix, as shown in Figure 2. It enables word vectors to contain as much semantic and grammatical information as possible. The construction method of word vector is: firstly, the co-occurrence matrix of words is constructed based on the corpus, and then the word vector is learned based on the co-occurrence matrix and GloVe model. Finally, the represented text is fed into the classifier according to selected features. Here, we discuss some representative classifiers in detail.

Fig. 2.

2.1.1 PGM-based Methods.

Probabilistic Graphical Models (PGMs) express the conditional dependencies among features in graphs, such as the Bayesian network [25]. It is a combination of probability theory and graph theory.

Naïve Bayes (NB) [8] is the simplest and most broadly used model based on applying Bayes’ theorem. The NB algorithm has an independent assumption: when the target value has been given, the conditions between text \(T=[T_{1}, T_{2},\ldots , T_{n}]\) are independent (see Figure 3). The NB algorithm primarily uses the prior probability to calculate the posterior probability \(\mathrm{P}\left(y \mid \mathrm{T}_{1}, T_{2}, \ldots , T_{n}\right)=\frac{p(y) \prod _{j=1}^{n} p\left(T_{j} \mid y\right)}{\prod _{j=1}^{\mathrm{n}} p\left(T_{j}\right)}\). Due to its simple structure, NB is broadly used for text classification tasks. Although the assumption that the features are independent is sometimes not actual, it substantially simplifies the calculation process and performs better. To improve the performance on smaller categories, Schneider [26] proposes a feature selection score method through calculating KL-divergence [27] between the training set and corresponding categories for multinomial NB text classification. Dai et al. [28] propose a transfer learning method named Naive Bayes Transfer Classification (NBTC) to settle the different distribution between the training set and the target set. It uses the EM algorithm [29] to obtain a locally optimal posterior hypothesis on the target set. NB classifier is also used for fake news detection [30], and sentiment analysis [31], which can be seen as a text classification task. Bernoulli NB, Gaussian NB and Multinomial NB are three popular approaches of NB text classification [32]. Multinomial NB performs slightly better than Bernoulli NB on few labeled datasets [33]. Bayesian NB classifier with Gaussian event model [32] has been proven to be superior to NB with multinomial event model on 20 Newsgroups (20NG) [34] and WebKB [35] datasets.

Fig. 3.

2.1.2 KNN-based Methods.

At the core of the K-Nearest Neighbors (KNN) algorithm [9] is to classify an unlabeled sample by finding the category with the most samples on the \(k\) nearest samples. It is a simple classifier without building the model and can decrease complexity through the fasting process of getting \(k\) nearest neighbors. Figure 4 showcases the structure of KNN. We can find \(k\) training texts approaching a specific text to be classified through estimating the in-between distance. Hence, the text can be divided into the most common categories found in \(k\) training set texts. The improvement of KNN algorithm mainly includes feature similarity [36], \(K\) value [37] and index optimization [38]. However, due to the positive correlation between model time/space complexity and the amount of data, the KNN algorithm takes an unusually long time on the large-scale datasets [39]. To decrease the number of selected features, Soucy et al. [40] propose a KNN algorithm without feature weighting. It manages to find relevant features, building the inter-dependencies of words by using a feature selection. When the data is extremely unevenly distributed, KNN tends to classify samples with more data. The Neighbor-Weighted K-Nearest Neighbor (NWKNN) [41] is proposed to improve classification performance on the unbalanced corpora. It casts a significant weight for neighbors in a small category and a small weight for neighbors in a broad class.

Fig. 4.

2.1.3 SVM-based Methods.

Cortes and Vapnik [42] propose Support Vector Machine (SVM) to tackle the binary classification of pattern recognition. Joachims [10], for the first time, uses the SVM method for text classification representing each text as a vector. As illustrated in Figure 4, SVM-based approaches turn text classification tasks into multiple binary classification tasks. In this context, SVM constructs an optimal hyperplane in the one-dimensional input space or feature space, maximizing the distance between the hyperplane and the two categories of training sets, thereby achieving the best generalization ability. The goal is to make the distance of the category boundary along the direction perpendicular to the hyperplane to be the largest. Equivalently, this will result in the lowest error rate of classification. Constructing an optimal hyperplane can be transformed into a quadratic programming problem to obtain a globally optimal solution. Choosing the appropriate kernel function [43] and feature selection [44] are of the utmost importance to ensure SVM can deal with nonlinear problems and become a robust nonlinear classifier. Furthermore, active learning [45] and adaptive learning [46] method are used for text classification to reduce the labeling effort based on the supervised learning algorithm SVM. To analyze what the SVM algorithms learn and what tasks are suitable, Joachims [47] proposes a theoretical learning model combining the statistical traits with the generalization performance of an SVM, analyzing the features and benefits using a quantitative approach. Transductive Support Vector Machine (TSVM) [48] is proposed to lessen misclassifications of the particular test collections with a general decision function considering a specific test set. It uses prior knowledge to establish a more suitable structure and study faster.

2.1.4 DT-based Methods.

Decision Trees (DT) [49] is a supervised tree structure learning method – reflective of the idea of divide-and-conquer – and is constructed recursively. It learns disjunctive expressions and has robustness for the text with noise. As shown in Figure 5, decision trees can be generally divided into two distinct stages: tree construction and tree pruning. It starts at the root node and tests the data samples (composed of instance sets, which have several attributes), and divides the dataset into diverse subsets according to different results. A subset of datasets constitutes a child node, and every leaf node in the decision tree represents a category. Constructing the decision tree is to determine the correlation between classes and attributes, further exploited to predict the record categories of unknown forthcoming types. The classification rules generated by the decision tree algorithm are straight-forward, and the pruning strategy [50] can also help reduce the influence of noise. Its limitation, however, mainly derives from inefficiency in coping with explosively increasing data size. More specifically, the Iterative Dichotomiser 3 (ID3) [51] algorithm uses information gain as the attribute selection criterion in the selection of each node – It is used to select the attribute of each branch node, and then select the attribute having the maximum information gain value to become the discriminant attribute of the current node. Based on ID3, C4.5 [52] learns to obtain a map from attributes to classes, which effectively classifies entities unknown to new categories. DT based algorithms usually need to train for each dataset, which is low efficiency [53]. Thus, Johnson et al. [54] propose a DT-based symbolic rule system. The method represents each text as a vector calculated by the frequency of each word in the text, and induces rules from the training data. The learning rules are used for classifying the other data, being similar to the training data. Furthermore, to reduce the computational costs of DT algorithms, Fast Decision-Tree (FDT) [55] uses a two-pronged strategy: pre-selecting a feature set and training multiple DTs on different data subsets. Results from multiple DTs are combined through a data-fusion technique to resolve the cases of imbalanced classes.

Fig. 5.

2.1.5 Integration-based Methods.

Integrated algorithms aim to aggregate the results of multiple algorithms for better performance and interpretation. Conventional integrated algorithms are bootstrap aggregation, such as RF [14], boosting such as the Adaptive Boosting (AdaBoost) [56] and XGBoost [15], and stacking. The bootstrap aggregation method trains multiple classifiers without strong dependencies and then aggregates their results. For instance, RF [14] consists of multiple tree classifiers wherein all trees depend on the value of the random vector sampled independently (depicted in Figure 5). It is worth noting that each tree within the RF shares the same distribution. The generalization error of an RF relies on the strength of each tree and the relationship among trees, and will converge to a limit with the increment of tree number in the forest. In boosting based algorithms, all labeled data are trained with the same weight to initially obtain a weaker classifier [57]. The weights of the data will then be adjusted according to the former result of the classifier. The training procedure will continue by repeating such steps until the termination condition is reached. Unlike bootstrap and boosting algorithms, stacking based algorithms break down the data into \(n\) parts and use \(n\) classifiers to calculate the input data in a cascade manner – Results from the upstream classifier will feed into the downstream classifier as input. The training will terminate once a pre-defined iteration number is targeted. The integrated method can capture more features from multiple trees. However, it helps little for short text. Motivated by this, Bouaziz et al. [58] combine data enrichment – with semantics in RFs for short text classification – to overcome the deficiency of sparseness and insufficiency of contextual information. In integrated algorithms, not all classifiers learn well. It is necessary to give different weights for each classifier. To differentiate contributions of trees in a forest, Islam et al. [59] exploit the Semantics Aware Random Forest (SARF) classifier, choosing features similar to the features of the same class, for extracting features and producing the prediction values.

Summary. The parameters of NB are more diminutive, less sensitive to missing data, and the algorithm is simple. However, it assumes that features are independent of each other. When the number of features is large, or the correlation between features is significant, the performance of NB decreases. SVM can solve high-dimensional and nonlinear problems. It has a high generalization ability, but it is sensitive to missing data. KNN mainly depends on the surrounding finite adjacent samples, rather than discriminating class domain to determine the category. Thus, for the dataset to be divided with more crossover or overlap of the class domain, it is more suitable than other methods. DT is easy to understand and interpret. Given an observed model, it is easy to deduce the corresponding logical expression according to the generated decision tree. The traditional method is a type of machine learning. It learns from data, which are pre-defined features that are important to the performance of prediction values. However, feature engineering is tough work. Before training the classifier, we need to collect knowledge or experience to extract features from the original text. The traditional methods train the initial classifier based on various textual features extracted from the raw text. Toward small datasets, traditional models usually present better performance than deep learning models under the limitation of computational complexity. Therefore, some researchers have studied the design of traditional models for specific domains with less data.

2.2 Deep Learning Models

The DNNs consist of artificial neural networks that simulate the human brain to automatically learn high-level features from data, getting better results than traditional models in speech recognition, image processing, and text understanding. Input datasets should be analyzed to classify the data, such as a single-label, multi-label, unsupervised, unbalanced dataset. According to the trait of the dataset, the input word vectors are sent into the DNN for training until the termination condition is reached. The performance of the training model is verified by the downstream task, such as sentiment classification, question answering, and event prediction. We show some DNNs over the years in Table 1, including designs that are different from the corresponding basic models, evaluation metrics, and experimental datasets.

Numerous deep learning models have been proposed in the past few decades for text classification, as shown in Table 1. We tabulate primary information – including publication years, venues, applications, code links, evaluation metrics, and experiment datasets – of main deep learning models for text classification. The applications in this table include Sentiment Analysis (SA), Topic Labeling (TL), News Classification (NC), Question Answering (QA), Dialog Act Classification (DAC), Natural Language Inference (NLI), and Relation Classification (RC). The multilayer perceptron [172] and the recursive neural network [173] are the first two deep learning approaches used for the text classification task, which improve performance compared with traditional models. Then, CNNs, Recurrent Neural Networks (RNNs), and attention mechanisms are used for text classification [101, 174, 175]. Many researchers advance text classification performance for different tasks by improving CNN, RNN, and attention, or model fusion and multi-task methods. The appearance of BERT [19], which can generate contextualized word vectors, is a significant turning point in the development of text classification and other NLP technologies. Many researchers [142, 176] have studied text classification models based on BERT, which achieves better performance than the above models in multiple NLP tasks, including text classification. Besides, some researchers study text classification technology based on Graph Neural Network (GNN) [155, 177] to capture structural information in the text, which cannot be replaced by other methods. Here, we classify DNNs by structure and discuss some representative models in detail.

2.2.1 ReNN-based Methods.

Traditional models cost lots of time on design features for each task. Furthermore, in the case of deep learning, the meaning of “word vectors” is different: each input word is associated with a fixed-length vector whose values are either drawn at random or derived from a previous traditional process, thus forming a matrix \(L\) called word embedding matrix which represents the vocabulary words in a small latent semantic space, of generally 50 to 300 dimensions. The Recursive Neural Network (ReNN) [173] can automatically learn the semantics of text recursively and the syntax tree structure without feature design, as shown in Figure 6. We give an example of ReNN based models. First, each word of input text is taken as the leaf node of the model structure. Then all nodes are combined into parent nodes using a weight matrix. The weight matrix is shared across the whole model. Each parent node has the same dimension with all leaf nodes. Finally, all nodes are recursively aggregated into a parent node to represent the input text to predict the label.

Fig. 6.

ReNN-based models improve performance compared with traditional models and save on labor costs due to excluding feature designs used for different text classification tasks. The Recursive AutoEncoder (RAE) [60] is used to predict the distribution of sentiment labels for each input sentence and learn the representations of multi-word phrases. To learn compositional vector representations for each input text, the Matrix-Vector Recursive Neural Network (MV-RNN) [62] introduces a ReNN model to learn the representation of phrases and sentences. It allows that the length and type of input texts are inconsistent. MV-RNN allocates a matrix and a vector for each node on the constructed parse tree. Furthermore, the Recursive Neural Tensor Network (RNTN) [64] is proposed with a tree structure to capture the semantics of sentences. It inputs phrases with different lengths and represents the phrases by parse trees and word vectors. The vectors of higher nodes on the parse tree are estimated by the equal tensor-based composition function. For RNTN, the time complexity of building the textual tree is high, and expressing the relationship between documents is complicated within a tree structure. The performance is usually improved, with the depth being increased for DNNs. Therefore, Irsoy et al. [66] propose a Deep Recursive Neural Network (DeepReNN), which stacks multiple recursive layers. It is built by binary parse trees and learns distinct perspectives of compositionality in language.

2.2.2 MLP-based Methods.

A MultiLayer Perceptron (MLP) [172], sometimes colloquially called “vanilla” neural network, is a simple neural network structure that is used for capturing features automatically. As shown in Figure 6, we show a three-layer MLP model. It contains an input layer, a hidden layer with an activation function in all nodes, and an output layer. Each node connects with a certain weight \(w_{i}\). It treats each input text as a bag of words and achieves high performance on many text classification benchmarks comparing with traditional models.

There are some MLP-based methods proposed by some research groups for text classification tasks. The Paragraph Vector (Paragraph-Vec) [67] is the most popular and widely used method, which is similar to the Continuous Bag-Of-Words (CBOW) [23]. It gets fixed-length feature representations of texts with various input lengths by employing unsupervised algorithms. Comparing with CBOW, it adds a paragraph token mapped to the paragraph vector by a matrix. The model predicts the fourth word by the connection or average of this vector to the three contexts of the word. Paragraph vectors can be used as a memory for paragraph themes and are used as a paragraph function and inserted into the prediction classifier.

2.2.3 RNN-based Methods.

The Recurrent Neural Network (RNN) [173] is broadly used for capturing long-range dependency through recurrent computation. The RNN language model learns historical information, considering the location information among all words suitable for text classification tasks. We show an RNN model for text classification with a simple sample, as shown in Figure 7. Firstly, each input word is represented by a specific vector using a word embedding technology. Then, the embedding word vectors are fed into RNN cells one by one. The output of RNN cells are the same dimension with the input vector and are fed into the next hidden layer. The RNN shares parameters across different parts of the model and has the same weights of each input word. Finally, the label of input text can be predicted by the last output of the hidden layer.

Fig. 7.

To diminish the time complexity of the model and capture contextual information, Liu et al. [79] introduce a model for catching the semantics of long texts. It is a biased model that parsed the text one by one, making the following inputs profit over the former and decreasing the semantic efficiency of capturing the whole text. For modeling topic labeling tasks with long input sequences, TopicRNN [83] is proposed. It captures the dependencies of words in a document via latent topics and uses RNNs to capture local dependencies and latent topic models for capturing global semantic dependencies. Virtual Adversarial Training (VAT) [178] is a useful regularization method applicable to semi-supervised learning tasks. Miyato et al. [85] apply adversarial and virtual adversarial training text and employ the perturbation into word embedding rather than the original input text. The model improves the quality of the word embedding and is not easy to overfit during training. Capsule network [179] captures the relationships between features using dynamic routing between capsules comprised of a group of neurons in a layer. Wang et al. [87] propose an RNN-Capsule model with a simple capsule structure for the sentiment classification task.

In the backpropagation process of RNN, the weights are adjusted by gradients, calculated by continuous multiplications of derivatives. If the derivatives are extremely small, it may cause a gradient vanishing problem by continuous multiplications. Long Short-Term Memory (LSTM) [180], the improvement of RNN, effectively alleviates the gradient vanishing problem. It is composed of a cell to remember values on arbitrary time intervals and three gate structures to control information flow. The gate structures include input gates, forget gates, and output gates. The LSTM classification method can better capture the connection among context feature words, and use the forgotten gate structure to filter useless information, which is conducive to improving the total capturing ability of the classifier. Tree-LSTM [1] extends the sequence of LSTM models to the tree structure. The whole subtree with little influence on the result can be forgotten through the LSTM forgetting gate mechanism for the Tree-LSTM model.

Natural Language Inference (NLI) [181] predicts whether one text’s meaning can be deduced from another by measuring the semantic similarity between each pair of sentences. To consider other granular matchings and matchings in the reverse direction, Wang et al. [182] propose a model for the NLI task named Bilateral Multi-Perspective Matching (BiMPM). It encodes input sentences by the BiLSTM encoder. Then, the encoded sentences are matched in two directions. The results are aggregated in a fixed-length matching vector by another BiLSTM layer. Finally, the result is evaluated by a fully connected layer.

2.2.4 CNN-based Methods.

Convolutional Neural Networks (CNNs) [18] are proposed for image classification with convolving filters that can extract features of pictures. Unlike RNN, CNN can simultaneously apply convolutions defined by different kernels to multiple chunks of a sequence. Therefore, CNNs are used for many NLP tasks, including text classification. For text classification, the text requires being represented as a vector similar to the image representation, and text features can be filtered from multiple angles, as shown in Figure 7. Firstly, the word vectors of the input text are spliced into a matrix. The matrix is then fed into the convolutional layer, which contains several filters with different dimensions. Finally, the result of the convolutional layer goes through the pooling layer and concatenates the pooling result to obtain the final vector representation of the text. The category is predicted by the final vector.

To try using CNN for the text classification task, an unbiased model of convolutional neural networks is introduced by Kim, called TextCNN [17]. It can better determine discriminative phrases in the max-pooling layer with one layer of convolution and learn hyperparameters except for word vectors by keeping word vectors static. Training only on labeled data is not enough for data-driven deep models. Therefore, some researchers consider utilizing unlabeled data. Johnson et al. [183] propose a CNN model based on two-view semi-supervised learning for text classification, which first uses unlabeled data to train the embedding of text regions and then labeled data. DNNs usually have better performance, but it increases the computational complexity. Motivated by this, a Deep Pyramid Convolutional Neural Network (DPCNN) [98] is proposed, with a little more computational accuracy, increasing by raising the network depth. The DPCNN is more specific than Residual Network (ResNet) [184], as all the shortcuts are exactly simple identity mappings without any complication for dimension matching.

According to the minimum embedding unit of text, embedding methods are divided into character-level, word-level, and sentence-level embedding. Character-level embeddings can settle Out-Of-Vocabulary (OOV) [185] words. Word-level embeddings learn the syntax and semantics of the words. Moreover, sentence-level embedding can capture relationships among sentences. Motivated by these, Nguyen et al. [186] propose a deep learning method based on a dictionary, increasing information for word-level embeddings through constructing semantic rules and deep CNN for character-level embeddings. Adams et al. [187] propose a character-level CNN model, called MGTC, to classify multi-lingual texts written. TransCap [188] is proposed to encapsulate the sentence-level semantic representations into semantic capsules and transfer document-level knowledge.

RNN based models capture the sequential information to learn the dependency among input words, and CNN based models extract the relevant features from the convolution kernels. Thus some works study the fusion of the two methods. BLSTM-2DCNN [77] integrates a Bidirectional LSTM (BiLSTM) with two-dimensional max pooling. It uses a 2D convolution to sample more meaningful information of the matrix and understands the context better through BiLSTM. Moreover, Xue et al. [189] propose MTNA, a combination of BiLSTM and CNN layers, to solve aspect category classification and aspect term extraction tasks.

2.2.5 Attention-based Methods.

CNN and RNN provide excellent results on tasks related to text classification. However, these models are not intuitive enough for poor interpretability, especially in classification errors, which cannot be explained due to the non-readability of hidden data. The attention-based methods are successfully used in the text classification. Bahdanau et al. [190] first propose an attention mechanism that can be used in machine translation. Motivated by this, Yang et al. [108] introduce the Hierarchical Attention Network (HAN) to gain better visualization by employing the extremely informational components of a text, as shown in Figure 8. HAN includes two encoders and two levels of attention layers. The attention mechanism lets the model pay different attention to specific inputs. It aggregates essential words into sentence vectors firstly and then aggregates vital sentence vectors into text vectors. The attention-based models can learn how much contribution of each word and sentence for the classification judgment, which is beneficial for applications and analysis through the two levels of attention.

Fig. 8.

The attention mechanism can improve the performance with interpretability for text classification, which makes it popular. There are some other works based on attention. LSTMN [112] is proposed to process text step by step from left to right and does superficial reasoning through memory and attention. BI-Attention [110] is designed for cross-lingual text classification to catch bilingual long-distance dependencies. Hu et al. [191] propose an attention mechanism based on category attributes for solving the imbalance of the number of various charges which contain few-shot charges. HAPN [124] is presented for few-shot text classification.

Self-attention [192] captures the weight distribution of words in sentences by constructing K, Q, and V matrices among sentences that can capture long-range dependencies on text classification. We give an example for self-attention, as shown in Figure 9. Each input word vector \(a_{i}\) can be represented as three n-dimensional vectors, including \(q_{i}\), \(k_{i,}\) and \(v_{i}\). After self-attention, the output vector \(b_{i}\) can be represented as \(\sum _{j}softmax(a_{ij})v_{j}\) and \(a_{ij}=q_{i} \cdot k_{j} / \sqrt {n}\). All output vectors can be parallelly computed. Lin et al. [114] used source token self-attention to explore the weight of every token to the entire sentence in the sentence representation task. To capture long-range dependencies, Bi-directional Block Self-Attention Network (Bi-BloSAN) [120] uses an intra-block Self-Attention Network (SAN) to every block split by sequence and an inter-block SAN to the outputs.

Fig. 9.

Aspect-Based Sentiment Analysis (ABSA) [31, 193] breaks down a text into multiple aspects and allocates each aspect a sentiment polarity. The sentiment polarity can be divided into three types: positive, neutral, and negative. Some attention-based models are proposed to identify the fine-grained opinion polarity towards a specific aspect for aspect-based sentiment tasks. ATAE-LSTM [194] can concentrate on different parts of each sentence according to the input through the attention mechanisms. MGAN [195] presents a fine-grained attention mechanism with a coarse-grained attention mechanism to learn the word-level interaction between context and aspect.

To catch the complicated semantic relationship among each question and candidate answers for the QA task, Tan et al. [196] introduce CNN and RNN and generate answer embeddings by using a simple one-way attention mechanism affected through the question context. The attention captures the dependence among the embeddings of questions and answers. Extractive QA can be seen as the text classification task. It inputs a question and multiple candidates answers and classifies every candidate answer to recognize the correct answer. Furthermore, AP-BILSTM [197] with a two-way attention mechanism can learn the weights between the question and each candidate answer to obtain the importance of each candidate answer to the question.

2.2.6 Pre-trained Methods.

Pre-trained language models [198] effectively learn global semantic representation and significantly boost NLP tasks, including text classification. It generally uses unsupervised methods to mine semantic knowledge automatically and then constructs pre-training targets so that machines can learn to understand semantics.

As shown in Figure 10, we give differences in the model architectures among the Embedding from Language Model (ELMo) [118], OpenAI GPT [199], and BERT [19]. ELMo [118] is a deep contextualized word representation model, which is readily integrated into models. It can model complicated characteristics of words and learn different representations for various linguistic contexts. It learns each word embedding according to the context words with the bi-directional LSTM. GPT [199] employs supervised fine-tuning and unsupervised pre-training to learn general representations that transfer with limited adaptation to many NLP tasks. Furthermore, the domain of the target dataset does not need to be similar to the domain of unlabeled datasets. The training procedure of the GPT algorithm usually includes two stages. Firstly, the initial parameters of a neural network model are learned by a modeling objective on the unlabeled dataset. We can then employ the corresponding supervised objective to accommodate these parameters for the target task. To pre-train deep bidirectional representations from the unlabeled text through joint conditioning on both left and right context in every layer, the BERT model [19], proposed by Google, significantly improves performance on NLP tasks, including text classification. BERT applies the bi-directional encoder designed to pre-train the bi-directional representation of depth by jointly adjusting the context in all layers. It can utilize contextual information when predicting which words are masked. It is fine-tuned by adding just an additional output layer to construct models for multiple NLP tasks, such as SA, QA, and machine translation. Comparing with these three models, ELMo is a feature-based method using LSTM, and BERT and OpenAI GPT are fine-tuning approaches using Transformer. Furthermore, ELMo and BERT are bidirectional training models and OpenAI GPT is training from left to right. Therefore, BERT gets a better result, which combines the advantages of ELMo and OpenAI GPT.

Fig. 10.

Transformer-based models can parallelize computation without considering the sequential information suitable for large scale datasets, making it popular for NLP tasks. Thus, some other works are used for text classification tasks and get excellent performance. RoBERTa [140], is an improved version of BERT, and adopts the dynamic masking method that generates the masking pattern every time with a sequence to be fed into the model. It uses more data for longer pre-training and estimates the influence of various essential hyperparameters and the size of training data. To be specific: (1) The training time is longer (a total of nearly 200,000 training set, nearly 1.6 billion training data have been seen), the batch size (8K) is larger, and the training data is more (30G Chinese training, including 300 million sentences and 10 billion words); (2) It removes the next sentence prediction (NSP) task; (3) It employs more extended training sequence; 4) It dynamically adjusts the masking mechanism and use the full word mask.

XLNet [138] is a generalized autoregressive pre-training approach. Unlike BERT, the denoising autoencoder with the mask is not used in the first stage, but the autoregressive LM is used. It maximizes the expected likelihood across the whole factorization order permutations to learn the bidirectional context. Furthermore, it can overcome the weaknesses of BERT by an autoregressive formulation and integrate ideas from Transformer-XL [200] into pre-training.

The BERT model has many parameters. In order to reduce the parameters, ALBERT [146] uses two-parameter simplification schemes. It reduces the fragmentation vector’s length and shares parameters with all encoders. It also replaces the next sentence matching task with the next sentence order task and continuously blocks fragmentation. When the ALBERT model is pre-trained on a massive Chinese corpus, the parameters are fewer than BERT and also have better performance. In general, these methods adopt unsupervised objective functions for pre-training, including the next sentence prediction, masking technology, and permutation. These target functions based on the word prediction demonstrate a strong ability to learn the word dependence and semantic structure [201].

BART [202] is a denoising autoencoder based on the Seq2Seq model, as shown in Figure 11(a). The pre-training of BART consists of two steps. Firstly, it uses a noise function to destroy the text. Secondly, the Seq2Seq model is used to reconstruct the original text. In various noise methods, the model obtains optimal performance by randomly shuffling the order of the original sentence and then using the first new text filling method. The new text filling method is replacing the text fragment with a single mask token. It uses only a specific masked token to indicate that a token is masked.

Fig. 11.

SpanBERT [203] is specially designed to better represent and predict spans of text, as shown in Figure 11(b). It optimizes BERT from three aspects and achieves good results in multiple tasks such as QA. The specific optimization is embodied in three aspects. Firstly, the span mask scheme is proposed to mask a continuous paragraph of text randomly. Secondly, Span Boundary Objective (SBO) is added to predict span by the token next to the span boundary to get the better performance to finetune stage. Thirdly, the NSP pre-training task is removed.

ERNIE [204] is based on the method of knowledge enhancement. It learns the semantic relations in the real world by modeling the prior semantic knowledge such as entity concepts in massive datasets. Specifically, ERNIE enables the model to learn the semantic representation of complete concepts by masking semantic units such as words and entities. It mainly consists of a Transformer encoder and task embedding. In the Transformer encoder, the context information of each token is captured by the self-attention mechanism, and the context representation is generated for embedding. Task embedding is used for tasks with different characteristics.

2.2.7 GNN-based Methods.

The DNN models like CNN get great performance on regular structure, but not for arbitrarily structured graphs. Some researchers study how to expand on arbitrarily structured graphs [205, 206]. With the increasing attention of Graph Neural Networks (GNNs), GNN-based models [207, 208] obtain excellent performance by encoding syntactic structure of sentences on semantic role labeling task [209], relation classification task [210], and machine translation task [211]. It turns text classification into a graph node classification task. We show a GCN model for text classification with four input texts, as shown in Figure 12. Firstly, the four input texts \(T=[T_{1}, T_{2}, T_{3}, T_{4}]\) and the words \(X=[x_{1}, x_{2}, x_{3}, x_{4}, x_{5}, x_{6}]\) in the text, defined as nodes, are constructed into the graph structures. The graph nodes are connected by bold black edges, which indicates document-word edges and word-word edges. The weight of each word-word edge usually means their co-occurrence frequency in the corpus. Then, the words and texts are represented through the hidden layer. Finally, the label of all input texts can be predicted by the graph.

Fig. 12.

The GNN-based models can learn the syntactic structure of sentences, making some researchers study using GNN for text classification. DGCNN [153] is a graph-CNN converting text to graph-of-words, having the advantage of learning different levels of semantics with CNN models. Yao et al. [155] propose the Text Graph Convolutional Network (TextGCN), which builds a heterogeneous word text graph for a whole dataset and captures global word co-occurrence information. To enable GNN-based models to underpin online testing, Huang et al. [159] build graphs for each text with global parameter sharing, not a corpus-level graph structure, to help preserve global information and reduce the burden. TextING [162] builds individual graphs for each document and learns text-level word interactions by GNN to effectively produce embeddings for obscure words in the new text.

Graph ATtention network (GAT) [212] employs masked self-attention layers by attending over its neighbors. Thus, some GAT-based models are proposed to compute the hidden representations of each node. The Heterogeneous Graph ATtention networks (HGAT) [213] with a dual-level attention mechanism learns the importance of different neighboring nodes and node types in the current node. The model propagates information on the graph and captures the relations to address the semantic sparsity for semi-supervised short text classification. MAGNET [166] is proposed to capture the correlation among the labels based on GATs, which learns the crucial dependencies between the labels and generates classifiers by a feature matrix and a correlation matrix.

Event Prediction (EP) can be divided into generated event prediction and selective event prediction (also known as script event prediction). EP, referring to scripted event prediction in this review, infers the subsequent event according to the existing event context. Unlike other text classification tasks, texts in EP are composed of a series of sequential subevents. Extracting features of the relationship among such subevents is of critical importance. SGNN [214] is proposed to model event interactions and learn better event representations by constructing an event graph to utilize the event network information better. The model makes full use of dense event connections for the EP task.

2.2.8 Others.

In addition to all the above models, there are some other individual models. Here we introduce some exciting models.

Siamese Neural Network. The siamese neural network [215] is also called a twin neural network (Twin NN). It utilizes equal weights while working in tandem using two distinct input vectors to calculate comparable output vectors. Mueller et al. [216] present a siamese adaptation of the LSTM network comprised of couples of variable-length sequences. The model is employed to estimate the semantic similarity among texts, exceeding carefully handcrafted features and proposed neural network models of higher complexity. The model further represents text employing neural networks whose inputs are word vectors learned separately from a vast dataset. To settle unbalanced data classification in the medical domain, Jayadeva et al. [217] use a Twin NN model to learn from enormous unbalanced corpora. The objective functions achieve the Twin SVM approach with non-parallel decision boundaries for the corresponding classes, and decrease the Twin NN complexity, optimizing the feature map to better discriminate among classes.

Virtual Adversarial Training (VAT) . Deep learning methods require many extra hyperparameters, which increase the computational complexity. VAT [218], regularization based on local distributional smoothness can be used in semi-supervised tasks, requires only some hyperparameters, and can be interpreted directly as robust optimization. Miyato et al. [85] use VAT to effectively improve the robustness and generalization ability of the model and word embedding performance.

Reinforcement Learning (RL) . RL learns the best action in a given environment through maximizing cumulative rewards. Zhang et al. [219] offer an RL approach to establish structured sentence representations via learning the structures related to tasks. The model has Information Distilled LSTM (ID-LSTM) and Hierarchical Structured LSTM (HS-LSTM) representation models. The ID-LSTM learns the sentence representation by choosing essential words relevant to tasks, and the HS-LSTM is a two-level LSTM for modeling sentence representation.

Memory Networks. Memory networks [220] learn to combine the inference components and the long-term memory component. Li et al. [221] use two LSTMs with extended memories and neural memory operations for jointly handling the extraction tasks of aspects and opinions via memory interactions. Topic Memory Networks (TMN) [169] is an end-to-end model that encodes latent topic representations indicative of class labels.

QA Style for Sentiment Classification Task. It is an interesting attempt to treat the sentiment classification task as a QA task. Shen et al. [222] create a high-quality annotated corpus. A three-stage hierarchical matching network was proposed to consider the matching information between questions and answers.

External Commonsense Knowledge. Due to the insufficient information of the event itself to distinguish the event for the EP task, Ding et al. [223] consider that the event extracted from the original text lacked common knowledge, such as the intention and emotion of the event participants. The model improves the effect of stock prediction, EP, and so on.

Quantum Language Model. In the quantum language model, the words and dependencies among words are represented through fundamental quantum events. Zhang et al. [224] design a quantum-inspired sentiment representation method to learn both the semantic and the sentiment information of subjective text. By inputting density matrices to the embedding layer, the performance of the model improves.

Summary. RNN computes sequentially and cannot be calculated in parallel. The shortcoming of RNN makes it more challenging to become mainstream in the current trend that models tend to have deeper and more parameters. CNN extracts features from text vectors through the convolution kernel. The number of features captured by the convolution kernel is related to its size. CNN is deep enough that, in theory, it can capture features at long distances. Due to insufficient optimization methods for parameters of the deep network and the loss of location information due to the pooling layer, the deeper layer does not bring significant improvement. Compared with RNN, CNN has parallel computing capability and can effectively retain location information for the improved version of CNN. Still, it has weak feature capture capability for long-distance. GNN builds a graph for text. When a valid graph structure is designed, the learned representation can better capture the structural information. Transformer treats the input text as a fully connected graph, with attention score weights on the edges. It is capable of parallel computing and is highly efficient in extracting features between different words by self-attention, solving short-term memory problems. However, the attention mechanism in Transformer is computation-heavy, especially when dealing with long sequences. Some improved models [146, 225] for computing complexity in Transformer have recently been proposed. Overall, Transformer is a better choice for text classification. Deep Learning consists of multiple hidden layers in a neural network with a higher level of complexity and can be trained on unstructured data. Deep learning can learn language features and master higher level and more abstract language features based on words and vectors. Deep learning architecture can learn feature representations directly from the input without too many manual interventions and prior knowledge. However, deep learning technology is a data-driven method that requires enormous data to achieve high performance. Although self-attention based models can bring some interpretability among words for DNNs, it is not enough comparing with traditional models to explain why and how it works well.

3 Datasets and Evaluation Metrics

3.1 Datasets

The availability of labeled datasets for text classification has become the main driving force behind the fast advancement of this research field. In this section, we summarize the characteristics of these datasets in terms of domains and give an overview in Table 2, including the number of categories, average sentence length, the size of each dataset, related papers, data sources to access, and applications.

3.1.1 Sentiment Analysis (SA).

SA is the process of analyzing and reasoning the subjective text within emotional color. It is crucial to get information on whether it supports a particular point of view from the text that is distinct from the traditional text classification that analyzes the objective content of the text. SA can be binary or multi-class. Binary SA is to divide the text into two categories, including positive and negative. Multi-class SA classifies text to multi-level or fine-grained labels. The SA datasets include Movie Review (MR) [226, 257], Stanford Sentiment Treebank (SST) [227], Multi-Perspective Question Answering (MPQA) [229, 258], IMDB [230], Yelp [231], Amazon Reviews (AM) [93], NLP&CC 2013 [111], Subj [250], CR [251], SS-Twitter [259], SS-Youtube [259], SE1604 [260], and so on. Here we detail several datasets.

MR. The MR is a movie review dataset, each of which corresponds to a sentence. The corpus has 5,331 positive data and 5,331 negative data. 10-fold cross-validation by random splitting is commonly used to test MR.

SST. The SST is an extension of MR. It has two categories. SST-1 with fine-grained labels with five classes. It has 8,544 training texts and 2,210 test texts, respectively. Furthermore, SST-2 has 9,613 texts with binary labels being partitioned into 6,920 training texts, 872 development texts, and 1,821 testing texts.

MPQA. The MPQA is an opinion dataset. It has two class labels and also an MPQA dataset of opinion polarity detection sub-tasks. MPQA includes 10,606 sentences extracted from news articles from various news sources. It should be noted that it contains 3,311 positive texts and 7,293 negative texts without labels of each text.

IMDB reviews. The IMDB review is developed for binary sentiment classification of film reviews with the same amount in each class. It can be separated into training and test groups on average, by 25,000 comments per group.

Yelp reviews. The Yelp review is summarized from the Yelp Dataset Challenges in 2013, 2014, and 2015. This dataset has two categories. Yelp-2 of these were used for negative and positive emotion classification tasks, including 560,000 training texts and 38,000 test texts. Yelp-5 is used to detect fine-grained affective labels with 650,000 training and 50,000 test texts in all classes.

AM. The AM is a popular corpus formed by collecting Amazon website product reviews [232]. This dataset has two categories. The Amazon-2 with two classes includes 3,600,000 training sets and 400,000 testing sets. Amazon-5, with five classes, includes 3,000,000 and 650,000 comments for training and testing.

3.1.2 News Classification (NC).

News content is one of the most crucial information sources which has a critical influence on people. The NC system facilitates users to get vital knowledge in real-time. News classification applications mainly encompass recognizing news topics and recommending related news according to user interest. The news classification datasets include 20 Newsgroups (20NG) [34], AG News (AG) [93, 234], R8 [235], R52 [235], Sogou News (Sogou) [136], and so on. Here we detail several datasets.

20NG. The 20NG is a newsgroup text dataset. It has 20 categories with the same number of each category and includes 18,846 texts.

AG. The AG News is a search engine for news from academia, choosing the four largest classes. It uses the title and description fields of each news. AG contains 120,000 texts for training and 7,600 texts for testing.

R8 and R52. R8 and R52 are two subsets which are the subset of Reuters [252]. R8 has 8 categories, divided into 2,189 test files and 5,485 training courses. R52 has 52 categories, split into 6,532 training files and 2,568 test files.

Sogou. The Sogou combines two datasets, including SogouCA and SogouCS news sets. The label of each text is the domain name in the URL.

3.1.3 Topic Labeling (TL).

The topic analysis attempts to get the meaning of the text by defining the sophisticated text theme. The topic labeling is one of the essential components of the topic analysis technique, intending to assign one or more subjects for each document to simplify the topic analysis. The topic labeling datasets include DBPedia [238], Ohsumed [239], Yahoo answers (YahooA) [93], EUR-Lex [240], Amazon670K [241], Bing [244], Fudan [245], and PubMed [261]. Here we detail several datasets.

DBpedia. The DBpedia is a large-scale multi-lingual knowledge base generated using Wikipedia’s most ordinarily used infoboxes. It publishes DBpedia each month, adding or deleting classes and properties in every version. DBpedia’s most prevalent version has 14 classes and is divided into 560,000 training data and 70,000 test data.

Ohsumed. The Ohsumed belongs to the MEDLINE database. It includes 7,400 texts and has 23 cardiovascular disease categories. All texts are medical abstracts and are labeled into one or more classes.

YahooA. The YahooA is a topic labeling task with 10 classes. It includes 140,000 training data and 5,000 test data. All texts contains three elements, being question titles, question contexts, and best answers, respectively.

3.1.4 Question Answering (QA).

The QA task can be divided into two types: the extractive QA and the generative QA. The extractive QA gives multiple candidate answers for each question to choose which one is the right answer. Thus, the text classification models can be used for the extractive QA task. The QA discussed in this paper is all extractive QA. The QA system can apply the text classification model to recognize the correct answer and set others as candidates. The question answering datasets include Stanford Question Answering Dataset (SQuAD) [246], TREC-QA [248], WikiQA [249], Subj [250], CR [251], MS MARCO [262], and Quora [263]. Here we detail several datasets.

SQuAD. The SQuAD is a set of question and answer pairs obtained from Wikipedia articles. The SQuAD has two categories. SQuAD1.1 contains 536 pairs of 107,785 Q&A items. SQuAD2.0 combines 100,000 questions in SQuAD1.1 with more than 50,000 unanswerable questions that crowd workers face in a form similar to answerable questions [264].

TREC-QA. The TREC-QA includes 5,452 training texts and 500 testing texts. It has two versions. TREC-6 contains 6 categories, and TREC-50 has 50 categories.

WikiQA. The WikiQA dataset includes questions with no correct answer, which needs to evaluate the answer.

MS MARCO. The MS MARCO contains questions and answers. The questions and part of the answers are sampled from actual web texts by the Bing search engine. Others are generative. It is used for developing generative QA systems released by Microsoft.

3.1.5 Natural Language Inference (NLI).

NLI is used to predict whether the meaning of one text can be deduced from another. Paraphrasing is a generalized form of NLI. It uses the task of measuring the semantic similarity of sentence pairs to decide whether one sentence is the interpretation of another. The NLI datasets include Stanford Natural Language Inference (SNLI) [181], Multi-Genre Natural Language Inference (MNLI) [265], Sentences Involving Compositional Knowledge (SICK) [266], Microsoft Research Paraphrase (MSRP) [267], Semantic Textual Similarity (STS) [268], Recognising Textual Entailment (RTE) [269], SciTail [270], etc. Here we detail several of the primary datasets.

SNLI. The SNLI is generally applied to NLI tasks. It contains 570,152 human-annotated sentence pairs, including training, development, and test sets, which are annotated with three categories: neutral, entailment, and contradiction.

MNLI. The MNLI is an expansion of SNLI, embracing a broader scope of written and spoken text genres. It includes 433,000 sentence pairs annotated by textual entailment labels.

SICK. The SICK contains almost 10,000 English sentence pairs. It consists of neutral, entailment, and contradictory labels.

MSRP. The MSRP consists of sentence pairs, usually for the text-similarity task. Each pair is annotated by a binary label to discriminate whether they are paraphrases. It respectively includes 1,725 training and 4,076 test sets.

3.1.6 Multi-Label (ML) Datasets.

In multi-label classification, an instance has multiple labels, and each label can only take one of the multiple classes. There are many datasets based on multi-label text classification. It includes Reuters [252], Reuters Corpus Volume I (RCV1) [255], RCV1-2K [255], Arxiv Academic Paper Dataset (AAPD) [117], Patent, Web of Science (WOS-11967) [271], AmazonCat-13K [272], BlurbGenreCollection (BGC) [273], etc. Here we detail several datasets.

Reuters. The Reuters is a popularly used dataset for text classification from Reuters financial news services. It has 90 training classes, 7,769 training texts, and 3,019 testing texts, containing multiple labels and single labels. There are also some Reuters sub-sets of data, such as R8, BR52, RCV1, and RCV1-v2.

RCV1 and RCV1-2K. The RCV1 is collected from Reuters News articles from 1996-1997, which is human-labeled with 103 categories. It consists of 23,149 training and 784,446 testing texts, respectively. The RCV1-2K dataset has the same features as the RCV1. However, the label set of RCV1-2K has been expanded with some new labels. It contains 2,456 labels.

AAPD. The AAPD is a large dataset in the computer science field for the multi-label text classification from website.¹ It has 55,840 papers, including the abstract and the corresponding subjects with 54 labels in total. The aim is to predict the corresponding subjects of each paper according to the abstract.

Patent Dataset. The Patent Dataset is obtained from USPTO,² which is a patent system grating U.S. patents containing textual details such title and abstract. It contains 100,000 US patents awarded in the real-world with multiple hierarchical categories.

WOS-11967. The WOS-11967 is crawled from the Web of Science, consisting of abstracts of published papers with two labels for each example. It is more shallow, but significantly broader, with fewer classes in total.

3.1.7 Others.

There are some datasets for other applications, such as SemEval-2010 Task 8 [274], ACE 2003-2004 [275], TACRED [276], and NYT-10 [277], FewRel [278], Dialog State Tracking Challenge 4 (DSTC 4) [279], ICSI Meeting Recorder Dialog Act (MRDA) [280], and Switchboard Dialog Act (SwDA) [281], and so on.

3.2 Evaluation Metrics

In terms of evaluating text classification models, accuracy and F1 score are the most used to assess the text classification methods. Later, with the increasing difficulty of classification tasks or the existence of some particular tasks, the evaluation metrics are improved. For example, evaluation metrics such as \(P@K\) and \(Micro\!-\!\!F1\) are used to evaluate multi-label text classification performance, and MRR is usually used to estimate the performance of QA tasks. In Table 3, we give the notations used in evaluation metrics.

Table 3.

Notations	Descriptions
\(TP\)	true positive
\(FP\)	false positive
\(TN\)	true negative
\(FN\)	false negative
\(TP_{t}\)	true positive of the \(t\) th label on a text
\(FP_{t}\)	false positive of the \(t\) th label on a text
\(TN_{t}\)	true negative of the \(t\) th label on a text
\(FN_{t}\)	false negative of the \(t\) th label on a text
\(\mathcal {S}\)	label set of all samples
\({Q}\)	the number of predicted labels on each text

Table 3. The Notations Used in Evaluation Metrics

3.2.1 Single-label Metrics.

Single-label text classification divides the text into one of the most likely categories applied in NLP tasks such as QA, SA, and dialogue systems [7]. For single-label text classification, one text belongs to just one catalog, making it possible not to consider the relations among labels. Here, we introduce some evaluation metrics used for single-label text classification tasks.

Accuracy and ErrorRate. The Accuracy and ErrorRate are the fundamental metrics for a text classification model. The \(Accuracy\) and \(ErrorRate\) are respectively defined as

\begin{equation} Accuracy =\frac{(TP+TN)}{N}, \end{equation}

(1)

\begin{equation} \quad ErrorRate = 1 - Accuracy =\frac{(FP+FN)}{N}. \end{equation}

(2)

Precision, Recall and F1. These are vital metrics utilized for unbalanced test sets, regardless of the standard type and error rate. For example, most of the test samples have a class label. \(F1\) is the harmonic average of \(Precision\) and \(Recall\). \(Precision\), \(Recall\), and \(F1\) as defined

\begin{equation} Precision =\frac{TP}{TP+FP}, \quad Recall =\frac{TP}{TP+FN}, \end{equation}

(3)

\begin{equation} F1 =\frac{2 Precision \times Recall }{ Precision +Recall}. \end{equation}

(4)

The desired results will be obtained when the accuracy, \(F1\) and \(Recall\) value reach 1. On the contrary, when the values become 0, the worst result is obtained. For the multi-class classification problem, the precision and recall value of each class can be calculated separately, and then the performance of the individual and whole can be analyzed.

Exact Match (EM). The EM [29] is a metric for QA tasks, measuring the prediction that matches all the ground-truth answers precisely. It is the primary metric utilized on the SQuAD dataset.

Mean Reciprocal Rank (MRR). The MRR [282] is usually applied for assessing the performance of ranking algorithms on QA and Information Retrieval (IR) tasks. \(MRR\) is defined as

\begin{equation} MRR=\frac{1}{Q} \sum _{i=1}^{Q} \frac{1}{{rank}(i)}, \end{equation}

(5)

where \({rank}(i)\) is the ranking of the ground-truth answer at answer \(i\)-th.

Hamming-Loss (HL). The HL [57] assesses the score of misclassified instance-label pairs where a related label is omitted or an unrelated label is predicted.

Among these single-label evaluation metrics, the Accuracy is the earliest metric that calculates the proportion of the sample size that is predicted correctly and is not considered whether the predicted sample is a positive or a negative sample. \(Precision\) calculates how many of the positive samples are actually positive, and the \(Recall\) calculates how many of the positive examples in the sample are predicted correctly. Furthermore, \(F1\) is the harmonic average of them, which is the most commonly used evaluation metrics.

3.2.2 Multi-label Metrics.

Compared with single-label text classification, multi-label text classification divides the text into multiple category labels, and the number of category labels is variable. These metrics are designed for single label text classification, which are not suitable for multi-label tasks. Thus, there are some metrics designed for multi-label text classification.

\(Micro\!-\!\!F1\) The \(Micro\!-\!\!F1\) [283] is a measure that considers the overall accuracy and recall of all labels. The \(Micro\!-\!\!F1\) is defined as:

\begin{equation} Micro\!-\!\!F1=\frac{2 {P}_{t} \times R_{t}}{{P}+{R}}, \end{equation}

(6)

where:

\begin{equation} P=\frac{\sum _{t \in \mathcal {S}} T P_{t}}{\sum _{t \in S} T P_{t}+F P_{t}},\quad R=\frac{\sum _{t \in S} T P_{t}}{\sum _{t \in \mathcal {S}} T P_{t}+F N_{t}}. \end{equation}

(7)

\(Macro\!-\!\!F1\) The \(Macro\!-\!\!F1\) [283] calculates the average \(F1\) of all labels. Unlike \(Micro\!-\!\!F1\), which sets even weight to every example, \(Macro\!-\!\!F1\) sets the same weight to all labels in the average process. Formally, \(Macro\!-\!\!F1\) is defined as:

\begin{equation} {Macro}\!-\!\!F1=\frac{1}{\mathcal {S}} \sum _{t \in \mathcal {S}} \frac{2 {P}_{t} \times R_{t}}{{P_{t}}+{R_{t}}}, \end{equation}

(8)

where:

\begin{equation} P_{t}=\frac{T P_{t}}{T P_{t}+F P_{t}},\quad R_{t}=\frac{T P_{t}}{T P_{t}+F N_{t}}. \end{equation}

(9)

In addition to the above evaluation metrics, there are some rank-based evaluation metrics for extreme multi-label classification tasks, including \(P@K\) and \(NDCG@K\).

Precision at Top K (P@K). The \(P@K\) [96] is the precision at the top k. For \(P@K\), each text has a set of \(\mathcal {L}\) ground truth labels \(L_{t}=\left\lbrace l_{0}, l_{1}, l_{2} \ldots , l_{\mathcal {L}-1}\right\rbrace\), in order of decreasing probability \(P_{t}=\) \([p_{0}, p_{1}, p_{2} \ldots , p_{Q-1}].\) The precision at \({k}\) is

\begin{equation} P@K= \frac{1}{k} \sum _{j=0}^{\min (\mathcal {L}, k)-1} {rel}_{L_{i}}\left(P_{t}(j)\right), \end{equation}

(10)

\begin{equation} \operatorname{rel}_{L}(p)= {\left\lbrace \begin{array}{ll}1 & \text{ if } p \in L \\ 0 & \text{ otherwise } \end{array}\right.}, \end{equation}

(11)

where \(\mathcal {L}\) is the number of ground truth labels or possible answers on each text and \(k\) is the number of selected labels on extreme multi-label text classification.

Normalized Discounted Cumulated Gains (NDCG@K). The \({NDCG}@{K}\) [96] is

\begin{equation} {NDCG} @ K=\frac{1}{{IDCG}\left({L}_{i}, k\right)}\sum _{j=0}^{n-1} \frac{r e l_{L_{i}}\left(P_{t}(j)\right)}{\ln (j+1)}, \end{equation}

(12)

where \(IDCG\) is ideal discounted cumulative gain and the particular rank position \(n\) is

\begin{equation} n=\min \left(\max \left(\left|P_{i}\right|,\left|L_{i}\right|\right), k\right). \end{equation}

(13)

Among these multi-label evaluation metrics, \(Micro\!-\!\!F1\) considers the number of categories, which makes it suitable for the unbalanced data distribution. \(Macro\!-\!\!F1\) does not take into account the amount of data that treats each class equally. Thus, it is easily affected by the classes with high Recall and Precision. When the number of categories is large or extremely large, either P@K or NDCG@K is used.

4 Quantitative Results

There are many differences between sentiment analysis, news classification, topic labeling, and natural language inference tasks, which can not be simplified modeled as a text classification task. In this section, we tabulate the performance of the main models given in their articles on classic datasets evaluated by classification accuracy, as shown in Table 4, including MR, SST-2, IMDB, Yelp.P, Yelp.F, Amazon.F, 20NG, AG, DBpedia, and SNLI.

Table 4.

We give the performance of NB and SVM algorithms from RNTN [64] due to that the less traditional text classification model has been an experiment on datasets in Table 4. The accuracy of NB and SVM are 81.8% and 79.4% on SST-2, respectively. We can see that, in the SST-2 data set with only two categories, the accuracy of NB is better than that of SVM. It may be because NB has relatively stable classification efficiency on new datasets. The performance is also stable on small datasets. Compared with the deep learning model, the performance of NB is lower. NB has the advantage of lower computational complexity than deep models. However, it requires manual classification features, making it difficult to migrate the model directly to other data sets.

For deep learning models, pre-trained models get better results on most datasets. It means that if you need to implement a text classification task, you can preferentially try pre-trained models, such as BERT, RoBERTa, and XLNET, etc., except MR and 20NG, which have not been experimented on BERT-based models. Pre-trained models are essential to NLP. It uses a deep model to learn a better feature of the text. It also demonstrates that the accuracy of NLP tasks can be significantly improved by a profound model that can be pre-trained from unlabeled datasets. For the MR dataset, the accuracy of RNN-Capsule [87] is 83.8%, obtaining the best result. It suggests that RNN-Capsule builds a capsule in each category for sentiment analysis. It can output words including sentiment trends indicating attributes of capsules with no applying linguistic knowledge. For 20NG dataset, BLSTM-2DCNN [77] gets 96.5% score with the best accuracy score. It may demonstrate the effectiveness of applying the 2D max-pooling operation to obtain a fixed-length representation of the text and utilize 2D convolution to sample more meaningful matrix information.

5 Future Research Challenges

Text classification – as efficient information retrieval and mining technology – plays a vital role in managing text data. It uses NLP, data mining, machine learning, and other techniques to automatically classify and discover different text types. Text classification takes multiple types of text as input, and the text is represented as a vector by the pre-training model. Then the vector is fed into the DNN for training until the termination condition is reached, and finally, the performance of the training model is verified by the downstream task. Existing models have already shown their usefulness in text classification, but there are still many possible improvements to explore.

Although some new text classification models repeatedly brush up the accuracy index of most classification tasks, they cannot indicate whether the model “understands” the text from the semantic level like human beings can. Moreover, with the emergence of the noise sample, the small sample noise may cause the decision confidence to change substantially or even lead to decision reversal. Therefore, the semantic representation ability and robustness of the model need to be proven in practice. Besides, the pre-trained semantic representation model represented by word vectors can often improve the performance of downstream NLP tasks. The existing research on the transfer strategy of context-free word vectors is still relatively preliminary. Thus, we conclude from a data, models, and performance perspective, text classification mainly faces the following challenges.

5.1 Challenges from Data Perspective

For a text classification task, data is essential to model performance, whether it is a traditional or deep learning method. The text data mainly studied includes multi-chapter, short text, cross-language, multi-label, less sample text. For the characteristics of these data, the existing technical challenges are as follows:

Zero-shot/Few-shot learning. Zero-shot or few-shot learning for text classification aims to classify text having no or few same labeled class data. However, the current models are too dependent on numerous labeled data. The performance of these models is significantly affected by zero-shot or few-shot learning. Thus, some works focus on tackling these problems. The main idea is to infer the features through learning kinds of semantic knowledge, such as learning relationships among classes [288] and incorporating class descriptions [170]. Furthermore, latent features generation [289], meta-Learning [106, 290, 291], and dynamic memory mechanism [292] are also efficient methods. Nevertheless, with the limitation of little unseen class data and different data distribution between seen class and unseen class, there is still a long way to go to reach the learning ability comparable to that of humans.

The external knowledge. As we all know, the more beneficial information is input into a DNN, the better the performance. For example, a question answering system incorporating commonsense knowledge can answer questions about the real world and help solve problems with incomplete information. Therefore, adding external knowledge (knowledge base or knowledge graph) [293, 294] is an efficient way to promote the model’s performance. The existing knowledge includes conceptual information [100, 127, 204], commonsense knowledge [223], knowledge base information [295, 296], general knowledge graph [170], and so on, which enhances the semantic representation of texts. Nevertheless, with the limitation of input scale, how and what to add for different tasks is still a challenge.

Special domain with many terminologies. Most of the existing models are supervised models, which over-rely on numerous labeled data. When the sample size is too small, or zero samples occur, the performance of the model will be significantly affected. New data set annotation takes a lot of time. Therefore, unsupervised learning and semi-supervised learning have great potential for text classification. Furthermore, texts in a particular field [297, 298], such as financial and medical texts, contain many specific words or domain experts intelligible slang, abbreviations, etc., which make the existing pre-trained word vectors challenging to work on.

The multi-label text classification task. Multi-label text classification requires full consideration of the semantic relationship among labels, and the embedding and encoding of the model is a process of lossy compression [299, 300]. Therefore, how to reduce the loss of hierarchical semantics and retain rich and complex document semantic information during training is still a problem to be solved.

5.2 Challenges from Model Perspective

Most existing structures of traditional and deep learning models are tried for text classification, including integration methods. BERT learns a language representation that can be used to fine-tune for many NLP tasks. The primary method is to increase data, improve computation power, and design training procedures for getting better results [301, 302, 303]. Understanding the tradeoff between data and compute resources and prediction performance is worth studying.

Text representation. The text representation method based on the vector space model is simple and effective in the text preprocessing stage. However, it will lose the semantic information of the text, so the application performance based on this method is limited. The proposed semantically based text representation method is too time-consuming. Therefore, the efficient semantically based text representation method still needs further research. In the text representation of text classification based on deep learning, word embedding is the main concept, while the representation unit is described differently in different languages. Then, a word is represented in the form of a vector by learning mapping rules through the model. Therefore, how to design adaptive data representation methods is more conducive to the combination of deep learning and specific classification tasks.

Model integration. Most structures of traditional and deep learning models are tried for text classification, including integration methods. RNN requires recursive step by step to get global information. CNN can obtain local information, and the sensing field can be increased through the multi-layer stack to capture more comprehensive contextual information. Attention mechanisms learn global dependency among words in a sentence. The transformer model is dependent on attention mechanisms to establish the depth of the global dependency relationship between the input and output. Therefore, designing an integrated model is worth trying to take advantage of these models.

Model efficiency. Text classification models based on deep learning are highly effective, such as CNNs, RNNs, and GNNs. However, there are many technical limitations, such as the depth of the network layer, regularization problem, network learning rate, etc. Therefore, there is still more broad space for development to optimize the algorithm and improve the speed of model training.

5.3 Challenges from Performance Perspective

The traditional model and the deep model can achieve good performance in most text classification tasks, but the anti-interference ability of their results needs to be improved [176, 304, 305]. How to realize the interpretation of the deep model is also a technical challenge.

The semantic robustness of the model. In recent years, researchers have designed many models to enhance the accuracy of text classification models. However, when there are some adversarial samples in the datasets, the model’s performance decreases significantly. Adversarial training is a crucial method to improve the robustness of the pre-training model. For example, a popular approach is converting attack into defense and using the adversarial sample training model. Consequently, how to improve the robustness of models is a current research hotspot and challenge.

The interpretability of the model. DNNs have unique advantages in feature extraction and semantic mining and have achieved excellent text classification tasks. Only a better understanding of the theories behind these models can accurately design better models for various applications. However, deep learning is a black-box model, the training process is challenging to reproduce, and the implicit semantics and output interpretability are poor. It makes the improvement and optimization of the model lose clear guidelines. Why does one model outperform another on one data set but underperform on others? What does the deep learning model learn? Furthermore, we cannot accurately explain why the model improves performance.

6 Conclusion

This paper principally introduces the existing models for text classification tasks from traditional models to deep learning. Firstly, we introduce some primary traditional models and deep learning models with a summary table. The traditional model improves text classification performance mainly by improving the feature extraction scheme and classifier design. In contrast, the deep learning model enhances performance by improving the presentation learning method, model structure, and additional data and knowledge. Then, we introduce the datasets with a summary table and evaluation metrics for single-label and multi-label tasks. Furthermore, we give the quantitative results of the leading models in a summary table under different applications for classic text classification datasets. Finally, we summarize the possible future research challenges of text classification.

Acknowledgments

Thanks for computing infrastructure provided by Huawei MindSpore platform.

Footnotes

https://arxiv.org/.

https://www.uspto.gov/.

References

[1]

Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proc. ACL, 2015. 1556–1566.

Abstract

1 Introduction

1.1 Major Differences and Contributions

1.2 Organization of the Survey

2 Text Classification Methods

2.1 Traditional Models

2.1.1 PGM-based Methods.

2.1.2 KNN-based Methods.

2.1.3 SVM-based Methods.

2.1.4 DT-based Methods.

2.1.5 Integration-based Methods.

2.2 Deep Learning Models

2.2.1 ReNN-based Methods.

2.2.2 MLP-based Methods.

2.2.3 RNN-based Methods.

2.2.4 CNN-based Methods.

2.2.5 Attention-based Methods.

2.2.6 Pre-trained Methods.

2.2.7 GNN-based Methods.

2.2.8 Others.

3 Datasets and Evaluation Metrics

3.1 Datasets

3.1.1 Sentiment Analysis (SA).

3.1.2 News Classification (NC).

3.1.3 Topic Labeling (TL).

3.1.4 Question Answering (QA).

3.1.5 Natural Language Inference (NLI).

3.1.6 Multi-Label (ML) Datasets.

3.1.7 Others.

3.2 Evaluation Metrics

3.2.1 Single-label Metrics.

3.2.2 Multi-label Metrics.

4 Quantitative Results

5 Future Research Challenges

5.1 Challenges from Data Perspective

5.2 Challenges from Model Perspective

5.3 Challenges from Performance Perspective

6 Conclusion

Acknowledgments

Footnotes

References

Cited By

Index Terms

Recommendations

Deep Learning--based Text Classification: A Comprehensive Review

Chinese text classification by the Naïve Bayes Classifier and the associative classifier with multiple confidence threshold values

Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

HTML Format

Login options

Full Access

Share

Share this Publication link

Share on social media

Affiliations