Empirical study on character level neural network classifier for Chinese text

Abstract

Character level models are drawing attention recently. A number of these models have been proposed and shown successful in Natural Language Processing tasks. While most of the models are experimented mainly on English, or other alphabetic languages, a number of problems arise when they applied these models to non-alphabetic language such as Chinese. In this study, we investigated the problems encountered when transferring these models to the Chinese and put forward some solutions. We propose a double embedding neural network model that is also character level and consists of both CNN and RNN with two separate embeddings. The model is applied to a fundamental Natural Language Processing task, text classification. Experiment results conducted on the Chinese corpus demonstrated that our character level neural network model performs just as well as or better than those word level classification models. Our model is able to reach 95.9% accuracy on a Chinese Fudan news dataset, which outperforms the state-of-the-art models.

Introduction

Text classification, the task of assigning predefined labels to text documents, is essential in many natural language processing applications, such as news filtering, information retrieval and sentiment analysis (Aggarwal and Zhai, 2012). Text classification on Chinese is challenging problem for many researchers because there is no natural delimiter in the Chinese text (Chang et al., 2008). To improve the performance of Chinese text classification, efforts must be made through two major tasks: feature selection and classifier selection.

State-of-the-art feature selection is based on bag-of-words (n-gram) model with some discriminative feature selectors such as MI, pLSA and LDA. However, these feature selection methods suffer from data sparsity and are unsatisfactory in capturing semantics of words, thus affect classification accuracy (Lai et al., 2015). Word embedding (Mikolov et al., 2013) addresses the data sparsity problem by learning a low dimensional vector for each word. Mikolov et al. showed that word embedding is able to infer the semantics and syntactics of words. There is usually a preliminary and important pre-processing step called sentence segmentation or word parsing, which may enhance the performance of NLP tasks (Chang et al., 2008, Chen et al., 2015, Liu et al., 2017, Tang et al., 2008) on Chinese. Yet the accuracy of sentence segmentation decreases when handling domain specific text or informally written text such as mirco-blogs. The error propagated to the model further affects the classification performance. An emerging set of models, namely, character level classification models (Conneau et al., 2016, Zhang et al., 2015), can avoid such a problem because sentence segmentation is eliminated in them. Moreover, the number of Chinese characters is less than that of words, so the sparsity issue can be less severe. Therefore, the models using embedding at character level models in Chinese are more suitable for Chinese than those word level based classification models.

The second task of text classification is selecting a machine learning model. Traditional models with strong baseline include Support Vector Machine (SVM), Naive Bayes (NB), Logistic Regression (LR), etc. (Wang and Manning, 2012). Recently, the neural network models have brought new insights to the various NLP tasks. Models applying neural network to word embedding without any syntactic or semantic knowledge are reported competitive to state-of-the-art models (Kim, 2014, Zhang et al., 2015). Convolution neural network (CNN) and Recurrent neural network (RNN) are two mainstream architectures of the neural network. Long short term memory (LSTM) (Hochreiter and Schmidhuber, 1997) is a popular choice for RNN. Combining the two models together resulted in better performance in some cases. Recurrent convolution neural network (Lai et al., 2015) applies a recurrent network over words to learn embedding then max-pooling layer to extract key features. C-LSTM (Zhou et al., 2015) firstly performs convolution operation over word embedding followed by a LSTM layer. These models stack one type of the neural network over another capturing semantics and feeding it to the other layer. Whereas in some models, a static embedding, obtained from an unsupervised neural language model, transfers information extracted from large text corpus to the task (Kim, 2014).

A comparative study of CNN and RNN (Yin et al., 2017) for Natural Language Processing from both theory and experiment concluded that CNNs are hierarchical architecture and RNNs are sequential architectures. For example, given embedding of an input sequence $e_1$, $e_2$, $e_3$, $e_4$, the CNN would use a window with size 2 and compute extra features, $f_1$ $=$ CNN($e_1$, $e_2$) and $f_2$ $=$ CNN($e_2$, $e_3$). Following such pattern higher order of features can be further computed. In the reference Zhang et al. (2015), convolution was performed on characters of an English word to compute a representation of the word. This can be done on Chinese too. The only difference is that since the Chinese do not have a natural delimiter, convolution is performed on the whole combination within a window size for Chinese, capturing word combinations and non-word combinations separately. On the other hand, RNN performs computation sequentially, which means, at a certain step, it takes in the current input and the previous output to compute an output. Such kind of computation indicates that the order of the sequence is considered (in language, we consider order of a language involves syntax). The CNN and RNN capture different types of structural information, the CNN for hierarchical and the RNN for sequential (Yin et al., 2017). In the task of character level text classification for Chinese, hierarchical structures such as word and phrase combination can be induced with CNN, while syntactical information can be learned using RNN. While embedding two kinds of knowledge into one vector may be challenging due to the irrelevancy of the knowledge, an appropriate representation may not be induced. Updating of the vector may go back and forth during training.

Given the problems mentioned above, this paper proposes a model that does not require any word segmentation when processing Chinese text data. Two embeddings are used to capture hierarchical and sequential information separately. The model consists of the CNN and RNN networks. Instead of stacking a certain type of networks, we have combined them in a concurrent manner, thus allowed the model to capture different types of information without interfering with each other. We conducted experiments using multiple character level models and traditional machine learning models on two Chinese datasets, Fudan News and Sougou News. We also compare our results with those reported from state-of-the-art researches as well.

The main contributions of this study are summarized as follows:

• We demonstrate that the character level classification models are not affected from the propagated error problem due to the fact that such a step is eliminated. Also, we show that the character level models have many advantages compared with the word level models.

• We present a Double Embedding Neural Network Classifier. It consists of the CNN and the RNN. We explain how to combine the two networks for allowing them to capture different types of information without interfering with each other.

• Our Double Embedding Neural Network Classifier model was applied to the Chinese Text classification task. Experiments show that this character level model outperforms the state-of-the-art word level models.