Hybrid neural conditional random fields for multi-view sequence labeling

Highlights

• We propose a hybrid neural CRF for multi-view sequence labeling, called MVCRF.

• Our model combines multi-view learning by utilizing consensus and complementary principles.

• We systematically compare the performance of MVCRF with other models.

• The experimental results show MVCRF achieves state-of-the-art performance.

Abstract

In traditional machine learning, conditional random fields (CRF) is the mainstream probability model for sequence labeling problems. CRF considers the relation between adjacent labels other than decoding each label independently, and better performance is expected to achieve. However, there are few multi-view learning methods involving CRF which can be directly used for sequence labeling tasks. In this paper, we propose a novel multi-view CRF model to label sequential data, called MVCRF, which well exploits two principles for multi-view learning: consensus and complementary. We first use different neural networks to extract features from multiple views. Then, considering the consistency among the different views, we introduce a joint representation space for the extracted features and minimize the distance between the two views for regularization. Meanwhile, following the complementary principle, the features of multiple views are integrated into the framework of CRF. We train MVCRF in an end-to-end fashion and evaluate it on two benchmark data sets. The experimental results illustrate that MVCRF obtains state-of-the-art performance:  ${\mathrm{F}}_1$ score 95.44% for chunking on CoNLL-2000, 95.06% for chunking and 96.99% for named entity recognition (NER) on CoNLL-2003.

Introduction

Sequence labeling problems such as named entity recognition (NER) and syntactic chunking are classical tasks in the field of natural language processing (NLP). The sequential data used in these tasks mostly contains different features, such as word feature and part-of-speech (POS), which may be obtained by diverse measuring modes or come from different feature extractors. We usually call them as multi-view data. For multi-view data, a naive method is concatenating multiple views directly into a single view, then using the single-view algorithms for subsequent processing. However, this approach may lead to the overfitting problem, and the unique statistical characteristics of each view cannot be fully exploited. The alternative method is only using one of the multiple views. Generally, the best performance cannot be achieved, either.

Multi-view learning (MVL) is an emerging direction and has made great developments in machine learning for recent years, which can solve the above problems well. It aims to improve generalization performance by making full use of information from multiple views. There is an increasing number of MVL methods proposed, which can be divided into three major categories [1]: co-training style algorithms [2], [3], [4], co-regularization style algorithms [5], [6], and margin consistency style algorithms [7], [8]. Without loss of generality, we can usually obtain better performance by adopting these MVL methods. Even if there is only a natural single view available, it is possible to further improve the model performance by manually generating multiple views, which reflects the huge advantage of MVL [2]. Recently, MVL has been a highly concerned topic in machine learning [9], [10], [11], [12], [13]. However, to the best of our knowledge, among the existing MVL methods, few of them can directly hand with sequence labeling problems. In this paper, we intend to propose a new kind of multi-view method to label sequential data.

The existing sequence labeling models include two main categories. One is the linear statistical models, such as hidden Markov models [14], maximum entropy Markov models [15], and conditional random fields (CRF) [16], [17], [18], [19]. The other is the non-linear neural networks based models. For the linear statistical models, CRF [16] is a popular model for fitting time sequences and excellent at the sequential data labeling tasks. CRF does not make independence assumptions on the observations. It focuses on the information in the sentence level instead of individual position. Therefore, CRF can more correctly catch the relationship within a sequence and a higher tagging accuracy is expected to achieve. With good properties, CRF has been widely used in sequence labeling tasks and obtains respectably good performance [16], [17], [18], [19]. Among the non-linear neural networks based models, the model based on the convolutional neural network (CNN) [20] was first presented for sequence labeling. Later, some sequence labeling models based on long short-term memory (LSTM) were proposed and made great success in sequence labeling [21], [22], [23].

In this paper, we develop a hybrid neural CRF for multi-view sequence labeling, named MVCRF. The model is based on the traditional CRF and adopts diverse neural networks for feature extraction of multiple views. Different from the available models for sequence labeling, the proposed model not only considers the correlation between neighborhood labels and jointly decodes the best sequence of labels, but also combines MVL by utilizing consensus and complementary principles [24]. MVCRF first takes each view of the sequential data as the input. Since neural networks have the ability to automatically extract features from data [25], then we adopt them to respectively extract features from multiple views in the proposed model. The extracted features are projected into a joint representation space. Inspired by the idea of co-regularization method [26], we regularize the log-likelihood via minimizing the distance between two views. In other words, we enforce the features from different views to be as close as possible by minimizing the distance, which reflects the consensus principle. Moreover, considering that each view may contain some specific information that not in other views, the features from different views are taken as the input to the CRF layer. At the last, CRF takes the role of making a structural prediction and outputting the best sequence labels.

The main contributions in this paper can be summarized as follows. Our work is to propose a hybrid neural CRF for multi-view sequence labeling. The key idea in our model is combining MVL to perform the sequence labeling. We first construct a joint representation space of different features, based on the consensus principle. Then we regularize the conditional probability distribution by the consistency of diverse views. Meanwhile, we also consider the complementary principle to take full use of the specific information for each view. We systematically compare the performance of MVCRF with other models. The experimental results show MVCRF achieves state-of-the-art performance on CoNLL-2000 and CoNLL-2003 benchmark data sets.

The rest of this paper is organized as follows. Section 2 reviews the related research. Section 3 proposes our MVCRF model. Section 4 reports experimental results on benchmark data sets and makes systematical comparisons. Finally, we draw conclusions and point out possible future work in Section 5.