Cross-domain sentiment classification via parameter transferring and attention sharing mechanism

Highlights

• Parameter transferring and attention sharing mechanism are proposed to effectively transfer sentiment knowledge through model transfer methods and effectively avoid overfitting.

• Model transfer is presented toto transfer the network parameters from source domain network to target domain network.

• Attention sharing mechanism is designed to share the attentional weights of different feature spaces in different domains.

• The prediction results show the validity and advantages of sentiment transfer strategy.

Abstract

Training data in a specific domain are often insufficient in the area of text sentiment classifications. Cross-domain sentiment classification (CDSC) is usually utilized to extend the application scope of transfer learning in text-based social media and effectively solve the problem of insufficient data marking in specific domains. Hence, this paper aims to propose a CDSC method via parameter transferring and attention sharing mechanism (PTASM), and the presented architecture includes the source domain network (SDN) and the target domain network (TDN). First, hierarchical attentional network with pre-training language model on training data, such as global vectors for word representation and bidirectional encoder representations from transformers (BERT), are constructed. The word and sentence levels of parameter transferring mechanisms are introduced in the model transfer. Then, parameter transfer and fine-tuning techniques are adopted to transfer network parameters from SDN to TDN. Moreover, sentiment attention can serve as a bridge for sentiment transfer across different domains. Finally, word and sentence level attention mechanisms are introduced, and sentiment attention is shared from the two levels across domains. Extensive experiments show that the PTASM-BERT method achieves state-of-the-art results on Amazon review cross-domain datasets.

Introduction

Classic text sentiment classification methods premise that domains between training and testing are independent and identically distributed. However, different domains have distribution differences under realistic conditions [1], [2]. The cross-domain sentiment classification (CDSC) task adopts source domain resources to achieve sentiment classification tasks in target domains [3], [4]. The utilization of CDSC not only extends the application scope of transfer learning in text-based social media but also promotes the classification effect of low-resource text sentiment classification tasks to solve the problem of insufficient marking samples in specific domains. In addition, CDSC is conducive to promoting the rapid development of industrial applications related to text-based sentiment analysis [5].

Deep learning methods have achieved exceptional results on text sentiment classification tasks in the past years but require abundant labeled training data. However, annotating domain-specific data is an extremely labor-intensive task [6]. The distribution of sentiment varies across different domains. Users tend to express the sentiment with diverse words in different domains [7], [8]. Thus, the expression of sentiment is domain-dependent. Generalizing classifiers trained in different domains is difficult; thus, specific sentiment transfer strategies must be introduced. Sentiment transfer across different domains aims to seek domain invariance as a bridge to achieve cross-domain transfer.

Documents in social media have a three-level semantic structure, that is, word-sentence-document. The sentiment of words in the document composition determines the sentiment of a sentence, while that sentences determines the overall sentiment of documents [9], [10], [11]. In addition, various words and sentences contribute differently to the overall sentimental expression [12]. The attention mechanism can effectively promote the effect of sequence-to-sequence models via performing weighted transformation [13], [14]. Intuition suggests that the word- and sentence-level attentional weights obtained from source domain training can guide the training of the target domain attentional weights. The most important words and sentences for sentiment decision-making considering attention mechanisms are obtained. Fig. 1 shows examples of word- and sentence-level attention visualizations for reviews in kitchen and electronics domains. For instance, the sentiment polarity of sentences 1 and 6 according to the words “happy” and “disappoint” can be easily determined. Similarly, the sentiment polarity of the review can be determined by using sentences 7 and 9. Simultaneously, document representations in different domains can share similar sentence and word attention models. For instance, in kitchen and electronics domains, sentences 4 and 11 have similar word attention weights, while samples (a) and (c) have similar sentence attention weights.

The two challenges should generally be addressed in CDSC. First, how can the structure and parameters of deep neural networks be transferred across different domains? The model transfer mechanism can be adopted to transfer the structure and parameters of the model. Meanwhile, fine-tuning strategies in deep transfer learning act as a key scientific issue for this subject. Second, how are attention mechanisms shared in neural network models trained across different domains? The sentiment attention mechanism can be used as a bridge to connect different domains. The word- and sentence- level attention mechanisms can be mutually guided and trained.

A CDSC method is proposed in this paper via parameter transferring and attention sharing mechanism (PTASM). First, the sentiment information of important words and sentences in the text is modeled through hierarchical attentional network (HAN), and a document-level distributed representation is determined. The pre-training language models, such as global vectors for word representation (Glove) and bidirectional encoder representations from transformers (BERT), are used as input for HAN. First, two HANs, namely, source domain network (SDN) and target domain network (TDN), are designed. Then, the word and sentence levels of the parameter transferring mechanisms are introduced in model transfer. In addition, word- and sentence-level attention alignment relationships across different domains are considered, and an effective cross-domain attention sharing mechanism is designed. Last, PTASM is validated with Glove and BERT on benchmark CDSC datasets. Experiments indicate that the PTASM-BERT method not only obtains a high cross-domain classification accuracy but also automatically learns the alignment degree of features between domains. Moreover, parameter transferring and attention sharing at the word and sentence levels are verified to be better than those at a single level.

The contributions of this paper can be summarized as follows.

• A CDSC method via parameter transferring and attention sharing mechanism, which can effectively transfer sentiment knowledge via model transfer methods and avoid overfitting, is proposed. Meanwhile, the attention sharing mechanism is utilized as a bridge across different domains.

• The model transfer strategy is adopted to transfer the network parameters. The attentional weights of different feature spaces for the attention sharing mechanism can guide one another, and the cosine distance of attentional weights is reduced.

• PTASM is validated on Amazon review datasets, and the parameter selection is experimentally verified. Experiments demonstrate that PTASM-BERT can improve the accuracy compared with several baseline methods.

The remainder of this paper is arranged as follows. Section 2 discusses model transfer methods and attention mechanism in sentiment analysis. Section 3 proposes the architecture of transferable neural networks framework and attention sharing mechanism. Section 4 introduces the experimental setup. Section 5 shows the experimental results and analyses of the proposed approach. Section 6 summarizes the paper and provides promising future works.