S map: Semisupervised aspect-based sentiment analysis with masked aspect prediction
Introduction
Sentiment analysis (SA) is a fundamental research topic in the information retrieval, and natural language processing communities [1], [2], [3]. Generally, SA aims to automatically detect the sentiment polarity, {e.g.,Positive, Neutral, Negative} from sentences such as those in reviews on restaurants, movies, and products. For example, given a restaurant review,“ Nice restaurant whose service needs to be improved”, SA aims to accurately detect its actual sentiment polarity Positive. Naturally, SA is potentially in demand in many real-world applications.
Unfortunately, the traditional SA task only concentrates on the sentiment polarity of the full text. We expect to analyze fine-grained aspect-based sentiment in many real scenarios to explore more valuable information. Retaking the example above, we may be more concerned with the sentiment target “service” whose polarity is negative rather than the polarity of the full text. In response to this demand, more attention has been recently paid to the emerging topic of Aspect-Based Sentiment Analysis (ABSA), whose aim is to automatically detect the sentiment polarity of certain aspects [4], [5], [6], [7], [8], [9], [10], [11].
Generally speaking, the first step is to collect fine-grained training datasets to resolve ABSA with machine learning techniques. Several human annotators should tag aspect words and their sentiment polarities for training sentences, as illustrated in Table 1. Collecting such training datasets is costly and much more expensive than traditional SA training data. Therefore the available ABSA training datasets are scarce and often contain very few training sentences, e.g., the volumes of the prevalent ABSA datasets Restaurant and Laptop from SemiEval 14 [12] are only 2282 and 3608, respectively. Such scarce ABSA training sentences contain limited supervised signals, limiting the performance upper bound of ABSA models.
To meet this demand, we take inspiration from the spirit of semi-Supervised learning (SSL) and accordingly attempt to train ABSA models by simultaneously leveraging a limited amount of expensive labeled sentences and more unlabeled-yet-cheaper sentences, yielding a topic of semi-supervised aspect-based sentiment analysis (SemiABSA). Formally, we are given a training dataset, which consists of a subset of labeled triplets and a subset of unlabeled sentences . Specifically, , , and denote the raw sentence, aspect word indicator, and one-hot sentiment polarity indicator, respectively, where represents its number of word tokens and is the number of sentiment polarities. We consider the inductive learning paradigm, where the aim is to train a predictive model from and apply it to predict any unseen sentence-aspect tuple . To our knowledge, very few studies have addressed this topic.
In this paper, we propose a novel self-training SemiABSA framework, namely Semi-Supervised aspect-based Sentiment analysis with Masked Aspect Prediction (S3map). To fully use unlabeled sentences, our basic idea is to form pseudo-aspect-specific sentence embeddings and pseudo-sentiment polarities and train the sentiment classifier with them in a self-training manner. Specifically, S map consists of 4 key modules: BERT-Encoder, Aspect-Discriminator, Aspect-Sentence-Encoder(AS-Encoder), and Sentiment-Predicter. First, the BERT-Encoder can be treated as the basic feature encoder for both sentences and tokens. Second, the Aspect-Discriminator is a BERT-Encoder-based masked aspect prediction task trained over labeled sentences, and it is used to identify the aspect words for unlabeled sentences. Third, with the pseudo-aspect in hand, we utilize the AS-Encoder to update the pseudo-aspect embedding and sentence embedding. Various structures such as GCN or attention networks can instantiate AS-Encoder. Then, we combine pseudo-aspect embedding and sentence embedding to form the pseudo-aspect-specific sentence embeddings. Finally, S map is jointly trained with labeled and pseudo-labeled sentences.
Empirically, we thoroughly investigate the potential of SemiABSA from various perspectives based on S map. First, we employ two prevalent ABSA collections of reviews on restaurants and laptops and four collections of unlabeled sentences from multiple domains, including reviews, daily social media posts, and encyclopedias. We generate synthetic SemiABSA datasets using pairwise combinations of labeled and unlabeled datasets. Accordingly, a total of 8 synthetic SemiABSA datasets are generated. We evaluate S map on these datasets. The empirical results demonstrate that S map can consistently improve performance by leveraging unlabeled sentences in various scenarios, even when labeled and unlabeled sentences are from different domains, and unlabeled sentences, i.e., encyclopedias, tend to be without any sentiment polarities. In addition, S map significantly outperforms the existing SemiABSA methods.
In summary, the major contributions of this paper are outlined below:
- •
We investigate the problem of SemiABSA and propose a novel framework named S3map.
- •
We propose a novel BERT-based MAP task to infer the aspect words of sentences.
- •
We conduct extensive experiments to indicate the effectiveness of S3map.
Section snippets
Sentiment analysis
Traditional SA methods mainly aim to automatically predict sentiment polarity, e.g., attitudes, and opinions, for full texts [13], [14]. From the development timeline of SA, the prior arts include rule-based systems [15], [16], shallow learning-based methods [17], [18] and deep learning-based methods [19], [20]. Generally, deep learning-based SA methods use neural networks to form discriminative text embeddings and can achieve promising performance. From the perspective of the network
Overall framework of S map
Overall, S map is built on the idea of self-training and it consists of 4 basic modules. (1) BERT-Encoder: We apply the pre-trained BERT Model as the basic encoder, which inputs a sentence and outputs the contextualized embeddings of all tokens . Each column denotes the embedding of the th word token. (2) Aspect Discriminator: This is used to identify the aspect words for unlabeled sentences (3) Aspect-Sentence-Encoder (AS-Encoder): This extracts the aspect
Experimental settings
In this section, we introduce the experimental settings, including datasets, parameter configuration of S map, and evaluation metrics.
Datasets. We employ two prevalent ABSA datasets Restaurant (Rest.) and Laptop (Lap.) from SemEval 2014 Task 4 [12], and four datasets of unlabeled sentences from various domains, including two review collections - (Yelp4 and Amazon,5) and two generic sentence
Results and analysis
In this section, we empirically evaluate the proposed S map method, and mainly attempt to answer the following questions:
- •
Q1: Can S map compete with the existing arts of semi-supervised learning and supervised learning?
- •
Q2: Can S map effectively improve the performance with auxiliary unlabeled sentences?
- •
Q3: Can S map be sensitive to the configurations of labeled and unlabeled sentences from various domains?
- •
Q4: Is S map is sensitive to different numbers of unlabeled sentences?
Conclusion
This work addresses the scarcity problem of fine-grained ABSA training sentences. To do this, we use the SemiABSA framework to simultaneously leverage labeled and unlabeled sentences for ABSA model training, and propose a novel method named S map. The proposed S map framework is built on the self-training paradigm, whose key idea is to generate pseudo-aspect words and pseudo-sentiment polarities for unlabeled sentences. Specifically, we propose a BERT-based MAP task to predict aspect words
CRediT authorship contribution statement
Zhiyao Yang: Investigation, Conceptualization, Methodology, Software, Validation, Writing – original draft, Writing – review & editing. Bing Wang: Conceptualization, Validation, Investigation, Writing – original draft, Writing – review & editing. Ximing Li: Writing – review & editing. Wenting Wang: Project administration. Jihong Ouyang: Project administration, Funding acquisition.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
We want to acknowledge support for this project from the National Natural Science Foundation of China (NSFC) (No. 62276113, No. 62006094), Scientific and Technological Developing Scheme of Jilin Province (No. 20180201003SF, No. 20190701031GH) and Energy Administration of Jilin Province (No. 3D516L921421).
References (68)
- et al.
MetaPro: A computational metaphor processing model for text pre-processing
Inf. Fusion
(2022) - et al.
Aspect-based sentiment analysis with attention-assisted graph and variational sentence representation
Knowl.-Based Syst.
(2022) - et al.
Semi-supervised target-oriented sentiment classification
Neurocomputing
(2019) - Y. Zhang, Y. Zhang, Tree Communication Models for Sentiment Analysis, in: Conference of the Association for...
- H. Tian, C. Gao, X. Xiao, H. Liu, B. He, H. Wu, H. Wang, F. Wu, SKEP: Sentiment Knowledge Enhanced Pre-training for...
- J. Barnes, R. Kurtz, S. Oepen, L. Øvrelid, E. Velldal, Structured Sentiment Analysis as Dependency Graph Parsing, in:...
- P. Chen, Z. Sun, L. Bing, W. Yang, Recurrent Attention Network on Memory for Aspect Sentiment Analysis, in: Conference...
- R. He, W.S. Lee, H.T. Ng, D. Dahlmeier, Effective Attention Modeling for Aspect-Level Sentiment Classification, in:...
- W. Xue, T. Li, Aspect Based Sentiment Analysis with Gated Convolutional Networks, in: Annual Meeting of the Association...
- X. Li, L. Bing, W. Lam, B. Shi, Transformation Networks for Target-Oriented Sentiment Classification, in: Annual...
A survey on classification techniques for opinion mining and sentiment analysis
Artif. Intell. Rev.
A survey of sentiment analysis in social media
Knowl. Inf. Syst.
Fuzzy rule based unsupervised sentiment analysis from social media posts
Expert Syst. Appl.
Document-level sentiment classification using hybrid machine learning approach
Knowl. Inf. Syst.
Sentiment analysis using deep learning approaches: An overview
Sci. China Inf. Sci.
Sentiment analysis using deep learning architectures: A review
Artif. Intell. Rev.
Modeling sentiment dependencies with graph convolutional networks for aspect-level sentiment classification
Knowl.-Based Syst.
The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection
IEEE Trans. Affect. Comput.
Cited by (8)
Aspect-based sentiment classification with aspect-specific hypergraph attention networks
2024, Expert Systems with ApplicationsPseudo dense counterfactual augmentation for aspect-based sentiment analysis
2023, NeurocomputingReconstructing graph networks by using new target representation for aspect-based sentiment analysis
2023, Knowledge-Based SystemsTraceNet: Tracing and locating the key elements in sentiment analysis
2023, Knowledge-Based SystemsAspect-Based Sentiment Analysis with Explicit Sentiment Augmentations
2024, Proceedings of the AAAI Conference on Artificial Intelligence
- 1
Contributing equally with the first author.