ASK-RoBERTa: A pretraining model for aspect-based sentiment classification via sentiment knowledge mining

Abstract

The main objective of aspect-based sentiment classification (ABSC) is to predict sentiment polarities of different aspects from sentences or documents. Recent research integrates sentiment terms into pretraining models whose accuracy impacts the ABSC performance. This paper introduces a sentiment knowledge-adaptive pretraining model (ASK-RoBERTa). A sentiment word dictionary is first built from general and field sentiment words. We develop a series of term and sentiment mining rules based on part-of-speech tagging and sentence dependency grammar. These mining rules consider word dependencies, compounding, and conjunctions. The pretraining model optimizes the mining rules to capture the dependency between aspects and sentiment words. Experimental results on multiple public benchmark datasets demonstrate the satisfactory performance of ASK-RoBERTa.

Introduction

Sentiment analysis and opinion mining offer valuable opportunities for the extraction and analysis of emerging patterns that appear as a result of the rapid development of social media communities [1]. The opportunity to automatically capture the general public’s sentiments about social events, political movements, marketing campaigns, and product preferences has attracted interest of both the scientific community and the business world [2]. When interacting with social media, people generally express their thoughts and opinions on a wide range of aspects. The goal of aspect-based sentiment classification (ABSC) is to discover sentiment polarities (e.g. positive, negative, neutral) for specified aspects within sentences and documents in contrast to conventional sentiment analysis that predicts the overall sentiment value of a given comment [3]. One of the primary benefits of ABSC approaches is their ability to extract the sentiment and polarity of a specific aspect from its context sentence. Different aspects of a given document can be studied, potentially revealing highlights of the underlying aspects, polarities, and meanings. ABSC datasets generally encompass a wide range of contexts, components, and sentiments. Consider the sentence “Boot time is superfast, but the battery life is poor”. This sentence combines contrasting polarities as the processing time will be satisfactory but under the implicit condition that battery life will be improved, which is not the case. This is a straightforward example of how ABSC grew in popularity due to its ability to extract useful information from textual comments.

Most early ABSC models employ a recursive neural network to improve sentiment classification accuracy by incorporating syntactic structural data [4]. ABSC models capture an effective representation of syntactic information in the hidden state layer [5], [6]. They are particularly effective at filtering out irrelevant words and aspect information [7], [8]. Although ABSC models have achieved significant performance, they generally lack the ability to mine sentiment dependencies between the aspect terms and contextual words. The significance of sentiment words in context is ignored, and this negatively affects ABSC performance.

Sentiment analysis based on machine and deep learning faces many challenges such as insufficient labelled data and poor generalization ability. Researchers have combined sentiment knowledge with supervised data to improve the model performance. For example, when extracting and analysing sentiments from texts, sentiment knowledge based on quality background dictionaries can capture fine-grained supervision information [9], [10]. Since sentiment dictionary knowledge is integrated into the modelling language, a word vector representation improves the performance of sentiment analysis tasks [11].

Aspect terms and sentiment words have recently been introduced in mask language model pretraining tasks that improves BERT performance (i.e., models based on multiple sentiment classification tasks) [12]. Moreover, graph convolutional networks can be derived using dependency trees and sentiment common sense and dependencies associated with specific aspects and terms [13]. Sentiment knowledge can be used as an effective auxiliary for identifying and explaining inherent dependencies between the aspect terms and sentiment words. In fact, in a given sentence, the sentiment polarity of a given aspect is determined by its own meaning and that of its related words. While recent research has achieved valuable performance in extracting sentiment knowledge, it still lacks fine-grained mining of aspect terms. This leads us to introduce a twofold approach, whose first objective is to introduce a set of general rules for extracting aspect term polarities. The second part incorporates a pretrained RoBERTa [14] model with optimized mask rules to better capture the dependency between the aspect terms and sentiment words.

The main contributions of this paper are as follows:

• A set of aspect mining rules based on part-of-speech tagging and sentence dependency grammar that are applied to aspects containing sentiment words. These rules consider word dependencies, compounds, and conjunctions to improve the overall accuracy.

• An aspect sentiment knowledge-adaptive pretraining model. The mask rules of the pretraining model are optimized to better capture the dependency between the aspects and sentiments.

• Extensive experiments were applied to four SemEval datasets to evaluate ASK-RoBERTa performance and the superiority of the proposed model against the baselines was demonstrated.

The rest of the paper is organized as follows. Section 2 presents ABSC-related works and sentiment knowledge, while Section 3 develops the main principles of our modelling approach. Section 4 presents the experimental setup and evaluation results. Finally, Section 5 concludes the paper and outlines future directions.