MASAD: A large-scale dataset for multimodal aspect-based sentiment analysis

doi:10.1016/j.neucom.2021.05.040

Neurocomputing

Volume 455, 30 September 2021, Pages 47-58

https://doi.org/10.1016/j.neucom.2021.05.040 Get rights and content

Abstract

Aspect-based sentiment analysis has obtained great success in recent years. Most of the existing work focuses on determining the sentiment polarity of the given aspect according to the given text, while little attention has been paid to the visual information as well as multimodality content for aspect-based sentiment analysis. Multimodal content is becoming increasingly popular in mainstream online social platforms and can help better extract user sentiments toward a given aspect. There are only few studies focusing on this new task: Multimodal Aspect-based Sentiment Analysis (MASA), which performs aspect-based sentiment analysis by integrating both texts and images. In this paper, we propose a mutimodal interaction model for MASA to learn the relationship among the text, image and aspect via interaction layers and adversarial training. Additionally, we build a new large-scale dataset for this task, named MASAD, which involves seven domains and 57 aspect categories with 38 k image-text pairs. Extensive experiments have been conducted on the proposed dataset to provide several baselines for this task. Though our models obtain significant improvement for this task, empirical results show that MASA is more challenging than textual aspect-based sentiment analysis, which indicates that MASA remains a challenging open problem and requires further efforts.

Section snippets

Introduction and motivation

Nowadays, as a major social media platform for expressing experiences and sharing the opinion about service, products, and travel, the Internet provides extensive contents of users’ opinion and sentiment about rich topics [1]. This information is expressed in multiple formats, such as reviews, tags, browser behavior, and shared media objects. The analysis of such information plays an essential role in the area of opinion mining, affective computing, and sentiment analysis. It can predict human

Related work

We divide the relevant work into two groups: aspect-based sentiment analysis and multimodal sentiment analysis. There are a large number of studies on the topic of aspect-based sentiment classification [4], [5], [7], [8], [12] and multimodal sentiment analysis [2], [13]. Here we mainly review the work that is most related to our research.

Task description

In this paper, we focus on the new task, namely multimodal aspect-level sentiment analysis (MASA), which aims to perform the aspect-based sentiment analysis [3], [5] based the textual and visual information. This task consists of two subtasks: Aspect Extraction (AE) and Aspect Polarity Prediction (AP). The goal of AE is to extract the aspect (e.g., dog, car) in the text-image pair. Then AP aims to judge the sentiment polarity for the given aspect via the text-image pair. Formally, the

Data collection

We collect and label our dataset based on the publicly available Visual Sentiment Ontology (VSO) dataset² [44] and Multilingual Visual Sentiment Ontology (MVSO) dataset³ [2], which are the largest available datasets for visual sentiment analysis. VSO dataset was collected from Flickr.⁴ We select Flickr since there is some existing work of multimedia research using it [44], [2], and to be specific,

Methods

In this section, we aim to find a straightforward architecture that provides good performance for the MASA task. In particular, we propose a multimodal aspect extraction (MMAE) method and a multimodal aspect polarity prediction (MMAP) method for aspect extraction (AE) and aspect polarity prediction (AP) respectively. The frameworks of these models are shown in Fig. 3. We first use text encoder and image encoder to extract the text and image representation via the state-of-the-art textual and

Experimental results and analysis

In this section, we conduct extensive experimental results on two subtasks, AE and AP on our MASAD dataset to evaluate our proposed methods and provide several baselines for this task. We first present the implementation details of our models. Then we discuss the experimental results of AE and AP. To be specific, we compare the results using text only, visual only, and their combination over seven domains of MASAD with various metrics. In addition, to verify the effectiveness of our model, we

Conclusion and future work

In this paper, we focus on the new task named multimodal aspect-based sentiment analysis (MASA), which performs aspect-based sentiment analysis based on the multimodal content on social media platforms. We propose a multimodal interaction model for this task, which learns the interaction between the aspect, text, and image effectively. Then we present and release a large-scale multimodal aspect-based sentiment analysis dataset named MASAD. The MASAD involves 57 aspects with seven domains and

CRediT authorship contribution statement

Jie Zhou: Writing - original draft, Conceptualization, Methodology, Software. Jiabao Zhao: Data curation, Methodology, Software. Jimmy Xiangji Huang: Visualization, Investigation, Supervision. Qinmin Vivian Hu: Methodology, Investigation. Liang He: Methodology, Validation, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

We greatly appreciate anonymous reviewers and the associate editor for their valuable and high quality comments that greatly helped to improve the quality of this article. This research is funded by the Science and Technology Commission of Shanghai Municipality (19511120200). This research is also supported by the Natural Sciences and Engineering Research Council (NSERC) of Canada and the York Research Chairs (YRC) program. Jie Zhou and Jiabao Zhao are first co-authors with equal contribution.

Jie Zhou is currently working toward the PhD degree at the Department of Computer Science and Technology, East China Normal University, China. His research interest includes sentiment analysis, aspect-level sentiment classification, retrieval model, and neural networks. He was awarded a scholarship from the China Scholarship Council, and received Top-3 in the KDD CUP Competition several times. Since 2016, he has published more than 10 referred papers in international conferences or journals,

References (54)

J. Zhou et al.
Position-aware hierarchical transfer model for aspect-level sentiment classification
Inf. Sci.
(2020)
J. Zhou et al.
Sk-gcn: Modeling syntax and knowledge via graph convolutional network for aspect-level sentiment classification
Knowl.-Based Syst.
(2020)
S. Poria et al.
Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis
Neurocomputing
(2017)
A. Graves et al.
Framewise phoneme classification with bidirectional lstm and other neural network architectures
Neural Networks
(2005)
Y. Lu et al.
Opinion integration through semi-supervised topic modeling
Proceedings of WWW
(2008)
B. Jou et al.
Visual affect around the world: a large-scale multilingual visual sentiment ontology
B. Liu
Sentiment analysis and opinion mining
Synthesis Lect Human Lang Technol
(2012)
J. Zhou et al.
Deep learning for aspect-level sentiment classification: Survey, vision and challenges
IEEE Access
(2019)
S. Manandhar
Semeval-2014 task 4: Aspect based sentiment analysis
Proceedings of SemEval
(2014)
M. Mitchell et al.
Open domain targeted sentiment
Proceedings of EMNLP
(2013)

M. Pontiki et al.

Semeval-2015 task 12: Aspect based sentiment analysis

Proceedings of SemEval

(2015)

M. Pontiki et al.

Semeval-2016 task 5: Aspect based sentiment analysis

Proceedings of SemEval

(2016)

M. Saeidi et al.

Targeted aspect based sentiment analysis dataset for urban neighbourhoods

Proceedings of COLING

(2016)

N. Xu et al.

Multi-interactive memory network for aspect based multimodal sentiment analysis

(2019)

J. Yu et al.

Adapting BERT for target-oriented multimodal sentiment classification

(2019)

J. Zhou et al.

Modeling multi-aspect relationship with joint learning for aspect-level sentiment classification

S. Poria et al.

Context-dependent sentiment analysis in user-generated videos

Proceedings of ACL

(2017)

J. Zhou et al.

A sentiment-aware pre-trained model for cross-domain sentiment analysis

S. Kiritchenko et al.

Nrc-canada-2014 Detecting aspects and sentiment in customer reviews

Proceedings of SemEval

(2014)

D.-T. Vo et al.

Target-dependent twitter sentiment classification with rich automatic features

Proceedings of IJCAI

(2015)

D. Tang et al.

Effective lstms for target-dependent sentiment classification

Proceedings of COLING

(2016)

R. He et al.

Exploiting document knowledge for aspect-level sentiment classification

Proceedings of ACL

(2018)

X. Li et al.

Deep multi-task learning for aspect term extraction with memory interaction

Proceedings of EMNLP

(2017)

C. Fan et al.

Convolution-based memory network for aspect-based sentiment analysis

J. Liu, Y. Zhang, Attention modeling for targeted sentiment, in: Proceedings of EACL, volume 2, 2017, pp....

S. Wang et al.

Target-sensitive memory networks for aspect sentiment classification

Proceedings of ACL

(2018)

W. Xue, T. Li, Aspect based sentiment analysis with gated convolutional networks, in: Proceedings of ACL, Association...

Cited by (36)

An empirical study of Multimodal Entity-Based Sentiment Analysis with ChatGPT: Improving in-context learning via entity-aware contrastive learning
2024, Information Processing and Management
Multimodal Entity-Based Sentiment Analysis (MEBSA) is an emerging task within sentiment analysis, with the objective of simultaneously detecting entity, sentiment, and entity category from multimodal inputs. Despite achieving promising results, most existing MEBSA studies requires a substantial quantity of annotated data. The acquisition of such data is both costly and time-intensive in practical applications. To alleviate the reliance on annotated data, this work explores the potential of in-context learning (ICL) with a representative large language model, ChatGPT, for the MEBSA task. Specifically, we develop a general ICL framework with task instructions for zero-shot learning, followed by extending it to few-shot learning by incorporating a few demonstration samples in the prompt. To enhance the performance of the ICL framework in the few-shot learning setting, we further develop an Entity-Aware Contrastive Learning model to effectively retrieve demonstration samples that are similar to each test sample. Experiments demonstrate that our developed ICL framework exhibits superior performance over other baseline ICL methods, and is comparable to or even outperforms many existing fine-tuned methods on four MEBSA subtasks.
Multi-grained fusion network with self-distillation for aspect-based multimodal sentiment analysis
2024, Knowledge-Based Systems
Aspect-based multimodal sentiment analysis (ABMSA) is an important branch of multimodal sentiment analysis. The goal of ABMSA is to use multimodal information to infer users’ sentiment polarity toward the targeted aspect for supporting corresponding decision-making. The existing ABMSA methods usually focus on exploring aspect-aware fine-grained interactions and demonstrate the benefits of integrating multimodal information. However, such approaches still suffer from the following limitations: (1) coarse-grained semantic information extraction is ignored, (2) the relevance between image and aspect is ignored, and (3) model-agnostic techniques to improve multimodal representations are not employed. To address these limitations, we propose a novel multimodal approach named multi-grained fusion network with self-distillation (MGFN-SD). This approach includes unimodal feature extraction, multi-grained representation learning, and self-distillation-based sentiment prediction. The multi-grained representation learning module extracts fine- and coarse-grained interactions based on the aspect–image relevance computation and the similarity calculation for dynamically filtering potential noise brought by the original image. In the sentiment prediction module, self-distillation is employed to transfer knowledge from both hard and soft labels to supervise the training process of each student classifier for improving the quality of multimodal representations. The experimental results on three benchmark datasets indicate the superiority, rationality, and robustness of MGFN-SD.
Atlantis: Aesthetic-oriented multiple granularities fusion network for joint multimodal aspect-based sentiment analysis
2024, Information Fusion
Joint Multi-modal Aspect-based Sentiment Analysis (JMASA) is a challenging task that seeks to identify all aspect-sentiment pairs from multimodal data. Current JMASA studies are insufficient in bridging the representational gap between textual and visual modalities. Additionally, they largely emphasize image feature extraction, neglecting the exploration of image presentation forms, like aesthetic characteristics. In this paper, we propose an Aesthetic-oriented Multiple Granularities Fusion Network for JMASA, termed Atlantis. This trident-shaped framework comprises three branches: Textual-vision Alignment Aspect-sentiment Extraction, Sentiment-aware Image Aesthetic Assessment, and Aesthetic-aware JMASA. Notably, the first two branches function as auxiliary learning tasks, with Textual-vision Alignment Aspect-sentiment Extraction aimed at bridging the representational gap between modalities, and Sentiment-aware Image Aesthetic Assessment dedicated to understanding the aesthetic attributes of images. Concurrently, the Aesthetic-aware JMASA dynamically integrates varied granular features from both branches to perform JMASA. To the best of our knowledge, this is the first aesthetic-oriented approach in the present field. Experimental results on two public datasets verify that Atlantis outperforms a series of prior strong methodologies and achieves a new state-of-the-art (SOTA) performance. The enhancement highlights Atlantis’s advanced capability in accurately identifying aspect-sentiment pairs with aesthetic features.
AMIFN: Aspect-guided multi-view interactions and fusion network for multimodal aspect-based sentiment analysis
2024, Neurocomputing
Aspect-based sentiment analysis (ABSA), which aims to analyze users’ sentiment towards the targeted aspect, has recently gained increasing attention due to its importance in supporting corresponding decision-makings in various tasks. Most existing ABSA studies primarily depend on only textual modality, but ignore the fact that in many cases the targeted aspect doesn’t appear in the sentence. Thus, multimodal ABSA is expected to alleviate this dilemma. However, most existing MABSA approaches still suffer from the following limitations: (1) ignoring the possible aspect-image irrelevant issue; (2) ignoring the coarse-grained interaction between the sentence and its associated image; (3) failing to simultaneously leverage multiple types of useful knowledge information. To address these issues, we propose an aspect-guided multi-view interactions and fusion network (AMIFN) for MABSA. Specifically, we utilize the multi-head attention mechanism to generate aspect-guided textual representation, which is used as the extended aspect semantic for guiding the subsequent aspect-related interactions. When exploring aspect-guided visual representation, we employ the image gate to dynamically filter potential noise introduced by the associated image to generate the final image representation. Meanwhile, the coarse-grained sentence-image interaction, which contains context and semantics information, and the syntactic dependencies, are leveraged for graph construction to obtain aspect-guided text-image interaction representations. Finally, the extracted multi-view interaction representations are integrated for sentiment classification. Extensive experimental results on three multimodal benchmark datasets demonstrate the superiority and rationality of AMIFN.
Progress, achievements, and challenges in multimodal sentiment analysis using deep learning: A survey
2024, Applied Soft Computing
Sentiment analysis is a computational technique that analyses the subjective information conveyed within a given expression. This encompasses appraisals, opinions, attitudes or emotions towards a particular subject, individual, or entity. Conventional sentiment analysis solely considers the text modality and derives sentiment by identifying the semantic relationship between words within a sentence. Despite this, certain expressions, such as exaggeration, sarcasm and humor, pose a challenge for automated detection when conveyed only through text. Multimodal sentiment analysis incorporates various forms of data, such as visual and acoustic cues, in addition to text. By utilizing fusion analysis, this approach can more precisely determine the implied sentiment polarity, which includes positive, neutral, and negative sentiments. Thus, the recent advancements in deep learning have boosted the domain of multimodal sentiment analysis to new heights. The research community has also shown significant interest in this topic due to its potential for both practical application and educational research. In light of this fact, this paper aims to present a thorough analysis of recent ground-breaking research studies conducted in multimodal sentiment analysis, which employs deep learning models across various modalities such as text, audio, image, and video. Furthermore, the article dives into a discussion of the multiple categories of multimodal data, diverse domains in which multimodal sentiment analysis can be applied, a range of operations that are integral to multimodal sentiment analysis, deep learning architectures, a variety of fusion methods, challenges associated with multimodal sentiment analysis, and the benchmark datasets in addition to the state-of-the-art approaches. The ultimate goal of this survey is to indicate the success of deep learning architectures in tackling the complexities associated with multimodal sentiment analysis.
Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis
2023, Information Processing and Management
Multi-modal Aspect-based Sentiment Analysis (MABSA) aims to forecast the polarity of sentiment concerning aspects within a given sentence based on the correlation between the sentence and its accompanying image. Comprehending multi-modal sentiment expression requires strong cross-modal alignment and fusion ability. Previous state-of-the-art (SOTA) models fail to explicitly align valuable visual clues with aspect and sentiment information in textual representations and overlook the utilization of syntactic dependency information in the accompanying text modality. We present CoolNet (Cross-modal Fine-grained Alignment and Fusion Network) to boost the performance of visual-language models in seamlessly integrating vision and language information. Specifically, CoolNet starts by transforming an image into a textual caption and a graph structure, then dynamically aligns the semantic and syntactic information from both the input sentence and the generated caption, as well as models the object-level visual features. Finally, a cross-modal transformer is employed to fuse and model the inter-modality dynamics.This network boasts advanced cross-modal fine-grained alignment and fusion capabilities. On standard benchmarks such as Twitter-2015 and Twitter-2017, CoolNet consistently outperforms state-of-the-art algorithm FITE with notable improvements in accuracy and Macro-F1 values. Specifically, we observe an improvement in accuracy and Macro-F1 values by 1.43% and 1.38% for Twitter-2015, and 0.74% and 0.88% for Twitter-2017, respectively, demonstrating the superiority of our CoolNet architecture.

View all citing articles on Scopus

Jiabao Zhao is currently working toward the PhD degree at the Department of Computer Science and Technology, East China Normal University, China. Her research interest includes few-shot learning, meta-learning, and neural networks. Since 2017, she has published several referred papers in international conferences or journals, such as ICME.

Jimmy Xiangji Huang received the Ph.D degree in information science from the City, University of London and was then a post doctoral fellow in the School of Computer Science, University of Waterloo, Canada. He is now a York Research Chair professor and the director of Information Retrieval & Knowledge Management Research Lab (IRLab), York University. He joined York University as an assistant professor, in 2003. He was awarded tenure and promoted to full professor in 2006 and 2011 respectively. He received the Deans Award for Outstanding Research in 2006, an Early Researcher Award, formerly the Premiers Research Excellence Award in 2007, the Petro Canada Young Innovators Award in 2008, the SHARCNET Research Fellowship Award in 2009, the Best Paper Award at the 32nd European Conference on Information Retrieval (ECIR 2010), United Kingdom, and LA&PS Award for Distinction in Research, Creativity, and Scholarship (established researcher) in 2015. Since 2003, he has published more than 230 refereed papers in journals (such as the ACM Transactions on Information Systems, the Journal of American Society for Information Science and Technology, Information Processing & Management, the IEEE Transactions on Knowledge and Data Engineering, Information Sciences, Information Retrieval, BMC Bioinformatics, BMC Genomics and BMC Medical Genomics), book chapters, and international conference proceedings (such as ACM SIGIR, ACM CIKM, KDD, ACL, COLING, IEEE ICDM, IJCAI and AAAI). He is a senior member of the IEEE & ACM Distinguished Scientist.

Qinmin Vivian Hu received her Ph.D. degree in Computer Science from York University, Toronto, Canada. She is now an Assistant Professor in the Department of Computer Science, Ryerson University, Toronto, Canada. Before that, she was an Associate Professor at East China Normal University, Shanghai, China, and a Postdoctoral Fellow at the MRI research facility, the Wayne State University, USA. She won the National NSERC Postdoctoral Fellowship as one of best Ph.D fellows in Canada in 2013. She has published more than 30 referred papers in top tier journals (i.e. IEEE Transactions on Knowledge and Data Engineering, ACM Transactions on Intelligent Systems and Technology) and conferences (i.e. AAAI, ACM SIGIR).

Liang He received his bachelor’s degree and PhD degree from the Department of Computer Science and Technology, East China Normal University, China. He is now a professor and the director of the Department of Computer Science and Technology, East China Normal University. His current research interest includes knowledge processing, user behavior analysis, and context-aware computing. He has been awarded the Star of the Talent in Shanghai. He is also a council member of the Shanghai Computer Society, a member of the Academic Committee, the director of the technical committee of Shanghai Engineering Research Center of Intelligent Service Robot, and a technology foresight expert of the Shanghai Science and Technology in focus areas. He has received the Shanghai Science and Technology Progress Award for 5 times, and won the first prize in 2013 and the second prize in 2015. He has obtained more than 10 patents, and published 2 monographs and more than 70 refereed papers in national and international journals and conference proceedings.

View full text

MASAD: A large-scale dataset for multimodal aspect-based sentiment analysis

Abstract

Section snippets

Introduction and motivation

Related work

Task description

Data collection

Methods

Experimental results and analysis

Conclusion and future work

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgements

Inf. Sci.

Knowl.-Based Syst.

Neurocomputing

Neural Networks

Opinion integration through semi-supervised topic modeling

Proceedings of WWW

Visual affect around the world: a large-scale multilingual visual sentiment ontology

Sentiment analysis and opinion mining

Synthesis Lect Human Lang Technol

Deep learning for aspect-level sentiment classification: Survey, vision and challenges

IEEE Access

Semeval-2014 task 4: Aspect based sentiment analysis

Proceedings of SemEval

Open domain targeted sentiment

Proceedings of EMNLP

Semeval-2015 task 12: Aspect based sentiment analysis

Proceedings of SemEval

Semeval-2016 task 5: Aspect based sentiment analysis

Proceedings of SemEval

Targeted aspect based sentiment analysis dataset for urban neighbourhoods

Proceedings of COLING

Multi-interactive memory network for aspect based multimodal sentiment analysis

Adapting BERT for target-oriented multimodal sentiment classification

Modeling multi-aspect relationship with joint learning for aspect-level sentiment classification

Context-dependent sentiment analysis in user-generated videos

Proceedings of ACL

A sentiment-aware pre-trained model for cross-domain sentiment analysis

Nrc-canada-2014 Detecting aspects and sentiment in customer reviews

Proceedings of SemEval

Target-dependent twitter sentiment classification with rich automatic features

Proceedings of IJCAI

Effective lstms for target-dependent sentiment classification

Proceedings of COLING

Exploiting document knowledge for aspect-level sentiment classification

Proceedings of ACL

Deep multi-task learning for aspect term extraction with memory interaction

Proceedings of EMNLP

Convolution-based memory network for aspect-based sentiment analysis

Target-sensitive memory networks for aspect sentiment classification

Proceedings of ACL