short-paper

Joint Visual-Textual Sentiment Analysis with Deep Neural Networks

Authors:

Jianchao YangAuthors Info & Claims

MM '15: Proceedings of the 23rd ACM international conference on Multimedia

Pages 1071 - 1074

https://doi.org/10.1145/2733373.2806284

Published: 13 October 2015 Publication History

Abstract

Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so on. Recently, social media users are increasingly using additional images and videos to express their opinions and share their experiences. Sentiment analysis of such large-scale textual and visual content can help better extract user sentiments toward events or topics. Motivated by the needs to leverage large-scale social multimedia content for sentiment analysis, we utilize both the state-of-the-art visual and textual sentiment analysis techniques for joint visual-textual sentiment analysis. We first fine-tune a convolutional neural network (CNN) for image sentiment analysis and train a paragraph vector model for textual sentiment analysis. We have conducted extensive experiments on both machine weakly labeled and manually labeled image tweets. The results show that joint visual-textual features can achieve the state-of-the-art performance than textual and visual sentiment analysis algorithms alone.

References

[1]

D. Borth, T. Chen, R. Ji, and S.-F. Chang. Sentibank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content. In ACM MM, pages 459--460, 2013.

Digital Library

[2]

D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In ACM MM, pages 223--232. ACM, 2013.

Digital Library

[3]

D. Cao, R. Ji, D. Lin, and S. Li. A cross-media public sentiment analysis system for microblog. Multimedia Systems, pages 1--8, 2014.

[4]

D. C. Cireşan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber. Flexible, high performance convolutional neural networks for image classification. In IJCAI, pages 1237--1242, 2011.

Digital Library

[5]

X. Hu, J. Tang, H. Gao, and H. Liu. Unsupervised sentiment analysis with emotional signals. In WWW, pages 607--618, 2013.

Digital Library

[6]

C. Hutto and E. Gilbert. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In ICWSM, 2014.

[7]

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.

[8]

D. Joshi, R. Datta, E. Fedorovskaya, Q.-T. Luong, J. Z. Wang, J. Li, and J. Luo. Aesthetics and emotions in images. IEEE Signal Processing Magazine, 28(5):94--115, 2011.

[9]

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097--1105, 2012.

Digital Library

[10]

Q. Le and T. Mikolov. Distributed representations of sentences and documents. In ICML, 2014.

Digital Library

[11]

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278--2324, 1998.

[12]

T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111--3119, 2013.

Digital Library

[13]

L.-P. Morency, R. Mihalcea, and P. Doshi. Towards multimodal sentiment analysis: Harvesting opinions from the web. In ICMI, pages 169--176, 2011.

Digital Library

[14]

B. O'Connor, R. Balasubramanyan, B. R. Routledge, and N. A. Smith. From tweets to polls: Linking text sentiment to public opinion time series. ICWSM, 11:122--129, 2010.

[15]

S. Siersdorfer, E. Minack, F. Deng, and J. Hare. Analyzing and predicting sentiment of images on the social web. In ACM MM, pages 715--718. ACM, 2010.

Digital Library

[16]

A. Tumasjan, T. O. Sprenger, P. G. Sandner, and I. M. Welpe. Predicting elections with twitter: What 140 characters reveal about political sentiment. ICWSM, 10:178--185, 2010.

[17]

M. Wang, D. Cao, L. Li, S. Li, and R. Ji. Microblog sentiment analysis based on cross-media bag-of-words model. In ICIMCS, pages 76:76--76:80. ACM, 2014.

Digital Library

[18]

Q. You, J. Luo, H. Jin, and J. Yang. Robust image sentiment analysis using progressively trained and domain transferred deep networks. In AAAI, 2015.

Digital Library

[19]

J. Yuan, S. Mcdonough, Q. You, and J. Luo. Sentribute: image sentiment analysis from a mid-level perspective. In WISDOM, page 10, 2013.

Digital Library

Cited By

Li GZeng XChen CZhou L(2024)Multi-attention Fusion for Multimodal Sentiment ClassificationProceedings of 2024 ACM ICMR Workshop on Multimodal Video Retrieval10.1145/3664524.3675360(1-7)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3664524.3675360
Phan HNguyen N(2024)A Fuzzy Graph Convolutional Network Model for Sentence-Level Sentiment AnalysisIEEE Transactions on Fuzzy Systems10.1109/TFUZZ.2024.336469432:5(2953-2965)Online publication date: 12-Feb-2024
https://dl.acm.org/doi/10.1109/TFUZZ.2024.3364694
Jahanbin KChahooki M(2024)A Hybrid Deep Implicit Neural Model for Sentiment Analysis via Transfer LearningIEEE Access10.1109/ACCESS.2024.342581912(131468-131486)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3425819
Show More Cited By

Index Terms

Joint Visual-Textual Sentiment Analysis with Deep Neural Networks
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Scene understanding
  2. Machine learning

Recommendations

Robust Visual-Textual Sentiment Analysis: When Attention meets Tree-structured Recursive Neural Networks
MM '16: Proceedings of the 24th ACM international conference on Multimedia

Sentiment analysis is crucial for extracting social signals from social media content. Due to huge variation in social media, the performance of sentiment classifiers using single modality (visual or textual) still lags behind satisfaction. In this ...
Cross-modality Consistent Regression for Joint Visual-Textual Sentiment Analysis of Social Multimedia
WSDM '16: Proceedings of the Ninth ACM International Conference on Web Search and Data Mining

Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so ...
Joint sentiment/topic model for sentiment analysis
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

Sentiment analysis or opinion mining aims to use automated tools to detect subjective information such as opinions, attitudes, and feelings expressed in text. This paper proposes a novel probabilistic modeling framework based on Latent Dirichlet ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '15: Proceedings of the 23rd ACM international conference on Multimedia

October 2015

1402 pages

ISBN:9781450334594

DOI:10.1145/2733373

General Chairs:
Xiaofang Zhou
The University of Queensland, Australia
,
Alan F. Smeaton
Dublin City University, Ireland
,
Qi Tian
The University of Texas at San Antonio, USA
,
Program Chairs:
Dick C.A. Bulterman
FXPAL, USA
,
Heng Tao Shen
The University of Queensland, Australia
,
Ketan Mayer-Patel
The University of North Carolina, USA
,
Shuicheng Yan
National University of Singapore, Singapore

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 October 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

MM '15

Sponsor:

SIGMM

MM '15: ACM Multimedia Conference

October 26 - 30, 2015

Brisbane, Australia

Acceptance Rates

MM '15 Paper Acceptance Rate 56 of 252 submissions, 22%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

67
Total Citations
View Citations
1,269
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)7

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li GZeng XChen CZhou L(2024)Multi-attention Fusion for Multimodal Sentiment ClassificationProceedings of 2024 ACM ICMR Workshop on Multimodal Video Retrieval10.1145/3664524.3675360(1-7)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3664524.3675360
Phan HNguyen N(2024)A Fuzzy Graph Convolutional Network Model for Sentence-Level Sentiment AnalysisIEEE Transactions on Fuzzy Systems10.1109/TFUZZ.2024.336469432:5(2953-2965)Online publication date: 12-Feb-2024
https://dl.acm.org/doi/10.1109/TFUZZ.2024.3364694
Jahanbin KChahooki M(2024)A Hybrid Deep Implicit Neural Model for Sentiment Analysis via Transfer LearningIEEE Access10.1109/ACCESS.2024.342581912(131468-131486)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3425819
Wang JYang YLiu KXie ZZhang FLi T(2024)CiteNet: Cross-modal incongruity perception network for multimodal sentiment predictionKnowledge-Based Systems10.1016/j.knosys.2024.111848295(111848)Online publication date: Jul-2024
https://doi.org/10.1016/j.knosys.2024.111848
Wang LPeng JZheng CZhao TZhu L(2024)A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learningInformation Processing & Management10.1016/j.ipm.2024.10367561:3(103675)Online publication date: May-2024
https://doi.org/10.1016/j.ipm.2024.103675
Lu QSun XGao ZLong YFeng JZhang H(2024)Coordinated-joint translation fusion framework with sentiment-interactive graph convolutional networks for multimodal sentiment analysisInformation Processing & Management10.1016/j.ipm.2023.10353861:1(103538)Online publication date: Jan-2024
https://doi.org/10.1016/j.ipm.2023.103538
Madan AParikh JJain RGupta AChaudhary AChadha DShubham (2023)Enhanced deep learning network for emotion recognition from GIFIntelligent Decision Technologies10.3233/IDT-22015817:2(415-433)Online publication date: 15-May-2023
https://doi.org/10.3233/IDT-220158
Das RSingh T(2023)Multimodal Sentiment Analysis: A Survey of Methods, Trends, and ChallengesACM Computing Surveys10.1145/358607555:13s(1-38)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3586075
Das RSingh T(2023)Image–Text Multimodal Sentiment Analysis Framework of Assamese News Articles Using Late FusionACM Transactions on Asian and Low-Resource Language Information Processing10.1145/358486122:6(1-30)Online publication date: 17-Feb-2023
https://dl.acm.org/doi/10.1145/3584861
Yin CZhang SZeng Q(2023)Hybrid Representation and Decision Fusion towards Visual-textual SentimentACM Transactions on Intelligent Systems and Technology10.1145/358307614:3(1-17)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1145/3583076
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten