Skip to main content
SpringerLink
Log in

Survey of visual sentiment prediction for social media analysis

  • Review Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Recent years have witnessed a rapid spread of multi-modality microblogs like Twitter and Sina Weibo composed of image, text and emoticon. Visual sentiment prediction of such microblog based social media has recently attracted ever-increasing research focus with broad application prospect. In this paper, we give a systematic review of the recent advances and cutting-edge techniques for visual sentiment analysis. To this end, in this paper we review the most recent works in this topic, in which detailed comparison as well as experimental evaluation are given over the cutting-edge methods. We further reveal and discuss the future trends and potential directions for visual sentiment prediction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Picard R W. Affective Computing. MIT Technical Report. 1995

    Google Scholar 

  2. Pang B, Lee L, Vaithyanathan S. Thumbs up?: sentiment classification using machine learning techniques. In: Proceedings of the ACL- 02 Conference on Empirical Methods in Natural Language Processing. 2002, 79–86

    Chapter  Google Scholar 

  3. Wilson T, Wiebe J, Hwa R. Just how mad are you? Finding strong and weak opinion clauses. In: Proceedings of National Conference on Artificial Intelligence. 2004, 761–767

    Google Scholar 

  4. Zhou L, Li B, Gao W, Wei Z, Wong K F. Unsupervised discovery of discourse relations for eliminating intra-sentence polarity ambiguities. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2011, 162–171

    Google Scholar 

  5. Hu M Q, Liu B. Mining and summarizing customer reviews. In: Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2004, 168–177

    Google Scholar 

  6. Wilson T, Wiebe J, Hoffmann P. Recognizing contextual polarity in phrase-level sentiment analysis. In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. 2005, 347–354

    Chapter  Google Scholar 

  7. Liu JJ, Cao Y B, Lin C Y, Huang Y L, Zhou M. Low-quality product review detection in sentiment summarization. In: Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 2007

    Google Scholar 

  8. McGlohon M, Glance N S, Reiter Z. Star quality: aggregating reviews to rank products and merchants. In: Proceedings of the International Conference on Weblogs and Social Media. 2010

    Google Scholar 

  9. Joshi M, Das D, Gimpel K, Smith N A. Movie reviews and revenues: an experiment in text regression. In: Proceedings of the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. 2010, 293–296

    Google Scholar 

  10. Feldman R, Rosenfeld B, Bar-Haim R, Fresko M. The stock sonar_ sentiment analysis of stocks based on a hybrid approach. In: Proceedings of the 23rd IAAI Conference on Artificial Intelligence. 2011

    Google Scholar 

  11. Zhang W B, Skiena S. Trading strategies to exploit blog and news sentiment. In: Proceedings of the International Conference on Weblogs and Social Media. 2010

    Google Scholar 

  12. Tumasjan A, Sprenger T O, Sandner P G, Welpe I M. Predicting elections with twitter: what 140 characters reveal about political sentiment. In: Proceedings of the International Conference onWeblogs and Social Media. 2010, 178–185

    Google Scholar 

  13. Chen B, Leilei Z, Daniel K, Dongwon L. What is an sentiment about? exploring political standpoints using sentiment scoring model. In: Proceeedings of AAAI Conference on Artificial Intelligence. 2010

    Google Scholar 

  14. Yano T, Smith N A. What’s worthy of comment? content and comment volume in political blogs. In: Proceedings of the International AAAI Conference on Weblogs and Social Media. 2010

    Google Scholar 

  15. Groh G, Hauffa J. Characterizing social relations via NLP-based sentiment analysis. In: Proceedings of the 5th International AAAI Conference on Weblogs and Social Media. 2011

    Google Scholar 

  16. Li B, Feng S, Xiong W, Hu W. Scaring or pleasing: exploit emotional impact of an image. In: Proceedings of the 20th ACM International Conference on Multimedia. 2012, 1365–1366

    Chapter  Google Scholar 

  17. Jia J, Wu S, Wang X H, Hu P Y, Cai L H, Tang J. Can we understand van Gogh’s mood?: learning to infer affects from images in social networks. In: Proceedings of the 20th ACM International Conference on Multimedia. 2012, 857–860

    Chapter  Google Scholar 

  18. Vonikakis V, Winkler S. Emotion-based sequence of family photos. In: Proceedings of the 20th ACM International Conference onMultimedia. 2012, 1371–1372

    Google Scholar 

  19. Siersdorfer S, Minack E, Deng F, Hare J. Analyzing and predicting sentiment of images on the social Web. In: Proceedings of the 18th ACM International Conference on Multimedia. 2010, 715–718

    Google Scholar 

  20. Borth D, Ji R, Chen T, Breuel T, Chang S F. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: Proceedings of the 21st ACM International Conference on Multimedia. 2013, 223–232

    Chapter  Google Scholar 

  21. Yuan J, Mcdonough S, You Q, Luo J. Sentribute: image sentiment analysis from a mid-level perspective. In Proceedings of the 2nd International Workshop on Issues of Sentiment Discovery and Opinion Mining. 2013

    Google Scholar 

  22. Cao D L, Ji R R, Lin A Z, Li S Z. Visual sentiment topic model based microblog image sentiment analysis. Multimedia Tools and Applications, 2014: 1–14

    Google Scholar 

  23. Ji R R, Yao H X, Liu W, Sun X S, Tian Q. Task dependent visual codebook compression, IEEE Transactions on Image Processing, 2012, 21(4): 2282–2293

    Article  MathSciNet  Google Scholar 

  24. Ji R R, Duan L Y, Yao H X, Xie L X, Rui Y, Gao W. Learning to distribute vocabulary indexing for scalable visual search. IEEE Transactions on Multimedia, 2013, 15(1): 153–166

    Article  Google Scholar 

  25. Ji R R, Gao Y, Liu W, Xie X, Qi Tian, Li X L. When location meets social multimedia: a comprehensive survey on location-aware social multimedia. ACM Transactions on Intelligent System and Technology, 2015, 6(1): 1–18

    Article  Google Scholar 

  26. Ji R R, Yao H X, Tian Q, Xu P F, Sun X S, Liu X M, Context-aware semi-local feature detector. ACM Transactions on Intelligent System and Technology, 2012, 3(3): 44–71

    Article  Google Scholar 

  27. Ji R R, Yu F X, Chang S F. Active query sensing for mobile location search. ACM Transactions on Multimedia Computing, Communications and Applications, 2012, 3S(8): 40–61

    Google Scholar 

  28. Ji R R, Gao Y, Zhong B N, Yao H X, Tian Q. Mining flickr landmarks bymodeling reconstruction sparsity. ACM Transactions onMultimedia Computing, Communications and Applications, 2011, 7S(1)

  29. Ji R R, Duan L Y, Chen J, Huang T J, Gao W. Mining compact 3D patterns for low bit rate mobile visual search. IEEE Transactions on Image Processing, 2014, 23(7): 3099–3113

    Article  MathSciNet  Google Scholar 

  30. You Q Z, Luo J B, JinH L, Yang J C. Robust image sentiment analysis using progressively trained and domain transferred deep networks. 2015, arXiv:1509.06041

    Google Scholar 

  31. Chen T, Yu F X, Chen J, CuiY, Chen Y Y, Chang S F. Object-based visual sentiment concept analysis and application. In: Proceedings of the ACM International Conference on Multimedia. 2014, 367–376.

    Google Scholar 

  32. Li L X, Cao D L, Li S Z, Ji R R. Sentiment analysis of Chinese micro-blog based on multi-modal correlation model. In: Proceedings of the 2015 IEEE International Conference on Image Processing. 2015, 4798–4802

    Chapter  Google Scholar 

  33. Chen F H, Gao Y, CAO D L. Multimodal hypergraph learning for microblog sentiment prediction. In: Proceedings of the 2015 IEEE International Conference on Multimedia and Expo. 2015, 1–6

    Chapter  Google Scholar 

  34. Wang M, Cao D L, Li L X, Li S Z, Ji R R. Microblog sentiment analysis based on cross-media bag-of-words model. In: Proceedings of International Conference on Internet Multimedia Computing and Service. 2014

    Google Scholar 

  35. You Q Z, Luo J B, Jin H L, Yang J C. Cross-modality consistent regression for joint visual-textual sentiment analysis. In: Proceedings of the 9th ACM International Conference on Web Search and Data Mining. 2016, 13–22

    Chapter  Google Scholar 

  36. Zhong B N, Yao H X, Chen S, Ji R R, Chin T J, Wang H Z. Visual tracking via weakly supervised learning from multiple imperfect oracles. Pattern Recognition, 2014, 47(3): 1395–1410

    Article  MATH  Google Scholar 

  37. Gao Y, Wang F L, Luan H B, Chua T S. Brand data gathering from live social media streams. In: Proceedings of ACM International Conference on Multimedia Retrieval. 2014

    Google Scholar 

  38. Gao Y, Wang M, Zha Z J, Shen J L, Li X L, Wu X D. Visual-textual joint relevance learning for tag-based social image search. IEEE Transactions on Image Processing, 2013, 22(1): 363–376

    Article  MathSciNet  Google Scholar 

  39. Gao Y, Wang M, Zha Z J, Tian Q, Dai Q H, Zhang N Y. Less is more: efficient 3-D object retrieval with query view selection. IEEE Transactions on Multimedia, 2011, 13(5): 1007–1018

    Article  Google Scholar 

  40. Gao Y, Tang J H, Hong R C, Yan S C, Dai Q H, Zhang N Y, Chua T S. Camera constraint-free view-based 3-D object retrieval. IEEE Transactions on Image Processing, 2012, 21(4): 2269–2281

    Article  MathSciNet  Google Scholar 

  41. Huang Y C, Liu Q S, Zhang S T, Metaxas D N. Image retrieval via probabilistic hypergraph ranking. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. 2010, 3376–3383

    Chapter  Google Scholar 

  42. Gao Y, Wang M, Zha Z J, Shen J L, Li X L, Wu X D. Visual-textual joint relevance learning for tag-based social image search. IEEE Transactions on Image Processing, 2013, 22(1): 363–376

    Article  MathSciNet  Google Scholar 

  43. Cai Z, Cao D L, Ji R R. Video (GIF) sentiment analysis using largescale mid-level ontology. 2015, arXiv:1506.00765

    Google Scholar 

  44. Wang Y L, Wang S H, Tang J L, Huan Liu H, Li B X. Unsupervised sentiment analysis for social media images. In: Proceedings of the 24th International Conference on Artificial Intelligence. 2015, 2378–2379

    Google Scholar 

  45. Narihira T, Borth D, Yu S X, Ni K, Darrell T. Mapping images to sentiment adjective noun pairs with factorized neural nets. 2015, arXiv preprint arXiv:1511.06838

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Cognitive Science Department, Xiamen University, Xiamen, 361005, China

    Rongrong Ji, Donglin Cao, Yiyi Zhou & Fuhai Chen

  2. Fujian Key Laboratory of the Brain-like Intelligent Systems, Xiamen, 361005, China

    Rongrong Ji, Donglin Cao, Yiyi Zhou & Fuhai Chen

Authors
  1. Rongrong Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Donglin Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yiyi Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fuhai Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Donglin Cao.

Additional information

Rongrong Ji received his PhD degree in computer science from Harbin Institute of Technology, China. He was a Postdoc research fellow in Columbia University, USA from 2010 to 2013. He is the author of over 50 tier-1 journals and conferences in IJCV, TIP, ICCV, CVPR, ACM Multimedia, IJCAI, AAAI, etc. Dr. Ji is the recipient of the Best Paper Award at ACM Multimedia 2011 and Microsoft Fellowship 2007. Besides, he is the awardee of the NSFC Excellent Young Scholars Program in 2014. He is currently a professor at the Department of Cognitive Science, School of Information Science and Engineering, Xiamen University, China. His research interests include image and video search, content understanding, mobile visual search and recognition, as well as interactive human-computer interface.

Donglin Cao is currently an assistant professor at the Department of Cognitive Science, School of Information Science and Engineering, Xiamen University, China. His research interest is cross-media information retrieval.

Yiyi Zhou is currently pursuing his PhD in computer science at Xiamen University, China. He received his MS degree from Durham University, UK and BE degree from Dalian Jiaotong University, China. His research interests include social media, multimedia analysis and machine learning.

Fuhai Chen received the BS degree in information and computing science from Xiamen University (XMU), China in 2014. He is currently working towards his master degree at XMU. His research interests include machine learning and computer scene understanding.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ji, R., Cao, D., Zhou, Y. et al. Survey of visual sentiment prediction for social media analysis. Front. Comput. Sci. 10, 602–611 (2016). https://doi.org/10.1007/s11704-016-5453-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-016-5453-2

Keywords

Search

Navigation