Skip to main content
SpringerLink
Log in

Visual Sentiment Analysis by Combining Global and Local Information

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

With the development of visual social networks, the sentiment analysis of images has quickly emerged for opinion mining. Based on the observation that the sentiments conveyed by some images are related to salient objects in them, we propose a scheme for visual sentiment analysis that combines global and local information. First, the sentiment is predicted from the entire images. Second, it is judged whether there are salient objects in an image or not. If there are, sub-images are cropped from the entire image based on the detection window of the salient objects. Moreover, a CNN model is trained for the set of sub-images. Predictions of sentiments from entire images and sub-images are then fused together to obtain the final results. If no salient object is detected in the images, the sentiment predicted directly from entire images is used as the final result. The compared experimental results show that the proposed approach is superior to state-of-the-art algorithms. It also demonstrates that reasonably utilizing the local information could improve the performance for visual sentiment analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Zhao S, Yao H, Yue G, Ding G, Chua TS (2018) Predicting personalized image emotion perceptions in social networks. IEEE Trans Affect Comput 9:526–540. https://doi.org/10.1109/TAFFC.2016.2628787

    Article  Google Scholar 

  2. Yang J, Sun M, Sun X (2017) Learning visual sentiment distributions via augmented conditional probability neural network. In: AAAI Conference on Artificial Intelligence, North America, February 2017. https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14506. Accessed 03 Apr 2019

  3. Kosti R, Alvarez JM, Recasens A, Lapedriza A (2017) Emotion recognition in context. In: IEEE conference on computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2017.212

  4. Zhao S, Yue G, Ding G, Chua TS (2017) Real-time multimedia social event detection in microblog. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2017.2762344

    Article  Google Scholar 

  5. Zhao S, Ding G, Huang Q, Chua TS, Bjorn WS, Kurt K (2018) Affective image content analysis: a comprehensive survey. In: IJCAI

  6. Joshi D, Datta R, Fedorovskaya E, Luong QT (2011) Aesthetics and emotions in images. IEEE Signal Process Mag 28:94–115. https://doi.org/10.1109/MSP.2011.941851

    Article  Google Scholar 

  7. Zhao S, Gao Y, Jiang X, Yao H, Chua TS, Sun X (2014) Exploring principles-of-art features for image emotion recognition. In: ACM int. conf. multimedia, pp 47–56. https://doi.org/10.1145/2647868.2654930

  8. Zhang H, Augilius E, Honkela T, Laaksonen J, Gamper H, Alene H (2011) Analyzing emotional semantics of abstract art using low-level image features. In: International conference on advances in intelligent data analysis X, LNCS, vol 7014, pp 413–423. https://doi.org/10.1007/978-3-642-24800-9_38

  9. Li B, Feng S, Xiong W, Hu W (2012) Scaring or pleasing: exploit emotional impact of an image. In: ACM international conference on multimedia, pp 1365–1366. https://doi.org/10.1145/2393347.2396487

  10. Liu N, Dellandra E, Tellez B, Chen L (2011) Associating textual features with visual ones to improve affective image classification. Springer, Berlin, pp 229–238. https://doi.org/10.1007/978-3-642-24600-5_23

    Book  Google Scholar 

  11. Lu X, Suryanarayan P, Li J, Newman MG, Wang JZ (2012) On shape and the computability of emotions. In: ACM international conference on multimedia, pp 229–238. https://doi.org/10.1145/2393347.2393384

  12. Yuan J, Mcdonough S, You Q, Luo J (2013) Sentribute: image sentiment analysis from a mid-level perspective. In: Proceedings of the second international workshop on issues of sentiment discovery and opinion mining. https://doi.org/10.1145/2502069.2502079

  13. Rao T, Xu M, Liu H, Wang J, Burnett I (2016) Multi-scale blocks based image emotion classification using multiple instance learning. IEEE Int Conf Image Process. https://doi.org/10.1109/ICIP.2016.7532434

    Article  Google Scholar 

  14. Borth D, Ji R, Chen T, Breuel T, Chang SF (2013) Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: 21st ACM international conference on Multimedia, pp 223–232. https://doi.org/10.1145/2502081.2502282

  15. Zhao S, Yao H, Yue G, Ji R, Ding G (2017) Continuous probability distribution prediction of image emotions via multi-task shared sparse regression. IEEE Trans Multimed 19:632–645. https://doi.org/10.1109/TMM.2016.2617741

    Article  Google Scholar 

  16. Zhao S, Ding G, Yue G, Xin Z, Tang Y, Han J, Yao H, Huang Q (2018) Discrete probability distribution prediction of image emotions with shared sparse learning. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2018.2818685

    Article  Google Scholar 

  17. Xu C, Cetintas S, Lee KC, Li LJ (2014) Visual sentiment prediction with deep convolutional neural networks. Eprint ArXiv arXiv:1411.5731

  18. You Q, Luo J, Jin H, Yang J (2015) Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: Twenty-ninth AAAI conference on artificial intelligence, pp 381–388. arXiv:1509.06041

  19. Wu L, Liu S, Jian M, Luo J, Zhang X, Qi M (2018) Reducing noisy labels in weakly labeled data for visual sentiment analysis. In: IEEE international conference on image processing, pp 1322–1326. https://doi.org/10.1109/ICIP.2017.8296496

  20. Zheng H, Chen T, Luo J (2016) When saliency meets sentiment: understanding how image content invokes emotion and sentiment, pp 630–634. arXiv:1611.04636

  21. Fan S, Jiang M, Shen Z et al (2017) The role of visual attention in sentiment prediction. In: Proceedings of the 2017 ACM on multimedia conference. ACM, pp 217–225. https://doi.org/10.1145/3123266.3123445

  22. Fan S, Shen Z, Jiang M, Koenig B, Xu J, Kankanhalli M, Zhao Q (2018) Emotional attention: a study of image sentiment and visual attention. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  23. You Q, Jin H, Luo J (2017) Visual sentiment analysis by attending on local image regions. In: AAAI conf. artif. intell, pp 231–237

  24. Yang J, She D, Sun M, Cheng MM, Rosin P, Wang L (2018) Visual sentiment prediction based on automatic discovery of affective regions. IEEE Trans Multimed 20:2513–2525. https://doi.org/10.1109/TMM.2018.2803520

    Article  Google Scholar 

  25. Zhang J, Sclaroff S, Lin Z, Shen X, Price B, Mech R (2016) Unconstrained salient object detection via proposal subset optimization. In: Computer vision and pattern recognition, pp 5733–5742. https://doi.org/10.1109/CVPR.2016.618

  26. You Q, Luo J, Jin H, Yang J (2016) Building a large scale dataset for image emotion recognition: the fine print and the benchmark. In: AAAI conf. artif. intel, pp 308–314. arXiv:1605.02677

  27. Simonyan K, Zisserman (2014) A very deep convolutional networks for large-scale image recognition. Comput Sci. arXiv:1409.1556v6

  28. Peng KC, Sadovnik A, Gallagher A, Chen T (2016) Where do emotions come from? Predicting the emotion stimuli map. In: IEEE international conference on image processing. https://doi.org/10.1109/ICIP.2016.7532430

  29. Chen T, Borth D, Darrell T, Chang SF (2014) Deep SentiBank: visual sentiment concept classification with deep convolutional neural networks. arXiv:1410.8586

Download references

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Beijing University of Technology, Beijing, China

    Lifang Wu, Mingchao Qi, Meng Jian & Heng Zhang

Authors
  1. Lifang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingchao Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meng Jian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Heng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Meng Jian.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was supported by National Natural Science Foundation of China (No. 61702022), Beijing Municipal Education Committee Science Foundation (No. KM201910005024), China Postdoctoral Science Foundation funded project (No. 2018T110019) and Beijing excellent young talent cultivation project (No. 2017000020124G075).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, L., Qi, M., Jian, M. et al. Visual Sentiment Analysis by Combining Global and Local Information. Neural Process Lett 51, 2063–2075 (2020). https://doi.org/10.1007/s11063-019-10027-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-019-10027-7

Keywords

Search

Navigation