Skip to main content
SpringerLink
Log in

Image sentiment prediction based on textual descriptions with adjective noun pairs

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

We aim to predict the sentiment related information reflected in images based on SentiBank, which is a library including Adjective Noun Pair (ANP) concept detectors for image sentiment analysis. Instead of using only ANP responses in images as mid-level features, we make full use of the ANPs’ textual sentiment. We first give each ANP concept in SentiBank a sentiment value by adding together the textual sentiment value of the adjective and that of the noun. Having detected the presence of ANPs in an image, we define an image sentiment value by computing the weighted sum of the textual sentiment values of ANPs describing this image with corresponding ANP responses as weights. On the one hand, we adopt a one-dimension logistic regression model to predict the sentiment orientation according to the image sentiment value. On the other hand, we use the ANP responses detected in an image as mid-level representations to train a regularized logistic regression classifier for sentiment prediction. We finally employ a late fusion algorithm to combine the prediction results from the two schemes. Experimental results reveal that the proposed method which takes into account the textual sentiment of ANPs improves the performance of SentiBank based image sentiment prediction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. http://visual-sentiment-ontology.appspot.com/

References

  1. Borth D, Ji R, Chen T, et al (2013a) Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: Proceedings of the ACM International Conference on Multimedia, p 223–232

  2. Borth D, Chen T, Ji R, et al (2013b) SentiBank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content. In: Proceedings of the ACM International Conference on Multimedia, p 459–460

  3. Cao D, Ji R, Lin D et al (2016) Visual sentiment topic model based microblog image sentiment analysis. Multimedia Tools & Applications 75(15):8955–8968

    Article  Google Scholar 

  4. Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2(3):27

    Google Scholar 

  5. Chen T, Yu FX, Chen J, et al (2014a) Object-based visual sentiment concept analysis and application. In: Proceedings of the ACM International Conference on Multimedia, p 367–376

  6. Chen T, Borth D, Darrell T, et al (2014b) DeepSentiBank: visual sentiment concept classification with deep convolutional neural networks. arXiv preprint arXiv: 1410.8586

  7. Chen YY, Chen T, Hsu WH, et al (2014c) Predicting viewer affective comments based on image content in social media. In: Proceedings of the ACM International Conference on Multimedia Retrieval, p 233–240

  8. Esuli A, Sebastiani F (2006) SentiWordnet: a publicly available lexical resource for opinion mining. In: Proceedings of LREC, p 417–422

  9. Jia J, Wu S, Wang X, et al (2012) Can we understand van gogh’s mood? learning to infer affects from images in social networks. In: Proceedings of the ACM International Conference on Multimedia, p 857–860

  10. Jou B, Bhattacharya S, Chang SF (2014) Predicting viewer perceived emotions in animated GIFs. In: Proceedings of the ACM International Conference on Multimedia, p 213–216

  11. Koh K, Kim SJ, Boyd SP (2007) An interior-point method for large-scale l1-regularized logistic regression. J Mach Learn Res 8(8):1519–1555

    MathSciNet  MATH  Google Scholar 

  12. Lai KT, Liu D, Chen MS et al (2015) Learning sample specific weights for late fusion. IEEE Trans Image Process 24(9):2772–2783

    Article  MathSciNet  Google Scholar 

  13. Machajdik J, Hanbury A (2010) Affective image classification using features inspired by psychology and art theory. In: Proceedings of the ACM International Conference on Multimedia, p 83–92

  14. Ng AY, Ngiam J, Foo CY, et al (2016) Unsupervised feature learning and deep learning. http://deeplearning.stanford.edu/wiki/index.php

  15. Pang B, Lee L (2008) Opinion mining and sentiment analysis. Inf Retr 2(1–2):1–135

    Google Scholar 

  16. Petz G, Karpowicz M, Fürschuß H, et al (2012) On text preprocessing for opinion mining outside of laboratory environments. In: Active Media Technology. Lecture Notes in Computer Science (LNCS), vol 7669 pp 618–629. doi:10.1007/978-3-642-35236-2_62

  17. Petz G, Karpowicz M, Fürschuß H et al (2015) Reprint of: computational approaches for mining user’s opinions on the web 2.0. Inf Process Manag 51(4):510–519

    Article  Google Scholar 

  18. Rasmussen CE (2006) Gaussian processes for machine learning. The MIT press, Cambridge

    MATH  Google Scholar 

  19. Shannon CE (2001) A mathematical theory of communication. ACM SIGMOBILE Mobile Computing and Communications Review 5(1):3–55

    Article  MathSciNet  Google Scholar 

  20. Thelwall M, Buckley K, Paltoglou G et al (2010) Sentiment strength detection in short informal text. J Am Soc Inf Sci Technol 61(12):2544–2558

    Article  Google Scholar 

  21. You Q, Luo J, Jin H, et al (2015) Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, p 381–388

  22. Yu FX, Cao L, Feris RS, et al (2013) Designing category-level attributes for discriminative visual recognition. In: Proceedings of 2013 I.E. Conference on Computer Vision and Pattern Recognition (CVPR), p 771–778

  23. Yuan J, Mcdonough S, You Q, et al (2013) Sentribute: image sentiment analysis from a mid-level perspective. In: Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining, Article number: 10

  24. Zhang H, Gönen M, Yang Z et al (2015) Understanding emotional impact of images using Bayesian multiple kernel learning. Neurocomputing 165:3–13

    Article  Google Scholar 

  25. Zhao S, Gao Y, Jiang X, et al (2014) Exploring principles-of-art features for image emotion recognition. In: Proceedings of the ACM International Conference on Multimedia, p 47–56

Download references

Acknowledgements

This work was supported by the Science and Technology Innovation Engineering Program for Shaanxi Provincial Key Laboratories under Grant 2013SZS15-K02, the Basis and Cutting-Edge Research Project of Science and Technology Department of Henan Province under Grant 142300410248 and the Key Scientific Research Plan of Higher Education Institutions of Henan Province under Grant 15A510041.

Author information

Authors and Affiliations

  1. School of Electronics and Information, Northwestern Polytechnical University, Xi’an, 710072, China

    Zuhe Li & Yangyu Fan

  2. School of Computer and Communication Engineering, Zhengzhou University of Light Industry, Zhengzhou, 450002, China

    Zuhe Li & Fengqin Wang

  3. Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an, 710119, China

    Weihua Liu

Authors
  1. Zuhe Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yangyu Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weihua Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fengqin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zuhe Li.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Z., Fan, Y., Liu, W. et al. Image sentiment prediction based on textual descriptions with adjective noun pairs. Multimed Tools Appl 77, 1115–1132 (2018). https://doi.org/10.1007/s11042-016-4310-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-4310-5

Keywords

Search

Navigation