Jianbo Yuan
Jiebo Luo
ABSTRACT
Visual content analysis has always been important yet challenging. Thanks to the popularity of social networks, images become an convenient carrier for information diffusion among online users. To understand the diffusion patterns and different aspects of the social images, we need to interpret the images first. Similar to textual content, images also carry different levels of sentiment to their viewers. However, different from text, where sentiment analysis can use easily accessible semantic and context information, how to extract and interpret the sentiment of an image remains quite challenging. In this paper, we propose an image sentiment prediction framework, which leverages the mid-level attributes of an image to predict its sentiment. This makes the sentiment classification results more interpretable than directly using the low-level features of an image. To obtain a better performance on images containing faces, we introduce eigenface-based facial expression detection as an additional mid-level attributes. An empirical study of the proposed framework shows improved performance in terms of prediction accuracy. More importantly, by inspecting the prediction results, we are able to discover interesting relationships between mid-level attribute and image sentiment.
- J. Bollen, H. Mao, and X. Zeng. Twitter mood predicts the stock market. Journal of Computational Science, 2(1):1--8, 2011.Google Scholar
Cross Ref
- E. Cambria and A. Hussain. Sentic album: content-, concept-, and context-based online personal photo management system. Cognitive Computation, 4(4):477--496, 2012.Google ScholarCross Ref
- R. Datta, D. Joshi, J. Li, and J. Z. Wang. Studying aesthetics in photographic images using a computational approach. In Computer Vision--ECCV 2006, pages 288--301. Springer, 2006. Google ScholarDigital Library
- R. Datta, D. Joshi, J. Li, and J. Z. Wang. Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys (CSUR), 40(2):5, 2008. Google ScholarDigital Library
- A. Esuli and F. Sebastiani. Sentiwordnet: A publicly available lexical resource for opinion mining. In Proceedings of LREC, volume 6, pages 417--422, 2006.Google Scholar
- A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth. Describing objects by their attributes. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 1778--1785. IEEE, 2009.Google ScholarCross Ref
- A. Hanjalic, C. Kofler, and M. Larson. Intent and its discontents: the user at the wheel of the online video search engine. In Proceedings of the 20th ACM international conference on Multimedia, pages 1239--1248. ACM, 2012. Google ScholarDigital Library
- P. Isola, J. Xiao, A. Torralba, and A. Oliva. What makes an image memorable? In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 145--152. IEEE, 2011. Google ScholarDigital Library
- B. J. Jansen, M. Zhang, K. Sobel, and A. Chowdury. Twitter power: Tweets as electronic word of mouth. Journal of the American society for information science and technology, 60(11):2169--2188, 2009. Google ScholarDigital Library
- J. Jia, S. Wu, X. Wang, P. Hu, L. Cai, and J. Tang. Can we understand van gogh's mood?: learning to infer affects from images in social networks. In Proceedings of the 20th ACM international conference on Multimedia, pages 857--860. ACM, 2012. Google ScholarDigital Library
- P. J. Lang, M. M. Bradley, and B. N. Cuthbert. International affective picture system (iaps): Technical manual and affective ratings, 1999.Google Scholar
- B. Li, S. Feng, W. Xiong, and W. Hu. Scaring or pleasing: exploit emotional impact of an image. In Proceedings of the 20th ACM international conference on Multimedia, pages 1365--1366. ACM, 2012. Google ScholarDigital Library
- D. Lundqvist, A. Flykt, and A. Öhman. The karolinska directed emotional faces-kdef. cd-rom from department of clinical neuroscience, psychology section, karolinska institutet, stockholm, sweden. Technical report, ISBN 91-630-7164-9, 1998.Google Scholar
- J. Machajdik and A. Hanbury. Affective image classification using features inspired by psychology and art theory. In Proceedings of the international conference on Multimedia, pages 83--92. ACM, 2010. Google ScholarDigital Library
- L. Marchesotti, F. Perronnin, D. Larlus, and G. Csurka. Assessing the aesthetic quality of photographs using generic image descriptors. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 1784--1791. IEEE, 2011. Google ScholarDigital Library
- M. R. Naphade, C.-Y. Lin, J. R. Smith, B. Tseng, and S. Basu. Learning to annotate video databases. In SPIE Conference on Storage and Retrieval on Media databases, 2002.Google Scholar
- A. Oliva and A. Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International journal of computer vision, 42(3):145--175, 2001. Google ScholarDigital Library
- B. O'onnor, R. Balasubramanyan, B. R. Routledge, and N. A. Smith. From tweets to polls: Linking text sentiment to public opinion time series. In Proceedings of the International AAAI Conference on Weblogs and Social Media, pages 122--129, 2010.Google Scholar
- V. Ordonez, G. Kulkarni, and T. L. Berg. Im2text: Describing images using 1 million captioned photographs. In Neural Information Processing Systems (NIPS), 2011.Google Scholar
- B. Pang and L. Lee. Opinion mining and sentiment analysis. Foundations and trends in information retrieval, 2(1-2):1--135, 2008. Google ScholarDigital Library
- G. Patterson and J. Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2751--2758. IEEE, 2012. Google ScholarDigital Library
- C. G. Snoek and M. Worring. Concept-based video retrieval. Foundations and Trends in Information Retrieval, 2(4):215--322, 2008. Google ScholarDigital Library
- D. Tao, X. Tang, X. Li, and X. Wu. Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 28(7):1088--1099, 2006. Google ScholarDigital Library
- M. Thelwall, K. Buckley, G. Paltoglou, D. Cai, and A. Kappas. Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology, 61(12):2544--2558, 2010. Google ScholarDigital Library
- A. Tumasjan, T. O. Sprenger, P. G. Sandner, and I. M. Welpe. Predicting elections with twitter: What 140 characters reveal about political sentiment. In Proceedings of the fourth international AAAI conference on weblogs and social media, pages 178--185, 2010.Google Scholar
- M. A. Turk and A. P. Pentland. Face recognition using eigenfaces. In Computer Vision and Pattern Recognition, 1991. Proceedings CVPR'91., IEEE Computer Society Conference on, pages 586--591. IEEE, 1991.Google ScholarCross Ref
- V. Vonikakis and S. Winkler. Emotion-based sequence of family photos. In Proceedings of the 20th ACM international conference on Multimedia, MM '12, pages 1371--1372, New York, NY, USA, 2012. ACM. Google ScholarDigital Library
- W. Wang and Q. He. A survey on emotional semantic image retrieval. In Image Processing, 2008. ICIP 2008. 15th IEEE International Conference on, pages 117--120. IEEE, 2008.Google ScholarCross Ref
- T. Wilson, J. Wiebe, and P. Hoffmann. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 347--354. Association for Computational Linguistics, 2005. Google ScholarDigital Library
- J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Computer vision and pattern recognition (CVPR), 2010 IEEE conference on, pages 3485--3492. IEEE, 2010.Google Scholar
- C. C. Yang and T. D. Ng. Terrorism and crime related weblog social network: Link, content analysis and information visualization. In Intelligence and Security Informatics, 2007 IEEE, pages 55--58. IEEE, 2007.Google ScholarCross Ref
- V. Yanulevskaya, J. Uijlings, E. Bruni, A. Sartori, E. Zamboni, F. Bacci, D. Melcher, and N. Sebe. In the eye of the beholder: employing statistical analysis and eye tracking for analyzing abstract paintings. In Proceedings of the 20th ACM international conference on Multimedia, MM '12, pages 349--358, New York, NY, USA, 2012. ACM. Google ScholarDigital Library
- S. Zhang, M. Yang, T. Cour, K. Yu, and D. N. Metaxas. Query specific fusion for image retrieval. In Computer Vision--ECCV 2012, pages 660--673. Springer, 2012. Google ScholarDigital Library
Index Terms
- Sentribute: image sentiment analysis from a mid-level perspective
Recommendations
Image sentiment prediction based on textual descriptions with adjective noun pairs
We aim to predict the sentiment related information reflected in images based on SentiBank, which is a library including Adjective Noun Pair (ANP) concept detectors for image sentiment analysis. Instead of using only ANP responses in images as mid-level ...
Structured low-rank inverse-covariance estimation for visual sentiment distribution prediction
AbstractVisual sentiment analysis has aroused considerable attention with the increasing tendency of expressing sentiments via images. Most previous studies mainly focus on predicting the most dominant sentiment categories of images while ...
Joint Visual-Textual Sentiment Analysis with Deep Neural Networks
MM '15: Proceedings of the 23rd ACM international conference on MultimediaSentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so ...
Comments