research-article

“I Have No Text in My Post”: Using Visual Hints to Model User Emotions in Social Media

Authors:

Sang-Wook KimAuthors Info & Claims

WWW '22: Proceedings of the ACM Web Conference 2022

Pages 2888 - 2896

https://doi.org/10.1145/3485447.3512009

Published: 25 April 2022 Publication History

Abstract

As an emotion plays an important role in people’s everyday lives and is often mirrored in their social media use, extensive research has been conducted to characterize and model emotions from social media data. However, prior research has not sufficiently considered trends of social media use—the increasing use of images and the decreasing use of text—nor identified the features of images in social media that are likely to be different from those in non-social media. Our study aims to fill this gap by (1) considering the notion of visual hints that depict contextual information of images, (2) presenting their characteristics in positive or negative emotions, and (3) demonstrating their effectiveness in emotion prediction modeling through an in-depth analysis of their relationship with the text in the same posts. The results of our experiments showed that our visual hint-based model achieved 20% improvement in emotion prediction, compared with the baseline. In particular, the performance of our model was comparable with that of the text-based model, highlighting not only a strong relationship between visual hints of the image and emotion, but also the potential of using only images for emotion prediction which well reflects current and future trends of social media use.

References

[1]

Barry J Babin, David M Hardesty, and Tracy A Suter. 2003. Color and Shopping Intentions: The Intervening Effect of Price Fairness and Perceived Affect. Journal of Business Research 56 (2003), 541–551.

[2]

Matteo Baldoni, Cristina Baroglio, Viviana Patti, and Paolo Rena. 2012. From Tags to Emotions: Ontology-driven Sentiment Analysis in The Social Semantic Web. Intelligenza Artificiale 6 (2012), 41–54.

[3]

David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research 3 (2003), 993–1022.

Digital Library

[4]

Damian Borth, Tao Chen, Rongrong Ji, and Shih-Fu Chang. 2013. Sentibank: Large-scale Ontology and Classifiers for Detecting Sentiment and Emotions in Visual Content. In Proc. ACM Int’l Conf. on Multimedia.

Digital Library

[5]

Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel, and Shih-Fu Chang. 2013. Large-Scale Visual Sentiment Ontology and Detectors Using Adjective Noun Pairs. In Proc. ACM Int’l Conf. on Multimedia.

Digital Library

[6]

David B Bracewell. 2008. Semi-Automatic Creation of An Emotion Dictionary Using Wordnet and Its Evaluation. In Proc. IEEE Int’l Conf. on Cybernetics and Intelligent Systems.

[7]

Lijuan Cai and Thomas Hofmann. 2003. Text categorization by boosting automatically extracted concepts. In Proc. ACM SIGIR Conf. on Research and Development in Informaion Retrieval.

Digital Library

[8]

Ming Chen, Lu Zhang, and Jan P Allebach. 2015. Learning Deep Features for Image Emotion Classification. In Proc. IEEE Int’l Conf. on Image Processing.

Digital Library

[9]

Mihaly Csikszentmihalyi and Eugene Halton. 1981. The Meaning of Things: Domestic Symbols and The Self. Cambridge University Press.

[10]

Munmun De Choudhury, Scott Counts, and Mary Czerwinski. 2011. Identifying relevant social media content: leveraging information diversity and user cognition. In Proceedings of the 22nd ACM conference on Hypertext and hypermedia. 161–170.

Digital Library

[11]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805(2018).

[12]

Paul Ekman. 1992. An Argument for Basic Emotions. Cognition & Emotion 6(1992), 169–200.

[13]

Carroll E Izard. 1991. The Psychology of Emotions. Springer Science & Business Media.

[14]

Carroll E Izard. 2007. Basic Emotions, Natural Kinds, Emotion Schemas, and A New Paradigm. Perspectives on Psychological Science 2 (2007), 260–280.

[15]

Bo Jiang, Yun Ling, and Jiale Wang. 2010. Tag Recommendation Based on Social Comment Network. International Journal of Digital Content Technology and its Applications 4(2010), 110–117.

[16]

Brendan Jou, Tao Chen, Nikolaos Pappas, Miriam Redi, Mercan Topkara, and Shih-Fu Chang. 2015. Visual affect around the world: A large-scale multilingual visual sentiment ontology. In Proceedings of the 23rd ACM international conference on Multimedia. 159–168.

Digital Library

[17]

Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International conference on machine learning. PMLR, 1188–1196.

[18]

Xin Lu, Reginald B Adams, Jia Li, Michelle G Newman, and James Z Wang. 2017. An Investigation into Three Visual Characteristics of Complex Scenes that Evoke Human Emotion. In Proc. IEEE Int’l Conf. on Affective Computing and Intelligent Interaction.

[19]

Xin Lu, Poonam Suryanarayan, Reginald B Adams Jr, Jia Li, Michelle G Newman, and James Z Wang. 2012. On Shape and The Computability of Emotions. In Proc. ACM Int’l Conf. on Multimedia.

Digital Library

[20]

Jana Machajdik and Allan Hanbury. 2010. Affective Image Classification Using Features Inspired by Psychology and Art Theory. In Proc. ACM Int’l Conf. on Multimedia.

Digital Library

[21]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781(2013).

[22]

Genevieve Patterson and James Hays. 2012. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2751–2758.

[23]

Kuan-Chuan Peng, Tsuhan Chen, Amir Sadovnik, and Andrew C Gallagher. 2015. A mixed bag of emotions: Model, predict, and transfer emotion distributions. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition.

[24]

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proc. Conf. on Empirical Methods in Natural Language Processing.

[25]

Gerard Salton and Michael J McGill. 1986. Introduction to Modern Information Retrieval. McGraw-Hill, Inc.

Digital Library

[26]

Aaron Smith, Monica Anderson, and Tom Caiazza. 2018. Social Media Use in 2018.

[27]

Qi Su, Xinying Xu, Honglei Guo, Zhili Guo, Xian Wu, Xiaoxun Zhang, Bin Swen, and Zhong Su. 2008. Hidden sentiment association in chinese web opinion mining. In Proc. ACM Int’l Conf. on World Wide Web.

Digital Library

[28]

H. Tankovska. 2021. Global social networks ranked by number of users 2021.

[29]

H. Tankovska. 2021. Social media platforms growth of MAU worldwide 2019-2021.

[30]

Peter D Turney. 2002. Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In Proc. 40th Annual Meeting on Association for Computational Linguistics.

[31]

Peter D Turney and Michael L Littman. 2003. Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems 21, 4 (2003), 315–346.

Digital Library

[32]

Princeton University. 2010. About WordNet. WordNet (2010).

[33]

Kate Vaisutis, Margot Brereton, Toni Robertson, Frank Vetere, Jeannette Durick, Bjorn Nansen, and Laurie Buys. 2014. Invisible Connections: Investigating Older People’s Emotions and Social Relations Around Objects. In Proc. ACM CHI Conf. on Human Factors in Computing Systems.

Digital Library

[34]

Patricia Valdez and Albert Mehrabian. 1994. Effects of Color on Emotions.Journal of Experimental Psychology: General 123 (1994), 394.

[35]

Maria V Valueva, NN Nagornov, Pave A Lyakhov, Georgiy V Valuev, and Nikolay I Chervyakov. 2020. Application of the residue number system to reduce hardware costs of the convolutional neural network implementation. Mathematics and Computers in Simulation 177 (2020), 232–243.

[36]

Pu Wang and Carlotta Domeniconi. 2008. Building semantic kernels for text classification using wikipedia. In Proc. ACM SIGKDD Int’l Conf. on Knowledge Discovery and Data mining.

Digital Library

[37]

Theresa Wilson, Janyce Wiebe, and Rebecca Hwa. 2004. Just how mad are you? Finding strong and weak opinion clauses. In Proc. AAAI Conf. on Artificial Intelligence.

[38]

Chung-Hsien Wu, Ze-Jing Chuang, and Yu-Chung Lin. 2006. Emotion Recognition from Text Using Semantic Labels and Separable Mixture Models. ACM Transactions on Asian Language Information Processing 5 (2006), 165–183.

Digital Library

[39]

Can Xu, Suleyman Cetintas, Kuang-Chih Lee, and Li-Jia Li. 2014. Visual Sentiment Prediction with Deep Convolutional Neural Networks. arXiv preprint arXiv:1411.5731(2014).

[40]

Peng Yang, Qingshan Liu, and Dimitris N Metaxas. 2010. Exploring facial expressions with compositional features. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 2638–2644.

[41]

Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. 2016. Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark. In Proc. AAAI Conf. on Artificial Intelligence.

[42]

Jianbo Yuan, Sean Mcdonough, Quanzeng You, and Jiebo Luo. 2013. Sentribute: image sentiment analysis from a mid-level perspective. In Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining. 1–8.

Digital Library

[43]

Sicheng Zhao, Yue Gao, Xiaolei Jiang, Hongxun Yao, Tat-Seng Chua, and Xiaoshuai Sun. 2014. Exploring Principles-of-Art Features for Image Emotion Recognition. In Proc. ACM Int’l Conf. on Multimedia.

Digital Library

[44]

Sicheng Zhao, Hongxun Yao, Yue Gao, Guiguang Ding, and Tat-Seng Chua. 2016. Predicting Personalized Image Emotion Perceptions in Social Networks. IEEE Transactions on Affective Computing(2016).

[45]

Sicheng Zhao, Xingxu Yao, Jufeng Yang, Guoli Jia, Guiguang Ding, Tat-Seng Chua, Bjoern W Schuller, and Kurt Keutzer. 2021. Affective image content analysis: Two decades review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).

Cited By

Santana JSantana MSampaio PBrito K(2024)Towards a Methodology for Analyzing Visual Elements in Social Media Posts of PoliticiansProceedings of the 17th International Conference on Theory and Practice of Electronic Governance10.1145/3680127.3680218(366-373)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1145/3680127.3680218
Qin YJu WWu HLuo XZhang M(2024)Learning Graph ODE for Continuous-Time Sequential RecommendationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.334939736:7(3224-3236)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TKDE.2024.3349397
Čepulionytė AToldinas JLozinskis B(2023)A Multilayered Preprocessing Approach for Recognition and Classification of Malicious Social Network MessagesElectronics10.3390/electronics1218378512:18(3785)Online publication date: 7-Sep-2023
https://doi.org/10.3390/electronics12183785
Show More Cited By

Index Terms

“I Have No Text in My Post”: Using Visual Hints to Model User Emotions in Social Media

Index terms have been assigned to the content through auto-classification.

Recommendations

Social media user classification: based on social capital expectation, susceptibility, and compulsion loop
ICEC '17: Proceedings of the International Conference on Electronic Commerce

Social media such as Facebook, Instagram and Twitter are originally developed as communication tools among individuals for private conversations. Through the platforms, people share photos, stories and news with their social media friends to interact ...
Emotions, Technology, and Social Media
Social Media Marketing: A Beginner Guide To Get Success In Your Business (Volume 1)

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '22: Proceedings of the ACM Web Conference 2022

April 2022

3764 pages

ISBN:9781450390965

DOI:10.1145/3485447

Editors:
Frédérique Laforest
INSA Lyon, France
,
Raphaël Troncy
EURECOM, France
,
Elena Simperl
King’s College London, UK
,
Deepak Agarwal
Pinterest, USA
,
Aristides Gionis
KTH Royal Institute of Technology, Sweden
,
Ivan Herman
W3C / retired
,
Lionel Médini
Université Lyon 1, France

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Research Foundation of Korea
Institute of Information & Communications Technology Planning & Evaluation grant funded by the Korea government (MSIT)

Conference

WWW '22

Sponsor:

SIGWEB

WWW '22: The ACM Web Conference 2022

April 25 - 29, 2022

Virtual Event, Lyon, France

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
634
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Santana JSantana MSampaio PBrito K(2024)Towards a Methodology for Analyzing Visual Elements in Social Media Posts of PoliticiansProceedings of the 17th International Conference on Theory and Practice of Electronic Governance10.1145/3680127.3680218(366-373)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1145/3680127.3680218
Qin YJu WWu HLuo XZhang M(2024)Learning Graph ODE for Continuous-Time Sequential RecommendationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.334939736:7(3224-3236)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TKDE.2024.3349397
Čepulionytė AToldinas JLozinskis B(2023)A Multilayered Preprocessing Approach for Recognition and Classification of Malicious Social Network MessagesElectronics10.3390/electronics1218378512:18(3785)Online publication date: 7-Sep-2023
https://doi.org/10.3390/electronics12183785
Wang JZhao SWu CAdams RNewman MShafir TTsachor R(2023)Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding EmotionProceedings of the IEEE10.1109/JPROC.2023.3273517111:10(1236-1286)Online publication date: Oct-2023
https://doi.org/10.1109/JPROC.2023.3273517
Yamamoto FKumamoto TNadamoto A(2022)Analysis of Behavioral Facilitation Tweets Considering the Emotion of Disaster Victims2022 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom)10.1109/ISPA-BDCloud-SocialCom-SustainCom57177.2022.00064(451-457)Online publication date: Dec-2022
https://doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom57177.2022.00064

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten