research-article

Smile to Me: Investigating Emotions and their Representation in Text-based Messaging in the Wild

Authors:

Romina Poguntke,

Albrecht Schmidt,

Stefan SchneegassAuthors Info & Claims

MuC '19: Proceedings of Mensch und Computer 2019

Pages 373 - 385

https://doi.org/10.1145/3340764.3340795

Published: 08 September 2019 Publication History

Abstract

Emotions are part of human communication shaping mimics and revealing feelings. Therefore, conveying emotions has been integrated in text-based messaging applications using emojis. While visualizing emotions in text messages has been investigated in prior work, we studied the effects of emotion sharing by augmenting the WhatsApp Web user interface. For this, we designed and developed four different visualizations to represent emotions detected through facial expression recognition of chat partners using a web cam. Investigating emotion representation and its effects, we conducted a four weeks longitudinal study with 28 participants being inquired via 48 semi-structured interviews and 64 questionnaires. Our findings revealed that users want to maintain control over their emotions, particularly regarding sharing, and their preference to view positive emotions avoiding unpleasant social situations. Based on that, we phrased four design recommendations stimulating novel approaches for augmenting chats.

References

[1]

Jussi Ängeslevä, Carson Reynolds, and Sile O'Modhrain. 2004. EmoteMail. In ACM SIGGPAPH 2004 Posters (SIGGRAPH '04). ACM, New York, NY, USA, 9-.

Digital Library

[2]

Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on. IEEE, 1--10.

[3]

M. S. Bartlett, G. Littlewort, I. Fasel, and J. R. Movellan. 2003. Real Time Face Detection and Facial Expression Recognition: Development and Applications to Human Computer Interaction. In 2003 Conference on Computer Vision and Pattern Recognition Workshop, Vol. 5. 53--53.

[4]

Vinay Bettadapura. 2012. Face expression recognition and analysis: the state of the art. arXiv preprint arXiv:1203.6722 (2012).

[5]

Tatiana Buhler, Carman Neustaedter, and Serena Hillman. 2013. How and Why Teenagers Use Video Chat. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, New York, NY, USA, 759--768.

Digital Library

[6]

Daniel Buschek, Alexander De Luca, and Florian Alt. 2015. There is more to typing than speed: Expressive mobile touch keyboards via dynamic font personalisation. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 125--130.

Digital Library

[7]

Saemi Choi, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2016. Typeface Emotion Analysis for Communication on Mobile Messengers. In Proceedings of the 1st International Workshop on Multimedia Alternate Realities. ACM, 37--40.

Digital Library

[8]

Karen Church and Rodrigo de Oliveira. 2013. What's up with what-sapp?: comparing mobile instant messaging behaviors with traditional SMS. In Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services. ACM, 352--361.

Digital Library

[9]

Pietro Cipresso, Daniela Villani, Claudia Repetto, Lucia Bosone, Anna Balgera, Maurizio Mauri, Marco Villamira, Alessandro Antonietti, and Giuseppe Riva. 2015. Computational psychometrics in communication and implications in decision making. Computational and mathematical methods in medicine 2015 (2015).

[10]

Roddy Cowie and Ellen Douglas-Cowie. 1996. Automatic statistical analysis of the signal and prosodic signs of emotion in speech. In Spoken Language, 1996. ICSLP 96. Proceedings., Fourth International Conference on, Vol. 3. IEEE, 1989--1992.

[11]

M. Denscombe. 2014. The Good Research Guide: For Small-scale Social Research Projects. McGraw-Hill Education.

[12]

Michael H Dickey, Molly McLure Wasko, Katherine M Chudoba, and Jason Bennett Thatcher. 2006. Do You Know What I Know? A Shared Understandings Perspective on Text-Based Communication. Journal of Computer-Mediated Communication 12, 1 (2006), 66--87.

[13]

Joan Morris DiMicco, Vidya Lakshmipathy, and Andrew Tresolini Fiore. 2002. Conductive Chat: Instant messaging with a skin conductivity channel. In Proceedings of Conference on Computer Supported Cooperative Work. Citeseer.

[14]

Paul Ekman and Harriet Oster. 1979. Facial expressions of emotion. Annual review of psychology 30, 1 (1979), 527--554.

[15]

Abdallah El Ali, Torben Wallbaum, Merlin Wasmann, Wilko Heuten, and Susanne CJ Boll. 2017. Face2Emoji: Using Facial Emotional Expressions to Filter Emojis. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 1577--1584.

Digital Library

[16]

Rana El Kaliouby and Peter Robinson. 2004. FAIM: Integrating Automated Facial Affect Analysis in Instant Messaging. In Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI '04). ACM, New York, NY, USA, 244--246.

Digital Library

[17]

Marc Fabri and DJ Moore. 2005. Is empathy the key? Effective communication via instant messaging. In Proceedings of the 11th EATA International Conference on Networking Entities.

[18]

Marc Fabri, DJ Moore, and DJ Hobbs. 2005. Empathy and enjoyment in instant messaging. In Proceedings of 19th British HCI group annual conference (HCI2005), Edinburgh, UK. 4--9.

[19]

Jackson Feijó Filho, Thiago Valle, and Wilson Prata. 2014. Non-verbal communications in mobile text chat: emotion-enhanced mobile chat. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services. ACM, 443--446.

Digital Library

[20]

Octavio Garcia, Jesús Favela, Guillermo Licea, and Roberto Machorro. 1999. Extending a collaborative architecture to support emotional awareness. CICESE, Mexico (1999).

[21]

Patrice Godefroid, James D. Herbsleb, Lalita Jategaonkar Jagadeesany, and Du Li. 2000. Ensuring Privacy in Presence Awareness: An Automated Verification Approach. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW '00). ACM, New York, NY, USA, 59--68.

Digital Library

[22]

Gerald J Gorn, Amitava Chattopadhyay, Tracey Yi, and Darren W Dahl. 1997. Effects of color as an executional cue in advertising: They're in the shade. Management science 43, 10 (1997), 1387--1400.

[23]

Mark Handel and James D. Herbsleb. 2002. What is Chat Doing in the Workplace?. In Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work (CSCW '02). ACM, New York, NY, USA, 1--10.

Digital Library

[24]

Mariam Hassib, Daniel Buschek, Paweł W. Wozniak, and Florian Alt. 2017. HeartChat: Heart Rate Augmented Mobile Chat to Support Empathy and Awareness. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 2239--2251.

Digital Library

[25]

Mariam Hassib, Mohamed Khamis, Stefan Schneegass, Ali Sahami Shirazi, and Florian Alt. 2016. Investigating User Needs for Bio-sensing and Affective Wearables. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, New York, NY, USA, 1415--1422.

Digital Library

[26]

Kiraz Candan Herdem. 2012. Reactions: Twitter based mobile application for awareness of friends' emotions. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. ACM, 796--797.

Digital Library

[27]

Javier Hernandez and Rosalind W Picard. 2014. SenseGlass: using google glass to sense daily emotions. In Proceedings of the adjunct publication of the 27th annual ACM symposium on User interface software and technology. ACM, 77--78.

Digital Library

[28]

Jonathan Klein, Youngme Moon, and Rosalind W. Picard. 2002. This computer responds to user frustration: Theory, design, and results. Interacting with computers 14, 2 (2002), 119--140.

[29]

Paul R Kleinginna and Anne M Kleinginna. 1981. A categorized list of emotion definitions, with suggestions for a consensual definition. Motivation and emotion 5, 4 (1981), 345--379.

[30]

Evdokimos I Konstantinidis, Antonis Billis, Theodore Savvidis, Stefanos Xefteris, and Panagiotis D Bamidis. 2017. Emotion Recognition in the Wild: Results and Limitations from Active and Healthy Ageing cases in a Living Lab. In eHealth 360°. Springer, 425--428.

[31]

Joonhwan Lee, Soojin Jun, Jodi Forlizzi, and Scott E. Hudson. 2006. Using Kinetic Typography to Convey Emotion in Text-based Interpersonal Communication. In Proceedings of the 6th Conference on Designing Interactive Systems (DIS '06). ACM, New York, NY, USA, 41--49.

Digital Library

[32]

Myungho Lee, Kangsoo Kim, Hyunghwan Rho, and Si Jung Kim. 2014. Empa Talk: A Physiological Data Incorporated Human-computer Interactions. In Proceedings of the Extended Abstracts of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI EA '14). ACM, New York, NY, USA, 1897--1902.

Digital Library

[33]

Fannie Liu, Laura Dabbish, and Geoff Kaufman. 2017. Supporting Social Interactions with an Expressive Heart Rate Sharing Application. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 77.

Digital Library

[34]

Hugo Liu, Henry Lieberman, and Ted Selker. 2003. A Model of Textual Affect Sensing Using Real-world Knowledge. In Proceedings of the 8th International Conference on Intelligent User Interfaces (IUI '03). ACM, New York, NY, USA, 125--132.

Digital Library

[35]

C. Ma, A. Osherenko, H. Prendinger, and M. Ishizuka. 2005. A chat system based on emotion estimation from text and embodied conversational messengers. In Proceedings of the 2005 International Conference on Active Media Technology, 2005. (AMT 2005). 546--548.

[36]

Daniel McDuff, Abdelrahman Mahmoud, Mohammad Mavadati, May Amr, Jay Turcot, and Rana el Kaliouby. 2016. AFFDEX SDK: A Cross-Platform Real-Time Multi-Face Expression Recognition Toolkit. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, New York, NY, USA, 3723--3726.

Digital Library

[37]

Daniel J McDuff, Javier Hernandez, Sarah Gontarek, and Rosalind W Picard. 2016. Cogcam: Contact-free measurement of cognitive stress during computer tasks with a digital camera. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4000--4004.

Digital Library

[38]

Nicola Millard and Linda Hole. 2008. In the Moodie: Using 'affective widgets' to help contact centre advisors fight stress. Affect and Emotion in Human-Computer Interaction (2008), 186--193.

Digital Library

[39]

Saif Mohammad. 2011. Colourful language: Measuring word-colour associations. In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics. Association for Computational Linguistics, 97--106.

Digital Library

[40]

Alena Neviarouskaya, Helmut Prendinger, and Mitsuru Ishizuka. 2007. Textual Affect Sensing for Sociable and Expressive Online Communication. In Affective Computing and Intelligent Interaction, Ana C. R. Paiva, Rui Prada, and Rosalind W. Picard (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 218--229.

Digital Library

[41]

Jan Novotny, Jan Dvorak, and Ondrej Krejcar. 2016. User based intelligent adaptation of five in a row game for android based on the data from the front camera. In International Conference on Augmented Reality, Virtual Reality and Computer Graphics. Springer, 133--149.

[42]

Mohammad Obaid, Charles Han, and Mark Billinghurst. 2008. "Feed the Fish": An Affect-aware Game. In Proceedings of the 5th Australasian Conference on Interactive Entertainment (IE '08). ACM, New York, NY, USA, Article 6, 6 pages.

Digital Library

[43]

James Ohene-Djan, Andrew Sammon, and Rachel Shipsey. 2006. Colour spectrumâĂ&Zacute;s of opinion: An information visualisation interface for representing degrees of emotion in real time. In Information Visualization, 2006. IV 2006. Tenth International Conference on. IEEE, 80--88.

Digital Library

[44]

Sharon Oviatt and Philip Cohen. 2000. Perceptual user interfaces: multimodal interfaces that process what comes naturally. Commun. ACM 43, 3 (2000), 45--53.

Digital Library

[45]

R. W. Picard. 1995. Affective Computing. Citeseer, Pennsylvania State University, PA, USA.

Digital Library

[46]

Robert Plutchik and Henry Kellerman. 1980. Emotion, Theory, Research, and Experience: Theory, Research and Experience. Academic press.

[47]

Ke-Chen Pong, Chi-An Wang, and Shuo Hsiu Hsu. 2014. GamIM: Affecting Chatting Behavior by Visualizing Atmosphere of Conversation. In CHI '14 Extended Abstracts on Human Factors in Computing Systems (CHI EA '14). ACM, New York, NY, USA, 2497--2502.

Digital Library

[48]

Microsoft Cognitive Services (previously Project Oxford). {n.d.}. Developer Guide. Retrieved July 28, 2017 from https://docs.microsoft.com/en-us/azure/cognitive-services/.

[49]

Karen Renaud, Judith Ramsay, and Mario Hair. 2006. 'You've got email!'... shall I deal with it now? Electronic mail from the recipient's perspective. International Journal of Human-Computer Interaction 21, 3 (2006), 313--332.

[50]

Claudia Repetto, Corina Sas, Cristina Botella, and Giuseppe Riva. 2017. AFFECTECH: Personal Technologies for Affective Health. In The 15th European Congress of Psychology, 2017.

[51]

Giuseppe Riva, Andrea Gaggioli, Daniela Villani, Pietro Cipresso, Claudia Repetto, Silvia Serino, Stefano Triberti, Eleonora Brivio, Carlo Galimberti, and Guendalina Graffigna. 2014. Positive technology for healthy living and active ageing. Active Ageing and Healthy Living: A Human Centered Approach in Research and Innovation as Source of Quality of Life 203 (2014), 44--56.

[52]

James A. Russell. 1980. A Circumplex Model of Affect. Journal of Personality and Social Psychology 39, 6 (1980), 1161--1178.

[53]

Corina Sas and Alan Dix. 2009. Designing for Reflection on Experience. In CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA '09). ACM, New York, NY, USA, 4741--4744.

Digital Library

[54]

Corina Sas, Alan Dix, Jennefer Hart, and Ronghui Su. 2009. Dramaturgical Capitalization of Positive Emotions: The Answer for Facebook Success?. In Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology (BCS-HCI '09). British Computer Society, Swinton, UK, UK, 120--129. http://dl.acm.org/citation.cfm?id=1671011.1671025

Digital Library

[55]

Corina Sas, Alan Dix, Jennefer Hart, and Ronghui Su. 2009. Emotional Experience on Facebook Site. In CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA '09). ACM, New York, NY, USA, 4345--4350.

Digital Library

[56]

Corina Sas, Muhammad Umair, and Muhammad Hamza Latif. 2018. Designing for Self-Regulation from both Pragmatic and Critical Design Lenses.

[57]

Corina Sas, Steve Whittaker, Steven Dow, Jodi Forlizzi, and John Zimmerman. 2014. Generating Implications for Design Through Design Research. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 1971--1980.

Digital Library

[58]

Jocelyn Scheirer, Raul Fernandez, Jonathan Klein, and Rosalind W Picard. 2002. Frustrating the user on purpose: a step toward building an affective computer. Interacting with computers 14, 2 (2002), 93--118.

[59]

Ben Shneiderman. 1997. Designing the User Interface: Strategies for Effective Human-Computer Interaction (3rd ed.). Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

Digital Library

[60]

Satyendra Singh. 2006. Impact of color on marketing. Management decision 44, 6 (2006), 783--789.

[61]

J. A. Smith. 2015. Qualitative Psychology: A Practical Guide to Research Methods. Sage.

[62]

A. Tat and M. S. T. Carpendale. 2002. Visualising human dialog. In Information Visualisation, 2002. Proceedings. Sixth International Conference on. 16--21.

[63]

Dzmitry Tsetserukou, Alena Neviarouskaya, Helmut Prendinger, Naoki Kawakami, Mitsuru Ishizuka, and Susumu Tachi. 2009. iFeel_IM! Emotion enhancing garment for communication in affect sensitive instant messenger. In Human Interface and the Management of Information. Designing Information Environments. Springer, 628--637.

Digital Library

[64]

Muhammad Umair, Muhammad Hamza Latif, and Corina Sas. 2018. Dynamic Displays at Wrist for Real Time Visualization of Affective Data. In Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems (DIS '18 Companion). ACM, New York, NY, USA, 201--205.

Digital Library

[65]

Fernanda B. Viégas and Judith S. Donath. 1999. Chat Circles. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99). ACM, New York, NY, USA, 9--16.

Digital Library

[66]

Hua Wang, Helmut Prendinger, and Takeo Igarashi. 2004. Communicating Emotions in Online Chat Using Physiological Sensors and Animated Text. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (CHI EA '04). ACM, New York, NY, USA, 1171--1174.

Digital Library

[67]

Beverly Woolf, Winslow Burleson, Ivon Arroyo, Toby Dragon, David Cooper, and Rosalind Picard. 2009. Affect-aware tutors: recognising and responding to student affect. International Journal of Learning Technology 4, 3-4 (2009), 129--164.

Digital Library

[68]

Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang. 2009. A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 1 (Jan 2009), 39--58.

Digital Library

[69]

Yu Zhu, Yan Li, Guowang Mu, and Guodong Guo. 2015. A study on apparent age estimation. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 25--31.

Digital Library

Cited By

Fimpel MFlach NReckzügel MMaurer B(2023)“Hey, can we talk?”: Exploring How Revealing Implicit Emotional Responses Tangibly Could Foster Empathy During Mobile TextingProceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3569009.3573124(1-7)Online publication date: 26-Feb-2023
https://dl.acm.org/doi/10.1145/3569009.3573124
Salaric PZitkus ECain R(2022)The Influence of Interaction Design on Relation Making: A Scoping ReviewHuman-Computer Interaction. User Experience and Behavior10.1007/978-3-031-05412-9_30(430-446)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-05412-9_30

Index Terms

Smile to Me: Investigating Emotions and their Representation in Text-based Messaging in the Wild
1. Human-centered computing
  1. Collaborative and social computing
    1. Empirical studies in collaborative and social computing
  2. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies

Recommendations

Remember to smile: design of a mobile affective technology to help promote individual happiness through smiling
PervasiveHealth '17: Proceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare

Wellbeing plays a central role in quality of life and encompasses aspects pertaining to mental and social wellbeing, as well as physical wellbeing and the absence of disease. Building upon the natural human understanding that smiling is an expression of ...
AVEC 2014: 3D Dimensional Affect and Depression Recognition Challenge
AVEC '14: Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge

Mood disorders are inherently related to emotion. In particular, the behaviour of people suffering from mood disorders such as unipolar depression shows a strong temporal correlation with the affective dimensions valence, arousal and dominance. In ...
AVEC 2013: the continuous audio/visual emotion and depression recognition challenge
AVEC '13: Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge

Mood disorders are inherently related to emotion. In particular, the behaviour of people suffering from mood disorders such as unipolar depression shows a strong temporal correlation with the affective dimensions valence and arousal. In addition, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MuC '19: Proceedings of Mensch und Computer 2019

September 2019

863 pages

ISBN:9781450371988

DOI:10.1145/3340764

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

H2020 European Research Council

Conference

MuC'19

MuC'19: Mensch-und-Computer

September 8 - 11, 2019

Hamburg, Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
294
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)8

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fimpel MFlach NReckzügel MMaurer B(2023)“Hey, can we talk?”: Exploring How Revealing Implicit Emotional Responses Tangibly Could Foster Empathy During Mobile TextingProceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3569009.3573124(1-7)Online publication date: 26-Feb-2023
https://dl.acm.org/doi/10.1145/3569009.3573124
Salaric PZitkus ECain R(2022)The Influence of Interaction Design on Relation Making: A Scoping ReviewHuman-Computer Interaction. User Experience and Behavior10.1007/978-3-031-05412-9_30(430-446)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-05412-9_30

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten