Abstract
This study proposes the design of a communication technique that uses graphical icons, including emojis, as an alternative to facial expressions in video calls. Using graphical icons instead of complex and hard-to-read video expressions simplifies and reduces the amount of information in a video conference. The aim was to facilitate communication by preventing quick and incorrect emotional delivery. In this study, we developed EmojiCam, a system that encodes the emotions of the sender with facial expression recognition or user input and presents graphical icons of the encoded emotions to the receiver. User studies and existing emoji cultures were applied to examine the communication flow and discuss the possibility of using emoji in video calls. Finally, we discuss the new value that this model will bring and how it will change the style of video calling.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
https://justadudewhohacks.github.io/face-api.js/docs/index.html (last accessed Feb. 12, 2021.).
- 2.
https://github.com/auduno/clmtrackr (last accessed Feb. 12, 2021.).
- 3.
https://github.com/nenadmarkus/picojs (last accessed Feb. 12, 2021.).
References
Non-verbal feedback and reactions - zoom help center. https://support.zoom.us/hc/en-us/articles/115001286183-Non-verbal-feedback-and-reactions-. Accessed 02 Aug 2021
Amaghlobeli, N.: Linguistic features of typographic emoticons in SMS discourse. Theor. Pract. Lang. Stud. 2 (2012). https://doi.org/10.4304/tpls.2.2.348-354
Derks, D., Fischer, A.H., Bos, A.E.: The role of emotion in computer-mediated communication: a review. Comput. Hum. Behav. 24(3), 766–785 (2008)
Ekman, P.: Basic Emotions, pp. 45–60. Wiley (1999). https://doi.org/10.1002/0470013494.ch3
El Kaliouby, R., Robinson, P.: FAIM: integrating automated facial affect analysis in instant messaging. In: Proceedings of the 9th International Conference on Intelligent User Interfaces, IUI 2004, pp. 244–246. Association for Computing Machinery, New York (2004). https://doi.org/10.1145/964442.964493
Goodman, L.A.: Snowball sampling. Ann. Math. Stat. 32(1), 148–170 (1961). http://www.jstor.org/stable/2237615
Herring, S., Dainas, A.: “nice picture comment!” graphicons in Facebook comment threads. In: Proceedings of the 50th Hawaii International Conference on System Sciences (2017)
Huang, J., et al.: Speed/Accuracy trade-offs for modern convolutional object detectors (2017)
Universal Multiple-Octet Coded Character Set (UCS). Standard, International Organization for Standardization, Geneva, CH, December 2003
Koh, J.I., Cherian, J., Taele, P., Hammond, T.: Developing a hand gesture recognition system for mapping symbolic hand gestures to analogous emojis in computer-mediated communication. ACM Trans. Interact. Intell. Syst. 9(1) (2019). https://doi.org/10.1145/3297277
Park, J., Barash, V., Fink, C., Cha, M.: Emoticon style: interpreting differences in emoticons across cultures. In: Seventh International AAAI Conference on Weblogs and Social Media (2013)
Pirzadeh, A., Wu, H.W., Bharali, R., Kim, B.M., Wada, T., Pfaff, M.S.: Designing multi-touch gestures to support emotional expression in IM, CHI EA 2014, pp. 2515–2520. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2559206.2581347
Rovers, A., van Essen, H.: HIM: a framework for haptic instant messaging. In: CHI 2004 Extended Abstracts on Human Factors in Computing Systems, CHI EA 2004, pp. 1313–1316. Association for Computing Machinery, New York (2004). https://doi.org/10.1145/985921.986052
Sato, W., Hyniewska, S., Minemoto, K., Yoshikawa, S.: Facial expressions of basic emotions in Japanese laypeople. Front. Psychol. 10, 259 (2019). https://doi.org/10.3389/fpsyg.2019.00259
Shin, H., Lee, J., Park, J., Kim, Y., Oh, H., Lee, T.: A tactile emotional interface for instant messenger chat. In: Smith, M.J., Salvendy, G. (eds.) Human Interface 2007. LNCS, vol. 4558, pp. 166–175. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73354-6_19
Tang, Y., Hew, K.F.: Emoticon, emoji, and sticker use in computer-mediated communication: a review of theories and research findings. Int. J. Commun. 13, 27 (2019)
Walther, J.B., D’addario, K.P.: The impacts of emoticons on message interpretation in computer-mediated communication. Soc. Sci. Comput. Rev. 19(3), 324–347 (2001)
Wang, H.C., Lai, C.T.: Kinect-taped communication: using motion sensing to study gesture use and similarity in face-to-face and computer-mediated brainstorming. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2014, pp. 3205–3214. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2556288.2557060
Zhou, R., Hentschel, J., Kumar, N.: Goodbye text, hello emoji: mobile communication on wechat in China. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 748–759 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Namikawa, K., Suzuki, I., Iijima, R., Sarcar, S., Ochiai, Y. (2021). EmojiCam: Emoji-Assisted Video Communication System Leveraging Facial Expressions. In: Kurosu, M. (eds) Human-Computer Interaction. Design and User Experience Case Studies. HCII 2021. Lecture Notes in Computer Science(), vol 12764. Springer, Cham. https://doi.org/10.1007/978-3-030-78468-3_42
Download citation
DOI: https://doi.org/10.1007/978-3-030-78468-3_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78467-6
Online ISBN: 978-3-030-78468-3
eBook Packages: Computer ScienceComputer Science (R0)