skip to main content
10.1145/3025453.3025953acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Emotion Actuator: Embodied Emotional Feedback through Electroencephalography and Electrical Muscle Stimulation

Published: 02 May 2017 Publication History

Abstract

The human body reveals emotional and bodily states through measurable signals, such as body language and electroencephalography. However, such manifestations are difficult to communicate to others remotely. We propose EmotionActuator, a proof-of-concept system to investigate the transmission of emotional states in which the recipient performs emotional gestures to understand and interpret the state of the sender.We call this kind of communication embodied emotional feedback, and present a prototype implementation. To realize our concept we chose four emotional states: amused, sad, angry, and neutral. We designed EmotionActuator through a series of studies to assess emotional classification via EEG, and create an EMS gesture set by comparing composed gestures from the literature to sign-language gestures. Through a final study with the end-to-end prototype interviews revealed that participants like implicit sharing of emotions and find the embodied output to be immersive, but want to have control over shared emotions and with whom. This work contributes a proof of concept system and set of design recommendations for designing embodied emotional feedback systems.

Supplementary Material

suppl.mov (pn4033p.mp4)
Supplemental video

References

[1]
Leonardo Angelini, Maurizio Caon, Denis Lalanne, Omar Abou khaled, and Elena Mugellini. 2015. Towards an Anthropomorphic Lamp for Affective Interaction. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, New York, NY, USA, 661--666.
[2]
V. H. Anh, M. N. Van, B. B. Ha, and T. H. Quyet. 2012. A real-time model based Support Vector Machine for emotion recognition through EEG. In 2012 International Conference on Control, Automation and Information Sciences (ICCAIS). 191--196.
[3]
MJ Asmussen, AZ Bailey, PJ Keir, J Potvin, T Bergel, and AJ Nelson. 2015. Combining Multiple Data Acquisition Systems to Study Corticospinal Output and Multi-segment Biomechanics. Journal of Visualized Experiments 107 (2015).
[4]
Peter Aspinall, Panagiotis Mavros, Richard Coyne, and Jenny Roe. 2013. The urban brain: analysing outdoor physical activity with mobile EEG. British Journal of Sports Medicine (2013).
[5]
Armando Barreto, Jing Zhai, and Malek Adjouadi. 2007. Non-intrusive Physiological Monitoring for Automated Stress Detection in Human-Computer Interaction. In International Workshop on Human-Computer Interaction. Vol. 4796. Springer, Berlin Heidelberg, 29--38.
[6]
Natalya N. Bazarova, Yoon Hyung Choi, Victoria Schwanda Sosik, Dan Cosley, and Janis Whitlock. 2015. Social Sharing of Emotions on Facebook: Channel Differences, Satisfaction, and Replies. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work (CSCW '15). ACM, New York, NY, USA, 154--164.
[7]
Danny Oude Bos. 2008. EEG-based emotion recognition: The Influence of Visual and Auditory Stimuli . http://hmi.ewi.utwente.nl/verslagen/capit a-selecta/CS-Oude Bos-Danny.pdf. (2008). {Online; accessed 19-Jan.-2016}.
[8]
Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25, 1 (1994), 49 -- 59.
[9]
Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza Bulut, Chul Min Lee, Abe Kazemzadeh, Sungbok Lee, Ulrich Neumann, and Shrikanth Narayanan. 2004. Analysis of Emotion Recognition Using Facial Expressions, Speech and Multimodal Information. In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI '04). ACM, New York, NY, USA, 205--211.
[10]
John T Cacioppo, Richard E Petty, Mary E Losch, and Hai Sook Kim. 1986. Electromyographic activity over facial muscle regions can differentiate the valence and intensity of affective reactions. Journal of personality and social psychology 50, 2 (1986), 260.
[11]
Ginevra Castellano, Loic Kessous, and George Caridakis. 2008. Affect and Emotion in HumanComputer Interaction. Springer-Verlag, Berlin, Heidelberg, Chapter Emotion Recognition Through Multiple Modalities: Face, Body Gesture, Speech.
[12]
Karen Church, Eve Hoggan, and Nuria Oliver. 2010. A Study of Mobile Mood Awareness and Communication Through MobiMood. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI '10). ACM, New York, NY, USA, 128--137.
[13]
Yanqing Cui, Jari Kangas, Jukka Holm, and Guido Grassel. 2013. Front-camera Video Recordings As Emotion Responses to Mobile Photos Shared Within Close-knit Groups. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 981--990.
[14]
Nele Dael, Marcello Mortillaro, and Klaus R Scherer. 2012. Emotion expression in body action and posture. Emotion 12, 5 (2012), 1085--1101.
[15]
Marianne Dainton and Brooks Aylor. 2002. Patterns of communication channel use in the maintenance of long distance relationships. Communication Research Reports 19, 2 (2002), 118--129.
[16]
Alwin de Rooij and Sara Jones. 2015. (E)Motion and Creativity: Hacking the Function of Motor Expressions in Emotion Regulation to Augment Creativity. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, New York, NY, USA, 145--152.
[17]
Anind K. Dey and Ed de Guzman. 2006. From Awareness to Connectedness: The Design and Deployment of Presence Displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, New York, NY, USA, 899--908.
[18]
Chris Dodge. 1997. The Bed: A Medium for Intimate Communication. In CHI '97 Extended Abstracts on Human Factors in Computing Systems (CHI EA '97). ACM, New York, NY, USA, 371--372.
[19]
Sandra E Duclos, James D Laird, Eric Schneider, Melissa Sexter, Lisa Stern, and Oliver Van Lighten. 1989. Emotion-specific effects of facial expressions and postures on emotional experience. Journal of Personality and Social Psychology 57, 1 (1989), 100--108.
[20]
Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion 6, 3--4 (1992), 169--200.
[21]
Paul Ekman and Wallace V Friesen. 1967. Head and body cues in the judgment of emotion: A reformulation. Perceptual and motor skills 24, 3 (1967), 711--724.
[22]
Petra Fagerberg, Anna Stahl, and Kristina Höök. 2003. Designing Gestures for Affective Input: An Analysis of Shape, Effort and Valence. MUM 2003 : proceedings of the 2nd International Conference on Mobile and Ubiquitous Multimedia, 10--12 December, 2003, Norrköping, Sweden 57--65 (2003), 57--65.
[23]
Allan Fong, Zahra Ashktorab, and Jon Froehlich. 2013. Bear-with-me: An Embodied Prototype to Explore Tangible Two-way Exchanges of Emotional Language. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). ACM, New York, NY, USA, 1011--1016.
[24]
Gary Garcia-Molina, Tsvetomira Tsoneva, and Anton Nijholt. 2013. Emotional Brain-computer Interfaces. Int. J. Auton. Adapt. Commun. Syst. 6, 1 (Dec. 2013), 9--25.
[25]
Daniel Gooch and Leon Watts. 2012. YourGloves, Hothands and Hotmits: Devices to Hold Hands at a Distance. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 157--166.
[26]
Erik Grönvall, Jonas Fritsch, and Anna Vallgrda. 2016. FeltRadio: Sensing and Making Sense of Wireless Traffic. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS '16). ACM, New York, NY, USA, 829--840.
[27]
Hatice Gunes and Massimo Piccardi. 2007. Bi-modal Emotion Recognition from Expressive Face and Body Gestures. Journal of Network and Computer Applications 30, 4 (Nov. 2007), 1334--1345.
[28]
Marc Hassenzahl, Stephanie Heidecker, Kai Eckoldt, Sarah Diefenbach, and Uwe Hillmann. 2012. All You Need is Love: Current Strategies of Mediating Intimate Relationships Through Technology. ACM Trans. Comput.-Hum. Interact. 19, 4, Article 30 (Dec. 2012), 19 pages.
[29]
J. A. Healey and R. W. Picard. 2005. Detecting stress during real-world driving tasks using physiological sensors. Trans. Intell. Transport. Sys. 6, 2 (June 2005), 156--166.
[30]
Kristina Höök. 2008. Affective Loop Experiences -- What Are They?. In Proceedings of the 3rd International Conference on Persuasive Technology (PERSUASIVE '08). Springer-Verlag, Berlin, Heidelberg, 1--12.
[31]
Thierry Keller and Andreas Kuhn. 2008. Electrodes for transcutaneous (surface) electrical stimulation. Journal of Automatic Control 18, 2 (2008), 35--45.
[32]
L Kiers, Didier Cros, KH Chiappa, and J Fang. 1993. Variability of motor potentials evoked by transcranial magnetic stimulation. EEG and Clinical Neurophysiology/Evoked Potentials Section 89, 6 (1993), 415--423.
[33]
Jina Kim, Young-Woo Park, and Tek-Jin Nam. 2015. BreathingFrame: An Inflatable Frame for Remote Breath Signal Sharing. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, New York, NY, USA, 109--112.
[34]
Ernst Kruijff, Dieter Schmalstieg, and Steffi Beckhaus. 2006. Using Neuromuscular Electrical Stimulation for Pseudo-haptic Feedback. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '06). ACM, New York, NY, USA, 316--319.
[35]
James D Laird. 1984. The real role of facial response in the experience of emotion: A reply to Tourangeau and Ellsworth, and others. 47, 4 (1984), 909--917.
[36]
Irene Levin. 2004. Living Apart Together: A New Family Form. Current Sociology (2004), 223--240.
[37]
Y. Liu, O. Sourina, and M. K. Nguyen. 2010. Real-Time EEG-Based Human Emotion Recognition and Visualization. In 2010 International Conference on Cyberworlds. 262--269.
[38]
Pedro Lopes, Alexandra Ion, Willi Mueller, Daniel Hoffmann, Patrik Jonell, and Patrick Baudisch. 2015a. Proprioceptive Interaction. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 939--948.
[39]
Pedro Lopes, Patrik Jonell, and Patrick Baudisch. 2015b. Affordance++: Allowing Objects to Communicate Dynamic Use. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2515--2524.
[40]
Pedro Lopes, Doăa Yüksel, Françis Guimbreti'ere, and Patrick Baudisch. 2016. Muscle-plotter: An Interactive System Based on Electrical Muscle Stimulation That Produces Spatial Output. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 207--217.
[41]
Danielle Lottridge, Nicolas Masson, and Wendy Mackay. 2009. Sharing Empty Moments: Design for Remote Couples. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 2329--2338.
[42]
Marion L. Chapman M. Carole Pistole, Amber Roberts. 2010. Attachment, relationship maintenance, and stress in long distance and geographically close romantic relationships. Journal of Social and Personal Relationships 27, 4 (6 2010), 535--552.
[43]
Maurizio Mauri, Valentina Magagnin, Pietro Cipresso, Luca Mainardi, Emery N Brown, Sergio Cerutti, Marco Villamira, and Riccardo Barbieri. Psychophysiological signals associated with affective states. In Engineering in Medicine and Biology Society, 2010 Annual International Conference of the IEEE (EMBC). 3563--3566.
[44]
Dan Nie, Xiao-Wei Wang, Li-Chen Shi, and Bao-Liang Lu. EEG-based emotion recognition during watching movies. In 2011 5th International IEEE/EMBS Conference on Neural Engineering (NER). 667--670.
[45]
Young-Woo Park, Kyoung-Min Baek, and Tek-Jin Nam. 2013. The Roles of Touch During Phone Conversations: Long-distance Couples? Use of POKE in Their Homes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1679--1688.
[46]
Timo Partala, Maria Jokiniemi, and Veikko Surakka. 2000. Pupillary Responses to Emotionally Provocative Stimuli. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (ETRA '00). ACM, New York, NY, USA, 123--129.
[47]
M. Perakakis and A. Potamianos. 2012. Affective evaluation of a mobile multimodal dialogue system using brain signals. In 2012 IEEE Spoken Language Technology Workshop (SLT). 43--48.
[48]
Arttu Perttula, Antti Koivisto, Riikka Mäkelä, Marko Suominen, and Jari Multisilta. 2011. Social Navigation with the Collective Mobile Mood Monitoring System. In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments (MindTrek '11). ACM, New York, NY, USA, 117--124.
[49]
Max Pfeiffer, Tim Duente, and Michael Rohs. 2016. Let Your Body Move: A Prototyping Toolkit for Wearable Force Feedback with Electrical Muscle Stimulation. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '16). ACM, New York, NY, USA, 418--427.
[50]
Max Pfeiffer, Tim Dünte, Stefan Schneegass, Florian Alt, and Michael Rohs. 2015. Cruise Control for Pedestrians: Controlling Walking Direction Using Electrical Muscle Stimulation. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2505--2514.
[51]
M. Pfeiffer and W. Stuerzlinger. 2015. 3D virtual hand pointing with EMS and vibration feedback. In 2015 IEEE Symposium on 3D User Interfaces (3DUI). 117--120.
[52]
Rosalind W Picard and Roalind Picard. 1997. Affective computing. Vol. 252. MIT press Cambridge.
[53]
David Prutchi and Michael Norris. 2005. Design and development of medical electronic instrumentation: a practical perspective of the design, construction, and test of medical devices. John Wiley & Sons. 344 pages.
[54]
John Riskind and Carolyn Gotay. 1982. Physical posture: Could it have regulatory or feedback effects on motivation and emotion? Motivation and Emotion 6, 3 (1982).
[55]
James A. Russell. 1980. A circumplex model of affect. 39, 6 (1980), 1161--1178.
[56]
Abraham Savitzky and Marcel JE Golay. 1964. Smoothing and differentiation of data by simplified least squares procedures. Analytical chemistry 36, 8 (1964).
[57]
Alexandre Schaefer, Frédéric Nils, Xavier Sanchez, and Pierre Philippot. 2010. Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers. Cognition and Emotion 24, 7 (2010).
[58]
Yu Shi, Natalie Ruiz, Ronnie Taib, Eric Choi, and Fang Chen. 2007. Galvanic Skin Response (GSR) As an Index of Cognitive Load. In CHI '07 Extended Abstracts on Human Factors in Computing Systems (CHI EA '07). ACM, New York, NY, USA, 2651--2656.
[59]
Magdalena W Sliwinska, Sylvia Vitello, and Joseph T Devlin. 2014. Transcranial magnetic stimulation for investigating causal brain-behavioral relationships and their time course. Journal of Visualized Experiments 89 (2014).
[60]
Paul Strohmeier, Juan Pablo Carrascal, Bernard Cheng, Margaret Meban, and Roel Vertegaal. 2016. An Evaluation of Shape Changes for Conveying Emotions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3781--3792.
[61]
Petra Sundström, Anna Sthl, and Kristina Höök. 2007. In Situ Informants Exploring an Emotional Mobile Messaging System in Their Everyday Practice. Int. J. Hum.-Comput. Stud. 65, 4 (April 2007), 388--403.
[62]
Emi Tamaki, Takashi Miyaki, and Jun Rekimoto. 2011. PossessedHand: Techniques for Controlling Human Hands Using Electrical Muscles Stimuli. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 543--552.
[63]
Dzmitry Tsetserukou and Alena Neviarouskaya. 2010. World's First Wearable Humanoid Robot That Augments Our Emotions. In Proceedings of the 1st Augmented Human International Conference (AH '10). ACM, New York, NY, USA, Article 8, 10 pages.
[64]
Harald G. Wallbott. 1998. Bodily expression of emotion. European Journal of Social Psychology 28, 6 (1998), 879--896.
[65]
David Watson, Lee A Clark, and Auke Tellegen. 1988. Development and validation of brief measures of positive and negative affect: The PANAS scales. 54, 6 (1988).
[66]
Julia Werner, Reto Wettach, and Eva Hornecker. 2008. United-pulse: Feeling Your Partner's Pulse. In Proceedings of the 10th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '08). ACM, New York, NY, USA, 535--538.
[67]
Matthew John Willis and Christian Martyn Jones. 2012. Emotishare: Emotion Sharing on Mobile Devices. In Proceedings of the 26th Annual BCS Interaction Specialist Group Conference on People and Computers (BCS-HCI '12). British Computer Society, Swinton, UK, UK, 292--297. http://dl.acm.org/citation.cfm?id=2377916.2377954
[68]
Zhihong Zeng, Maja Pantic, Glenn Roisman, Thomas S Huang, and others. 2009. A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions. IIEEE Transactions on Pattern Analysis and Machine Intelligence 31, 1 (2009), 39--58.

Cited By

View all
  • (2024)Shared Bodily Fusion: Leveraging Inter-Body Electrical Muscle Stimulation for Social PlayProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660723(2088-2106)Online publication date: 1-Jul-2024
  • (2024)Co-Presence, Connection and Co-Creation: Building Real-time Cross-person Neurofeedback InteractionsACM SIGGRAPH 2024 Talks10.1145/3641233.3664334(1-2)Online publication date: 18-Jul-2024
  • (2024)EmoEcho: Designing Emotion Mimicry Mechanics for Enhancing Social Engagement in Digital GamesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650864(1-6)Online publication date: 11-May-2024
  • Show More Cited By

Index Terms

  1. Emotion Actuator: Embodied Emotional Feedback through Electroencephalography and Electrical Muscle Stimulation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. affect display
    2. affective computing
    3. eeg.
    4. emotion
    5. emotion sharing
    6. ems

    Qualifiers

    • Research-article

    Funding Sources

    • European Research Council

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)99
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Shared Bodily Fusion: Leveraging Inter-Body Electrical Muscle Stimulation for Social PlayProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660723(2088-2106)Online publication date: 1-Jul-2024
    • (2024)Co-Presence, Connection and Co-Creation: Building Real-time Cross-person Neurofeedback InteractionsACM SIGGRAPH 2024 Talks10.1145/3641233.3664334(1-2)Online publication date: 18-Jul-2024
    • (2024)EmoEcho: Designing Emotion Mimicry Mechanics for Enhancing Social Engagement in Digital GamesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650864(1-6)Online publication date: 11-May-2024
    • (2024)SynCocreate: Fostering Interpersonal Connectedness via Brainwave-Driven Co-creation in Virtual RealityExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3648650(1-5)Online publication date: 11-May-2024
    • (2024)Understanding User Acceptance of Electrical Muscle Stimulation in Human-Computer InteractionProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642585(1-16)Online publication date: 11-May-2024
    • (2024)An Immersive and Interactive VR Dataset to Elicit EmotionsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345620230:11(7343-7353)Online publication date: Nov-2024
    • (2024)A multimodal driver monitoring benchmark dataset for driver modeling in assisted driving automationScientific Data10.1038/s41597-024-03137-y11:1Online publication date: 30-Mar-2024
    • (2024)New haptic systems for elicit emotions in audio-visual events for hearing impaired people.Procedia Computer Science10.1016/j.procs.2024.05.137237:C(533-543)Online publication date: 24-Jul-2024
    • (2023)SleepyFlora: Supporting Sleep Sharing and Augmentation over a Distance for Social Bonding across Time ZonesCompanion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3584931.3606961(76-80)Online publication date: 14-Oct-2023
    • (2023)Mirror Hearts: Exploring the (Mis-)Alignment between AI-Recognized and Self-Reported EmotionsExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585607(1-7)Online publication date: 19-Apr-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media