skip to main content
10.1145/3385956.3422122acmconferencesArticle/Chapter ViewAbstractPublication PagesvrstConference Proceedingsconference-collections
abstract

AffectivelyVR: Towards VR Personalized Emotion Recognition

Published: 01 November 2020 Publication History

Abstract

We present AffectivelyVR, a personalized real-time emotion recognition system in Virtual Reality (VR) that enables an emotion-adaptive virtual environment. We used off-the-shelf Electroencephalogram (EEG) and Galvanic Skin Response (GSR) physiological sensors to train user-specific machine learning models while exposing users to affective 360° VR videos. Since emotions are largely dependent on interpersonal experiences and expressed in different ways for different people, we personalize the model instead of generalizing it. By doing this, we achieved an emotion recognition rate of 96.5% using the personalized KNN algorithm, and 83.7% using the generalized SVM algorithm.

References

[1]
Rosa María Baños, Cristina Botella, Mariano Alcañiz, Víctor Liaño, Belén Guerrero, and Beatriz Rey. 2004. Immersion and emotion: their impact on the sense of presence. Cyberpsychology & behavior 7, 6 (2004), 734–741.
[2]
Rosa María Baños, Víctor Liaño, Cristina Botella, Mariano Alcañiz, Belén Guerrero, and Beatriz Rey. 2006. Changing induced moods via virtual reality. In International Conference on Persuasive Technology. Springer, 7–15.
[3]
Anthony J Bishara and James B Hittner. 2012. Testing the significance of a correlation with nonnormal data: comparison of Pearson, Spearman, transformation, and resampling approaches.Psychological methods 17, 3 (2012), 399.
[4]
Rafael A Calvo and Sidney D’Mello. 2010. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on affective computing 1, 1 (2010), 18–37.
[5]
Carlos Carreiras, Ana Priscila Alves, André Lourenço, Filipe Canento, Hugo Silva, Ana Fred, 2018. BioSPPy: Biosignal Processing in Python.
[6]
Charles Darwin and Phillip Prodger. 1998. The expression of the emotions in man and animals. Oxford University Press, USA.
[7]
Arindam Dey, Thammathip Piumsomboon, Youngho Lee, and Mark Billinghurst. 2017. Effects of sharing physiological states of players in a collaborative virtual reality gameplay. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 4045–4056.
[8]
Ruo-Nan Duan, Xiao-Wei Wang, and Bao-Liang Lu. 2012. EEG-based emotion recognition in listening music by using support vector machine and linear dynamic system. In International Conference on Neural Information Processing. Springer, 468–475.
[9]
Kunal Gupta, Ryo Hajika, Yun Suen Pai, Andreas Duenser, Martin Lochner, and Mark Billinghurst. 2019. In AI We Trust: Investigating the Relationship between Biosignals, Trust and Cognitive Load in VR. In 25th ACM Symposium on Virtual Reality Software and Technology. 1–10.
[10]
Kunal Gupta, Ryo Hajika, Yun Suen Pai, Andreas Duenser, Martin Lochner, and Mark Billinghurst. 2020. Measuring Human Trust in a Virtual Assistant using Physiological Sensing in Virtual Reality. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 756–765.
[11]
Daekyo Jeong, Sangbong Yoo, and Jang Yun. 2019. Cybersickness analysis with eeg using deep learning algorithms. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 827–835.
[12]
Jinwoo Kim, Woojae Kim, Heeseok Oh, Seongmin Lee, and Sanghoon Lee. 2019. A deep cybersickness predictor based on brain signal analysis for virtual reality contents. In Proceedings of the IEEE International Conference on Computer Vision. 10580–10589.
[13]
Bahareh Nakisa, Mohammad Naim Rastgoo, Dian Tjondronegoro, and Vinod Chandran. 2018. Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors. Expert Systems with Applications 93 (2018), 143–155.
[14]
Mehdi Naseriparsa and Mohammad Mansour Riahi Kashani. 2014. Combination of PCA with SMOTE resampling to boost the prediction rate in lung cancer dataset. arXiv preprint arXiv:1403.1949(2014).
[15]
Rosalind W Picard. 2000. Affective computing. MIT press.
[16]
Thammathip Piumsomboon, Youngho Lee, Gun A Lee, Arindam Dey, and Mark Billinghurst. 2017. Empathic mixed reality: Sharing what you feel and interacting with what you see. In 2017 International Symposium on Ubiquitous Virtual Reality (ISUVR). IEEE, 38–41.
[17]
Ben Somers, Tom Francart, and Alexander Bertrand. 2018. A generic EEG artifact removal algorithm based on the multi-channel Wiener filter. Journal of neural engineering 15, 3 (2018), 036007.
[18]
Goran Udovičić, Jurica Ðerek, Mladen Russo, and Marjan Sikora. 2017. Wearable emotion recognition system based on GSR and PPG signals. In Proceedings of the 2nd International Workshop on Multimedia for Personal Health and Health Care. 53–59.

Cited By

View all
  • (2024)Towards Emotionally Intelligent Virtual Environments: Classifying Emotions through a Biosignal-Based ApproachApplied Sciences10.3390/app1419876914:19(8769)Online publication date: 28-Sep-2024
  • (2024)Physiology-Driven Empathic Large Language Models (EmLLMs) for Mental Health SupportExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651132(1-5)Online publication date: 11-May-2024
  • (2024)CAEVR: Biosignals-Driven Context-Aware Empathy in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337213030:5(2671-2681)Online publication date: 4-Mar-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
VRST '20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology
November 2020
429 pages
ISBN:9781450376198
DOI:10.1145/3385956
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2020

Check for updates

Author Tags

  1. EEG
  2. Emotion Recognition
  3. GSR
  4. Machine Learning
  5. Personalized
  6. Physiology
  7. Virtual Reality

Qualifiers

  • Abstract
  • Research
  • Refereed limited

Conference

VRST '20

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)103
  • Downloads (Last 6 weeks)6
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Towards Emotionally Intelligent Virtual Environments: Classifying Emotions through a Biosignal-Based ApproachApplied Sciences10.3390/app1419876914:19(8769)Online publication date: 28-Sep-2024
  • (2024)Physiology-Driven Empathic Large Language Models (EmLLMs) for Mental Health SupportExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651132(1-5)Online publication date: 11-May-2024
  • (2024)CAEVR: Biosignals-Driven Context-Aware Empathy in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337213030:5(2671-2681)Online publication date: 4-Mar-2024
  • (2024)Virtual Reality and Emotional Responses: A Comprehensive Literature Review on Theories, Frameworks, and Research GapsITM Web of Conferences10.1051/itmconf/2024630102263(01022)Online publication date: 13-Feb-2024
  • (2024)EEG-based affective computing in virtual reality with a balancing of the computational efficiency and recognition accuracyComputers in Human Behavior10.1016/j.chb.2023.108085152:COnline publication date: 14-Mar-2024
  • (2023)SensoryScape: Context-Aware Empathic VR PhotographySIGGRAPH Asia 2023 XR10.1145/3610549.3614596(1-2)Online publication date: 28-Nov-2023
  • (2023)EEG-Based Brain-Computer Interactions in Immersive Virtual and Augmented Reality: A Systematic ReviewProceedings of the ACM on Human-Computer Interaction10.1145/35932267:EICS(1-33)Online publication date: 19-Jun-2023
  • (2023)Heightened Empathy: A Multi-user Interactive Experience in a Bioresponsive Virtual RealityACM SIGGRAPH 2023 Immersive Pavilion10.1145/3588027.3595599(1-2)Online publication date: 23-Jul-2023
  • (2023)VRdoGraphy: An Empathic VR Photography Experience2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00352(1013-1014)Online publication date: Mar-2023
  • (2023)Virtual Reality for Emotion Elicitation – A ReviewIEEE Transactions on Affective Computing10.1109/TAFFC.2022.318105314:4(2626-2645)Online publication date: 1-Oct-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media