Skip to main content
SpringerLink
Log in

Tracing Stress and Arousal in Virtual Reality Games Using Players’ Motor and Vocal Behaviour

  • Conference paper
  • First Online:
Computer-Human Interaction Research and Applications (CHIRA 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1996))

  • 127 Accesses

Abstract

In this study, we tackle the integration of stressors and voice interaction in a Virtual Reality game to assess players’ arousal and stress levels. The selected game genre and its characteristic components are used as a basis to create stress-inducing elements. Additionally, a voice interaction module has been created using a voice assistant called Minerva. The module allows for real-time detection and recording of players’ emotional responses based on variations in pitch and intensity of speech. The game consists of a single level divided into four areas with increasing levels of stress. The experiment involved 16 volunteer students who played the game while their prosodic and behavioral movement data were collected. Participants also completed questionnaires and produced ratings to assess their perceived stress and arousal levels. The collected data were analyzed to evaluate the effectiveness of the real-time estimation of arousal and stress.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://store.facebook.com/it/quest/products/quest-2/.

  2. 2.

    https://docs.unity3d.com/ScriptReference/Windows.Speech.DictationRecognizer.html.

References

  1. Akçay, M.B., Oğuz, K.: Speech emotion recognition: emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers. Speech Commun. 116, 56–76 (2020)

    Article  Google Scholar 

  2. Balzarotti, S., Piccini, L., Andreoni, G., Ciceri, R.: “I know that you know how I feel’’: Behavioral and physiological signals demonstrate emotional attunement while interacting with a computer simulating emotional intelligence. J. Nonverbal Behav. 38, 283–299 (2014)

    Article  Google Scholar 

  3. Banse, R., Scherer, K.R.: Acoustic profiles in vocal emotion expression. J. Pers. Soc. Psychol. 70(3), 614 (1996)

    Article  Google Scholar 

  4. Bänziger, T., Scherer, K.R.: The role of intonation in emotional expressions. Speech Commun. 46(3–4), 252–267 (2005)

    Article  Google Scholar 

  5. Barrett, L.F.: The theory of constructed emotion: an active inference account of interoception and categorization. Soc. Cogn. Affect. Neurosci. 12(1), 1–23 (2017)

    Google Scholar 

  6. Barrett, L.F., Bliss-Moreau, E.: Affect as a psychological primitive. Adv. Exp. Soc. Psychol. 41, 167–218 (2009)

    Article  Google Scholar 

  7. Barrett, L.F., Satpute, A.B.: Historical pitfalls and new directions in the neuroscience of emotion. Neurosci. Lett. 693, 9–18 (2019)

    Article  Google Scholar 

  8. Boccignone, G., Conte, D., Cuculo, V., Lanzarotti, R.: Amhuse: a multimodal dataset for humour sensing. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, pp. 438–445 (2017)

    Google Scholar 

  9. Boersma, P.: Praat, a system for doing phonetics by computer. Glot. Int. 5(9), 341–345 (2001)

    Google Scholar 

  10. Bone, D., Lee, C.C., Narayanan, S.: Robust unsupervised arousal rating: a rule-based framework with knowledge-inspired vocal features. IEEE Trans. Affect. Comput. 5(2), 201–213 (2014)

    Article  Google Scholar 

  11. Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)

    Article  Google Scholar 

  12. Brambilla, S., Boccignone, G., Borghese, N., Ripamonti, L.A.: Between the buttons: stress assessment in video games using players’ behavioural data. In: Proceedings of the 6th International Conference on Computer-Human Interaction Research and Applications - CHIRA, pp. 59–69. INSTICC, SciTePress (2022). https://doi.org/10.5220/0011546400003323

  13. Burkhart, M.C., Brandman, D.M., Franco, B., Hochberg, L.R., Harrison, M.T.: The discriminative Kalman filter for Bayesian filtering with nonlinear and nongaussian observation models. Neural Comput. 32(5), 969–1017 (2020)

    Article  MathSciNet  Google Scholar 

  14. Busso, C., Lee, S., Narayanan, S.: Analysis of emotionally salient aspects of fundamental frequency for emotion detection. IEEE Trans. Audio Speech Lang. Process. 17(4), 582–596 (2009)

    Article  Google Scholar 

  15. Cohen, S., Kamarck, T., Mermelstein, R.: A global measure of perceived stress. J. Health Soc. Beh. 385–396 (1983)

    Google Scholar 

  16. Corcoran, A.W., Pezzulo, G., Hohwy, J.: From allostatic agents to counterfactual cognisers: active inference, biological regulation, and the origins of cognition. Biol. Philos. 35(3), 32 (2020)

    Article  Google Scholar 

  17. Csikszentmihalyi, M., Csikszentmihalyi, M.: Toward a psychology of optimal experience. Flow and the foundations of positive psychology: the collected works of Mihaly Csikszentmihalyi, pp. 209–226 (2014)

    Google Scholar 

  18. D’Amelio, A., Patania, S., Buršić, S., Cuculo, V., Boccignone, G.: Inferring causal factors of core affect dynamics on social participation through the lens of the observer. Sensors 23(6), 2885 (2023)

    Article  Google Scholar 

  19. Friston, K.: The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11(2), 127–138 (2010)

    Article  Google Scholar 

  20. Frommel, J., Fischbach, F., Rogers, K., Weber, M.: Emotion-based dynamic difficulty adjustment using parameterized difficulty and self-reports of emotion. In: Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play, pp. 163–171 (2018)

    Google Scholar 

  21. Frommel, J., Schrader, C., Weber, M.: Towards emotion-based adaptive games: emotion recognition via input and performance features. In: Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play, pp. 173–185 (2018)

    Google Scholar 

  22. Hirst, D.J., de Looze, C.: Measuring Speech. Fundamental Frequency and Pitch., pp. 336–361. Cambridge University Press (2021)

    Google Scholar 

  23. Juslin, P.N., Scherer, K.R.: Vocal Expression of Affect. Oxford University Press, Oxford (2005)

    Book  Google Scholar 

  24. Khalil, R.A., Jones, E., Babar, M.I., Jan, T., Zafar, M.H., Alhussain, T.: Speech emotion recognition using deep learning techniques: a review. IEEE Access 7, 117327–117345 (2019)

    Article  Google Scholar 

  25. Kim, Y., Provost, E.M.: Emotion classification via utterance-level dynamics: a pattern-based approach to characterizing affective expressions. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3677–3681. IEEE (2013)

    Google Scholar 

  26. Lebois, L.A., Hertzog, C., Slavich, G.M., Barrett, L.F., Barsalou, L.W.: Establishing the situated features associated with perceived stress. Acta Psychol. 169, 119–132 (2016)

    Article  Google Scholar 

  27. Linson, A., Parr, T., Friston, K.J.: Active inference, stressors, and psychological trauma: a neuroethological model of (mal) adaptive explore-exploit dynamics in ecological context. Behav. Brain Res. 380, 112421 (2020)

    Article  Google Scholar 

  28. Nogueira, P.A., Torres, V., Rodrigues, R., Oliveira, E., Nacke, L.E.: Vanishing scares: biofeedback modulation of affective player experiences in a procedural horror game. J. Multimodal User Interfaces 10(1), 31–62 (2016)

    Article  Google Scholar 

  29. Pallavicini, F., Ferrari, A., Pepe, A., Garcea, G., Zanacchi, A., Mantovani, F.: Effectiveness of virtual reality survival horror games for the emotional elicitation: preliminary insights using resident evil 7: biohazard. In: Antona, M., Stephanidis, C. (eds.) UAHCI 2018. LNCS, vol. 10908, pp. 87–101. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-92052-8_8

    Chapter  Google Scholar 

  30. Peters, A., McEwen, B.S., Friston, K.: Uncertainty and stress: why it causes diseases and how it is mastered by the brain. Prog. Neurobiol. 156, 164–188 (2017)

    Article  Google Scholar 

  31. Quigley, K.S., Lindquist, K.A., Barrett, L.F.: Inducing and measuring emotion and affect: tips, tricks, and secrets. In: Reis, H.T., Judd, C.M. (eds.) Handbook of Research Methods in Social and Personality Psychology, pp. 220–252. Cambridge University Press, New York (2014)

    Chapter  Google Scholar 

  32. Russell, J.A.: Core affect and the psychological construction of emotion. Psychol. Rev. 110(1), 145 (2003)

    Article  Google Scholar 

  33. Schell, J.: The Art of Game Design: A book of lenses. CRC Press (2008)

    Google Scholar 

  34. Scherer, K.R.: Vocal communication of emotion: a review of research paradigms. Speech Commun. 40(1–2), 227–256 (2003)

    Article  Google Scholar 

  35. Schulkin, J., Sterling, P.: Allostasis: a brain-centered, predictive mode of physiological regulation. Trends Neurosci. 42(10), 740–752 (2019)

    Article  Google Scholar 

  36. Schuller, D.M., Schuller, B.W.: A review on five recent and near-future developments in computational processing of emotion in the human voice. Emot. Rev. 13(1), 44–50 (2021)

    Article  Google Scholar 

  37. Shah Fahad, M., Ranjan, A., Yadav, J., Deepak, A.: A survey of speech emotion recognition in natural environment. Digit. Signal Proc. 110, 102951 (2021)

    Article  Google Scholar 

  38. Singh, Y.B., Goel, S.: A systematic literature review of speech emotion recognition approaches. Neurocomputing 492, 245–263 (2022)

    Article  Google Scholar 

  39. Solms, M., Friston, K.: How and why consciousness arises: some considerations from physics and physiology. J. Conscious. Stud. 25(5–6), 202–238 (2018)

    Google Scholar 

  40. Vachiratamporn, V., Legaspi, R., Moriyama, K., Numao, M.: Towards the design of affective survival horror games: an investigation on player affect. In: 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, pp. 576–581 (2013)

    Google Scholar 

  41. Yang, W., Rifqi, M., Marsala, C., Pinna, A.: Physiological-based emotion detection and recognition in a video game context. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2018)

    Google Scholar 

Download references

Acknowledgement

This work has been partially supported by EC H2020 ESSENCE project, Grant number 101016112.

Author information

Authors and Affiliations

  1. University of Milan, 20133, Milan, Italy

    Susanna Brambilla, Giuseppe Boccignone, N. Alberto Borghese, Eleonora Chitti, Riccardo Lombardi & Laura A. Ripamonti

Authors
  1. Susanna Brambilla
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giuseppe Boccignone
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. N. Alberto Borghese
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eleonora Chitti
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Riccardo Lombardi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Laura A. Ripamonti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Susanna Brambilla .

Editor information

Editors and Affiliations

  1. Instituto Superior Técnico, IT - Institute of Telecommunications, Lisbon, Portugal

    Hugo Plácido da Silva

  2. University of Turin, Turin, Italy

    Pietro Cipresso

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Brambilla, S., Boccignone, G., Borghese, N.A., Chitti, E., Lombardi, R., Ripamonti, L.A. (2023). Tracing Stress and Arousal in Virtual Reality Games Using Players’ Motor and Vocal Behaviour. In: da Silva, H.P., Cipresso, P. (eds) Computer-Human Interaction Research and Applications. CHIRA 2023. Communications in Computer and Information Science, vol 1996. Springer, Cham. https://doi.org/10.1007/978-3-031-49425-3_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-49425-3_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-49424-6

  • Online ISBN: 978-3-031-49425-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation