Skip to main content
SpringerLink
Log in

Towards instrumental quality assessment of multisensory immersive experiences using a biosensor-equipped head-mounted display

  • Research Article
  • Published:
Quality and User Experience Aims and scope Submit manuscript

Abstract

Multisensory virtual reality (VR) experiences, which include haptic and/or olfactory feedback, are growing in popularity. Notwithstanding, an understanding of their impact on perceived quality of experience (QoE), and on several QoE factors, such as sense of presence, immersion, realism, engagement, emotions, and cybersickness, is still limited. Here, we explore the use of a biosensor-equipped VR head-mounted display (termed sHMD) to measure correlates of these QoE factors directly from the collected neurophysiological signals. An experiment with 11 participants was performed and showed that adding haptic and olfactory feedback significantly improved the users’ motivations, sense of presence, task engagement, and overall QoE relative to a conventional audio-visual immersive experience. Moreover, several metrics derived from the biosignals showed significant correlations with subjective measures of presence, realism, emotions, and overall QoE. These findings suggest that wearables could potentially be used to track QoE-related measures, thus allowing for personalized multisensory experiences to be created “on the fly”.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Amini Gougeh R, Falk TH (2022) Multisensory immersive experiences: a pilot study on subjective and instrumental human influential factors assessment. In: 2022 14th International conference on quality of multimedia experience (QoMEX), IEEE, pp 1–6

  2. Arguel A, Lockyer L, Chai K et al (2019) Puzzle-solving activity as an indicator of epistemic confusion. Front Psychol 10:163

    Article  Google Scholar 

  3. Bakdash JZ, Marusich LR (2017) Repeated measures correlation. Front Psychol 8:456

    Article  Google Scholar 

  4. Benali-Khoudjal M, Hafez M, Alexandre JM, et al (2003) Thermal feedback model for virtual reality. In: MHS2003. Proceedings of 2003 international symposium on micromechatronics and human science (IEEE Cat. No. 03TH8717), IEEE, pp 153–158

  5. Bernardi L, Leuzzi S, Radaelli A et al (1994) Low-frequency spontaneous fluctuations of rr interval and blood pressure in conscious humans: a baroreceptor or central phenomenon? Clin Sci 87(6):649–654

    Article  Google Scholar 

  6. Boster JS (2005) Emotion categories across languages. In: Handbook of categorization in cognitive science. Elsevier, pp 313–352

  7. Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Therapy Exp Psychiatr 25(1):49–59

    Article  Google Scholar 

  8. Brockmyer JH, Fox CM, Curtiss KA et al (2009) The development of the game engagement questionnaire: a measure of engagement in video game-playing. J Exp Soc Psychol 45(4):624–634

    Article  Google Scholar 

  9. Brooks J, Lopes P, Amores J, et al (2021) Smell, taste, and temperature interfaces. In: Extended abstracts of the 2021 CHI conference on human factors in computing systems, pp 1–6

  10. Cassani R, Moinnereau MA, Ivanescu L et al (2020) Neural interface instrumented virtual reality headsets: toward next-generation immersive applications. IEEE Syst Man Cybern Magaz 6(3):20–28

    Article  Google Scholar 

  11. Castaneda D, Esparza A, Ghamari M et al (2018) A review on wearable photoplethysmography sensors and their potential future applications in health care. Int J Biosens Bioelectron 4(4):195

    Google Scholar 

  12. Cavdar SK, Taskaya-Temizel T, Musolesi M et al (2020) A multi-perspective analysis of social context and personal factors in office settings for the design of an effective mobile notification system. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4(1):1–38

    Article  Google Scholar 

  13. Chalmers A, Ferko A (2008) Levels of realism: From virtual reality to real virtuality. In: Proceedings of the 24th spring conference on computer graphics, pp 19–25

  14. Chatterjee R, Bandyopadhyay T (2016) Eeg based motor imagery classification using svm and mlp. In: 2016 2nd international conference on computational intelligence and networks (CINE), IEEE, pp 84–89

  15. Cheok AD, Karunanayaka K (2018) Virtual taste and smell technologies for multisensory internet and virtual reality. Springer

    Book  Google Scholar 

  16. Cho D, Ham J, Oh J et al (2017) Detection of stress levels from biosignals measured in virtual reality environments using a kernel-based extreme learning machine. Sensors 17(10):2435

    Article  Google Scholar 

  17. Comşa IS, Saleme EB, Covaci A et al (2019) Do i smell coffee? the tale of a 360 mulsemedia experience. IEEE MultiMedia 27(1):27–36

    Google Scholar 

  18. Delorme A, Makeig S (2004) Eeglab: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 134(1):9–21

    Article  Google Scholar 

  19. Elliott AC, Woodward WA (2007) Statistical analysis quick reference guidebook: With SPSS examples. Sage

  20. Feick M, Kleer N, Tang A, et al (2020) the virtual reality questionnaire toolkit. In: The 33st annual ACM symposium on user interface software and technology adjunct proceedings. Association for Computing Machinery, New York, NY, USA, UIST ’20 Adjunct, https://doi.org/10.1145/3379350.3416188, event-place: Minneapolis, Minnesota, USA

  21. Feng M, Dey A, Lindeman RW (2016) The effect of multi-sensory cues on performance and experience during walking in immersive virtual environments. In: 2016 IEEE virtual reality (VR), IEEE, pp 173–174

  22. Freeman FG, Mikulka PJ, Prinzel LJ et al (1999) Evaluation of an adaptive automation system using three EEG indices with a visual tracking task. Biol Psychol 50(1):61–76

    Article  Google Scholar 

  23. Giraldo S, Ramirez R (2013) Brain-activity-driven real-time music emotive control. In: The 3rd international conference on music & emotion, Jyväskylä, Finland, June 11-15, 2013, University of Jyväskylä, Department of Music

  24. Halbig A, Latoschik ME (2021) A systematic review of physiological measurements, factors, methods, and applications in virtual reality. Front Virtual Real 2:89

    Article  Google Scholar 

  25. Honegger F, Feng Y, Rauterberg M (2021) Multimodality for passive experience: effects of visual, auditory, vibration and draught stimuli on sense of presence. J Univ Comput Sci 27(6):582–608

    Google Scholar 

  26. Hussain N, Ghinea G (2022) Guidelines for evaluating wearables’ quality of experience in a mulsemedia context. Multimed Tools Appl pp 1–32

  27. Ishiguro T, Suzuki C, Nakakoji H et al (2019) Immersive experience influences eye blink rate during virtual reality gaming. Polish Psychol Bullet 50(1):49–53

    Google Scholar 

  28. Jasper HH (1958) The ten-twenty electrode system of the international federation. Electroencephalogr Clin Neurophysiol 10:370–375

    Google Scholar 

  29. Jezler O, Gatti E, Gilardi M, et al (2016) Scented material: changing features of physical creations based on odors. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems, pp 1677–1683

  30. Karimi Moridani M, Setarehdan SK, Motie Nasrabadi A et al (2016) Non-linear feature extraction from hrv signal for mortality prediction of icu cardiovascular patient. J Med Eng Technol 40(3):87–98

    Article  Google Scholar 

  31. Kerruish E (2019) Arranging sensations: smell and taste in augmented and virtual reality. Sens Soc 14(1):31–45

    Article  Google Scholar 

  32. Kim K, Bruder G, Welch G (2018) Blowing in the wind: increasing copresence with a virtual human via airflow influence in augmented reality. In: International conference on artificial reality and telexistence and eurographics symposium on virtual environments

  33. Kothe CAE, Jung TP (2016) Artifact removal techniques with signal reconstruction. US Patent App. 14/895,440

  34. Kourtesis P, Argelaguet F, Vizcay S, et al (2022) Electrotactile feedback applications for hand and arm interactions: a systematic review, meta-analysis, and future directions. IEEE transactions on haptics

  35. Kreimeier J, Hammer S, Friedmann D, et al (2019) Evaluation of different types of haptic feedback influencing the task-based presence and performance in virtual reality. In: Proceedings of the 12th ACM international conference on pervasive technologies related to assistive environments, pp 289–298

  36. Ku J, Jang HJ, Kim KU et al (2005) Experimental results of affective valence and arousal to avatar’s facial expressions. CyberPsychol Behav 8(5):493–503

    Article  Google Scholar 

  37. Laurans G, Desmet PM (2017) Developing 14 animated characters for non-verbal self-report of categorical emotions. J Des Res 15(3–4):214–233

    Google Scholar 

  38. Lee J, Hwang S, Kim K, et al (2022) Auditory and olfactory stimuli-based attractors to induce reorientation in virtual reality forward redirected walking. In: CHI conference on human factors in computing systems extended abstracts, pp 1–7

  39. Libriandy E, Arlini Puspasari M (2020) Immersive virtual reality and gamification evaluation on treadmill exercise by using electrophysiological monitoring device. In: 2020 The 6th international conference on industrial and business engineering, pp 191–195

  40. Marusich LR, Bakdash JZ (2021) rmcorrshiny: a web and standalone application for repeated measures correlation. F1000Research 10

  41. Mäyrä F, Ermi L (2011) Fundamental components of the gameplay experience. Digarec Ser 6:88–115

    Google Scholar 

  42. Melo M, Gonçalves G, Monteiro P, et al (2020) Do multisensory stimuli benefit the virtual reality experience? a systematic review. IEEE transactions on visualization and computer graphics

  43. Mesfin G, Saleme EB, Ademoye OA et al (2020) Less is (just as good as) more-an investigation of odor intensity and hedonic valence in mulsemedia QOE using heart rate and eye tracking. IEEE Transact Multimed 23:1095–1105

    Article  Google Scholar 

  44. Michela A, van Peer JM, Brammer JC, et al (2022) Deep-breathing biofeedback trainability in a virtual-reality action game: a single-case design study with police trainers

  45. Moinnereau MA, Oliveira A, Falk TH (2022) Human influential factors assessment during at-home gaming with an instrumented vr headset. In: 2022 14th international conference on quality of multimedia experience (QoMEX), IEEE, pp 1–4

  46. Moinnereau MA, Oliveira AAD, Falk TH (2023) Quantifying time perception during virtual reality gameplay using a multimodal biosensor-instrumented headset: a feasibility study. Front Neuroergonomics 4(1189):179

    Google Scholar 

  47. Montano N, Porta A, Cogliati C et al (2009) Heart rate variability explored in the frequency domain: a tool to investigate the link between heart and behavior. Neurosci Biobehav Rev 33(2):71–80

    Article  Google Scholar 

  48. Mousavi M, Jen YH, Musa SNB (2013) A review on cybersickness and usability in virtual environments. In: Advanced engineering forum, Trans Tech Publ, pp 34–39

  49. Munyan BG III, Neer SM, Beidel DC et al (2016) Olfactory stimuli increase presence in virtual environments. PloS one 11(6):e0157,568

    Article  Google Scholar 

  50. Murray N, Qiao Y, Lee B et al (2013) Age and gender influence on perceived olfactory & visual media synchronization. In: 2013 IEEE international conference on multimedia and expo (ICME), IEEE, pp 1–6

  51. Murray N, Lee B, Qiao Y et al (2016) Olfaction-enhanced multimedia: A survey of application domains, displays, and research challenges. ACM Comput Surv (CSUR) 48(4):1–34

    Article  Google Scholar 

  52. Mycka M, Czajka A (2018) Similar but not the same: application of EEG frontal alpha asymmetry to game research. Homo Ludens 1(11):173–187

    Google Scholar 

  53. Nacke LE, Stellmach S, Lindley CA (2011) Electroencephalographic assessment of player experience: a pilot study in affective ludology. Simul Gaming 42(5):632–655

    Article  Google Scholar 

  54. Obrist M, Velasco C, Vi CT, et al (2016) Touch, taste, & smell user interfaces: the future of multisensory hci. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems, pp 3285–3292

  55. Peck TC, Gonzalez-Franco M (2021) Avatar embodiment. a standardized questionnaire. Front Virtual Real 1:575,943

    Article  Google Scholar 

  56. Perkis A, Timmerer C, Baraković S, et al (2020) QUALINET white paper on definitions of immersive media experience (IMEx). arXiv preprint arXiv:2007.07032

  57. Perret J, Vander Poorten E (2018) Touching virtual reality: a review of haptic gloves. In: ACTUATOR 2018; 16th international conference on new actuators, VDE, pp 1–5

  58. Pezent E, Israr A, Samad M, et al (2019) Tasbi: Multisensory squeeze and vibrotactile wrist haptics for augmented and virtual reality. In: 2019 IEEE world haptics conference (WHC), IEEE, pp 1–6

  59. Pratticò FG, Calandra D, Piviotti M, et al (2021) Assessing the user experience of consumer haptic devices for simulation-based virtual reality. In: 2021 IEEE 11th international conference on consumer electronics (ICCE-Berlin), IEEE, pp 1–6

  60. Raheel A, Majid M, Alnowami M et al (2020) Physiological sensors based emotion recognition while experiencing tactile enhanced multimedia. Sensors 20(14):4037

    Article  Google Scholar 

  61. Ranasinghe N, Jain P, Thi Ngoc Tram N, et al (2018) Season traveller: multisensory narration for enhancing the virtual reality experience. In: Proceedings of the 2018 CHI conference on human factors in computing systems, pp 1–13

  62. Regal G, Voigt-Antons JN, Schmidt S et al (2019) Questionnaires embedded in virtual environments: reliability and positioning of rating scales in virtual environments. Qual User Exp 4(1):1–13

    Article  Google Scholar 

  63. Slater M, Sanchez-Vives MV (2016) Enhancing our lives with immersive virtual reality. Front Robot AI 3:74

    Article  Google Scholar 

  64. Slater M, Wilbur S (1997) A framework for immersive virtual environments (five): Speculations on the role of presence in virtual environments. Presence Teleoperators Virtual Environ 6(6):603–616

    Article  Google Scholar 

  65. Souchet AD, Philippe S, Lourdeaux D et al (2021) Measuring visual fatigue and cognitive load via eye tracking while learning with virtual reality head-mounted displays: a review. Int J Human Comput Interact 38(9):1–24

    Google Scholar 

  66. Suhaimi NS, Yuan CTB, Teo J, et al (2018) Modeling the affective space of 360 virtual reality videos based on arousal and valence for wearable eeg-based vr emotion classification. In: 2018 IEEE 14th international colloquium on signal processing & its applications (CSPA), IEEE, pp 167–172

  67. Toet A, van Erp JB (2019) The emojigrid as a tool to assess experienced and perceived emotions. Psych 1(1):469–481

    Article  Google Scholar 

  68. Toet A, Heijn F, Brouwer AM, et al (2019) The emojigrid as an immersive self-report tool for the affective assessment of 360 vr videos. In: International conference on virtual reality and augmented reality, Springer, pp 330–335

  69. Union I (2008) ITU-T recommendation p. 910: Subjective video quality assessment methods for multimedia applications

  70. Van Gent P, Farah H, Van Nes N et al (2019) Heartpy: a novel heart rate algorithm for the analysis of noisy signals. Transportation research part F: traffic psychology and behaviour 66:368–378

    Article  Google Scholar 

  71. Velasco C, Obrist M (2020) Multisensory experiences: where the senses meet technology. Oxford University Press, Oxford

    Book  Google Scholar 

  72. Velasco C, Obrist M, Petit O et al (2018) Multisensory technology for flavor augmentation: a mini review. Front Psychol 9:26

    Article  Google Scholar 

  73. Wang Q, Kang B, Kristensson PO (2022) Supporting playful rehabilitation in the home using virtual reality headsets and force feedback gloves. In: 2022 IEEE conference on virtual reality and 3D user interfaces (VR), IEEE, pp 504–513

  74. Weech S, Kenny S, Barnett-Cowan M (2019) Presence and cybersickness in virtual reality are negatively related: a review. Front Psychol 10:158

    Article  Google Scholar 

  75. Welch P (1967) The use of fast fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Transact Audio Electroacoust 15(2):70–73

    Article  Google Scholar 

  76. Witmer B, Singer M (1998) Measuring presence in virtual environments: a presence questionnaire. Presence 7(3):225–240

    Article  Google Scholar 

  77. van Zyl H, Meiselman HL (2015) The roles of culture and language in designing emotion lists: comparing the same language in different english and spanish speaking countries. Food Qual Prefer 41:201–213

    Article  Google Scholar 

Download references

Acknowledgements

The wish would like to acknowledge funding from the Natural Sciences and Engineering Council (NSERC) of Canada (RGPIN-2021-03246).

Author information

Authors and Affiliations

  1. INRS-EMT, University of Québec, Montréal, Québec, Canada

    Reza Amini Gougeh & Tiago H. Falk

Authors
  1. Reza Amini Gougeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tiago H. Falk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Reza Amini Gougeh.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Amini Gougeh, R., Falk, T.H. Towards instrumental quality assessment of multisensory immersive experiences using a biosensor-equipped head-mounted display. Qual User Exp 8, 9 (2023). https://doi.org/10.1007/s41233-023-00062-7

Download citation

  • Received:

  • Published:

  • DOI: https://doi.org/10.1007/s41233-023-00062-7

Keywords

Search

Navigation