Multisensory Immersive Experiences: From Monitoring of Human Influential Factors to New Applications in Healthcare

While virtual and extended reality applications are on the rise, existing experiences are not fully immersive, as only two senses (audio-visual) are typically stimulated. In this keynote talk, I will describe our ongoing work on developing multisensory immersive experiences, which combine auditory, visual, olfactory, and haptic/somatosensory stimuli. I will show the impact that stimulating more senses can have on user quality of experience, sense of presence and immersion, and engagement levels.

Moreover, with multisensory experiences, monitoring human influential factors is crucial, as the perception of sensory stimuli can be very subjective (e.g., while a smell can be pleasant for some, it can be unpleasant for others). To this end, I will also describe our work on instrumenting virtual reality headsets with biosensors to allow not only for automated (remote) monitoring of human behaviour and tracking of human influential factors, but to also develop new markers of user experience, such as a multimodal time perception metric or a cybersickness metric.

Lastly, I will describe some new applications of multisensory experiences that we are developing for healthcare and well-being. I will start with the use of immersive multisensory nature walks for mental health and describe two ongoing projects, one with patients with post-traumatic stress disorder and another with nurses suffering from burnout. I will conclude with a description of the use of multisensory priming for motor-imagery based neurorehabilitation for stroke survivors.