Daniel Sjölie
ABSTRACT
The combination of virtual reality (VR) and brain measurements is a promising development of HCI, but the maturation of this paradigm requires more knowledge about how brain activity is influenced by parameters of VR applications. To this end we investigate the influence of two prominent VR parameters, 3d-motion and interactivity, while brain activity is measured for a mental rotation task, using functional MRI (fMRI). A mental rotation network of brain areas is identified, matching previous results. The addition of interactivity increases the activation in core areas of this network, with more profound effects in frontal and preparatory motor areas. The increases from 3d-motion are restricted to primarily visual areas. We relate these effects to emerging theories of cognition and potential applications for brain-computer interfaces (BCIs). Our results demonstrate one way to provoke increased activity in task-relevant areas, making it easier to detect and use for adaptation and development of HCI.
- Aguirre, G.K., Detre, J.A., Alsop, D.C., and D'Esposito, M. The parahippocampus subserves topographical learning in man. Cereb. Cortex 6, 6 (1996), 823--829.Google Scholar
Cross Ref
- Backman, A. Colosseum3D-Authoring Framework for Virtual Environments. In Proc. EUROGRAPHICS Workshop IPT & EGVE Workshop, (2005), 225--226. Google ScholarDigital Library
- Barsalou, L.W. Grounded Cognition. Annual Review of Psychology 59, 1 (2008), 617--645.Google ScholarCross Ref
- Beck, L., Wolter, M., Mungard, N., Kuhlen, T., and Sturm, W. Combining Virtual Reality and Functional Magnetic Resonance Imaging (fMRI): Problems and Solutions. In HCI and Usability for Medicine and Health Care. 2007, 335--348. Google ScholarDigital Library
- Blanke, O. and Arzy, S. The Out-of-Body Experience: Disturbed Self-Processing at the Temporo-Parietal Junction. Neuroscientist 11, 1 (2005), 16--24.Google ScholarCross Ref
- Born, R.T. and Bradley, D.C. Structure and Function of Visual Area MT. Annual Review of Neuroscience 28, 1 (2005), 157--189.Google ScholarCross Ref
- Christou, G., Law, E.L., Green, W., and Hornbaek, K. Challenges in evaluating usability and user experience of reality-based interaction. In Proc. CHI 2009 Extended Abstracts, ACM (2009), 4811--4814. Google ScholarDigital Library
- Cohen, M.S., Kosslyn, S.M., Breiter, H.C., et al. Changes in cortical activity during mental rotation A mapping study using functional MRI. Brain 119, 1 (1996), 89--100.Google ScholarCross Ref
- Constantinidis, C. and Wang, X. A neural circuit basis for spatial working memory. The Neuroscientist: A Review Journal Bringing Neurobiology, Neurology and Psychiatry 10, 6 (2004), 553--65.Google Scholar
- Curtis, C.E. and D'Esposito, M. Persistent activity in the prefrontal cortex during working memory. Trends in Cognitive Sciences 7, 9 (2003), 415--423.Google ScholarCross Ref
- Cutrell, E. and Tan, D.S. BCI for passive input in HCI. In Proc. CHI 2008 Workshop on Brain-Computer Interfaces for HCI and Games, (2008). Google ScholarDigital Library
- Decety, J. and Jeannerod, M. Mentally simulated movements in virtual reality: does Fitt's law hold in motor imagery? Behavioural Brain Research 72, 1-2 (1995), 127--134.Google ScholarCross Ref
- Dolan, R. Neuroimaging of Cognition: Past, Present, and Future. Neuron 60, 3 (2008), 496--502.Google ScholarCross Ref
- Ericsson, K.A., Prietula, M.J., and Cokely, E.T. The making of an expert. Harvard business review 85, 7/8 (2007), 114.Google Scholar
- Eriksson, J., Larsson, A., and Nyberg, L. Item-specific Training Reduces Prefrontal Cortical Involvement in Perceptual Awareness. Journal of Cognitive Neuroscience 20, 10 (2008), 1777--1787. Google ScholarDigital Library
- Farrer, C. and Frith, C.D. Experiencing Oneself vs Another Person as Being the Cause of an Action: The Neural Correlates of the Experience of Agency. NeuroImage 15, 3 (2002), 596--603.Google ScholarCross Ref
- Fitts, P.M. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology 47, 6 (1954), 381--391.Google ScholarCross Ref
- Friston, K. and Stephan, K. Free-energy and the brain. Synthese 159, 3 (2007), 417--458.Google ScholarCross Ref
- Girouard, A., Hirshfield, L.M., Solovey, E., and Jacob, R.J.K. Using functional Near-Infrared Spectroscopy in HCI: Toward evaluation methods and adaptive interfaces. In Proc. CHI 2008 Workshop on Brain-Computer Interfaces for HCI and Games, (2008).Google Scholar
- Girouard, A. Adaptive brain--computer interface. In Proc. CHI 2009 Extended Abstracts, ACM Press (2009), 3097--3100. Google ScholarDigital Library
- Hawkins, J. On Intelligence. Owl Books, 2005. Google ScholarDigital Library
- Hawkins, J. and George, D. Hierarchical Temporal Memory. Numenta Inc., 2006.Google Scholar
- Hirshfield, L.M., Solovey, E.T., Girouard, A., et al. Using Brain Measurement to Evaluate Reality Based Interactions. In Proc. CHI 2009 Workshop on Challenges in the Evaluation of Usability and User Experience in Reality Based Interaction, (2009), 19.Google Scholar
- Hirshfield, L.M., Solovey, E.T., Girouard, A., et al. Brain measurement for usability testing and adaptive interfaces: an example of uncovering syntactic workload with functional near infrared spectroscopy. In Proc. CHI 2009, ACM Press (2009), 2185--2194. Google ScholarDigital Library
- Jacob, R.J.K., Girouard, A., Hirshfield, L.M., et al. Reality-based interaction: a framework for post-WIMP interfaces. In Proc. CHI 2008, ACM Press (2008), 201--210. Google ScholarDigital Library
- Jäncke, L. Virtual reality and the role of the prefrontal cortex in adults and children. Frontiers in Neuroscience 3, 1 (2009).Google ScholarCross Ref
- Kilner, J.M., Friston, K.J., and Frith, C.D. Predictive coding: an account of the mirror neuron system. Cognitive Processing 8, 3 (2007), 159--66.Google ScholarCross Ref
- Kosslyn, S.M., Ganis, G., and Thompson, W.L. Neural foundations of imagery. Nat Rev Neurosci 2, 9 (2001), 635--642.Google ScholarCross Ref
- Lécuyer, A., Lotte, F., Reilly, R.B., Leeb, R., Hirose, M., and Slater, M. Brain--Computer Interfaces, Virtual Reality, and Videogames. Computer 41, 10 (2008), 66--72. Google ScholarDigital Library
- Minnery, B.S. and Fine, M.S. Neuroscience and the future of human-computer interaction. Interactions 16, 2 (2009), 70--75. Google ScholarDigital Library
- Mourao-Miranda, J., Ecker, C., Sato, J.R., and Brammer, M. Dynamic Changes in the Mental Rotation Network Revealed by Pattern Recognition Analysis of fMRI Data. Journal of Cognitive Neuroscience 21, 5 (2009), 890--904. Google ScholarDigital Library
- Parasuraman, R. and Wilson, G.F. Putting the brain to work: neuroergonomics past, present, and future. Human Factors 50, 3 (2008), 468--474.Google ScholarCross Ref
- Postma, A. and Barsalou, L.W. Spatial working memory and imagery: From eye movements to grounded cognition. Acta Psychologica 118, 2--3 (2009).Google Scholar
- Rizzo, A.A., Schultheis, M., Kerns, K.A., and Mateer, C. Analysis of assets for virtual reality applications in neuropsychology. Neuropsychological Rehabilitation 14, 1 (2004), 207--239.Google ScholarCross Ref
- Schacter, D.L., Addis, D.R., and Buckner, R.L. Remembering the past to imagine the future: the prospective brain. Nat Rev Neurosci 8, 9 (2007), 657--661.Google ScholarCross Ref
- Shepard, R.N. and Metzler, J. Mental Rotation of Three-Dimensional Objects. Science 171, 3972 (1971), 701--703.Google Scholar
- Sjölie, D., Bodin, K., Eriksson, J., and Janlert, L. Using brain imaging to assess interaction in immersive VR. In Proc. CHI 2009 Workshop on Challenges in the Evaluation of Usability and User Experience in Reality Based Interaction, (2009), 23.Google Scholar
- Sweetser, P. and Wyeth, P. GameFlow: a model for evaluating player enjoyment in games. Comput. Entertain. 3, 3 (2005), 3--3. Google ScholarDigital Library
- Tagaris, G.A., Kim, S., Strupp, J.P., Andersen, P., U?urbil, K., and Georgopoulos, A.P. Mental Rotation Studied by Functional Magnetic Resonance Imaging at High Field (4 Tesla): Performance and Cortical Activation. Journal of Cognitive Neuroscience 9, 4 (1997), 419--432. Google ScholarDigital Library
Index Terms
- Effects of interactivity and 3D-motion on mental rotation brain activity in an immersive virtual environment
Recommendations
Virtual reality with fMRI: a breakthrough cognitive treatment tool
The impact of virtual reality (VR) has been felt in a wide range of fields over the past 10 years. VR has been shown to be an effective treatment for anxiety, phobia, pain, posttraumatic stress disorder, stress inoculation training, and drug and alcohol ...
Effects of electrical brain stimulation on brain indices and presence experience in immersive, interactive virtual reality
AbstractThe subjective presence experience in virtual reality (VR) is associated with distinct brain activation patterns. Particularly, the dorsolateral prefrontal cortex (DLPFC) seems to play a central role. We investigated the effects of electric brain ...
Virtual reality with fMRI: a breakthrough cognitive treatment tool
The impact of virtual reality (VR) has been felt in a wide range of fields over the past 10 years. VR has been shown to be an effective treatment for anxiety, phobia, pain, posttraumatic stress disorder, stress inoculation training, and drug and alcohol ...
Reviews
Jeanine M. Meyer
Sj?lie et al. describe in this paper the examination of brain scans produced by functional magnetic resonance imaging (fMRI) on subjects solving puzzles. They discuss their work within the context of building virtual reality applications, prior studies using brain scanning, and theories of how the brain operates. While in the fMRI unit, each subject is fitted with glasses and shown a computer-generated three-dimensional (3D) scene. The task is to determine if two sets of connected blocks represent the same or different physical objects-that is, if one object can be rotated and translated to match the other. The puzzle is presented in three different ways: the still condition displays a static picture, the auto condition has the objects rotated by the application, and the interactive condition provides buttons for the subject to rotate the objects. For each of these three conditions, the puzzle is presented using standard and stereo projections. Puzzles are presented in random order and the time between tasks is used as the baseline. The scans reveal what part of the brain is active during each task. The pictures produced by the fMRI are encoded in such a way as to determine if and how scans made under different conditions and of different subjects are distinct. The analysis omits the cases when a subject made a mistake. Sj?lie et al. describe the encoding and the statistical analysis involved in making these determinations, but more explanation would have been helpful. The authors report that there were no significant effects for different conditions; this was confirmed by post-scan interviews-after all, mental rotation probably occurs for all three conditions. They do imply that the scans for all ten subjects were similar enough to make generalizations. They also note that their findings were consistent with prior work on 3D rotation tasks. The strongest effect appears to be the effect of interactivity. Anyone who builds virtual reality systems or is interested in concepts of cognitive theories such as presence, prediction, and mental workload will benefit from a careful study of this paper and by reading the many articles referenced. Online Computing Reviews Service
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments