Lost in space: multisensory conflict yields adaptation in spatial representations across frames of reference

Abstract

According to embodied cognition, bodily interactions with our environment shape the perception and representation of our body and the surrounding space, that is, peripersonal space. To investigate the adaptive nature of these spatial representations, we introduced a multisensory conflict between vision and proprioception in an immersive virtual reality. During individual bimanual interaction trials, we gradually shifted the visual hand representation. As a result, participants unknowingly shifted their actual hands to compensate for the visual shift. We then measured the adaptation to the invoked multisensory conflict by means of a self-localization and an external localization task. While effects of the conflict were observed in both tasks, the effects systematically interacted with the type of localization task and the available visual information while performing the localization task (i.e., the visibility of the virtual hands). The results imply that the localization of one’s own hands is based on a multisensory integration process, which is modulated by the saliency of the currently most relevant sensory modality and the involved frame of reference. Moreover, the results suggest that our brain strives for consistency between its body and spatial estimates, thereby adapting multiple, related frames of reference, and the spatial estimates within, due to a sensory conflict in one of them.

Appendices

Appendix 1: IPQ evaluation

The IPQ assesses presence in virtual realities on three different scales and allows to quantify the degree of immersion experienced by the participants within the VR. The igroup consortium provides reference data from different VR setups. We compared our data to setups that also used a head-mounted display. The reference data set comprised 24 mean values for the three scales.

Due to a software issue, only 21 of the 33 participants completed the IPQ questionnaire in our study. We checked whether the results exceeded those of the reference data. The results of the respective t-tests are shown in Table 6; the data are shown in Fig. 10.

Table 6 T-Test results for the different IPQ scales

Fig. 10

Scores for the different IPQ scales in the observed (light gray) and the reference (dark gray) data. Significant differences (indicated with an asterisk) were found for the spatial presence scale. The scores show that the experimental setup yielded comparable immersion and realism judgments as reference setups, while spatial presence was improved

With respect to involvement and realism, the data are comparable to the reference data. In case of spatial presence, the results are significantly improved compared to the reference data [t(33.4) = 1.76, p < 0.05]. Together, the results show a sufficient degree of immersion. Improvements with respect to spatial presence dovetail with other results that showed enhanced spatial perception in VR when participants were equipped with a body model (Mohler et al. 2008), or when they could interact with the VR via bodily motion (Schroeder et al. 2016).

Appendix 2: Collected petals

To check whether the participants complied with the manual task, the amount of petals picked was subjected to a separate analysis. In general, participants complied with the task, collecting 4.5 petals on average per trial. However, there were considerable individual differences, leading to a rather high standard deviation of 1.4 petals. To further check for learning effects and effects due to the induced drift, the number of collected petals was analyzed with a 2 × 2 repeated measure ANOVA using R (R Core Team 2016) and the ez package (Lawrence 2015). We considered the factors block and offset condition. The experiment was divided into two blocks; the according factor had two levels. To allow a straightforward analysis of the different offset conditions, we aggregated over the variations in the depth axis (see Fig. 5b), such that the resulting factor had only two levels (visual offsets to the left or to the right).

The results of the analysis are shown in Table 7. There was a considerable learning effect. In the second block, participants collected significantly more petals than in the first block (M = 3.9 vs. M = 5.1). Furthermore, participants collected more petals when the hands were visually shifted to the right (M = 4.2), than when they were shifted to the left (M = 4.8). This unwanted effect is most likely due to the asymmetric layout of the scene (see Fig. 3). Visual offsets to the right were compensated by moving the hands to the left. This allows a more convenient trajectory through the task space, because the hands operate in the center of the tracking range with the flower physically slightly to the left and the basket to the right of the center of the tracking range. For visual offsets to the left, the trajectory through the task space is less convenient. In this case, the shifts are compensated by placing the hands to the right of the center of the tracking range, such that the hands had to be moved even further to the right in order to reach the basket.

Table 7 ANOVA table for the number of collected petals