Keywords

1 Introduction

In our previous study, we aimed to evaluate virtual reality (VR) performance from an ergonomics viewpoint and instigated the task for both the real and virtual environments to consider the requirements for developments in the virtual environment [1]. The difference between the performance of the real and virtual environments were considered from the perspective of the sense of agency (SoA). In particular, the VR performance was evaluated based on the two-step account of the agency model proposed by Synofzik et al. [2].

Meanwhile, the SoA and related concepts such as sense of embodiment (SoE), sense of body ownership (SoBO), and sense of self-location (SoSL) have been discussed in recent research in the field of VR. With respect to these concepts, Kilteni et al. considered SoE for understanding the artificial body and pointed out, based on review results, that the three concepts of SoBO, SoA, and SoSL were associated with the concept of embodiment [3].

Recently, VR systems, mainly applied to computer games, have become easily available and widely used by young people. Meanwhile, multipurpose haptic devices for the virtual environment, such as the SPIDAR system, have been developed for everyday training tasks [4]. Therefore, the wide range of use contexts and user requirements should be considered to effectively design and evaluate the systems. However, the measurement of objective quality of VR systems from the user’s experience has not been widely considered, although some questionnaires for evaluating the user’s experience based on the SoE have been proposed [5]. This study aims to investigate the measures relating to the user performance for tasks such as erroneous behavior; in this regard, the relationships between the erroneous behavior and user experience are considered from the perspective of the SoE.

2 Method

To investigate the effect of VR systems on the user performance, we took the task that we designed in our previous study, the rod tracking task (RTT), and carried out experiments with participants undertaking the task both in reality and the virtual environment. Then we investigated and compared the participants’ performances and experiences in the RTT. The experiment was between subject design and the procedures in the both conditions were almost the same.

2.1 Rod Tracking Task for Different Environments

To observe the user’s SoE, we used the rod tracking task because the characteristics of the RTT, which was created in our previous study, are well understood [1].

When executing the RTT, the user is required to grasp a rod with their right hand and attempt to move the rod between the ends of a curved slit in a panel, without making contact with the sides of the slit. The panel is installed in front of the user and is rotated anticlockwise by 45° with respect to the user, as shown in Fig. 1.

Fig. 1.
figure 1

Experimental scene of a participant executing the RTT in reality. Two electrodes for the EMG are attached to the participants’ lower arm and near the base of their fifth finger.

Full size image

In the virtual environment, the user can be affected by various factors arising from the design of the VR, such as a sudden vibration of the rod on contact with the sides of the slit. Therefore, the performance in a normal situation refers to the performance in reality, and must be compared to the performance in the virtual environment. We set up a VR test for the RTT, as well as the apparatus for the same task to be executed in reality, as shown in Fig. 2. In addition, the experimental task should immerse the user, because the user’s performance depends on their attention to their own behavior. Therefore, successful execution of the RTT requires a certain level of skill and the task must interest participants to some extent.

Fig. 2.
figure 2

The real slit panel and the rod for the RTT in reality (a) and its equivalent virtual scene in VR (b) [1].

Full size image

The slit width, the size of the panel, and the rod’s diameter of 10 mm were decided based on the required difficulty of the task (see Fig. 3). We concretely determined that the RTT needed to be difficult enough that an inexperienced participant was required to repeatedly practice around ten times before successfully executing the task in reality. The sine-curved slit and the direction of installation of the panel were also chosen from several patterns based on the required difficulty of the RTT. Furthermore, our previous study found that moving the rod in section B of the slit was the most difficult region for the participants and produced the greatest number of contacts, because it was difficult to see the gap between the rod and the slit as shown in Fig. 2. Therefore, in this study we have assumed that the user’s SoE is lost or reduced in section B.

Fig. 3.
figure 3

Specification of the slit panel. The width of the slit panel is 20 mm and the track is divided into four sections (A–D) for convenience [1].

Full size image

2.2 Experimental Equipment for Reality

To observe the participant’s performance, we used the experimental equipment for the RTT composed of several devices, as shown in Fig. 4. To record the movement of the right hand, a motion sensor (Leap Motion) connected to a personal computer (DELL XPS 8700) was used. The movement was sampled and recorded as three-dimensional coordinates at 10 Hz. Meanwhile, to observe the participant’s intentional and unintentional muscle tension while manipulating the rod, surface electromyograms (EMGs) on the muscle abductor digiti minimi (on the fifth finger) and muscle flexor carpi ulnaris (on the lower arm) were sampled and recorded at 100 Hz using a multi-telemeter (Nihon-Koden WEB-9500). Additionally, the signal recording contact between the rod and the slit panel was sent to the PC via a USB I/O terminal (Contec AIO-160802AY-USB) and to a multi-telemeter, as well as illuminating a red LED indicator for the benefit of the participant.

Fig. 4.
figure 4

Connection diagram of the devices recording the participant’s performance in reality [1].

Full size image

2.3 Experimental Equipment for the Virtual Environment

The new VR environment developed for this study for investigating human performance consisted of two user interfaces including a haptic interface (SPIDAR-HS2) and a head-mounted display (HTC VIVE PRO). SPIDAR-HS2, the human-sized haptic user interface for VR, controlled the rod’s two endpoints. It consisted of the controllers and eight motor modules including a motor, a threaded pulley, and an encoder for reading the yarn winding of each pulley. The motor modules provided a sense of force, as well as detecting the positions and angle of the end effectors; in other words, they detected the rod movement. The other experimental conditions and the specifications of SPIDAR-HS2 are shown in Table 1, and further details of the specifications of SPIDAR can be found in reference [4].

Table 1. Specifications of the SPIDER-HS2 used for the RTT.
Full size table

In the case of the experiment in the virtual environment, the movement of the rod was recorded using SPIDAR-HS system and the surface EMGs on the same two body parts described above were also recorded using the multi-telemeter (Nihon Koden WEB-9500).

The physics of the rod and slit panel are similar to those of the real objects, especially visually, but are somewhat different in terms of the haptic sense, because of being in a virtual environment. For instance, the rod as the end effector of SPIDAR-HS2 did not fall owing to gravity during the task. Furthermore, SPIDAR-HS2 presented occasional vibrations to the user’s kinesthetic sense when the rod came in contact with the sides of the slit, because the haptic interface could recreate a sensation closer to a real sense than the real kinesthetic and haptic sense. In this regard, we consider it a kind of erroneous motion, which can occur in any haptic device. Instead, we tried to observe the user’s performance when the unnatural feedbacks were perceived in the virtual environment controlled by the many devices.

2.4 Questionnaire for Investigating the SoE in RTT

From the perspective of SoE, it is assumed that SoA, SoSL, and SoBO, as the elements of SoE, can affect the participant’s performance. To consider the participants’ performances in the virtual environment, therefore, we obtained the subjective data using the SoE questionnaire, which we designed based on previous studies. Many kinds of questionnaires for investigating SoE have been proposed in recent years. For instance, Mar and Tabitha proposed an embodiment questionnaire for investigating avatar embodiment in an immersive virtual environment [5]. The foci of the embodiment questionnaire were covered by six characteristics such as body ownership, agency and motor control, tactile sensations, location of the avatar’s body, the avatar’s external appearance, and response to external stimuli. For the RTT trials in a virtual environment, however, the participants looked at the movement of the rod which they controlled using SPIDAR-HS2. In other words, the participants could not look at their hand both in reality and on their avatar in the virtual environment while executing the RTT; therefore, we selected nine statements from the embodiment questionnaire for the RTT as executed in both reality and the virtual environment. In addition, to construct the questionnaire for assessing the SoE for the RTT, we arranged the relevant statements based on the characteristics of the RTT. The nine questionnaire statements were composed of three categories, SoBO, SoA, and SoSL; these are shown in Table 2.

Table 2. The ten questionnaire items for investigating the participant’s subjective experience in RTT.
Full size table

The questionnaire was given to the participants at the end of the experiment. In this regard, Mar and Tabitha suggested that it should be clear that the questions are related to the participants’ experience during the experiment; therefore, we instructed the participants to answer the questionnaire while recalling the situations described in each item using a seven-point Likert-scale. The Likert-scale ranged from strongly disagree (−3) to disagree (−2), somewhat disagree (−1), neither agree nor disagree (0), somewhat agree (+1), agree (+2), and strongly agree (+3).

In this study, the SoE score for the RTT was composed of the SoBO, SoA, and SoSL scores, which were estimated as follows:

  • SoBO score = \( ( - {\text{Q1}} - {\text{Q2}} - {\text{Q3}})/ 3 \)

  • SoA score = (Q4 + Q5 − Q6 – Q7)/4

  • SoSL score = (Q8 – Q9)/2

  • SoE score = (−Q1 − Q2 − Q3)/3 + (Q4 + Q5 − Q6 – Q7)/4 + (Q8 – Q9)/2.

In other words, we assume the SoBO, SoA, and SoSL have equal impact on the SoE, ranging from −3 to +3, and the SoE score reflects the participants’ experience.

2.5 Experimental Procedure in Reality

The 17 participants were male students ranging from 21 to 23 years of age and gave their informed consent for participation in advance. All participants were right handed and had no prior experience of the RTT.

A trial of the RTT in this experiment was to pull the rod from the far end of the slit to the end closest to the participant along the track. Subsequently, the participant had to push the rod from the close end to the far end and had to avoid contact of the rod with the sides of the slit except at both ends. The speed at which the participant moved the rod was chosen by the participant; as long as the participant held his/her head as steady as possible during the trial, as shown in Fig. 1.

Before the experiment, the participants were measured by the EMGs on the muscle abductor digiti minimi (on the fifth finger) and muscle flexor carpi ulnaris (on the lower arm) under a resting situation for ten seconds. After that, they repeated the trial for three days until they could avoid contact of the rod with the slit; however, the participants could take a rest and relax for a while if necessary and the trial was limited to fifteen times a day.

Using the experimental equipment, we recorded the movement of the participant’s hand and the participant’s surface EMGs. The contact signal, which was generated when the rod and slit panel came into contact, was recorded by both the PC and the multi-telemeter. The participants’ opinions about most difficult section in the slit panel and the reason why the rod came into contact with the slit edges were recorded in an interview after every trial, if the participants could recall. At the end of the experiment, the participants reported their experiences and answered the aforementioned questionnaire.

2.6 Experimental Procedure in the Virtual Environment

We selected twelve male students ranging from 22 to 25 years of age as participants. The participants were different from those participating in the real-world experiment. The participants were right handed and had no prior experience doing the RTT in a real-world environment, but had executed other tasks in a virtual environment using the other SPIDAR system and a head-mounted display. In this regard, the participants were used to performing the RTT in virtual environments and we assumed they had accrued sufficient skill to perceive the virtual environment to some extent.

A trial of the RTT in this experiment was the same as in the real-world experiment; that is, they pulled the end effector like the rod from the far end of the slit to the near end along the track and then pushed the rod from the near end to the far end. During both parts of the task, they had to avoid contact of the rod with the slit edges except at both ends. The speed at which the participant moved the rod was decided by the participant; however, the participant was required to hold their head as steady as possible during the trial in the virtual environment.

Before the experiment, the respective participants were measured by the EMGs on the muscle abductor digiti minimi (on the fifth finger) and muscle flexor carpi ulnaris (on the lower arm) under a resting situation for ten seconds. After measuring the EMGs, the participants repeated the trial until their performance was improved up to a maximum of ten trials, and the participants could take a rest and relax as required.

Using the experimental equipment, we recorded the changing position and angle of the end effector as well as the participant’s surface EMGs. The contact signal, which was generated by the PC when the rod and slit panel made contact, was also recorded by the multi-telemeter via USB I/O terminal. In addition, the participants’ opinions, especially about the experience executing the RTT, were recorded in interviews and the participants answered the questionnaire at the end of experiment.

3 Results

3.1 Improvement of the Performance of the RTT in Reality

Our previous study showed that it was most difficult for the participants to move the rod without contact during section B of the slit because of the reason described before; however, it was not clear that the repetition of the RTT in real-world conditions resulted in the improvement of performance in section B. Thus, at first, we investigate the improvement of the performance within 3 days in the real-world for the RTT. Figure 5 shows that the number of instances of contact between the rod and the slit edges decreases day by day. The 95% confidence interval of the number of contact events in section B on the third day was 3.9 ± 2.0. However, the number of contact events in section B decreases insufficiently when compared to the other sections.

Fig. 5.
figure 5

Average number of contact events at the last trial for the RTT of each day in the real-world experiments (n = 17).

Full size image

In the real-world experiment, only one of the seventeen participants could complete the RTT without contact in the first day; however, almost of the participants made contact between the rod and the slit edges. However, nine of the seventeen participants performed successfully in section B on the third day. That is, the nine participants made contact between the rod and the slit less than once a trial. In other words, in the real-world case, almost all of the participants were able to make progress controlling the rod in the RTT.

3.2 Number of Contacts in the Virtual Environment

The RTT in the virtual environment was more difficult for the other twelve participants and a greater number of contact events were observed in their last trial, excepting one participant. The number of contact events, therefore, can serve as a measure for evaluating the performance at each section of the slit panel for both environments of the RTT. This is shown in Fig. 6.

Fig. 6.
figure 6

Number of contact events during the last trial of the first day in each section of the slit panel, during trials in reality (n = 17) and in the virtual environment (n = 12).

Full size image

Figure 6 shows that it is significantly more difficult for participants to avoid contact in section B than any other section, regardless of the RTT’s environment (p < .05). Furthermore, it can also be observed that the performance in the RTT in the virtual environment was unimproved by the last trial. Therefore, we infer that the factors existing in the virtual environment can affect performance and make the RTT more difficult in section B than it is in the real world.

3.3 Experiences of the RTT in Real and in the Virtual Environment

The participants’ experiences, estimated using the questionnaire, were compared between the real and virtual environments. The 95% confidence interval of the SoE score in the virtual environment was 5.0 ± 0.9, whereas the SoE score in the real environment was 8.4 ± 0.4. The scores of each of the three components of SoE for the RTT are shown in the Fig. 7. Figure 7 also shows that the SoA score has a strong effect on the SoE score.

Fig. 7.
figure 7

Comparison of the average scores of the three components of SoE for the RTT trials in the two different environments.

Full size image

3.4 Performance After Contact with Slit Edges

In our previous study, we measured the amplitudes of the EMGs at the surface of two body parts to evaluate the participants’ skill in the RTT in both environments. The EMGs were calculated using the root-mean-square (RMS). The standard deviations of RMS EMGs at the surface of the two body parts were also used to understand the SoA. We tried to determine the effect of the SoA on the performance in detail. The standardized RMS EMG waveforms of the participants’ two body parts were obtained by dividing the RMS EMG signals during the RTT by the participant’s averaged resting EMS EMG signals. The general trends of multiple participants’ RMS EMG waveforms were obtained by averaging the standardized RMS EMG waveforms. Figure 8 shows two averaged standardized RMS EMG waveforms for two seconds after contact in section B of the slit; these wave forms are smoothed by the moving average method. The average standardized RMS EMG waveform in the real environment (a) was obtained by averaging forty samples from the first trials involving the seventeen real-world participants, and the average waveform in the virtual environment (b) was obtained by averaging eight samples from the first trials involving the twelve virtual-environment participants.

Fig. 8.
figure 8

Comparison of two trends of the RMS EMG waveforms over a period of two seconds after participants had made contact between the rod and the slit edges: in reality (a) and in the virtual environment (b).

Full size image

Figure 8 indicates that the amplitude of the EMG decreases after contact is made, and the duration at local minimum differs between the real and virtual environments. Therefore, the times between contact and the amplitude of the RMS EMG dropping to a local minimum were measured, and the averages of the forty real-world samples and eight virtual-environment samples were compared. It was found that the average duration in the virtual environment case is statistically longer than in the real case. Figure 9 shows the results from the RMS EMG waveforms on the fifth finger, which represents the muscle tension for dexterous manipulation of the rod.

Fig. 9.
figure 9

Duration between contact and the time of local minimum of the RMS EMG amplitude.

Full size image

To explain the difference in these durations, we propose a kind of processing time for recognizing the situation, from when the contact was perceived to when the participant moves to the next action. From the viewpoint of motor readiness, the average of the duration in the real world could be valid, and could suggest that cognitive processes after contact progress smoothly in a short time. In other words, it is possible to say that the participants required more time for cognition in the virtual environment than in a similar real-world situation.

4 Discussion

We propose that the following two major factors affected the participants’ SoE in the RTT and the task environment: one was the difficulty of understanding the gap be-tween the rod and the slit because of the physical characteristics of the RTT, and the other was the unnatural feedback, mainly from the haptic device, in the virtual environment. We could observe the influence of the factors using the questionnaire we designed for the RTT. That is, the SoE score was lower in the virtual environment, especially the SoA score (see Fig. 7). Additionally, factors such as the characteristics of the RTT had an effect on the participants’ performance: controlling and moving the rod during section B produced more contact events than in the other sections, regardless of the task environment (see Fig. 6). The unnatural feedbacks in the virtual environment also caused more contact events; this is clear because the number of contact events in the virtual environment was significantly more than in the real world (see Fig. 6). With respect to this point, almost of the participants pointed out the unnatural feelings or feedbacks produced by the haptic devices; however, they acquired the skill of controlling the rod in the virtual environment, despite the difficulty of moving the rod through section B. In other words, they experienced the unnatural sensations such as the vibration of the rod by the haptic devices, but they learned to understand these sensations, and they improved at executing the RTT. In this sense, based on the two-step account of the agency model for explaining the mechanism of SoA [2], it is concluded that the participant’s SoA could not make a successful perceptual representation in a feeling of agency, and the judgement of agency needed more time to achieve the propositional representation to understand the situation in the virtual environment. Therefore, in the virtual environment, the duration between contact and the RMS EMG amplitude dropping to a local minimum was longer in the virtual environment, as shown in Fig. 9. Moreover, the duration could be an objective measure for evaluating the RTT from the viewpoint of SoA.

Based on the above, we can say that in the virtual environment, especially the VR system we experimentally developed, unnatural or erroneous feedback from the VR system led the user’s unsuccessful cognitive processes to decide what action is appropriate for the task in the situation.

5 Conclusion

In this study, we investigated the effect of the factors relating to the rod tracking task using a VR system on the user’s performance from the perspective of SoE, especially SoA. We have successfully explained the waveforms of RMS EMGs in the period after the rod made contact with the slit edges, based on the two-step account of the agency model for SoA. Concretely, unnatural or erroneous feedback from the VR system decreased the user’s situational awareness and required more time for decision-making. Furthermore, those undesirable feedbacks could make the user’s subjective experience worse, such as decreasing the level of SoA then SoE in VR. Consequently, the effect of the artificial haptic characteristics on virtual reality performance could be revealed.

Based on the above, we propose that the waveforms of RMS EMGs could be useful as a measure of the duration required for awareness of situation and decision-making through this study; however, further research is required to confirm the validity of the measure.