Study of Virtual Reality Performance Based on Sense of Agency

Abstract

In recent years, virtual reality (VR) technology has been applied to various needs and problems. However, there are as yet few guidelines for providing VR from user’s characteristics. Therefore, we aimed to make design guideline for VR from an ergonomics viewpoint and by observing the characteristics of the user’s performance in a virtual environment during a task. Thus, the task was designed to be performed in both in reality and a virtual environment. Using the task, we observed undesirable performances by the user which could be affected by aspects of the virtual environment. First, 15 participants completed the task in reality and their performance was measured based on the movement of their hand and a surface electromyogram on their fifth finger and lower arm. We therefore obtained the characteristics of the task. Second, seven participants performed the task in a virtual environment and their performance was observed. When analyzing the results, we found undesirable performance by the participants. We therefore consider the unusual phenomena related to aspects of our VR to be based on the concept of a sense of agency (SoA). Consequently, we estimated that knowledge or predefined significance is limited in a virtual environment and it is not useful to perform tasks in VR and keep SoA to some extent. In this context, we confirmed that introducing the concept of SoA is useful when explaining performance in VR. However, our conceptual consideration should be confirmed in further research.

1 Introduction

Recently, virtual reality (VR) technology has been applied to an increasing variety of human–computer interactions, offering realistic an operating experience for training, amusement, and other uses. VR includes visual and audial user interfaces, as well as haptic devices such as data gloves stimulating our haptic or kinesthetic senses. These user interfaces offer a greater sense of realism and feeling of immersion. The VR technology produced by high-specification computers used to be appropriate for every task. However, designing cost-effective VR appropriate to an objective is now a necessary aspect of VR design. Furthermore, VR should be coordinated based on not only system performance but also from a user’s viewpoint. However, there are as yet few guidelines for providing VR from an ergonomic viewpoint. With this in mind, we set out to produce ergonomics-based VR design guidelines. However, the characteristics of a user’s cognitive and physical performance in recent virtual environments were not clear. Consequently, concepts to explain and describe a user’s performance had to be considered.

In VR, the avatar is the projection of the user and performs or manipulates objects in the virtual environment; however, the user experiences and learns skills through performing tasks in the virtual environment rather than the avatar. In this context, Forbes-Pitt [1] argued the context of playing a 3D game from the perspective of agency theory, and proposed that the ego agent becomes enhanced and is inseparable from the machine and that the ago agent feels whole in the world of the game. The ego agent, in every human’s social world, refers to the self-knowledge attendant in the awareness of one’s own mental abilities. As well as when playing 3D games, therefore, the ego agent of a user in a virtual environment could be inseparable from VR technology and may feel whole in the virtual environment. This viewpoint suggests that it is appropriate to regard users who move their bodies to manipulate objects in virtual environments rather than those who interact with VR.

In case of evaluating VR, including user interfaces stimulating our senses and computer systems creating and controlling virtual environments, the sense of immersion is an understandable concept. Adams (2014) classified the sense of immersion into three types [2]. However, immersion is a higher-level concept and quantitative measure or metrics for evaluation of the sense of immersion into virtual environment has not yet been confirmed.

The sense of Agency (SoA), a concept related to the sense of immersion, has been focused on by Frith [3] in the field of the neuropsychology of schizophrenia. Synofzik et al. [4] also discussed whether the empirical comparator model explaining the sensorimotor control mechanism and SoA and concluded that a framework of a “two-step account of agency” could allow investigation of the concept of SoA when understanding SoA and its disruptions in schizophrenia, as shown in Fig. 1.

Fig. 1.

Two-step account of agency Synofzik et al. [4].

Fig. 2.

Experimental scene of a participant executing the RTT in reality. Two electrodes for the EMG are attached to the participants’ lower arm and near the base of their fifth finger.

Fig. 3.

The real slit panel and the rod for the RTT in reality (left hand side) and its equivalent virtual scene in VR (right hand side).

Synofzik et al. [4] explain that the extent to which the feeling and judgement of agency contribute to the overall SoA depends on the context and task requirements, respectively. The feeling of agency is produced by a sub-personal weighting process of different action-related perceptual and motor cues, and forms the overall judgement of agency.

Action-related authorship indicators, such as feed-forward cues, proprioception, and sensory feedback produce which feeling of agency contributing to the overall SoA, depend on the context and the task requirements. In this sense, a SoA deficit could be the result of incongruence between action-related authorship indicators when the user acts, due to an imperfect virtual environment. However, SoA relates not only to the feeling of agency but also to the judgement of agency. A mismatch between different authorship indicators triggers (i) a primary basic feeling of not being the initiator of an event and (ii) a second interpretative mechanism which looks for the best explanation, resulting in a specific belief formation about the origins of the change in perception. In this sense, a user who is confused by an unexplainable situation in the virtual environment has a reduced SoA until the user finds the best explanation. However, we have little knowledge of the characteristics of the influence on performance of a reduced SoA in virtual environments. Thus, we tried to observe user’s performance as characterized by the SoA in the virtual environment.

Focusing on the SoA relating to human–computer interaction (HCI), Sato and Yasuda [5] defined a sense of self-agency as the sense that I am the one who is generating an action, and focused on the degree of discrepancy resulting from comparison between predicted and actual sensory feedback through experiment. Limerick et al. [6] propose that SoA refers to the experience of controlling both one’s body and the external environment and considered human–computer interaction (HCI) using the concept of SoA.

Limerick’s study [6] also reviews some previous empirical studies which examine HCI issues. In these studies, the conditions of audial, visual, or tactile stimuli were assigned as independent variables and the participant’s SoA-related performance was measured as the dependent variable under proper laboratory conditions. However, when considering the design of VR from a user’s viewpoint, we need to know the user’s psychological condition when their SoA becomes impaired while executing the task in the virtual environment, as this unusual condition commonly occurs due to either malfunction of VR or system design problems.

In this study, we designed an experimental task for users in order to observe the user’s conditions and tried to consider the user’s performances based on the concept of SoA.

2 Creating Rod Tracking Task for Different Environments

To observe user performance in a virtual environment, we first created a task named the rod tracking task (RTT) and set up the similar experimental environment in reality and a virtual environment to compare performance in these environments.

2.1 Design Concept for Making Task in VR

It was assumed that the user can look for the best explanation even if the user is inexperienced and cannot perform as intended in reality; however, it is hard to find an appropriate explanation when the user faces an unexperienced situation in the virtual environment. In this case, it could occur that the judgement of agency relating to SoA is difficult. To observe this phenomenon, we created the Rod Tracking Task (RTT) and observed the user’s performance. When executing the RTT, the user is required to grasp a rod by the right hand and effort to move the rod back and forth without contact from end to end in a slit panel. The slit panel is installed in front of the user and was rotated anticlockwise by 45 degrees with respect to the user, as shown in Fig. 4.

Fig. 4.

Specification of the slit panel. The width of the slit panel is 20 mm and the track are divided into four sections (A–D) for convenience.

The RTT was designed considering the following requirements: firstly, the users’ performance must be different between normal situations and under unusual conditions. In this regard, the user could be affected by various factors arising from the design of VR. Therefore, the performance under normal situation refers to the performance in reality and must be compared to the performance in the virtual environment. We then set up VR for the RTT, as well as apparatus for a similar task in reality, as shown in Fig. 3. Secondly, the experimental task should immerse oneself because the user’s performance depends on the attention to the user’s own behavior. Therefore, successful performance of the RTT requires a certain level of skill and the task must interest participants to some extent.

The slit width and the size of the slit panel were decided based on the difficulty of the task (see Fig. 4). Regarding the difficulty, we concretely determined that the RTT needs a level of difficulty such that an inexperienced participant was required to repeatedly practice around ten times before achieving the task in reality. The sine-curved slit and the direction of installation of the slit panel was also chosen from several patterns based on the difficulty of the RTT.

2.2 Experimental Equipment for Reality

To observe the participant’s performance, the experimental equipment for the RTT was composed of the several devices as shown in Fig. 5. To record the movement of the right hand, a motion sensor (Leap Motion) connected to a personal computer (DELL XPS 8700) was used. The movement was sampled and recorded as three-dimensional coordinates at 10 Hz. The participant’s surface electromyogram (EMG) on the muscle abductor digiti minimi (on the fifth finger) and muscle flexor carpi ulnaris (on the lower arm) were sampled and recorded at 100 Hz using a multi-telemeter (Nihon Koden WEB-9500). Furthermore, the signal of contact between the rod and the slit panel was sent to the PC via an USB I/O terminal (Contec AIO-160802AY-USB) and to a multi-telemeter, as well as illuminating a red LED indicator for the benefit of the participant.

Fig. 5.

Connection diagram of the devices recording the participant’s performance in reality.

2.3 Experimental Equipment for Our Virtual Environment

The VR environment we made for investigating human performance consisted of two user interfaces including a haptic interface (SPIDER-HS) and a head-mounted display (HTC VIVE). SPIDER-HS, the human-sized haptic user interface for VR, consisted of controllers and eight motor modules including a motor, a threaded pulley, and an encoder for reading the yarn winding of each. The motor modules also presented a force sense as well as detecting the positions and angle of end effectors i.e. the rod moved by the participant, as shown in Fig. 6. Other experimental conditions and the specifications of SPIDER-HS were shown in Table 1 and further details of the specifications of SPIDER system are introduced in reference [7].

Fig. 6.

Experimental scene shows the participant grasps a rod on eight strings to eight motor modules of SPIDER-HS. The head-mounted display presents the scene shown on the right-hand side of Fig. 3.

Table 1. Specifications of the SPIDER-HR used for the RTT.

In the case of the experiment in the virtual environment, the movement of the rod was recorded using SPIDER-HS system and the surface EMGs on the same two body parts described above were also recorded using the multi-telemeter (Nihon Koden WEB-9500).

Characteristics of Our Virtual Environment.

The physics of the rod and the slit panel was similar to that of the real object, especially visually, but somewhat different in terms of haptic sense because of being in a VR environment. For instance, the rod as the end effector of the SPIDER-HS did not fall due to gravity during the task.

3 Evaluation of the RTT in Reality

The objective of evaluating the RTT was to clarify the characteristics of the RTT in reality using the real experimental equipment described above and then to evaluate the performance for the RTT in the virtual environment was based on the knowledge of the characteristics of the RTT.

3.1 Method

Participants.

We selected 15 male students ranging from 21 to 23 years of age and participants gave their informed consent for participation in advance. All participants were right handed and had no experienced of the RTT.

Experimental Task and Procedure.

A trial of the RTT in this experiment was to pull the rod from the end of the slit to the end of the near side along the track; after that, the participant had to push the rod from the end of near side to the far side and had to avoid contact of the rod with the slit panel except at both ends of the slit. The speed at which the participant moved the rod was chosen the respective participants, as long as the participant held the head as steady as possible during the trial, as shown in Fig. 2. The respective participants repeated the trial ten times; however, the participants could take a rest and relax for a while if necessary.

Using the experimental equipment, we recorded the movement of the participant’s hand and the participant’s surface EMGs. The contact signal which was generated when the rod and slit panel were contacted was recorded by both the PC and the multi-telemeter. Furthermore, the participants’ opinions, especially about which section was most difficult, were recorded in an interview after all the trials were complete.

3.2 Result

Number of Contacts.

As the result of trials, the participants accomplished the RTT in around five attempts. However, the average number of contacts in each section (see Fig. 3) throughout all trials is shown in Fig. 7.

Fig. 7.

Number of contacts in each section during trials in reality (n = 15).

Figure 7 represents that it is significantly more difficult for participants to avoid contact in section-B. However, we also confirmed that the relation between the number of contacts and the direction of movement of the rod was insignificant (p =.45). We can therefore infer that the section of the slit panel was the main factor affecting the number of contacts.

Participants’ Opinions.

The results of interviewing the participants indicated that the section subjectively judged most difficult was section-B, as shown in Table 2. Almost of participants found that section B was difficult, and the section-C was easy, despite the length of the slit at both section B and C was the same. Participants said that the width of the slit at section-C was perceivable, but that it was hard to see the slit at section-B and it was therefore difficult to judge the appropriate course when moving the rod. (see the left hand side of Fig. 4)

Table 2. Number of subjects who found the section was difficult or easy (n = 15).

Relation Between EMG and Skill at the RTT

The envelope of the surface EMG signal at the fifth finger and lower arm were calculated by root-mean-square (RMS) and the envelopes of each were compared between the case of inexperienced trial and the trial during which participants accomplished the RTT. Figure 8 shows that the difference between the two situations for each surface EMG is insignificant. However, the results suggest that participants manipulated the rod with their fifth finger rather than by using their lower arm.

Fig. 8.

RMS EMGs at the different two body parts (n = 15).

Time for the RTT.

The speed at which the participant moved the rod was at the discretion of the respective participants. thus, we observed that they were tried to accomplish the RTT in any way they wished. The time taken to finish the trial did not shorten but lengthened significantly (p = .02).

3.3 Discussion

The above results show that characteristics of the RTT. In particular, it is difficult to avoid contact in section-B because it is hard to perceive the slit and consequently hard to discern the position of the rod. In other words, there were inexperienced participants who could not obtain feed-forward visual cues to make a perceptual representation. However, the main point to note is that all participants could achieve the RTT in reality. According to the framework of two-step account of agency [7], this result explains that the feeling of agency has been produced, judgement of agency has formed and therefore that a sense of agency was stable until the end of all trials through repeated practice.

4 Observing the Performance of the RTT in VR

The aim of this study was to observe several performances during unusual situations in the virtual environment and to consider the performance based on a SoA. We set up the virtual environment for the RTT and conducted an experiment using seven participants and the equipment for VR described above.

4.1 Method

Participants.

We selected seven male students ranging from 21 to 23 years of age as participants. All participants were right handed and had no experience in doing the RTT in reality but had experienced performing other tasks in a virtual environment using the SPIDER system and a head-mounted display. In this regard, the participants were used to performing experimental tasks in virtual environments and we assumed they had accrued sufficient skill to perceive the virtual environment to some extent.

Experimental Task and Procedure.

A trial of the RTT in this experiment was the same as in the prior experiment in reality. That is, they pulled the end effector like the rod from the end of the slit to the near side along the track and then pushed the rod from the near side to the far side. During both sections of the task they had to avoid contact of the rod with the slit panel except as both ends. The speed at which the participant moved the rod was decided by the respective participants. However, the participant was required to hold their head as steady as possible during the trial in the virtual environment. The respective participants repeated the trial until their performance was improved up to a maximum of ten trials, and the participants could take a rest and relax as required.

Using the experimental equipment, we recorded the changing position and angle of the end effector as well as the participant’s surface EMGs. The contact signal, which was generated by PC when the rod and slit panel made contact, were also recorded by the multi-telemeter. Furthermore, the participants’ opinions, especially about which section was most difficult, were recorded in an interview after all the trials were completed.

4.2 Results

All participants had no experience of the RTT in reality; therefore, the different conditions in reality and in the virtual environment were not particularly noticed by any of the participants. In this sense, they could concentrate on the RTT in the virtual environment.

Number of Contacts.

No participants accomplished the RTT within 10 trials. According to the average number of contacts in each section, the difference between the number of contacts in each section is insignificant; however, it is similar to the result from the experiment in reality that the number of contact in section-B was the highest of all sections (see Fig. 9). In other words, to move the rod through section-B was the most difficult for the participants in the virtual environment as well as in reality, though, the number of contacts in each section was higher in VR than in reality.

Fig. 9.

Number of contacts at each section during a trial in the virtual environment (n = 7).

Participants’ Opinions.

The results of interviewing the participants indicated that five out of seven participants found that section B was difficult. A participant pointed out not only the visual problems in section B mentioned by participants in the reality experiment, but also several issues relating to haptic sense; e.g. a participant said that they could not perceive and understand which surface contacted to the rod from the visual and the haptic sense. The other participants said that they felt unnatural force in section B. In this sense, we assumed they experienced abnormal perception in section B.

Deviation of RMS EMG Before and After Contact at Section B.

From the participants’ opinions, we investigated the performance at section-B in all trials and found that there were twelve cases in which the rod continued to contact with the surface of slit at section-B regardless of the participant. In these cases, the standard deviation (SD) of RMS EMG at the surface of two body part before and after contact were significantly different, as shown in Fig. 10.

4.3 Discussion

Trying to explain the observed undesirable performance based on the framework of SoA shown in Fig. 1, the results suggest that the participants made a strong effort to control the rod continuously when the participants faced those cases. According to their opinions, we assumed that the participant tried to create a perceptual representation mainly based on haptic sensory feedback and attempted to move the rod in various directions because they could not obtain visual feed-forward cues and produce a perceptual representation. They could not produce a feeling of agency, and consequently the bottom-up mechanism was disabled, and concrete judgement of agency was not formed. Thus, the extent of SoA in those situations was lower than before and the lower SoA led to undesirable performance.

Fig. 10.

Standard deviation of RMS EMG before and after contact (n = 7).

We assumed that there were several factors affecting feeling of agency in VR, such as incompleteness of visual feedback and haptic feedback despite participants having knowledge about their characteristics, which differed from those in the real environment. In this regard, it was estimated that forming propositional representation took more time than producing a perceptual representation because perceptual representation from unintentional sensory information was produced more quickly than processing conceptual capacities and attitudes such as beliefs or desires shown in Fig. 1. This indicates that knowledge or predefined significance was limited in virtual environment and could not be useful to perform tasks in VR and consequently retain a SoA to some extent.

5 Conclusion

In this study, we designed a VR environment and a task named the RTT. The RTT was performed experimentally in reality and in VR, and we investigated the problem concerning user performance in VR. The results of our investigation were considered based on a framework of SoA, and we found several points to consider when designing VR. Therefore, we concluded that the framework of SoA could be useful when determining design principles for VR. However, our conceptual considerations should be confirmed in further research.