1 Introduction

The virtual reality technology can provide various experiences by reproducing the space. In the generation of experiences, it is usually necessary first to reproduce the space of the external 3D space as an environment. By acting on its own in the reproduced environment, the user can gain experience in a specific environment. Such an experience enables the user to know the past three-dimensional environment (space) through its own actions, and it is extremely effective in giving practically close knowledge as obtained by a person experienced at that time.

However, it is the current user who senses the environment and acquires those knowledge, and the experience is not the same sensory experience as what the past people felt when acting. In order to pass on the actual past actions of other people and the sensation they received at that time, the past experiences themselves must be handed down. There are various levels of experiences, from physical motion in space to those with deep thought, however it is a very difficult goal at any level to pass on such experiences themselves [1]. Especially, it is a difficult task to transfer high presence experiences involving physical activity and experiences based on dexterous physical skills including sport motion. Physical motion is the base of all human activity, and if we can share that experience, exchanging the exclusive value that an excellent person possesses will increase the average activity level of human beings. It is expected to accelerate the development of human society.

2 Re-living Experience

Conventionally, linguistic expressions in books etc. or images such as movies have been used as media for transferring general experiences (see Table 1). Both of them can cover a wide range of activities as an expression of the events of experience, however they can not include the sense of bodily motion itself through the audiovisual channels. They are distinctive in psychological depictions and in depicting the world of the first person or third person viewpoint. In books etc., experiences are created internally by interpretation and imagination based on memory. In recent years, movies extended to multiple senses have emerged [2]. This includes presentation of vestibular stimulation, airflow, scent, mist, tactile sensation to the back and feet, in addition to stereoscopic images and stereophonic sounds. By the extended movie, it is possible to greatly enhance the presence of the space. In addition, the sensation of riding on the same vehicle in the movie is typically rendered. In such a movie, the tactile sensation of the viewer is stimulated by the device that applies pressure and vibration implemented in the seat. The tactile sensation received by an arbitrary person in the movie is also reproduced. Such a multisensory presentation contrihoweveres to raising the sensation of presence as a part of the story production, however it lacks essential parts as a re-living experience. An ordinary movie is not composed mainly of depiction from a first person viewpoint which is a unique characteristic as a re-living experience.

Table 1. Media and contents providing re-living experience
Full size table

To reproduce the actual physical experience of someone, it is necessary to present the changing visual field as the result of the viewpoint motion during his/her action. In addition, the sensation of the body motion executed at the time of the past action must be presented. In particular, if advanced motor skills are to be re-lived, it is necessary to present the sensation of the position and posture of the body in action and the muscular strength control for each part of the body when performing it. These features stem from the fact that it is a reproduction of the first person’s perspective. Strictly speaking, it will be a reproduction based on all cognitive events during the life of an individual, however this is impossible to realize. The important element is the sensation of the body due to the action and the visual perception according to the change of the view point. This is the conclusion from that the fundamental policy is to reproduce the body sensation that always existed in the actual experience.

As described above, there is a difficulty in the physical re-living experience which regenerates after others on the basis of the body. It is a different way of perception from ordinary external perception. The difference is that the perception during the action is modified corresponding to the control input of the physical motion. For example, in visual perception, there is a saccade suppression in which the input is modified during the motion of the eyeball [3]. There is a fundamental difficulty that it is required to regenerate the situation of such embodied perception [4, 5] by others, performed in different physical conditions. This introduces implementation difficulties from several viewpoints.

First, there is difficulty in visual expression. Showing high-quality images at remote places imparts the sensation of presence of the environment, especially if resolution and viewing angle are large, however it is insufficient for the body-dependent first person’s sight. It is difficult to make the image perceived by the user (the reliving person) precisely identical to the one seen by others at the level of scan path. To control the motion of eyeballs is harder than to measure the scan path of the past observer. However, it is possible to acquire a specific task skill of the visual processing by tracking the gaze point of the other person, although the scan path is inaccurate where the movements of the eyeballs do not match between the user and the past person. In that case, the scope of application is considered limited. By recording and playing back the video, the re-living visual experience can be provided where the user perceives what was going on in the outside world at the actions of others. However, in order to render it closer to the first person’s experiences, its characteristics must be introduced. Specifically, it is necessary to introduce the vection that is invoked in the first person activity such as walking in addition to the physical property from which the motion of visual field is created. As a method to visually introduce such physical feature in walking, perturbation of visual field was investigated [6]. More directly, rendering the image of one’s own body is considered effective. However, the effect of the body image has not been clarified sufficiently when it is observed from the first person viewpoint.

Second, there is a difficulty in presentation of the body sensation. Actual physical motion is the basis of human activity, however the degrees of freedom of movement and the amount of space are so large that its measurement and reproduction are difficult to implement. A motion capture system of a fixed measurement space is required to obtain accurate body motion data of others. Although a wearable motion sensor is convenient to measure a free body motion, it is still difficult to achieve sufficient accuracy so that the outdoor behavior for a long duration is a difficult problem at present. Furthermore, in order to reproduce the sensation during physical motion, it is necessary to evoke sensations of limbs, hands, trunk, head similar to others. For this purpose, the re-living user should do the same motion as others, however it is almost impossible to execute the same motion in the real space. What virtual reality technology has mainly developed up to now is the virtual expression of the outside world to the user, however in the next second stage (2.0), the virtual expression of self body sensation is considered to make its feature. The sensation of physical motion is represented by proprioception and skin sensation, however vestibular, visual and auditory sensations also play an important role. It is necessary to synthesize and provide integrated stimuli to these sensory organs, which is a difficult technical problem, especially in terms of high degrees of freedom of motion and its speed.

Thirdly, there is a problem of representation of agency that is the subjective sense of body movement. Re-living other person’s behavior is to receive the whole sensation perceived including body sensation. However, general actions are originated by oneself, and the resultant sensations are utilized in the actions. In other words, there is a loop of active movement in which active motion produces the sensation and the next motion is generated based on the sensation. This is incompatible with the fact that the re-living is a projection of replicated sensation. In order for the re-living to be equivalent to the experience of oneself including his/her physical motion, the same sensation as others must be received as, at the same time, the result of active motion of the user. In other words, causality must be reversed as that from the sensation to the agency. Consequently, the information obtained in the re-living experience should approach to that from his own experience, otherwise it is considered that the user is not able to reproduce that motion by oneself (especially in the case of advanced motor skills). Although this is a mere re-examination as a third party, it is necessary to become your own experience, however the problem of the active movement loop remains.

3 Hybrid Re-living: A New Experience Presentation

There is the difficulty mentioned above for the re-living experience involving physicality. For this reason, we decided to extend its scope as follows. Although the brain process in which the first person experience related to physical motion is created is not elucidated to a sufficient level, this is the process where a consistent memory of subjective actions is created. The extent of matching to the other’s experiential data can take up to a level of theoretically perfect identity from that is only perceived as being in the same environment. Among them, except for the ideal state that can fully be recognized as one’s own experience, the both of the perception of other person’s experience and the perception of one’s own experience coexist at the same time. Thus, it is a re-living experience in a hybrid state. Furthermore, in terms of rendering of the visual avatar introduced to realize the re-living experience, duality of physical body perception is also recognized. It is duality of an avatar’s body and the self body, and duality of first person drawing and third person drawing. In addition, there is duality of passivity and activity in the re-living process.

Thus, following four hybrid properties, or duality is an issue of design.

  • Self and the other person

  • Self body and the avatar’s body

  • First person and third person viewpoints

  • Activeness and passiveness

3.1 Reproduction of First-Person Experience from Third-Person Data

Experience is a consistent representation of the outside world and the self body depending on a physical body. It is thought that the perception of the other person’s state is performed by using the perception mechanism of own body [7]. However, at the same time it is clear that it is not their own physical body state, as there is a gap in perceptual identity. Therefore, it is a problem of virtual reality 2.0 to make equivalent perceived state even if its own body perception is not exactly the same as the other’s. In the beginning, the goal of the virtual reality was to make the perception of the external virtual world equivalent to the actual one, and the accuracy of the drawing of the outside world was sought for that. In the perception of the re-living experience, since the goal is to make the perception equivalent to that perceived by the other person of the outside world and the body, it is a hybrid perception state assuming the existence of others. On that premise, it is necessary to draw constructively the outside world information and the bodily information corresponding to the behavior of the other at an appropriate level.

Table 2. Action and result in three levels (Act: activation, Rst: result)
Full size table

The Table 2 represents experiential elements at each concept level of vision, body, and agency. In order for a series of actions to have meaning, a structure of High level is necessary, however since the VR technology operates at a perception level, the Low, Mid levels are the targets of rendering. For these, the VR system gives the user the properly operated input for the experience reconstruction.

Regarding visual perception, we record and present the audiovisual information of the outside world, however as shown in the above discussion, the Low level is difficult to intervene, so we provide information recorded in the Mid level. The image of the remote place from the viewpoint of a person flows along with the walking that is the basis of the person’s action. If the image in the direction of the head only was recorded, the user see simply the playback of the image. On the other hand, if an omni-directional image is recorded, the active head rotation is allowed to the user in the viewing experience. In both cases, the optic flow by walking provides an important part of the sensation of motion, and it has a great effect on the transmission of behavior.

As for the body, by adding movement stimuli to the vestibular sensation and the limbs, the sensation of a moving body can be created. Although the degrees of freedom of the motion are considerably restricted in the implementation technology, the sensation of walking and running which are the basis of a human spatial motion, can be generated to a considerable extent. Because walking and running are of high importance in human motion, automation (rhythm generation, standardization of motion pattern) of the execution control of the body movement is implemented in the CNS. Therefore, it is considered that the perception of body movement may be easily replaced by that from an external force input. Motion of the hand based on a visual feedback control requires a good design for the device that reproduces the movement sensation of the hand which is thought to be more difficult to be automated than walking and running.

Avatar expression of the body is a powerful clue that connects visual and body sensations. When rendering an avatar, it is possible to choose whether it is a model of yourself or a model of another person, and whether to render from first person viewpoint or third person viewpoint. In the case of rendering from the first person viewpoint, by placing the virtual camera near the eyeball of the head of the avatar, the body of the avatar can be observed by looking downward. In the case of rendering from the third person viewpoint, although it is free where to place the viewpoint, it is appropriate to set a field of view from which the avatar’s body can be observed from the back of the avatar. In the case of the third person viewpoint, the motion state such as walking is clearly understood, and the impression of the walking experience in the recorded environment becomes clear under the condition that the entire body of the person or another person is in the field of view. In the case of the first person viewpoint, the view is almost the environment of the outside world, however if you swing hands largely during walking you can see the hand. If you look down, you can see your feet, however while walking you are looking at the front almost so the visual impression of the body is not great.

Agency of physical motion is obtained by driving your own body by yourself and getting matched change of the result. It is almost always established in a normal experience. Although, we do not have to be conscious of it, in the case of experiences driven by your own body, it is obviously a necessary relationship. In a re-living experience, this agency poses a problem. The goal is that the user’s body motion is the same as the other’s motion, however the user himself can not perform it. By externally driving the user’s body motion, it may be possible to bring the motion trajectory closer, however it is no longer a self-driving experience. As with other duality, a meaning can be added to this agency problem by introducing the continuity of the level of the agency in the spectrum of the re-living experience. Partially introducing active selection into passively transmitted environment and body data is considered to contribute to the activity of the experience. Under the multiple duality described above, the quality of the re-living experience is considered to be determined.

3.2 Implementation and Evaluation

Multisensory Passive Stimulation. By giving passive body motion of walking as well as images of the environment, it is possible to present the sense of walking and driving to a considerable extent. The presentation of body sensation is based on external stimulation to the body. If no activity is given, the user on the seat simply receives multi-sensory presentation. Under this condition, the movement stimulation to the body is captured by the vestibular sensation, the proprioception, and the skin sensation. This provides the whole bodily sensation considerably similar to the normal body motion (walking etc.). Audiovisual information is presented in monocular omni-directional images recorded at remote locations, and the user was allowed the head rotation. Figure 1 is a multi-sensory presentation system (FiveStar) operating at the exhibition of a conference. This system provided a physical re-living experience whose data was recorded by the omni-directional camera during walking in a remote place (Toronto and Niagara fall, Canada).

It was revealed in the stimulus presentation experiment that these multiple sensory stimuli each have an effect of enhancing walking sensation. The vestibular stimulation by the motion seat can present lifting and driving sensation of the left and right legs with pitching/rolling and lifting to the front, back, left and right. Foot motion display represents the sensation of locomotion with alternating up/down and back/forth movements of the legs. One of the characteristics of these motion expressions is that very small amount of motion is appropriate as compared with the actual motion during walking motion. This is related to the problem of the activity. The sensation of passive motion is felt greater than the motion sensation evoked as result of active motion.

Fig. 1.
figure 1

Multisensory presentation system (minimized system) at a conference exhibit.

Full size image

Avatar Rendition. Presenting a visual body, an avatar, as shown in Fig. 2 makes the perception of motion state in the remote environment clearer. This is the image of a scene where the avatar based on the 3D measurement of the experiment participant is walking in the city of Toronto. We constructed a virtual tour that walks Toronto and Niagara for about 3 min with a screen composition that the view point follows just behind this avatar. The realism of the experience of this virtual tour was evaluated by 80 participants of the virtual reality academic conference. As a questionnaire of presence, a visual analogue scale that spans from the conference venue (0.0) to the Niagara falls (1.0) was used. The result of presence evaluation was at a value of 0.738 (standard error 0.016) which means that the sensation evaluated was closer to the sensation of being in Niagara.

Fig. 2.
figure 2

Participant’s avatars walking virtually in Toronto.

Full size image

Furthermore, using these individualized avatars, we presented the city view with a walking avatar from the viewpoint as first person and third person shown in Fig. 3. The stimulus was only visual information (not multisensory) with a duration of 60 s. The participant was sitting on a stationary seat. We added a condition with active commitment of the participant who was allowed to stop the movie by aborting the arm swing. Using these avatars, we evaluated walking sensation and body ownership when the viewpoint was changed as first person and third person like Fig. 3. As a result, it is suggested that the first person’s avatar rendering was more effective in improving the walking sensation of the participant. Body ownership rating was almost the same as the rating of walking sensation. There was no significant difference as to whether the avatar was self or another person. On the other hand, the active condition where the movie was stopped by aborting the arm swing had higher walking sensation and body ownership than the condition that the movie was played without arm swing.

Fig. 3.
figure 3

Avatars walking in Toronto.

Full size image

4 Conclusion

In this paper, we presented a new structure and implementation examples to reproduce the body sensation of others as a next generation (ver. 2.0) virtual reality experience, When presenting the sensation of motion of others to pass on bodily experience, it is necessary to consider duality in several respects. The duality was found in the virtual experience where both self and others are present at the same time, and where there is a difference between your true body and the avatar’s body posture and movement, and where both viewpoints of the first person viewpoint playback and the third person view playback are useful. It is also the problem of activeness of real experience and passivity of re-living experience. Because of these aspects of duality, the re-living experience is required to have hybrid properties as its basic characteristics. It is a challenge to optimize this hybridity in the design. An implementation by a multi-sensory display including an omni-directional movie and bodily stimulus revealed preliminary knowledge about body presentation by an avatar and placement of viewpoint. As activeness of physical motion and agency are important characteristics of the bodily experience, resolution of incompatibility with the passivity of re-living experience is one of the future tasks.