Keywords

1 Introduction

Virtual Reality (VR) is about to revolutionize the world of media and is considered as one of its latest developments to the most promising defining digital media [3]. VR allows the user to immerse and interact with alternative realities in a computer-generated 3D environment [1, 2]. The degree of how much one immerses in the VR world can vary. Full immersion requires tools such as a head mounted display (HMD) that can generate a 3D visual and audio environment. Also, input devices such as a data glove, which gives the user the feeling that they perceive the fictitious world as their real one [4].

The research and application of VR technology in education has enriched the way you teach and learn [5]. Today we can work in virtual environments (VE) not only in the areas of scientific visualization, monitoring of complex technical objects or entertainment, but above all in simulations and training. Computer simulations have been shown to be effective instruments for teaching. Most simulation interactions in learning environments today focus on the mouse and keyboard controls as a computer interface. However, there are other effective ways to extend learning simulations to more physical and embodied modalities, which have unique spatial characteristics that enable sophisticated knowledge representation in virtual environments [6, 7].

E-learning technology improves traditional teaching techniques. Abstract concepts can be illustrated and understood better. It breaks the limitations of space and time. Lessons can be learned by students anytime, anywhere. VR supports e-learning and brings it closer to the real learning environment. In the e-learning field, practical approaches can be learned almost realistically with immediate feedback [8]. Recent research has shown that motivation to learn through virtual learning environments is increased [9].

We use storytelling to further increase that motivation and create a higher level of immersion in the VR world for the user. Storytelling is used in teaching to better communicate learning content to students. Learning content is linked to a story, making it easier for students to remember what they have learned [10]. Also, the natural interaction with virtual characters increases user immersion. An important principle in storytelling is to involve the audience in the story. They experience the story and do not just hear it. Storytelling can combine learning and emotional wellbeing to help build sustainable knowledge [11].

Guest students from other universities often find it difficult to find their way around the new environment and to find important premises. For this reason, we have built a virtual environment of our university, so that these students can do a virtual walk around from their university. This is to ensure that they can learn where at the host university certain rooms such as the deanery are located. In this paper we want to showcase that the learning process in virtual reality is much more effective when storytelling is used.

The goal of the experiment was to find a specific room in the fastest possible time. Therefore, one group had to find this way with the help of storytelling and the other group without. An important challenge which VR content creators face, is the interaction between viewer agency and narrative guidance, as viewers step into a \({360}^{\circ }\) environment and have the freedom to choose where to look. We can trigger curiosity with sound, light, interaction, or movement so viewers naturally turn in a certain direction. In addition, remembering things is made easier by involving emotional experience in virtual storytelling.

First, we will summarize related work and confirm that storytelling is a missing task as a tool to guide the viewers attention in the navigation process. Afterwards, we will propose our used material and methods for this experiment. Furthermore, we will explain our results in more detail. Finally, we will give a short outlook and discuss our future work.

2 Related Works

The study of storytelling in VR involves understanding e-learning concepts, and methods of directing attention in \({360}^{\circ }\) virtual environments.

2.1 Storytelling in E-Learning

The way stories are told is crucial to how much we remember details. Stories have the ability to form memories and thoughts like no other form of modern communication. To simplify remembrance of important details, abstract story skeletons are built that are central to the narrative process [12, 13].

The dissertation by Oaks [14] examines the learning behavior of college students. The goal of this study was to find out what, if any, effects storytelling as a pedagogical method of information retention. Above all, it focused on whether the storytelling variant is more effective than the conventional lecture method to positively influence the recall of lesson content. It has been found that the subjects who have been instructed in the storytelling method have significantly better ability to remember learning content than the students who have been taught the same content in a more traditional lecture method.

In this paper, we want to take the approach that storytelling is more effective than traditional methods to remember learning content to the next level. Storytelling triggers emotions that make it easier to remember. Through a higher immersion which is achieved with the use of virtual reality in a \({360}^{\circ }\) environment, storytelling should anchor learning content even deeper into the memory.

2.2 Directing Attention in VR

One of the main problems we have to face in the virtual world is that in a \({360}^{\circ }\) canvas the viewer has the freedom to look and is not restricted to a frame view when watching a movie or playing a game. This makes it all the more difficult to direct the viewer’s attention to where the action is happening. We also do not want the user to miss important story content because he is looking in the wrong direction. In addition, we want to make the experience as immersive as possible and not force it to direct in a desired direction, but the viewer should get the feeling if we direct his attention that he will do it of his free will.

Sheikh et al. [15] have tested various techniques in their work to direct the user’s attention in virtual reality in a desired direction. In doing so, they analyzed qualitative and quantitative data from these tests to evaluate the effectiveness of the different attention-directing techniques. They examined in various ways how the user responds to audio or visual cues. It was found that using both cues at the same time was the most successful in the tests to direct the viewer in the desired direction. The advantage of audio is that its use, even if not fully spatialised, indicates to the viewer that something is happening and his attention is already being tempted. But with visual cues, subjects often followed the cues, but not to the very end.

They also tested subjective experience and collected data on preferences at what distance the scenes in the \({360}^{\circ }\) environment happen. The results of the participants showed that the distance had no effect on following the action. The enjoyment of watching the video and the degree of immersion were adversely affected if the distance to the action was less than or more than three meters. The distance of three meters provided a good balance of being close enough to enhance immersion, but far enough that not to penetrate into the private space of the user.

In this paper, we want to use visual and acoustic cues to support them with storytelling elements. By being involved into the story, the user’s attention can be more easily directed in a desired direction as our test-subjects seek the right path and immerse more and more into the VR world. By taking a role as an actor in the \({360}^{\circ }\) environment, we try to direct their attention to the right path, which the viewer is supposed to remember. The fact that each junction serves as a node to reestablish the correct track, the viewer should be helped in these places by storytelling in order to trigger a recall more effectively than in the control condition without storytelling elements.

3 Method

Since we often get visited by guests from partner universities, we have provided them with a virtual environment, which allows them to conduct a virtual tour beforehand from their home in order to be able to find important rooms such as the cafeteria more quickly. The rather modern design of our faculty makes all corridors look identical, making the search for the desired location difficult. In order to make these places more easy to find, we also want to show the way from the entrance to the desired location in a video. In this paper, we want to compare two approaches. One video shows the search process using storytelling, the other video without. By this approach we want to check which method is better suited to remember a certain route in the short- and longterm.

3.1 Storytelling

Virtual characters are important components in virtual storytelling. The natural interaction with virtual people allows the user to immerse more into the virtual environment and makes the viewer feel like a part of this virtual world. We distinguish two methods of plot generation [16]:

  1. (1)

    Character-driven storytelling focuses on the inner conflict of a character. The viewer is more concerned with the personal development of the characters and their attitudes and think about the reasons for making decisions and how these in turn change the form of action and the story as a whole.

  2. (2)

    Plot-based storytelling places greater emphasis on the plot itself, as well as plot twists and outer conflicts. The goals of the stories focus more on the development of a situation. The user chooses between different plotlines throughout the story.

Since the goal of the video was to memorize the right path in an unknown environment, we opted for a plot-based approach that traditionally presents the storyline in a predefined way. However, this process lacks the refinement of character behavior. Since we only see the \({360}^{\circ }\) movie and do not interact with the characters it contains, this is not relevant to us.

3.2 Storyboard

For our experiment, we shot the \({360}^{\circ }\) video as a single take. We wanted to avoid that the user is torn by scene change from the VR world. We have used narrative anchors in certain places with the help of storytelling in order to let the user look in the desired direction and memorize junctions. As a pretest, we have provided the viewer with a simple and essential task for a student, which the user has to fulfill in the context of a story. They are supposed to find the machine with which you can charge the chip card from the canteen for the cafeteria. The starting point is the entrance to the university. In order to guide the viewer to the right room or giving important hints, people in the video interact with the viewer. The path to the cards charger has six junctions where we want to direct the attention through storytelling as seen in Fig. 1. In the first scene, the viewer sees two students talking about wanting to eat in the canteen. One student remembers that he/she has no more money left on the card and says he’s/she’s going to the charger. This person speaks to the viewer and asks if he or she wants to do the same and takes him/her over the stairs to the next floor (Scene 1). This is the point where the viewer gets involved in the story.

On the next floor, the student talks to the spectator and tells the subject that they are going to race down the long hallway and meet up on the next floor, showing the viewer the way to run (Scene 2). In order to optically control the viewer with the help of light, the viewer sees light sources on the stairs at the end of the hallway, which implies to take this way up (Scene 3).

Arrived on the next floor, the student comes to meet the test subject. The actor admits that the spectator won the race, which gives the audience a sense of achievement and captivates the viewer more in the story (Scene 4). In addition, the actor tells the viewer that he/she should go on, because the actor is too exhausted from the race. In the next scene, the viewer meets a game master congratulating the viewer on his/her victory. The game master shows three possible ways that the viewer can follow (Scene 5). On the basis of cards the spectator pulls the number for the next way. By picking number three, the viewer continues on the path on a long hallway again. Attention can be directed acousitcally, as sounds come out of a room as you approach an important door. That is why acoustic elements are only executed when the viewer enters a certain area in our video (Scene 6). The test-subject hears a conversation between two people discussing how much money they should load onto their chip card. Once the viewer enters the door, he/she sees the cards charger at the end of the room and the video is finished.

Fig. 1.
figure 1

Storyboard of the video with storytelling.

Full size image

3.3 Participants

Subjects were recruited from students from various faculties of the University of Wuerzburg. All data used was collected at the University of Applied Sciences Wuerzburg-Schweinfurt. 30 people participated in the study. 15 people were shown the video with storytelling and the other 15 a video with the same process only without the storytelling elements such as light, acoustics or the participation of actors. Both groups then had to find their way from the university entrance to the cards charger.

3.4 Materials

The video was shot with the GoPro Fusion Black 360. It takes a video of \({180}^{\circ }\) via two internal cameras and then stitches them into a \({360}^{\circ }\) film. To this view, this camera also takes on spatial audio. So that subjects can watch the video, the HTC Vive Pro is used as a virtual reality glasses. Also, the subjects wear the Tobii Pro Glasses 2 eye tracking glasses for the second part of the test. This high definition camera captures a full HD video of what is happening in front of the participant, as well as its eye movements and orientation.

3.5 Process

First, the subjects got the HTC Vive Pro glasses and watched two initial videos for acclimatisation purposes. After that, both the storytelling group and the group without it got notified about the task of watching the following video and memorizing the way to the card charger. After watching the video, they answered the immersion questionnaire by Jennet et al. [17] and then set off to find their destination again. The test persons were given the eye-tracking glasses, with which we tracked their gaze behavior during the search. In addition, metrics such as the number of steps and duration for the entire search process were also collected. Finally, a concluding interview was conducted on the experiences of the entire experiment.

4 Results

The descriptive results of the experiment are shown in Table 1 and Figs. 2 and 3. The variant with storytelling was far more effective than the variant without. 93% of the subjects with storytelling have found their way back. In the conventional one, the success rate was only 66%. Even the average time and number of steps in finding the way was less in the group with storytelling than in the group without. The difference between the average of the steps is not so great because the subjects lingered on the site for a long time thinking about the way without storytelling.

Table 1. Average time and steps as well as the success rate at completion of the search process with and without storytelling.
Full size table

It was interesting to observe that of the presented storytelling elements the junction with the game master was the most effective. From the personal interviews it has become apparent that this part was best remembered, because to the subjects this was very surprising and unexpected. By using visual and audio cues at the same time, it was easier for the viewer to take the path to the left of the stairs the actor had shown before the race started. The fact that the viewer has been told by the actor that the way is at the end of the corridor, the subjects were more easily guided by the light cues. By using spatial audio, all subjects were able to find their orientation to the right door, which made the subsequent search process easier, because they could remember which direction they were turning when hearing the sound. Participants were prompted for their thoughts and emotions on the videos by being asked to rate their experience using the immersion questionnaire [17] and rate between 1 represents ‘not at all’ and 5 ‘a lot’. When comparing the ratings, the variant with storytelling always performed better than the variant without. Many factors have contributed to better memorizing the path through storytelling. Subjects who answered the Immersion questionnaire have been given more attention by the task. The personal appeal of the actor to the user increased the involvement. The urge to win was therefore also higher and the fun factor through the various elements was also raised. Ultimately, the increased immersion and the other mentioned emotional stimuli leads to an increased motivation to find the way and remember this better than without storytelling.

Fig. 2.
figure 2

Evaluation of the immersion questionnaire of the subjects with and without storytelling.

Full size image

Heat maps and gaze plots are data visualizations that can reveal important data about gaze movements and their behavior. Heat charts show how the gaze spreads across the stimulus and can simultaneously represent the gaze movements of several people. Gaze plots on the other hand can represent gaze fixations and the order of gaze movements. Through data visualizations and personal interviews, we found out that, at the recollection of the path, 73% of the subjects in their head went through the video sequence. This gaze plot depicts a looking pattern for this reason.

The subjects generally viewed the game master as he spoke to them where also in both visualizations increased residence time can be determined. Afterwards, their gaze mostly went through the path ways, which are responsible for the outer viewpoints.

Fig. 3.
figure 3

Heat map and gaze plot of the gaze samples from the scene with the game master.

Full size image

5 Future Work

There are many approaches to explore. After this pretest, we will examine how sustainable the learning success of indoor navigation through storytelling is. This means that the same subjects should each repeat the search process after longer intervals in order to test the mentioned metrics. Furthermore, we want to test the cognitive load in the learning process to find out how high the load is when remembering the way or how the interaction is when you have to remember the way and must follow a story at the same time.

In addition, one could test how effective different types of directing attention methods are in a \({360}^{\circ }\) environment at each junction. A reaction test could then be used to check how quickly the attention is directed in the desired direction in order to stay on the right path.

To give the viewer more freedom to find a path, one could support the search process with interactive storytelling. The search process would no longer be limited to a single path, but would give the viewer several options for targeting. Interacting with virtual people would totally immerse a user into what might impact the learning process. The story would be co-created by the user because each action section depends on his decision and is not fixed [18].