The Shared View Paradigm in Asymmetric Virtual Reality Setups

2.1 Asymmetric Virtual Reality Setups

The importance of awareness of other users within virtual environments is already accepted [3]. Benford and Fahlén [2] describe this importance at the example of a virtual environment where all users use the same hardware to access the environment. They propose a spatial model of awareness where the view of users, the focus, is regarded as highly relevant for the interaction between them. The model is implemented in several systems, such as MASSIVE [23], where they show how different users interact concerning the avatar representation and their degree of immersion. Furthermore, work by Billinghurst and Kato [4] shows the relevance of the focus for AR. They argue that a rising degree of immersion diminishes the focus of users and restricts the cooperation within such symmetric immersive virtual environments (see [50] for further reference to asymmetry in the field of VR). Further work by Billinghurst et al. [5] describes the shared space concept, which is the application of AR in the area of computer-supported cooperative work. They point out that visual cues of users can increase the communication bandwidth and improve the performance to complete collaborative tasks. These insights suggest that the knowledge about what other users can see is of high relevance for asymmetric VR setups, as well, and that users may benefit from this knowledge. This motivates the research of visualizations of the learners’ view in an asymmetric setup during runtime.

Early work in the area of multi-user environments by Broll [6] proposes a trainer-trainee-system. Here, both the trainer and the trainee are part of the immersive environment. They experience it by using one VR HMD each. On the contrary, Roo and Hachet [47] investigate in multi-user environments that combine different mixed reality (MR) [38] modalities. In their work, users can choose between six different hardware setups, ranging from physical objects over projector-based AR to fully immersive VR HMDs. In all setups, users interact with the virtual world with spatial actions. Consequently, the authors describe necessary awareness considerations in such a setup to support the users. They propose to render what the HMD shows on an interactive picture so that users with a lower immersive experience can perceive what the fully immersed users see. None of their user roles utilizes a common desktop PC to get insights into the scene.

Peter et al. [41] propose a dedicated tool for a specific user role in asymmetric VR setups that they call VR-guide. As the name implicates, VR-guides try to guide immersed users through the experience of a virtual environment. The authors illustrate VR-guides in a scenario like the one we illustrated in Section 1. VR-guides use desktop PCs to get insights in the VR scene of the immersed VR users. They distinguish five areas that their proposed tool supports VR-guides in. Two of them relate to the actual view of learners. At first, they provide the optional feature to adopt the learners’ view by rendering what the users see on the desktop PC. Second, they mention features for monitoring learners. The authors state that additional visualizations, such as the exploration state of the scene, would also be helpful. However, the authors focus on implementing and evaluating interactive highlighting techniques in this paper, so that alternative solutions for the shared view paradigm remain unmentioned.

Recent work by Horst et al. [28], [29] also explores multi-user environments that incorporate both users that are fully immersed and users that utilize desktop PCs, like the VR-guide. They propose a texture-based solution to visualize avatars of co-located users within their virtual surroundings and render them on a 2D screen at runtime. A Kinect [37] camera and its associated real-time segmentation algorithm [49] are utilized to capture the learners’ appearances from a third-person point of view (POV). They track the camera using an external device to merge the physical space of the camera with the virtual space of a virtual camera and composite the textures using depth compositing. They provide educators the freedom to relocate the physical camera and get insights on both the virtual scene and the learner in it. As the authors focus on realistic avatar-representations, abstract view or gaze visualizations are not considered in this work.

2.2 Gaze Visualization and Virtual Reality

There also exists work about gaze visualization relating to VR technology. Stellmach et al. [51] transfer established gaze visualizations techniques into three-dimensional virtual environments. These are advanced 3D scan paths and 3D attentional maps (heat maps), which they visualize based on pre-recorded eye-tracking data. Löwe et al. [35] propose specialized visualizations and an according analytics framework to evaluate head movement and gaze data for immersive video technology. They include view similarity visualization that highlights viewing areas. Clay et al. [7] propose measurement and visualization techniques for exploration data and behavior data of learners. They record basic variables based on a fixed frame rate and save it for later evaluation. Within their visualizations, they include certain hit points in the 3D space at which users have looked a certain amount of time. They color-code these points based on distance measures from which they have been looked at. These works provide insights into the virtual content and the view of learners on a 2D screen. They enable experts to analyze the data a posteriori to the actual usage of the VR technology. The perception of gaze data during an ongoing experiment is not mentioned by the authors.

Figure 1

A 3D scene (a), the original POV of the learner and the educator (b and c), and five illustrations (d–h) that show the POV of the educator during the visualization of the learner’s view.

We have already mentioned the existence of work that utilizes common view visualizations in collaborative VR setups [52], [42], [43], [44], [33]. Lee et al. [33] propose a view in view technique that enables users with AR glasses or VR HMDs to perceive each other’s view. They render a view plane within the 3D environment so that both immersed users get a certain amount of what Lee et al. call ‘view awareness’ of each other. Tait and Billinghurst [52] use a view frustum visualization within the virtual environment to model the view and the head pose of remote users. Piumsomboon et al. [42], [43] make use of similar view frustum visualizations to give their participants awareness cues of what their collaborating co-users are seeing. Furthermore, Piumsomboon et al. [44] combine this visualization with avatar representations of the co-users. However, these works either focus on visualizations for fully immersed users or utilize the techniques as a side aspect and means to an end within their work. The visualization techniques seen in these works remain largely unexplored. Both established and novel techniques must be evaluated to conclude the suitability within an asymmetric VR setup.

3.1 View Visualizations

View visualizations are visual cues that support educators in our asymmetric VR learning scenario to understand what the learners are currently seeing. We consider five visualizations, illustrated in Figure 1d–h. For all of them, we allow the educator to navigate through the virtual scene freely. All view visualizations are only visible for educators – learners experience the VR without any additional visualizations.

The first technique we investigate is the volume technique (Figure 1d). Volume makes use of the camera intrinsics that are configured within the virtual camera that renders the view of the learner for the HMD. Based on these parameters and the camera position, a 3D view frustum is rendered in the view of the educator to illustrate which parts of the scene the learner can see at this moment. The length of the volume should be adjusted to a semantically reasonable length considering the content of the scene. For example, an indoor scene might only need a shorter volume than an exterior scene. Objects of interest could be farther away but still be visible for the learners outdoors, whereas an overlong frustum might cut walls through which they cannot see. As a help to find a reasonable length, level of detail (LOD) information could be used as they also include semantic information of the scene author and are already used to describe a variety of 3D scene properties [53], [14], [48].

View in view is a straightforward technique that shows the field of view (FOV) of learners in one corner of the educator’s FOV (Figure 1e). The FOV in a VR is the area of the virtual world that is seen by a user at a given time. The aspect ratio of the image overlay must be adjusted compared to the original HMD texture. Otherwise, the image texture could either be too small to perceive details or too big so that it covers a major portion original educator’s FOV. These adjustments depend on the ratio and resolution of the desktop screen and the HMD screen. By providing users a resizable window, they can adjust it during use.

Figure 2

Three illustrations that depict our proposed techniques for visualizing the exploration-state of a scene on a desktop screen.

The monochrome technique transfers the information that learners see from their point of view (POV) within the FOV of the educator by coloring it. Pixels that show areas that the learner can see are displayed in the original color, whereas all areas that are solely visible to the educator are depicted in a monochrome color palette. Therefore, this technique visualizes the intersections of the two views, as illustrated in Figure 1f.

Minimap (Figure 1g) displays a top-down map in one corner of the desktop screen. The learner’s gaze is represented by a triangular shape on this map, in the style of a 2D view frustum. In this technique, we also explicitly visualize the view of the educator in the same way as the learner’s view. It can help educators to orient themselves in the virtual world with respect to the learner. This technique is also able to show intersecting areas.

Figure 1h shows blueprint – a technique that uses a similar top-down map as minimap. Again, both views are visualized. Instead of showing textured scene objects, blueprint presents outlined shapes in uniform colors for selected objects of the scene. This map holds a certain abstraction so that users can concentrate on important objects rather than seeing all elements of a scene. Therefore, the blueprint approach requires a selection of important aspects of the scene before usage. These aspects can be landmarks of the scene that help educators to orient themselves within the scene, but also important exhibits that should be explored by the learner. Scene authors must choose which of the objects should be visualized in consultation with the educators. Based on this consultation, the scene author creates the blueprint map.

All the above-mentioned techniques are non-exclusive. They provide the opportunity to be combined. For window-based visualizations, such as view in view, minimap, and blueprint, multiple windows can be arranged on the desktop screen. The scene-based visualizations (monochrome and volume) can be rendered simultaneously in the same view. A combination of window-based and scene-based visualizations is a concatenation of these techniques.

3.2 Exploration Visualizations

Educators can be supported by providing them insights in the exploration-state of the scene while collaborating within the asymmetric VR setup. We explore three different exploration visualizations that are suited for usage during runtime on a desktop screen.

The minimap fog technique draws on common methodology from the area of third-person/top-down strategy games, called fog of war. Here, a map is placed on top of the educators’ FOV (Figure 2a). It is similar to the minimap or the blueprint visualizations which can both be used as a basis for this technique. The map is initially covered with gray ‘fog’. Only regions that have already been discovered by the learner are shown in their original coloring. The fog is removed little by little when the learner explores the 3D scene. Therefore, educators can see which parts of the scene were not explored yet and can guide the learners to these areas to ensure a full exploration.

Fog of war is usually used on a 2D map, as stated above. But we also transferred the methodology to the three-dimensional space for visualizing the exploration-state of the scene. We call this technique 3D fog. It is illustrated in Figure 2b. Despite projecting the map metaphor on a 2D plane, we fill the 3D room with fog and remove it in the areas that have been seen by the learner. Like the volume technique (Figure 1d), a view volume has to be defined which makes the fog disappear on intersecting with the volume boundaries. As for the volume technique, the size of the underlying view frustum has to be defined with respect to key aspects of the virtual scene. If the frustum is too big, areas that have not been actively explored might be revealed too soon. If it is too small, areas already observed might not have been marked. For this technique, it is also important that the fog is not too dense even at long distances. Three-dimensional fog otherwise may leave educators seeing only a black screen when they look into directions that have not been seen by the learner.

Figure 3

Four screenshots of our prototype that illustrate the investigated view visualizations. View in view and monochrome are shown as a combined version in (a).

The last exploration technique is halo (Figure 2c). Here, halo-effects are placed within the 3D scene. A halo signals that the area where it is placed is not yet explored by the learner. When explored, the according halo vanishes or tints. Again, a certain view volume is needed to specify when a section of the 3D scene is explored. Compared to its predecessors it does not visualize the entire area that has been explored but focuses on distinct exhibits or points of interest, like the blueprint abstraction mentioned before. Similar to blueprint, these distinct points have to be set before usage.

4.1 View Implementation

View in view (Figure 3a) was implemented using Unity UI components to create a canvas in the upper left corner. In addition to the camera for the educator, we created a second camera based on the synchronized data from the learner’s scene. This camera renders to a Unity rendertexture which then is projected on the UI canvas. For our prototype, we adjusted the canvas to a fixed size. We use the aspect ratio of the desktop screen as the determining factor and used 1/3 of both its width and height. The fixed size ensured that educators both were able to see details within the canvas and having enough space to navigate and perceive their own POV.

Based on the synchronized camera intrinsics, we create a view frustum object for the volume visualization (Figure 3b). This volume is rendered with a semi-transparent material so that the visualization does not occlude objects that learners currently see. We used a distance of 2 m for the ‘far clipping plane’ of the frustum to portray the visual attention range of the learner. This value was an approximation appropriate for the combined indoor/outdoor scene that we used.

The monochrome technique was implemented with three render textures and two shaders. Two of the textures reflect the current view of the educator – one colored and one post-processed in monochrome color. The third is a binary black and white mask which is rendered again from the educators’ POV. The first two textures are composited together with one of the shaders using the mask which contains the intersection information. The intersection reflects the parts of the scene that are seen by both the educator and the learner. The second shader is used to create the mask texture. Here, we utilize a view frustum, such as the volume implementation. This time it is not rendered, but it is used for calculating and determining which parts of the scene are seen by the learner. These parts of the scene are then rendered in the mask texture, in white, while the remaining areas of the textures are ignored and left black. When combined altogether, the generated texture presents the intersections of both views colored. The rest of the scene is displayed in monochrome (Figure 3a). This texture is used as the final texture that is rendered on the desktop screen using Unity’s Blit() method.

We implemented minimap (Figure 3d) by capturing the scene with a separate orthographic camera. It was placed vertically above the position of the desktop player. One challenge was related to the indoor area of our scene and the actual height of the camera position. Two solutions could be pursued to meet this challenge. First, the map camera could have been placed on a height that would cover an appropriate area of the scene with moderate orthographic camera frustum measures that determine the viewing volume for Unity cameras. But in our case there were objects in the outdoor areas that had a height larger than the height of the second floor. So, to capture these objects from above, the camera had to be at least as high as these objects were tall. Therefore, some objects like the roof, the gable, and parts of the second floor had to be put on a separate Unity layer. This layer is marked not to be rendered by the map camera since the camera follows the educator’s position. Otherwise, objects on the lowest floor would have been occluded by the roof of the second floor. The second solution would place the camera in a fixed and short distance above the educator’s head position and then crank up the size of the camera to cover a larger part of the scene. This counteracts that the roof or the second floor would occlude underlying objects, but objects larger than the camera height would be cut off on the map texture. Therefore, we decided to implement the first solution and customized the scene by creating a separate Unity layer for elements that should be ignored by the map camera.

A similar methodology as for minimap was used for implementing the blueprint technique (Figure 3c). We placed Unity standard primitives with a blue material at positions of important objects of the scene. These primitives then were scaled accordingly, and they were put on a separate Unity layer which was exclusively rendered by the map camera.

4.2 Exploration Implementation

For the halo exploration (Figure 4a), we utilized the Unity halo component on empty game objects. This component controls the size, color, and position. We positioned the halos manually at places of interest, analogous to the places where we positioned primitives for the blueprint visualization. For larger objects, we did not place halos at the volumetric centers of the objects but at suitable positions next to them, like in front of a chalkboard or on top of a table.

The minimap fog implementation (Figure 4b) utilized the map from the blueprint technique. The minimap technique itself could have been used alternatively, but to have fewer scene objects and less visual clutter on the abstract blueprint map, we decided to use this one. To visualize the fog, we spawned a set of coherent semi-transparent cubes that were only visible for the map camera. We controlled the resolution of the fog by the size of the cubes. The fog cubes were positioned perpendicularly between the map camera and the object primitives. A view volume of a given size was positioned on the same height. The fog cubes were disabled once they came in contact with the volume.

Figure 4

Screenshots of the three proposed exploration state visualizations. 3D Fog is illustrated in two separate pictures. On the left, the educator stands within the explored area looking towards non-explored space, whereas on the right, the user stands in unexplored areas and looks towards already explored ones.

We used a similar methodology to visualize the 3D Fog (Figure 4c/4d). Instead of creating one layer of flat cubes, we spawned cubes continuously attached to each other as a uniform honeycomb that filled the entire 3D scene space. We utilized a 3D volume as described above to disable the cubes that are intersecting with the view of the learner. Again, the resolution of the fog can be adjusted by the actual size of the cubes. In our prototype, we used a size of approximately 1 m² as we experienced several seconds loading time for this visualization with a higher amount of fog cubes. The actual FOV of a educator that stands in an already explored area and looks towards an unexplored one is shown in Figure 4c. The reverse case, where a educator stands in unexplored areas and looks towards explored areas is shown in Figure 4d. These pictures illustrate the importance of using non-additive fog elements. Looking through multiple additive textures otherwise would obfuscate farther objects and envelop the user’s view in black.

5.1 User Study

The user study involved 21 unpaid, voluntary participants (aged between 21 and 58 years with Ø 31,25 and SD 12,95, 5 females). Their VR experience was captured on a 4-point scale with Ø 1,15 and SD 0,91, where 0 means they never have used VR technology, and 3 means they regularly use VR. On that basis, we classify the participants of our study as non-specialists in the field of VR.

We used a 24-inch screen for our study. The prototype, as described in Section 4, ran on a gaming computer with an i7 processor, a GeForce GTX 1070 GPU, and 8 GB RAM. An Oculus Go was used as VR technology in this setup. The study was performed within a controlled laboratory environment. The real-world setup is illustrated in Figure 5.

Figure 5

A photo of the real-world setup of our study. One of the experimenters used the Oculus Go HMD while the participant used the view visualization prototype.

The procedure of the study was as follows. At first, participants were welcomed and then informed about the asymmetric VR learning scenario and challenges for educators of knowing what learners currently observe. They were introduced briefly to the user interface of the prototype and the keyboard controls. Thereafter, an experimenter assigned the participants the task to retrace and understand the view of an experimenter who took on the learner role and would use the VR technology and explore the virtual scene. We used a randomized block design to encounter the impact of a possible learning process on the scores throughout the study. So, each participant experienced the visualization techniques in different orders. The participants used each of the five view visualization techniques for about 4 minutes and observed an experimenter that explored a VR scene with several points of interest, as described in the prototyping section. They were asked to fill out a questionnaire in between each of the visualizations.

This questionnaire incorporated the questions Q1–Q4 and the abbreviated version of the AttrakDiff questionnaire [27], [25]. Our participants used a version of the questionnaire which was translated into their native language. We used a 7-point semantic differential scale for each of the items in the questionnaire to capture the scores. As all techniques that we explored have the potential to be used in a combination, we asked the participants to enable and disable the view visualizations until they reached a configuration that they found most suitable after their experiences they have made.

At the end of the user study, the participants were requested to fill out a post-study questionnaire that asked them to bring each of the visualizations in order from best to worst. This brief post-study questionnaire was also used to capture demographic data. A single session of the study was performed within the time frame of one hour.

5.2 Analysis of the Results

We performed non-parametric and dependent Friedman tests [17] on the items Q1–Q4 to test the data on significant differences. With a threshold for statistical significance of 5 %, all seven tests show that there were significant differences with p≤0.00001. We conducted further post-hoc tests (after Conover [8]) to identify between which techniques the significant differences occurred. Table 1 shows the output of the Friedman and the post-hoc tests. It also shows the absolute mean values for each of the techniques regarding each aspect, rounded up on four decimal places. The p-values from the post-hoc tests are shown in Table 2.

Regarding the perception of the line of sight (A1), Table 1 depicts that view in view, volume and minimap have similarly high values and all differ significantly from the lower-rated blueprint and monochrome techniques. The field of view aspect (A2) was rated above 5 points in the mean for view in view and volume, above 4 points for minimap and below 2.5 points for blueprint and monochrome. View in view differs significantly from the remaining techniques, whereas minimap differs significantly from all other techniques, and blueprint and monochrome differ from the first three. So, view in view and volume were rated significantly higher than the other visualizations for A2. The third aspect for the view visualization (reliability, A3) shows that view in view was given the highest mean score of 5.3333. View in view also differs significantly from the other four visualizations. Volume and minimap rank second having higher values than blueprint and monochrome. Volume and minimap also differ significantly from the latter two. Regarding the disturbance of the view visualization (A4), volume received the highest score. The score differs significantly from the remaining four techniques. The lowest value at this aspect was obtained by the monochrome technique with 3.000. It only differs significantly from the volume technique.

Figure 6

The portfolio presentation compares the five view visualization techniques regarding hedonic and pragmatic aspects.

We also analyzed the outcomes of the AttrakDiff questionnaires and compared the view visualization techniques regarding A5. We used descriptive statistics as suggested by the authors of the questionnaire [27], [25]. The portfolio presentation of the view visualization scores (Figure 6) arranges the five techniques regarding their hedonic and pragmatic characteristics [27], [26]. It shows that the minimap technique is closest to the ‘desired’ region. Next to it are volume and view in view. They scored lower hedonic but higher pragmatic quality. Blueprint and monochrome are farther behind in both values and lie within and next to the ‘neutral’ field.

The word-pair visualization of the view visualization techniques (Figure 7) gives further insights into the single items of the AttrakDiff questionnaire. Besides a more detailed reflection of the portfolio presentation, it shows some special features and anomalies of the techniques. Monochrome has two larger amplitudes at the items ‘confusing – predictable’ and ‘unimaginative – creative’. Except for one, all items for monochrome were negatively rated compared to the neutral value of 3 points. However, our participants rated the creativity of the visualization as the highest of all five techniques. For view in view we can observe some larger negative deviations at the items ‘ugly – attractive’, ‘tacky – stylish’ and ‘unimaginative – creative’. All view in view items lie above 3 points, whereas all the blueprint items are located left of the 3-point-line of the graphic. The lines of minimap and volume run visually similar. Six items of the volume technique were rated higher than for minimap and four items of minimap were rated above the corresponding volume items. The largest gaps of these two techniques are located at two items of concerning the pragmatic quality ‘complicated – simple’ ‘impractical – practical’, where the volume visualization obtained higher ratings and one item of the hedonic quality category, where minimap was preferred by the participants (‘cheap – premium’).

Figure 7

The description of word-pairs for the view visualization techniques shows the mean ratings of the single items that the AttrakDiff consists of.

The combination and individual sorting tasks were analyzed, as well. The according data is presented in Table 3 and shows that the combination ‘view in view & volume’ was preferred by ten participants, which was nearly half the number of participants. The second most preferred combination was ‘volume & minimap’ with five preferences. The sorting task was transferred on a scale that ranged from 0 (last place) to 4 points (first place). The results are illustrated in Figure 8. The box-whisker plot shows that the interquartile range of the volume and view in view techniques begin directly at the highest score limit.

Figure 8

The graphic shows the individually preferred view visualizations.

5.3 Discussion of the Results

All Friedman tests have shown significant differences. Based on these results, we reject all null hypotheses H1₀–H4₀ and accept the corresponding alternative hypotheses H1_a–H4_a. The techniques differ regarding the proposed quality criteria A1–A4.

Further results of our user study show how the view visualization techniques differ. The three techniques view in view, volume and minimap were more suitable for observing what learners see within the asymmetric immersive VR setup than blueprint or monochrome. For the aspects A1–A3, the first-mentioned techniques performed significantly better than their alternatives. Regarding the visual disturbance of the view visualizations (A4), the volume visualization was perceived significantly less disturbing by our participants than the other four. The AttrakDiff analysis supports this claim, as it shows a considerably high product character (A5) score for volume, especially regarding the pragmatic usability.

The single preference rating in Figure 8 suggests that our participants liked view in view and volume most. Furthermore, this trend is established with the combination preferences (Table 3). Here we see, that these two were not only chosen over the remaining three visualizations, but they were clearly preferred to be used together, as well.

The study generally shows that future applications similar to our use-case should choose between the volume and the view in view visualization or even use these two techniques combined. Both individual preferences and measures of the questionnaire items support this claim. Using a combination of multiple view visualizations is additionally motivated by the study results since only three of our twenty-one participants chose to use just one visualization, while 15 preferred a combination of two, and also three preferred even a combination of three techniques.