Virtual Reality Based Space Operations – A Study of ESA’s Potential for VR Based Training and Simulation

Abstract

This paper presents the results of a study the authors conducted together over a year in order to identify key issues of ESA’s (European Space Agency) potential for a deployment of Virtual Reality training environments within space operations. Typically, ESA simulates several operations using DES like systems that need to be linked to a VR environment for training purposes. Based on the second generation of VR equipment and development tools the paper describes a holistic design approach from scenario development through design decisions on SW and HW choices until the final development of a PoC for a virtual lunar base that might simulate the metabolism of a lunar base. Here the idea was to mirror the mass- and energy-flows within a lunar base in order to maintain an environment, in which astronauts can live and work and to establish a tool that supports the training of astronauts for operating such a lunar base, the one likely next step of human space exploration beyond the International Space Station as identified by ESAs decision makers. In the end, we have realized a PoC for a fire emergency case on a lunar base allowing astronauts being trained in a fully simulated and integrated environment. The system could be tested and evaluated in two set-ups, first using classical VR controllers, second, using recent VR glove technology.

1 Motivation

“There’s life in the old dog yet” might describe the “second life” of Virtual Reality (VR) based systems and thus several new endeavors for the realization of VR based training and simulation systems. In the last 3–4 years, we have been facing a second wave of VR technology conquering consumer markets and industrial applications. This is, by no means, only been pushed by new hardware and affordable pricing models, but also new technology stacks, that have been designed and developed specifically for interactive applications being deployable on any potential device or platform. Coming with the miniaturization of sensory and its modalities, new ways of perceiving the environment, new forms of interaction and new ways to distribute and deploy interactive content have been realized. Here, the precision and easiness of technology deployment might have been a third pushing factor for the revival of VR based systems, leading to a new hype of its use. The potential of precise, low cost VR and new forms of HCI also inspired new approaches for training and simulation environments. Therefore, the European Space Agency (ESA) started an activity to establish a study on the deployment of VR technology for Space Operations, its training and coupling to the internal simulation backend. This paper presents a holistic approach for the design and realization of a Virtual Reality (VR) based workplace that enables ESAs decision makers, astronauts or operators to conduct interactive explorations of different situations using recent technologies in view of the realization of a virtual lunar base.

2 Background

ESA identified “lunar exploration” as one likely next step of human space exploration beyond the International Space Station (ISS), which operates in Low Earth Orbit. One option is the establishment of a crewed lunar base for long-duration human presence on the Moon. For studying and optimizing the metabolism of a lunar base (i.e. the mass- and energy-flows within the lunar base in order to maintain an environment, in which astronauts can live and work) and to establish a tool that supports the training of astronauts for operating such a lunar base, the use of a virtual lunar base appeared to be a promising option [1]. In this scenario, one major challenge is the coupling of ESA’s simulation backend to a real-time responsive, immersive VR environment. Typically, the simulation backend offers services based on discrete event simulation models (DES) that provide access to run-time data of space systems (Space Systems Simulators Infrastructure (SIMULUS)), or functions and services specifically serving the needs of spacecraft monitoring and control systems (MCS - Mission Operations Support Infrastructure (MICONYS)). Although many of the DES systems do offer the possibility of interaction, they lack the ability to place the user of the simulator into the center of operations that might be realized through an immersive 3D representation of a simulated scenario [2]. The potential to directly link operations simulation to an immersive virtual reality (VR) environment, allowing users to interactively change the simulation while in process, opens new ways for exploring complex interactions between model users, objects and operations being simulated. In previous works, [3] examined the promise, at that point in time, VR presented to developers of DES models, though initial enthusiasm has been thwarted by the tremendous overhead in 3D content creation and the limitations of VR. Nevertheless, the authors established the notion of VR-based DES or VRSIM. A comprehensive overview on VRSIM or VR-DES technologies is given in [4], concluding that mainly industrial manufacturing systems in the view of smart factories have been realized as testbeds for VR environments with a focus on the whole value chain optimization for rapid decision making in simulated “what-if scenarios”. The authors also identified future endeavors that will be focusing on new sensory equipped and networked environment with many more real-time (big) data that might need to be pushed into a VR environment. Although many best-practices have been established [4], any coupling to existing infrastructure components comes with its own peculiarities and specific problems imposing new challenges on VR based resp. “user-in-the-loop” exploration of DES generated data.

3 Overall Approach

3.1 Ideation and Scenario Development

We were starting our study with ideation and concept creation sessions with several ESA stakeholders involved (i.e. ESA’s space operation center – ESOC, ESA‘s astronaut training center – EAC, and ESA’s research and technology center – ESTEC), to draw on different scenarios of use and derived use cases. The findings should provide the basis on the choice of technology and the basis for the technology planning. During the ideation sessions, stakeholders were interviewed to freely discuss potential scenarios independent of the available technologies and possible use cases. A maximum of 20 persons with different backgrounds and different profiles in ESA’s operations were asked on their ideas and on the use of this technology. Resulting scenarios have been grouped into main categories such as training, operations and planning. In a second step for each of the scenarios, representative use cases for the use of new technologies have been elaborated which lead to the identification of actors, their interactions, and the interaction with ESA’s ground systems resp. involved simulation modules from the ESA’s infrastructure (Fig. 1).

Fig. 1.

Ideation and use case definition at ESOC/ESTEC/EAC premises (mid: high level scenarios and grouping, right use case definition describing (fltr) actors, tasks and interactions, simulation modules involved)

As a result, some of the scenarios within training should be mentioned here as those have been identified by several ESA units to be used for a potential proof of concept implementation. Here ideas grouped around: HW failure of a lunar rover and its recovery by an astronaut; HW failure in MCS Situation – problem occurs at MCS/fire or HW failure; training of personnel on recovery actions - A new set of astronauts arrive on lunar base, familiarizing themselves with existing infrastructure, inspecting the base, and astronaut has to assemble parts of VLB, e.g. new modules, antennas, parts of rover, etc. Training & familiarization of «any device» in VR assembly/disassembly of device in «micro or zero gravity» Moon Base operations – Astronaut performs maintenance task on lunar base: monitoring status of devices; performing actions; ground carries out “in parallel” involving bidirectional communication base and ground. The scenario of the astronaut-training center focused on an emergency fire case, in which astronauts have to be trained on how to deal with exceptional emergency operations, like rescue and evacuation, communication & coordination with ground, extinguish fire in module.

3.2 Selection of Technologies

VR

In a subsequent step, the choice of technologies has been taken in which recent technologies (a comprehensive overview of STAR technologies can be found in [5]) have been chosen in order to establish the VR infrastructure for the VLB. Here, following the taxonomy of [5], we have focused on the deployment of a stationary head mounted displays such as HTC Vive (see Fig. 2, top). For manual tasks we will evaluate ManusVR¹ gloves (see Fig. 2, right) as input device for gesture tracking and manual interactions as an alternative to the HTC Vive VR controllers (see Fig. 2, left). These controllers are shipping with the HTC Vive and are based on their lighthouse tracking solution. For VR application development, we used Unity3D².

Fig. 2.

VR set-up for the VLB implementation, top: HTC Vive HMD, left: Vive controllers, right: manusVR gloves.

Simulation

While there are different simulation environments in use all over ESA, we identified SIMSAT as a good candidate for the PoC, because it is widely used and there are already interfaces for different other components relevant in different identified scenarios. In addition, this decision allowed us to use behavioral models made for the lunar base.

3.3 Realization

For the realization of the VR-DES coupling, we have defined an indirect connection of the VR environment to the SIMULUS/SIMSAT infrastructure of ESA. Here, the interface leverages a python adapter to connect via the CORBA API provided by SIMSAT. On the other side, the adapter connects to a message broker. This message broker is used to exchange messages between the SIMSAT adapter and the VR. Because its message protocol is open and the messages are JSON encoded, this interface can be used for other clients in the future. Possibilities are for example web based status displays or augmented reality systems, which are planned for a future study.

In the VR environment, a single node directly connects to the message broker, and handles the communication between special interactive nodes and the simulation. The system has been designed in a way that allows scene developers to set up a new VR scenario without any programming on the simulation connection side.

The first prototypical implementation has been realized and was defined within the framework of the EAC training program (see next chapter).

4 Implementation of a ‘Virtual Lunar Base’ PoC

For the final development of a proof-of-concept (PoC) showcase, we have chosen in agreement with several ESA stakeholders a scenario in which they deemed VR as most beneficial. Thus, together with EAC/ESTEC and ESOC, we have been further developing an astronaut training scenario with related use cases, which has been identified in stage one of the project.

4.1 Scenario

The developed scenario simulates an emergency in a lunar habitat in which an astronaut – the habitant – has to follow a standard procedure typically applied in orbital stations such as ISS. Here, a fire is detected in one of the racks inside the lunar base, and a habitant has to follow the procedures to extinguish it. These procedures are in place to minimize the risk for the responder as well as other habitants of the lunar base. Several objects to handle this task are available to the user. Some of these objects are connected to the simulation, while others only exist in VR to allow a trainer to verify that certain steps in the procedure are taken. In order to keep the scenario evaluation friendly, some simplifications where made to allow untrained evaluation participants to solve the tasks as well. For example, a device used to measure combustion products in an enclosure simply shows a temperature. The following list contains some examples of objects, which can be found in the VR scenario. Some of these are invisible to the user, some are stationary and some can be carried around.

• Fire Extinguisher: A gas cylinder containing a finite amount of CO². It has a mountable nozzle to deliver the gas directly into a rack. The user can pick it up, mount the nozzle and operate it.

• Fire Zone: Invisible volume inside of a rack, which communicates with the simulation. The user can only interact with this zone via the extinguisher or the meter.

• Thermometer: The device is inspired by the CSA-CP used on the ISS. To simplify handling, the probing lance is directly connected to the device, rather than via a hose.

• Lights and Buttons: These directly communicate via messages with the simulation. There are environment lights, alarm lights and switches for light, ventilation and energy.

• O² Mask: The use of this item only allows the trainer to check if procedures are followed.

• Torches: These can be picked up and used to illuminate the room.

The O² mask and torches are examples for objects, which have no connection to the simulation at all. The mask is worn to signal the trainer that this self-protection step in the procedures is followed. The torches help the user to navigate the lunar base when the lights are switched of, but this interaction happens purely in VR.

The trainer is able to modify simulation parameters and can check how his trainees do react in the presented scenario. Possible modifications for the trainer include:

• Selection between multiple fire zones,

• Automatic fire alarm or user notification via “Green Card” (message like “you smell a burning odor”),

• Failed automatic deactivation of the inter module ventilation,

• Severity of the fire, which determines if it can be stopped by removing power from the rack or an extinguisher has to be used.

The evaluation section contains a description of a use case, which has been chosen to test how fast a new user can handle the given tasks.

4.2 Interaction Design and Multi User Set-Up

To help new users to understand how to interact in this virtual environment, we included interactive interaction courses for controllers and gloves. Figure 3 on the left shows such interactive instructions for the gloves. These courses lead the users through all possible forms of interaction and show which buttons/gestures can be used.

Fig. 3.

Trainee perspective. Left: Calibration and interaction tutorial. Right: Sharing the space with a second trainee, who is represented by his HMD and controllers

Gloves and Controllers

One goal of the PoC was to have a platform, on which different interaction devices could be compared. During the workshop, we identified the Manus VR gloves as a more natural interaction device for VR, so the PoC was develop to be usable with both, Vive controllers and Manus VR gloves.

Trainer View

To observe what is happening in VR, the trainer can choose between different view modes: First Person, Overview and Follow. For First Person, the trainer shares the same camera perspective as the user. For Overview and Follow, the trainer gets a view from above the lunar base, with all structure above the users floor removed (see Fig. 4, right). For Overview, this view contains the whole base, while for Follow, the view is zoomed in on the user. When working with multiple Users, these options are available for every user.

Fig. 4.

Trainer Views. Left to right: Simulation interface, ground control telemetry and VR spectator view.

The trainer can also manipulate the simulation parameters to put his trainees in different situations. The left and center images in Fig. 4. Trainer Views. Left to right: Simulation interface, ground control telemetry and VR spectator view. Figure 4 show shortcuts to trigger common behavior in the simulation and a telemetry screen. He can also directly write into simulation parameters to create unique situations.

Multiple Users in VR

The tasks in the training session can be shared between multiple users. To allow this happens at two different levels: Simulation and VR. For the direct VR to VR connection, the capabilities of the VR-Engine where used. This allows the user to see each other in VR (represented by their interaction devices: HMD and Controllers/Gloves, see Fig. 3), and to synchronize the position of objects that can be picked up.

Object states that are controlled by the simulation do not need to be synchronized externally. Both VR instances react on the same messages, thus their objects have the same state. Therefore, if one user uses a light switch, his VR instance sends this message to the simulation and both VR instances receive the change of the lamp state.

5 User Studies and Evaluation

5.1 Methodology

In order to validate our solutions we conducted specific validation sessions with different ESA stakeholders addressing the fire emergency scenario in a simulated VR environment of the virtual lunar base. The system we were setting up offered the use of controllers for interaction, and, as it was required by the PoC, the use of gloves for an interactive manipulation of objects. Therefore, several validation sessions were conducted in two iterations. The simulation backend was controlled by a trainer that supervised the trainee during each session.

We decided on an easy to explain use case, which was tested by volunteers from the ESA staff (mostly ESOC and EAC members). Since the test users (trainee) came from different backgrounds and weren’t necessary involved in the virtual lunar base project we constructed a set of tasks that were easy to explain without getting into too much detail about the environment.

To prepare the participants, they were given time to familiarize with the controllers while the trainer explained the function of different objects necessary for the tasks ahead. After that, the participants solved the tasks, while the trainer took time and gave verbal support if needed.

After this stage, the participant proceeded to repeat the experience with the gloves as input device. Since the setup for the gloves is significantly more difficult than for the controllers, we also measured the time it took the users to step through the glove calibration process.

An interactive guide then explained the participant which gestures are used to trigger the interactions learned with the controller, followed by a moment of time to test these in the environment. After this familiarization phase, the known tasks were repeated using the gloves.

Finally, the users had to fill in a questionnaire following the design mentioned below. We then could gather the feedback remotely and could analyze the users reaction to the developed PoC.

User Experience (UX)

We aligned our feedback design to recent research results provided by the HCI community. However, in contrast to only focus on pure usability aspects of the PoC, we aimed at analyzing the user experience (UX). In contrast to usability procedures, UX tries to balance instrumental and non-instrumental qualities that are subjectively perceived by the user. Thüring and Mahlke [6] claim, that the three central components of UX are represented by the perception of instrumental and non-instrumental qualities that trigger emotional reactions when a person uses or has used a product. They highlight that traditional usability testing with a focus on removing interaction problems, which cause stress, does not interfere enthusiasm for a product compulsory. Therefore, not only pragmatic aspects such as utility or usability of a product but also hedonic aspects such as stimulation, identification or evocation might influence the user perception. Those can then be measured by emotional reactions such as pleasure, satisfaction and develops an appeal towards the product.

Thus, the model of [6] guided the design of our questionnaire and the choice of specific aspects for “quantizing” user experience aspects. Due to the manifold of (non-)instrumental qualities, the selection has been based on “subjective” priorities for the purpose of this study. We have been focussing on three selected aspects of UX proposed by [7]:

• perspicuity as a measure of learnability – “How easy to understand is the user interface?”

• efficiency as a measure to what degree the user can use the interface with a high level of productivity – “Is the workload for the interaction with the VR environment reduced to a minimum?”

• stimulation as measure for the stimulus of a user, as a “driving power” for interaction and learning of new skills – “Does the VR environment captivate the user?”

Questionnaire

An online questionnaire suited as basis to gather feedback from the test subjects. The intention was to collect qualitative as well as quantitative data during each validation session. The time required to conduct certain tasks following the standard procedure is considered as one of the quantitative parameters within the subsequent evaluation for the execution with controllers and gloves, the elevation of the non-instrumental and instrumental aspects for perspicuity, efficiency, and stimulation the others. Here, we mapped the aspects to a Likert scale with 7 features ranging from one to another extreme. For a later assessment, the scale was subdivided into five intervals that are allocated to different levels of maturity [8]. The neutral interval contains values between [−0.8; 0.8 (To achieve a “good” the interval has been defined to [0.8; 2] resp. “poor” (−0.8; −2]. Extreme values at the end of the scale are rated as either “very poor” or “very good” (Fig. 5).

Fig. 5.

Instrumental and non-instrumental aspects elevated in this study. Mapping them onto a Likert scale following a typical user experience questionnaire layout [8]

The qualitative data blocks contained five questions with pre-formulated answers following the layout in [7] and a free text area where subjects could comment. This data blocks should reflect on the user reactions supporting the capturing of certain UX aspects. We asked:

• 1. What do you think about the handling of the VR Scenario using the devices?

• 2. What do you think about the time it took to use the devices?

• 3. What do you think about your experience with the VR scenario?

• 4. What do you think about the functions offered by the VR environment and the presented scenario?

• 5. Was every function available that you wish to have?

5.2 Validation Sessions

Place and Time

The validation sessions were conducted during a two-week period at the premises of ESOC in Darmstadt and the European Astronaut Training Center (EAC) in Cologne.

Participants

The participants were volunteers from ESOC and EAC. Roughly ~2/3 were male. The youngest participant was 20 and the oldest 55, with a median at 30. About 10% claimed to be experienced VR users, while the rest divided equally into users with some and with no experience. In total, we had received 17 filled in questionnaires.

Test Cases

The volunteers had to fulfil the following tasks that have been presented at the initial stage of the validation session and the methodology described above. Figure 6 shows some of these tasks:

• 1. You will be alerted by the trainer about high temperature readings in a specific rack.

• 2. Take the temperature probe and verify the high temperatures in the specified rack.

• 3. Switch off the power circuit for the rack.

• 4. Pick up the oxygen mask and mount it onto your head.

• 5. Pick up the fire extinguisher and mount the barrel to the valve Sect. 

• 6. Empty the extinguisher into the rack.

• 7. Use the Temperature probe to verify that the fire is extinguished and alert the trainer.

Fig. 6.

Evaluation Tasks – Left to Right: checking temperature, switching off power, putting on the oxygen mask and extinguishing the fire

5.3 Results

Quantitative Analysis

We have gathered the feedback from the Likert scale and normed then according to the described procedure. Interesting wise, it clearly shows high values for the controllers. Although the median values are not too far from each other, the variance for using glove control is much higher, to the upper 75% percentile as well as to the lower 25% percentile in several aspects. Especially the lower whiskers, which range to “bad” indicate that the users were not very satisfied using glove control. The perspicuity of glove control has been perceived as much more complicated than using the controllers. Also, users have not been convinced to execute the tasks in an efficient way as compared to the execution of the tasks using the controllers (Fig. 7).

Fig. 7.

Instrumental and non-instrumental aspects for controller and gloves

The summary chart Fig. 8 visualizes the aggregated mean values over the Likert scale results for each aspect.

Fig. 8.

Comparison of the averaged results for perspicuity, efficiency and stimulation with controllers and gloves

Timing of the Tasks

We have captured the timing for execution of the tasks in both system settings. In both settings, the trainer defined a fire location in order to start the simulation. After the trainee indicated a “ready” to the trainer, the trainer started the time recording. It was stopped as soon as the tasks have been fully executed and the fire simulation has been extinguished resp. the temperature went below a lower degree threshold. Figure 9 shows the results of the timing. In addition, here, the volunteers managed to solve the tasks faster using the controllers rather than the gloves. However, median values are not too distant; the discrepancy to the upper percentile (0.75) and upper whisker is more significant for the gloves. A clear timing advantage has been observed for the controllers.

Fig. 9.

Results of the time recording to solve all tasks

Conclusion

While all users had their experience with controllers first, we expected to see faster times with gloves, since the tasks are only a repetition. While the overall feedback is positive, the feedback indicates a more positive response to and a faster execution time using the controller interaction. Taking into account, an average calibration time of 1:52 min for glove control, the controller interaction became more accepted. This quantitative feedback is supported by the results of the qualitative data blocks.

Qualitative Analysis

Testing with the Controllers

While 94% rated the experience as completely useful and interesting, only 75% found it to have all functionality they wanted to have. While one user missed feature in the physical modelling, others wished to have the means to do moon walks or have simulated smoke in the habitat.

Only 6% deemed the handling of the VR to be too demanding to focus on the content, 41% needed some time before they were able to focus and over 50% could had no problems to use it from the start.

82% of participants were satisfied with the time it took them to solve the tasks, and the remaining 18% had solvable problems to some extent. None of the participants rated these problems high enough to say they prevented them from consuming the content.

From the free text feedback, most of the comments referred to aspects of the virtual lunar base that were excluded from the evaluation scenario. Others made clear that we are missing a clear optical distinction between indicators and switches, likewise, that there is no way for the user to know if he is hitting the fire zone with his extinguisher nozzle. Another request was for a more game like interaction, where objects are collected and can be directly at hand if needed, instead of carrying every object on its own. One participant asked for direct movement instead of teleportation, which let us know that when explaining teleportation, we should tell how it helps to prevent motion sickness in many users.

Testing with Gloves

30% of users felt that the glove interaction was too complicated to focus on the content, and another 60% needed some time before they could focus. Only a third was satisfied with the time it took them to solve the tasks.

In the free text feedback, some participants showed that they enjoyed the more natural interaction, once they familiarized themselves enough with the gloves. Other where astonished that the gloves were not easier to use than the controllers were. Some users complained that not all interactions using the gloves are “natural”, which is to be expected for teleportation, but also grabbing an object can hardly be experienced in a natural way without haptic feedback.

6 Conclusion

This paper presented the results of a one-year study in order to elevate the potential of ESA’s space operations being linked to a simulated training environment. We have realized a virtual lunar base that aimed at establishing a training environment for “lunar exploration”, which has been identified by ESA as one likely next step of human space exploration beyond the ISS. We managed to couple the DES system backend of ESA’s simulation environment to a highly interactive virtual environment, in which astronauts can be trained for certain tasks that a trainer could design and develop. The exemplified PoC realization focused on a dedicated training scenario that has been developed together with ESA’s astronaut trainers as well as further stakeholders at the different premises of ESA. The overall decision chain and communication flow between lunar base, mission control and the link to ESA’s telemetry simulation backend has been realized. A fully-fledged fire emergency simulation can be performed in the VR environment, leveraging several physical simulation results as well as ESA’s SIMSAT backend services. In order to validate the different possibilities for user interaction using VR controllers as well as recent glove solutions (here ManusVR), dedicated validation sessions have been set-up, the feedback of volunteers gathered and analyzed. Our initial hypothesis that the comfortable ManusVR gloves might be more attractive to the volunteers and might lead to a better task performance over VR controller could not be kept. Instead, the overall performance of volunteers dropped behind in several instrumental and non-instrumental aspects of user perception compared to VR controllers. More frustratingly, the timings the volunteers needed to execute a range of tasks using classical controllers are lower than with gloves. Adding the calibration time that each user needed for using the gloves to even get started, only a weak recommendation for a similar system set-up with gloves can be given.