Keywords

1 Introduction

One goal of Augmented Reality (AR) training is to inject virtual objects and events into a live environment to allow trainees to acquire the targeted knowledge, skills, and abilities in a highly realistic environment. Two primary reasons to apply AR within a training program are to reduce the cost associated with live training entities and to create stressful conditions that could cause a significant level of risk if completed live. The goals and benefits of Virtual Reality (VR) training are a similar balance of creating highly realistic and potentially stressful conditions while controlling the risk and cost of training. The process of inducing stress within a controlled training environment is important when preparing learners to perform for high-stress conditions. Alternatively, there is utility in increasing levels of perceived stress and associated arousal within AR and VR environments in order to engage learners.

A variety of methods of inducing stress during training have been developed and evaluated [1–4] and resilience training and stress inoculation programs rely on those applications to elicit target states during training. For example, the U.S. Air Force’s Stress Inoculation Training (SIT) program trains battlefield airmen to first understand the negative effects of stress, learn to detect and control stress responses, and finally practice skills taught under realistic high stress conditions [5]. Guidance followed during this program suggests that stressor intensity should be incrementally increased as task proficiency is demonstrated [6]. This approach ensures that trainees are not overwhelmed early which could interfere with skill acquisition while continuing to push learners operate under more and more realistic high-stress conditions [6]. The greatest challenge with creating training that balances learner stress levels lies in the ability to track learner stress levels over time.

This paper presents an approach to objectively measure stress levels during AR and VR training and reviews applications of the approach to optimize and evaluate training programs. The presented technology leverages a physiology-based stress classification algorithm that has the capability to objectively and unobtrusively capture real-time individualized stress data in a mobile environment with over 95 % accuracy. While originally designed to support Veteran mental health therapy, the algorithm, which uses a wrist-worn device to collect human physiology state provides a continuous realtime stress measure that can be used to optimize AR and VR training. The approach provides a more effective option to alternatives, including subjective stress ratings or cortisol analysis post-training.

Two key applications of the stress classifier are presented in this paper. The first application is the creation of adaptive AR and VR training systems that adjust stressors that are presented to learners based on their state and performance. The goal of this application is to balance stress with training progression during AR and VR training exercises. The second application included is the objective evaluation of AR and VR training event fidelity based on objective trainee responses, with the goal of optimizing future training events. In addition to reviewing frameworks to support the application of stress evaluation in AR and VR environments, a comparative evaluation of stress evaluation techniques is presented.

2 Measuring Stress

The most important and difficult step in creating a training platform that can adapt based on trainee stress is the effective evaluation of stress. Early measurements of psychological stress have relied upon written scales that leverage a battery of validated questions to elicit a rating of stress for each individual. These scales were initially developed to support clinical psychologists and fall within two general categories of measures: Measures of stressful life events that have occurred; and measures of the subjective evaluation of perceived stress and/or ability to cope with stress.

The use of life event scales date back to Holmes and Rahe’s 1967 Schedule of Recent Experiences (SRE) which includes a checklist of 43 stressful life events such as the death of a spouse, divorce, or being fired from work [7]. Each of the life events that were experienced on that scale are counted and the total is used to provide a relative stress level for the person being evaluated. Since the inception of the SRE, longer batteries and a variety of specialized batteries have been developed for special groups such as children [8] or combatants within the Gulf War [9]. Because the scales that are used require a level of customization to the events that occur within the targeted population, this approach to measuring stress has the potential to lack consistency or sensitivity.

A second approach to measuring stress involves the use of perceived stress measures. The most commonly used perceived stress measure is the Perceived Stress Scale (PSS) [10]. The PSS includes questions that measure of how unpredictable, uncontrollable, and overloaded respondents lives are in combination with direct evaluations of experienced stress levels and is available in a four, ten, and fourteen item batteries. Although perceived stress measures are more generalizable than life event scales and can be gathered more efficiently using the four item form, they still require some level of self-evaluation and cannot be continuously gathered without breaking the flow of training.

2.1 Physiological Stress Monitoring

To support the capabilities to unobtrusively modify training based on trainee stress and performance or objectively evaluate the effectiveness of a stressor to create targeted trainee states, it is critical to move away from written batteries and towards physiological measures of stress and negative arousal. Advances in wearable physiological sensors have made it possible to measure human states that are capable of quantifying stress [11, 12], including cardiovascular and respiratory measures and electrodermal activity. Because most tasks that AR and VR training are beneficial to train require physical activity, there is risk that those measures could contain noise during training. Particularly, because the same physiological states that are affected by increases in the sympathetic nervous system activity (which allows them the be effective stress measures [13]), also tend to fluctuate due to physical activity or environmental conditions, there is a need to cast a wider net when classifying stress, and take into account a variety of physiological and environmental sates simultaneously. Table 1 provides and overview of physiological states that are particularly useful for classifying stress, examples of conditions that could create noise in each data type, and references of previous efforts to apply each measure to classify stress.

Table 1. Physiological states, features, and sources of noise for stress classification
Full size table

Researchers recently developed a physiological classifier of stress that leverages a combination of skin conductance, cardiovascular features, skin temperature, and physical movement collected from an Empatica E3 band to serve as the core of a clinical stress therapy tool [19]. After collecting EDA and HRV data for participants under baseline conditions and under stressed conditions (created using the Trier Social Stress Test (TSST) protocol), a linear classifier (see Fig. 1) was trained to differentiate psychological stress vs. non-stressed conditions at over 95 % accuracy. In addition to leveraging multiple features listed in Table 1, the researchers leveraged gyroscopes to detect movement and skin temperature to detect temperature to discount components of the classifier when significant levels of noise were present.

Fig. 1.
figure 1

Decision boundary for linear stochastic gradient descent- trained E3 classifier applied (left; reprinted from [19] with permission) and Empatica E3 band (right)

Full size image

3 Leveraging Stress Measures to Improve Adaptive Training

Two of the key characteristics that are evaluated in AR and VR training applications are immersion and presence. Although throughout the history of VR and AR development there have been various interpretations of the terms, immersion generally refers to the level of fidelity of the sensory cues used within the environment while presence is a subjective response of the person experiencing the environment [20]. By this definition, it is apparent that fidelity can be objectively evaluated by comparing the visual, auditory, haptic, and olfactory cues present in a VR or AR environment to those in the live environment. Presence is a more difficult to objectively measure construct as it relies on the evaluation of the person experiencing the environment. To target this evaluation and move away from subjective ratings or presence, physiological data, including those that are described above to measure stress (e.g. change in heart rate and skin conductance) have been studies and demonstrated a correlation with presence measures over 15 years ago [21, 22].

In addition to measuring the sense of realism that VR and AR systems can instill, the use of physiological stress measures in combination with real-time performance can be used to drive adaptive training. Adaptive training has the potential to significantly optimize training efficiency [23] and based on the varied response of learners to stress exposure training, the process of adapting training based on individual responses to stressors holds even more potential. Originally designed as a clinical intervention for patients that required coping mechanisms for conditions like anger and phobic reactions, the goal of stress exposure training is to alleviate the negative effects of stress on performance by preparing personnel to perform tasks effectively under high-demand, high-stress conditions [24].

Figure 2 outlines a simplified stress appraisal process and includes two paths of stress response, one associated with the availability of effective coping mechanisms that results in normal performance and one that is associated with a stress appraisal that leads to a decrease in normal performance. VR and AR Stress Exposure Training seeks to build effective coping mechanisms through repeated presentation of stressors. In the therapy domain, this process is used to target a single phobia or condition with continuous support of a therapist throughout the process. When preparing for a wide array of stressors, such as preparing for battlefield stressors in an active combat environment, this task becomes more complicated because individuals react differently to environmental stressors based on past experiences and their appraisal of the situation [25].

Fig. 2.
figure 2

Appraisal process and effects of stress

Full size image

In order for the stress exposure therapy process to continually present new stressors within training to (1) determine the conditions that must be targeted in training and (2) repeat the exposure in a controlled environment that allows the development of effective coping mechanisms, it is critical that a performance and stress measurement feedback loop be created and used to drive future presentations of stressors. Stress training aims to teach the necessary skills for maintaining effective task performance under stress conditions and to enhance familiarity within the environment. Research on stress exposure training and the effects of stress provide the following guidance to the development of adaptive AR and VR training frameworks:

  • Virtual training scenarios must continually be adapted to add various types of stress within the environment in order for trainees to develop coping mechanisms.

  • In order ensure effective transfer of the coping strategies to the live environment, stress conditions must be similar to those found in the live environment.

  • Because the goal of stress exposure training is to ensure effective task performance under stress, training must take into account the real-time performance of trainees to ensure that use of the system does not lead to negative training.

Figure 3 outlines an adaptive stressor process diagram that can be followed to control stressor presentation within VE and AR training platforms. The approach and architecture requires four key components in order to effectively drive stress training:

Fig. 3.
figure 3

Adaptive stress training process diagram

Full size image

  • Performance Evaluation- The ultimate goal of training is to improve learner performance. Although stress exposure training has the secondary goal of training performance under high stress conditions, optimizing performance must remain as the core goal of training. In order to ensure that negative training is avoided when inducing stress, it is critical to take into account trainee performance on targeted training objectives prior to evaluating learner stress states. The goal of the evaluation is for learner performance to remain high (or bounce back when performance drops) when new stressors are presented within the training scenario. For this to effectively work in a closed adaptive system, the performance evaluation metrics should be coded within the AR/VR training environment.

  • Stress State Evaluation- Physiological stress monitoring provides a real-time and objective method to continuously track stress and the response to stressors within the AR/VR training environment. A wrist-worn stress monitoring approach similar to the one described above has the potential to evaluate stress when learners are in an environment that allows them to move around, such as an AR training space.

  • Trainee State Classification- Once trainee performance and stress state is calculated, it’s important to merge the data to support the determination of overall trainee state and the appropriate response of the training system to meet the targeted training goals. For example, the combination of good performance and a low stress state should be used to trigger a new stressor within the scenario while a combination of high stress and low performance should be used to reduce/remove the stressors present to allow trainee performance to normalize and avoid negative training.

  • Stressor Activation/Deactivation Methods- In order for scenario adaptations to be applied based on performance and stress states, it is necessary to include hooks into the code to activate and deactivate stressors in real time. To meet this need, modular stressors that can be activated in real-time (e.g. reducing visual acuity of the scene, integrating additional enemies, etc.) must be designed and scripts developed to trigger them based on the evaluated trainee state.

By applying a closed adaptive training cycle that can modify AR and VR training scenarios in real-time it is possible to create an ever-changing environment that pushes learners to create coping mechanisms for environmental stressors that they are expected to operate during the presentation of in a live transfer environment. The approach allows a variety of potential stressors to be presented in close succession and only continuing the focus on those that require additional coping strategies. If this approach is scaled to groups of learners, it is possible to garner knowledge regarding the effectiveness of each stressor to trigger targeted stress states on a more generalizable scale. This capability to evaluate stressor effectiveness is a second key benefit of objectively measuring stress within AR and VR training environments. By measuring the response to cues across trainees, it is possible to classify or order stressors based on their effectiveness of created targeted stress states. By evaluating additional characteristics of each trainee during this evaluation, it is possible to further classify the effectiveness of various stressor cues based on trainee characteristics such as expertise level or past experience with particular cues/conditions in a live environment. This subclassification of training stressor effectiveness has the potential to optimize the order and presentation of scenario stressors based on each trainee’s specific experiences.

In order for the evaluation of stressor cues to be supported in real-time, an infrastructure must be developed to precisely track the presentation of cues in the training environment. Specifically, in addition to objective measures of stress, the following characteristics must be tracked in order to develop models of stressor effectiveness:

  • Cue activation tracking- Within an AR environment stress induction cues can be triggered live or as an augmented component of the scenario. In order to evaluate the effects of cues and combination of cues accurately, a tracking system must be instantiated to track when live and augmented cues are triggered.

  • Trainee characteristics- By tracking a database of trainee characteristics in addition to the stress response to various stressors as they are presented, it is possible to create a more precise prediction of the cues that will be responded to in similar ways during future training.

The approach of leveraging physiological data in combination with performance data to drive adaptive training augmentation has demonstrated significant value in previous research. For example, research has suggested that skin conductance response (SCR) as a measure of arousal can be used in combination with a contextual understanding of the training tasks that learners are completing to predict learning gains and modify adaptive tutoring systems [26]. Further research conducted at the Army Research Lab led to the development of an architecture, similar to that presented in Fig. 3, of personalized adaptive training that leverages a combination of trainee physiological state and performance to drive training adaptations [27].

4 Future Research

The research into adaptive training platforms that leverage physiological stress states to drive training augmentation shows potential to optimize training. In order to meet the potential of the approach, additional research is needed to improve stress classification and the application of stress evaluations in the adaptive training domain. Two particular areas that require additional research include the application of deep understanding of the effects of stress on learning and memories and the development of personalized measures of stress and ruggedized sensor arrays that further improve stress classification accuracy.

Learning and memory development occurs in stages, including initial memory encoding, consolidation, retrieval, and reconsolidation. Research shows that the presence of stress affects each stage of the memory/learning process in different ways [28–36]. Although factors aside from the timing of stress during the learning, recall, and reconsolidation process have effects on the process, the general understanding of the effects of stress on each stage of the process are outlined in Fig. 4. Future research on the integration of stress within VE an AR training environments should apply these basic constructs of learning and memory to create micro-adaptive training that not only adapts training to present the correct cues and conditions (as objectively evaluated), but also present them at the correct time. The goal of this application should be to prime learners to retain and consolidate new information or break the retrieval or reconsolidation of negatively trained actions and memories.

Fig. 4.
figure 4

Effects of stress on learning and recall process

Full size image

A second avenue of research that is critical to improve the capabilities of VE and AR training systems to support adaptive stress training is the enhancement of classifiers of stress that are used to drive adaptations. Specifically, there is a need to develop approaches and systems to automate the development of personalized classifiers of stress and systems to evaluate the effects of stress training across studies and as learners move to a transfer environment. Finally, to meet this need, there is a need to develop new noninvasive hardware platforms that can measure the core features that can be used to classify stress while accounting for the noise that is associated with measuring stress in unpredictable environments. By continuing research in these domains, it will be possible to merge the power of adaptive training within AR and VR environments and objective stress measurement to better prepare people for high risk and high stress jobs and potentially reduce the impact of negative stress on people in those positions.