Keywords

1 Introduction

Interaction is a dynamic adaptation of behavior and actions between actors. It requires effective communication, cooperation, and coordination to adequately interpret the other’s goals, expectations, and reactions. Humans possess such abilities and naturally apply these in social interaction, machines do not. But in its attempt to revolutionize the interaction between humans and machines, the field of Augmented Cognition (AugCog) has long recognized the need for technology to adapt to the state of the operator.

Most AugCog systems use physiological measures to detect critical cognitive states and trigger adaptation strategies to address the problem state and restore or augment performance. However, mere detection of a critical state may indicate a need for adaptation – for example to address excessive workload – but reveals little about the type of adaptation that would be appropriate with respect to the given situation [1]. Also, adaptation strategies in demonstrated AugCog systems are often hard-coded, i.e., whenever a critical operator state X is detected, the machine triggers a predetermined strategy AX to address the problem. But without accounting for context, it is likely that adaptations are triggered or withdrawn at inopportune moments, potentially disrupting or confusing the user. The associated cognitive costs may include task switching issues, situation awareness problems, and workload increases. In these cases, adaptations may even have a negative impact on performance, outweighing the benefit of adaptation altogether. This inconsiderate application of adaptation strategies has been labeled ‘brute force mitigation’ [2]. As a first step to addressing this problem, we have identified various aspects of where and how context sensitivity is essential for effective adaptation in AugCog systems:

  1. 1.

    When assessing operator state, it is important to realize that there are various states that have been found to be interdependent [3]. Adaptation based on a single problem state may lead to oversimplification by addressing the symptom rather than the cause. For example, attention may be modulated by workload, fatigue, and even the emotional state, but depending on the modulating state, attention may be impacted in very different ways. For an adaptation strategy to be effective, context is necessary to understand the relationship between the detected problem state and related states.

  2. 2.

    Cognitive states are hypothetical constructs (cf. [4]) that cannot be manipulated directly. To be able to impact cognitive state X, adaptation strategy AX requires an appropriate environmental variable for manipulation. For example, a plausible way to mitigate high workload could be to reduce the number of simultaneous tasks. However, this strategy will only work if the number of tasks is sufficiently high. If task load is low and workload is up for other reasons (e.g. lack of experience), this strategy will not be effective. Thus, context information is essential for determining the adequacy of an adaptation strategy.

  3. 3.

    The adaptation should be capable of adapting itself to the situation and the user [5]. As situation and context evolve, a once adequate adaptation strategy may become inadequate. For example, brute force adaptation may automate a task the user has already started to perform or attempt to restore global situation awareness by shifting the user’s focus away from a new high-priority task.

2 Advanced Dynamic Adaptation Management (ADAM)

To overcome challenges with brute force adaptation, we have developed an “Advanced Dynamic Adaptation Management” component (ADAM). Instead of triggering a static adaptation AX to respond to a critical cognitive state X, the goal of ADAM is to select, configure, and trigger adequate adaptation when and where needed.

Dynamic adaptation management requires extensive amounts of task and user context. Therefore, we also developed a diagnostic engine – our Realtime ASsessment of Multidimensional User State (RASMUS) – that implements a multidimensional approach to user state assessment. RASMUS diagnoses up to six cognitive and emotional problem states, of which three cognitive states – workload, attention, and fatigue – were recently validated. RASMUS evaluates a multitude of individual and task-related indicators known to impact these cognitive states, as well as physiological and behavioral reactions. Based on this information, it returns user states assumed to be critical and reports which indicators have likely contributed to this state on a second-by-second basis. RASMUS diagnostics enable technology to detect not only when the human operator’s performance declines and what cognitive states are insufficient, but also indicates contextual factors (system state, task state, user state as indicated by physiological and behavioral metrics) that may have contributed to the situation. More detail on RASMUS is provided in another article within this volume [6].

ADAM assumes that declines in performance are symptoms caused by underlying cognitive problem states. For example, task omission may be caused by excessive workload, a lack of attention, or even motivational issues. Accordingly, ADAM assumes that task performance can be restored by addressing that cognitive problem state through adaptation. According to Breton and Bossé [7], humans should receive support when their “cognitive capabilities are not sufficient to adequately perform the task” (pp. 1–4). Thus, ADAM assumes a need for adaptation when a performance decrement occurs in conjunction with at least one critical cognitive state. The condition of a performance decrement was included to allow operators to self-adapt to the problem state, considering that “having an adaptive system working together with an adaptive operator will likely be unsuccessful. An adaptive system is more likely to work successfully when it starts reallocating tasks as soon as the operator is no longer able to adapt properly to changing task demands” ([8], p. 10). Also, intervening too early may favor complacency (“based on an unjustified assumption of satisfactory system state,” [9], p. 23) and hinder development of resilience and coping strategies.

For adaptation to be triggered, ADAM also requires that the pool of adaptation strategies includes a strategy that is (a) capable of addressing the cognitive problem state and (b) suitable in the given context. For example, it is only possible to automate tasks if enough tasks are present). ADAM then draws from a pool of candidate strategies and configures them based on RASMUS diagnostic output. This dynamic approach is expected to avoid much of the potential cognitive cost associated with brute force adaptations. Once a need for adaptation is detected, dynamic adaptation management involves five steps (Fig. 1) that are explained in detail below.

Fig. 1.
figure 1

Five steps of dynamic adaptation management

Full size image

Step 1: Determine an adaptation objective.

Adaptation objectives are rather abstract descriptions of cognitive manipulations that describe how a certain cognitive problem state could be manipulated in a way that it is no longer critical. ADAM will select an adequate adaptation objective based on the diagnosed problem state. For example, when a performance problem occurred along with a state of high workload, ADAM’s objective may be to decrease task load to address the problem. In contrast, if the performance problem coincided with fatigue, a further decrease in task load would be counterproductive and ADAM may instead attempt to increase operator activation and arousal. There may be different strategies available for achieving each objective.

Step 2: Select adaptation strategy.

From a pool of available adaptations, ADAM selects a strategy that is designed to address the adaptation objective and that is suitable under current conditions.

Each strategy is associated with at least one adaptation objective. All adaptations able to achieve the adaptation objective determined in Step 1 become candidate strategies. Known prerequisites for all candidate strategies are then evaluated to determine the best strategy under current circumstances. Given that adaptation cannot directly affect cognitive states, one prerequisite is the availability of a manipulable variable that is related to the cognitive state to be addressed. Moreover, it is possible that other conditions must be met in order for a strategy to work. If prerequisites are not fulfilled, the strategy is deemed unsuitable in the given context and further strategies from the pool will be evaluated for their suitability.

As an example for strategy-specific prerequisites, consider that RASMUS diagnostics report a decline in performance in conjunction with critical workload. RASMUS also indicates that a high number of simultaneous tasks contributed to the critical workload condition. A rather intuitive adaptation objective in this case is to decrease workload by reducing the number of simultaneous tasks. Available candidate adaptation strategies to address this objective are automation and task sequencing. Both require multiple tasks to be simultaneously active (which is the case here). Adaptive automation would take over certain tasks until performance is restored but would only be useful if a certain minimum number of tasks must be processed at that time and at least one of these can be automated. In contrast, task sequencing would schedule the tasks to distribute them more evenly over time, but is only appropriate if certain tasks have a lower priority than others.

Step 3: Configure adaptation strategy to suit task and cognitive state.

An adaptation strategy may contain contextual parameters which can be used to tailor it further to the state of the task and the user. For example, a task sequencing strategy could be based on task priority or urgency. A cueing strategy aimed at refocusing the operator’s attention could consider his current focus to determine cue location or select the best modality for cue presentation (e.g. visual if eyes are on-screen, auditive if eyes are off-screen). Task urgency could also be used to determine cue salience.

Step 4: Trigger adaptation strategy.

Once dynamically selected and configured, the adaptation strategy is activated in the information display, altering human-machine interaction in a way that impacts the cognitive problem state and serves the adaptation objective.

Step 5: Monitor the impact of the adaptation.

Monitoring the effects of adaptation with respect to task performance and cognitive state changes will determine whether and how adaptation should be continued. If the adaptation objective was accomplished and the underlying problem states are no longer present, it is important to withdraw adaptation, as inappropriate continuation could have negative effects on the operator and task performance. For example, an inadequate adaptation may occupy cognitive resources, interrupt a high priority task, or refocus the operator’s attention away from it. There are four possible monitoring results with different implications for adaptation management (Table 1).

Table 1. Implications of monitoring the impact of an adaptation
Full size table

3 Proof-of-Concept Implementation

3.1 Task Environment

For an initial demonstration of ADAM’s capabilities, an adaptation manager was implemented for a naval anti-air-warfare simulation (Fig. 2). The simulation includes four tasks that were simplified for learnability but designed to maintain the essential cognitive demands of the real-world task.

Fig. 2.
figure 2

User interface of the experimental task (Color figure online)

Full size image

During the simulation the operator is to perform all occurring tasks with a focus on keeping the Identification Safety Range (ISR) around the ownship clear of threats. Tasks with different priorities (detailed in Table 2) occur at scripted times throughout the scenario. In the case of multiple simultaneous tasks, users were instructed to perform higher priority tasks first. RASMUS diagnostics report an instance of critical performance whenever a task is not completed within the time limits indicated in Table 2. In addition to performance, three cognitive states are monitored in the proof-of-concept system:

Table 2. Task descriptions and properties (numbers in parentheses refer to display areas marked in Fig. 2)
Full size table
  1. 1.

    (high) workload

  2. 2.

    (incorrect) attentional focus

  3. 3.

    passive task-related (TR) fatigue (cf. [10])

These cognitive states, further detailed in [6], were chosen because of their relevance to the task and because RASMUS diagnostics were successfully validated for these states in a recent experiment [11]. Along with every critical cognitive state, RASMUS reports the status of contextual indicators that contributed to the state diagnosis. Table 3 shows the contextual indicators considered for cognitive state detection.

Table 3. Contextual indicators associated with cognitive problem states
Full size table

Using this diagnostic setup, the dynamic adaptation management must distinguish between the five critical state combinations depicted in Fig. 3 to adapt human-machine interaction in a context-adequate manner. High workload and passive TR fatigue are mutually exclusive; however, attentional problems can certainly occur in conjunction with the two other states.

Fig. 3.
figure 3

Five critical state combinations to be considered by dynamic adaptation management

Full size image

3.2 Adaptations

In our proof-of-concept system, the adaptation objective to be pursued is determined based on cognitive problem states reported at the time of critical performance. To demonstrate dynamic selection of an adaptation based on diagnosed cognitive state problems, an adequate adaptation objective was formulated for each cognitive problem state (Fig. 4).

Fig. 4.
figure 4

Context-sensitive selection and configuration of adaptation strategies in the prototype system

Full size image

  • The adaptation objective for high workload is reduction of task load. This can be achieved, for example, by temporarily reducing the number of tasks to be processed.

  • The adaptation objective for incorrect attentional focus is to redirect the user’s attention to the highest priority task.

  • The adaptation objective for passive TR fatigue is activation of the operator to combat his passivity. The activation of the user can be accomplished by increasing arousal.

When performance decrements occur, the system dynamically selects from two implemented strategies – automation and cueing. Depending on the adaptation objective, the appropriate strategy is invoked dynamically. To also demonstrate context-sensitive configuration, task context and operator metrics are used to configure the cueing strategy in three variants: visual cue, auditory cue, and the combination of visual and auditory cue.

Automation.

Automation changes the extent of human involvement in the task by taking over tasks previously performed by a human operator. Automation is particularly helpful when operator workload is high. Under low workload conditions, however, it may have undesirable effects such as boredom and fatigue [12]. Therefore, automation can be used to address the adaptation objective for the high workload (i.e. reduce task load) but is unsuitable strategy to address the other adaptation objectives.

For our proof-of-concept system we chose to implement simple dual mode automation: Under normal circumstances, the operator is responsible for all tasks. Only when high workload is diagnosed along with critical performance, all low priority tasks (identification of contacts outside the ISR) are automated. This frees up cognitive resources that can be used to perform higher-priority tasks. It has also been recommended that safety-critical tasks remain with the human to maintain situation awareness [13]. To prevent automation errors and avoid complacency effects (e.g., [14]), all automated identifications are marked with a badge (Fig. 5) and must be manually verified by the user for the badge to disappear. In high workload situations, the user can thus focus on the most important tasks; when less taxed, he can verify the automatically assigned identities, distributing task load more evenly. Future iterations of the automation strategy could be enabled to dynamically select the task to be automated and choose between different levels of automation.

Fig. 5.
figure 5

Automated identification; contacts with automatically assigned identities are marked with red badges (Color figure online)

Full size image

Cueing.

This strategy manipulates the user’s attention by enhancing the information presentation. Often, attention is shifted by increasing the saliency of relevant information objects. Cues are not limited to the visual modality, but can also be implemented as auditive, tactile, or multimodal cues, as well as in different levels of intrusiveness to enable context-sensitive configuration. However, cueing should also be used with caution, as cued attention may be narrowed to such an extent that other high-priority information may not be perceived [15]. Also, task interruption, as initiated by refocused attention, has unfavorable effects on stress and performance, in particular with regard to complex or cognitively demanding tasks [16, 17].

To demonstrate context-sensitive configuration of an adaptation strategy two cueing variants – visual and auditive cueing – were implemented and are activated depending on the adaptation objective.

Passive TR fatigue, often caused by prolonged monotonous tasks, can lead to performance problems if the operator drifts off or becomes distracted. In this case, the adaptation objective is to address the passivity by activating the user. A visual cue may not be effective here as it may not even be perceived. Thus, if a performance problem occurs along with passive TR fatigue, an auditory cue in the form of an intrusive alarm tone is triggered to indicate the presence of an unnoticed task and increase the operator’s arousal.

In contrast, the occurrence of a performance problem along with an incorrect attentional focus indicates that the operator is attentive, albeit not focused on the most important task. Thus, the adaptation objective is to refocus the operator’s attention to the highest-priority task. In this case, use of a visual cue is appropriate because it can be assumed that the user’s attention is directed at the task environment and all tasks in the simulation concern visual information objects. The visual cue indicates the presence and location of a task more important than the one currently attended to. It consists of an arrow that is attached to the mouse pointer and always points in the direction of the information object with the highest priority task (Fig. 6).

Fig. 6.
figure 6

Cueing (visual configuration); the arrow points to the highest-priority contact (4071) (Color figure online)

Full size image

3.3 Monitoring of Adaptation Effects

After an adaptation was triggered, performance and cognitive state criteria are continuously monitored to examine whether a strategy was successful in achieving the associated adaptation objective. In the proof-of-concept implementation, monitoring results determine the withdrawal of adaptation strategies.

In the case of automation, new contacts are no longer automatically identified when the tasks that caused the performance problem no longer exist or when workload is no longer critically high. However, the identity of automatically identified tracks and their badges are preserved in order to avoid inconsistencies in the information presentation. Reversing automatically assigned identities to “unknown” would cause a sudden increase in task load which could lead to undesired oscillation effects in the adaptation management [18].

The visual cue has achieved its objective of shifting the operator’s attention when the contact with the highest priority task is selected. It is then deactivated. The objective of the auditory cue (i.e. activate the operator and alert him to the presence of a task) is achieved when the user resumes task processing. Consequently, the audible alarm is turned off as soon as the user selects a relevant information object.

4 User Feedback

To obtain initial feedback and user insights regarding the effectiveness and adequacy of the dynamic adaptation, we asked three subjects (m36, m34, f31) that had already participated in our validation experiment to perform the task and then complete a survey.

Participants were not informed about the functionality of the adaptive features. It was only mentioned that under certain circumstances the system will automatically identify contacts and participants were asked to manually confirm automatic identifications as soon as workload permits. After performing the task, participants completed a questionnaire to assess their subjective impression of effectiveness and adequacy of adaptations. Further questions asked for potential improvements and other qualitative feedback. Due to the small number of participants, the results were only evaluated qualitatively.

4.1 Occurrence and Perception of Adaptations

Only two subjects triggered all implemented adaptations. One subject did not experience an auditory cue because no performance problems occurred along with diagnosed fatigue. Although subjects were not informed about the adaptive features, they recognized all adaptations. However, two subjects overestimated, and one subject underestimated how often adaptations were triggered (Table 4).

Table 4. Number of adaptations triggered (subject estimates in parenthesis)
Full size table

4.2 Perceived Effects on Performance

Subjects perceived the effects of adaptation differently: One subject felt that adaptations generally had a positive effect on performance and the presence of the adaptations for the overall task was helpful. One subject perceived some aspects enhanced and others impaired performance. One respondent indicated that the adaptations in his opinion had no effect on performance.

Feedback can be further differentiated when considering the individual strategies (Table 5). None of the participants found the automation strategy disturbing, two people rated it helpful. The visual cue was perceived as very helpful by two subjects, while one perceived it as “somewhat disturbing”. One person perceived the alarm sound as “somewhat disturbing” but pointed out that this impression was linked to the specific signal tone (similar to an alarm clock buzzer). Also, in this case the auditory cue was triggered when an unidentified contact was overlooked in a noncritical location. The respondent criticized the high intrusiveness despite the low priority of the task, and suggested that tone and volume should be tailored to the priority of the task.

Table 5. Ratings of perceived effects on performance
Full size table

4.3 Appropriateness of Adaptation Timing

Due to the cognitive costs associated with inadequate adaptation, an important aspect of adaptation management is adequate timing. Adaptations are only useful in situations when the benefits outweigh these costs. The participants’ overall impression was that the timing was appropriate (Table 6). With regard to individual strategies, the vast majority of ratings indicated appropriate timing. The “very inappropriate” rating for the auditory cue followed a programming bug that made it impossible to engage a hostile contact. Although the subject had performed the task correctly, the system diagnosed a performance problem and triggered the auditory cue. The same subject rated the timing of the automatic identification “somewhat inappropriate”, commenting that he was not aware of the adaptation and was surprised when he noticed the changes. Considering that automation is only invoked under high workload conditions, it is likely that workload-induced attentional tunneling [19] inhibited the perception of automated changes in peripheral contacts.

Table 6. Ratings of perceived appropriateness of adaptation timing
Full size table

4.4 Appropriateness of Adaptation Withdrawal

Another question addressed the withdrawal of the adaptation. Since the automated identification of contacts was not reversed for consistency reasons, withdrawal could not be assessed for this strategy. For the remaining strategies, two participants found the duration of adaptation to be appropriate (Table 7). One person indicated a positive overall impression but rated the withdrawal of the cues as somewhat inappropriate. In the case of the visual cue, the subject felt that the arrow disappeared too quickly.

Table 7. Ratings of perceived appropriateness of adaptation withdrawal
Full size table

5 Conclusions

With our prototype implementation of context-sensitive adaptation, we were able to demonstrate the concept and feasibility of near real-time selection and configuration of adaptation strategies and have thus made important steps towards a truly dynamic adaptation management. Our proof-of-concept implementation was limited in that only one adaptation objective was assigned to each critical cognitive state, and only one adaptation strategy was implemented for each adaptation objective, but conceptually ADAM supports multiple adaptation objectives per problem state and multiple strategies per adaptation objective. Future work will expand the prototype system to include more adaptation objectives and strategies, consider additional diagnostic results in the selection of adaptations, and enable truly dynamic configuration of strategies in real-time. Once these enhancements are implemented, we plan to conduct empirical evaluations to validate the effectiveness of dynamic adaptation management and quantify the effects on the performance of operators.

Dynamically invoked adaptations were generally triggered as expected and predominantly well received by survey participants. Some characteristics perceived as disruptive must be attributed to a technical problem; others can be addressed by slight changes to the strategies or trigger rules. Despite the small number of survey participants, initial lessons learned can be drawn from the results and the qualitative feedback we received: Although participants mostly felt that adaptations were triggered at appropriate times, two subjects noted that their occurrence was sometimes perceived as random. It was not intuitively clear to them why and under what circumstances certain adaptations appeared. This demonstrates a need for providing users of adaptive systems with sufficient understanding of the system’s behavior. Another lesson learned is that perception and appraisal of adaptations is very individual: an adaptation praised as particularly helpful by one subject may be perceived as disturbing by another. If the system was provided with such personal preferences, this information could be used just like other contextual information to further individualize adaptation.

The combination of multidimensional user state diagnostics and dynamic adaptation management offers potential far beyond what was described in this paper. Both components were developed as generalizable concepts that we seek to apply to different application areas. We expect particular benefits for the operation of highly automated task environments and safety-critical systems where performance decrements and critical cognitive states may have serious consequences. The approach could also be used for adaptive training applications to individualize training on the fly based on real-time context information about the task, trainee state, and performance.