1 Introduction

At the Institute of Flight Systems, the concept of Manned-Unmanned Teaming (MUM-T) is a well-established approach to improve future military aviation. Guiding multiple reconnaissance Unmanned Aerial Vehicle (UAV) from the cockpit yields many advantages including information gain, flexibility and safety. But the additional tasks of operating the UAV and process the information of their sensors increase the workload of the cockpit crew. If the demand exceeds the available mental resources (i.e. workload peak), e.g. in time-critical multi-tasking situations, it results in performance decrements and human errors [1]. In the aviation domain, automation has always been a solution to reduce workload. But shifting more tasks from the human to (conventional) automation brings other problems: Loss of situation awareness, workload peaks due to high demanding cognitive tasks, which remain with the human, and “automation induced workload peaks” due to clumsy automation [2].

To counteract this, Onken and Schulte introduce two modes of cognitive automation [3, 4]. This automation is able to handle and support cognitive tasks, to avoid exceeding mental resources on the one hand and too little or wrong human involvement on the other hand. One mode is, to establish a cognitive agent (, see Fig. 1) onboard of each UAV which facilitates a task-based (instead of parameter-based) UAV guidance [5]. The agent is able to understand high-level commands, formulated as tasks, and reports the results back on the same task-based level. Several studies show, that these cognitive agents enable the crew of a manned helicopter to guide multiple UAV from the cockpit while performing complex mission scenarios [6].

Fig. 1.
figure 1

Work system of the manned-unmanned teaming configuration in the human-agent teaming notation from [4].

Full size image

As second mode, we introduce a cognitive agent (, see Fig. 1) as associate system to support the crew adapted to their mental state. The goal is, to prevent workload peaks and human errors while keeping the crew “in the loop” to prevent out-of-the-loop problems [7], complacency and automation bias [8]. Therefore, the associate system takes the role of an artificial, restrained-behaving co-pilot. Onken and Schulte describe a guideline for this restrained behavior as a set of rules [3]. The basic rule is, to let the crew do their job as long as possible without the associate system intervening. However,

  1. 1.

    if the attention of the human is not on the objectively most important task, guide the human’s attention to that task; 


  2. 2.

    if, nevertheless, the human is overtaxed, transform the task situation into one, the human can handle again. And only 


  3. 3.

    if the human is in principle not able to perform the task and the cost of a failure would be too high, adopt the task as a last resort.

These escalating behavior rules link the type and amount of assistance to the attention and mental state of the crew. Therefore, the adaptive associate system needs a context-rich representation of mental workload [9]. It is not sufficient to know if the workload is high, but in addition it is necessary to know the reason, i.e. the causal task situation, which leads to high workload. Only the knowledge about the task situation enables the agent to support the crew task-based, also for cognitive tasks.

2 Task-Based Operationalization of the Mental State as Precursor of Adaptive Assistance

As basis for the workload-adaptive associate system we introduce a context-rich definition of mental workload, as described in [9]. It includes

  • the currently pursued work objective and the resulting tasks, which are necessary to achieve this objective, i.e. plan;

  • the set of tasks, which the operator currently executes, i.e. activity;

  • the demand on mental resources, which is necessary to execute the activity and

  • behavior patterns, which the operator typically shows during the task execution and variations from these patterns [10].

2.1 Task Model

We operationalize this definition of mental workload with tasks. Therefore, we developed a hierarchical task model which contains the domain knowledge of our application, i.e. all tasks which can occur during a MUM-T helicopter mission [11]. This machine-readable task model enables the associate system

  • to have sufficient knowledge of the domain, similar to the knowledge of the human crew. This includes the demand on mental resources, which is necessary to execute a specific task, constraints for tasks and relations between tasks;

  • to communicate with the crew in a natural manner, since humans communicate very efficient by using tasks as expression for very complex situations;

  • to communicate in the same efficient way between different modules of the associate system;

  • to know different variants of task sharing between the crew and the automation. According to [1], working memory load can be reduced by automating tasks. That is, increasing the level of automation, means to decrease the involvement and therefore the taskload of the human operator. The Levels of Automation (LoA) in our task model define, how the associate system can reduce the crews taskload by automating a specific abstract task higher (like the mission planning task “PilotPlanMission” in Fig. 2).

    Fig. 2.
    figure 2

    Excerpt of the task model, which shows the abstract task “PilotPlanMission” and its three different Levels of Automation, marked with red stars. As higher the LoA is, as less the human involvement and therefore taskload of the crew. The image does not show the subtasks of lower levels of the task model.

    Full size image

2.2 Plan

Based on the tasks of the task model, the plan is a dynamically generated sequence of mission tasks (, see example mission plan in Fig. 3), which fulfills the mission objective (, see Fig. 3). In our application, the plan depends on the objective of the MUM-T helicopter mission and constraints like terrain, air spaces and others. To plan and schedule this sequence, we use a mixed-initiative approach, where the pilot is in charge to plan the mission for reasons of plan situation awareness, transparency and trust. The mixed-initiative mission planner (MIP) supports the planning process by intervening on own initiative in case of threat avoiding, missing tasks or optimization [12]. The MIP knows different levels of automation (the three levels depicted in Fig. 2), which the associate system can use to simplify planning related problems.

Fig. 3.
figure 3

Example of a transport helicopter mission. The order Transportation Mission can be fulfilled by the complex mission tasks on the bottom.

Full size image

2.3 Activity

Beside the plan, the activity is a key element in our definition of the crew’s mental state. Therefore, our activity determination observes the crew to identify the current activity on-line [13]. It uses different measurement sensors – gaze tracking and manual interaction tracking as well as speech recognition – and the knowledge about observable evidences from the task model. It combines all simultaneously observed evidences by using an evidential reasoning approach derived from Dempster-Shafer theory to assign each evidence to a task from the task model [13, 14]. The activity is the set of elementary tasks (, see Fig. 2), which the crew executes at a given moment. We assume, that if the percentage, which supports the execution of a specific task (blue belief value in Fig. 4) is greater than 0.5, the task is part of the activity. Therefore, in the example in Fig. 4, CommunicateIntern and FlyTransitFriendManual is the current activity of the pilot.

Fig. 4.
figure 4

Screenshot of an example result of the activity determination. Here, the pilot is flying a manual transit flight and communicating via the intercom at the same time (taken from [13]). The belief value of CheckRadio and CommunicateATC is too low to be part of the activity.

Full size image

2.4 Demand on Mental Resources

To be able to estimate the demand on mental resources, which is necessary to execute the entire activity, the task model stores the demand on mental resources for each single task as demand vector [15]. After identifying the activity, our resource assessment combines all related demand vectors by using the conflict matrix of Wickens’ Multi-Resource Theory to estimate an overall workload value [11, 15, 16] (Fig. 5). Due to the eight separately stored components of the demand vector for the different resources (visual-spatial, visual-verbal, auditory-spatial, auditory-verbal, cognitive-spatial, cognitive-verbal, response-manual and response-vocal), our method provides information, which resource of the pilot leads to a workload peak. And, as mentioned before, from the activity determination, the associate system knows which task situation leads to that workload peak.

Fig. 5.
figure 5

Screenshot of the resource assessment, which shows the mental resources usage on the three stages – perception, cognition and response – as well as the total mental workload estimation over the last 60 s for the activity shown in Fig. 4.

Full size image

3 Concept and Implementation of Adaptive Assistance

The context-rich representation of mental workload is the basis for adaptive interventions, which support the crew adapted to their mental state for specific task situations.

Figure 6 depicts the functional architecture of the associate system. The process of inferring adaptive interventions consists of three phases. Before the associate system is able to identify the trigger for assistance – neglected task, (predicted) workload peaks and critical events – it preprocesses the mental state (Phase ① in Fig. 6). Thereby, it uses the plan to project the mental state into the future, to

Fig. 6.
figure 6

Functional architecture of the associate system: “This is inside the little in Fig. 1.” The process of inferring adaptive interventions has three phases: Mental state preprocessing (1), intervention decision (2) and intervention execution (3).

Full size image

  • know which elementary tasks are necessary to fulfill the plan and

  • identify situations of high workload in the future, which may occur during the plan execution.

In the second step, the associate system identifies trigger for adaptive assistance (“critical states”) and plans the intervention (Phase ② in Fig. 6). To implement the adaptive assistance, the associate system uses the human-dialog interface, which is part of the human-machine interface. In addition, the associate system uses other (cognitive) automation (e.g. the mixed-initiative planner or the adaptive crew sensor interaction) to simplify task situations or adopt tasks via the automation dialog interface (Phase ③ in Fig. 6). For detailed information about the process of identifying trigger, the decision process and stages of intervention see [17].

3.1 Phase 1: Identifying Trigger for Adaptive Assistance

For predicting future mental states and identifying resource conflicting task situations as well as neglected tasks, the associate system refines the plan – which is part of the “input” mental workload (see grey box “Mental Workload” in Fig. 6) – to a detailed plan on elementary task level (which is the lowest level of the hierarchical task model). It uses the task relationships from the task model to identify all elementary tasks, which belong to a specific mission task. Figure 7 shows the mission task EnterHOA, which is the task of entering the helicopter mission area, and its subtasks after the refinement process.

Fig. 7.
figure 7

Excerpt of the task model displaying the hierarchical sub-task relationships of the mission task EnterHOA. The green boxes indicate completed, the red not completed tasks. If the associate system adopts a task, the box becomes magenta. (Color figure online)

Full size image

Because the activity determination expresses the activity on the same level of elementary tasks, the associate system is able to match the activity with the detailed plan and check the completed tasks (green task-boxes in Fig. 7). All tasks which are planned, but not executed timely are neglected task and trigger for adaptive assistance (red task-boxes in Fig. 7).

In a second step, the associate system schedules all elementary tasks. Therefore, it uses the execution time and task constraints from the task model. This results in a task timeline, as Fig. 8 shows. The associate system simulates the execution of all future task situations and estimate the needed demand on mental resources (red graph curve in Fig. 8). A situation in which the crew has to perform many tasks in parallel and therefore their demand on mental resources exceeds a threshold is a workload peak (red marked period in Fig. 8) and trigger for adaptive assistance.

Fig. 8.
figure 8

Timeline of the mission task EnterHOA and its subtasks (the first 11 grey task-boxes) and the mission task PlanMission and its subtasks. Due to many parallel tasks, the associate system predicts a workload peak (red marked time period). (Color figure online)

Full size image

The third trigger for adaptive assistance are critical events, like changes of the tactical situation, failures of the helicopter systems or threatened UAV. The associate system cannot predict such events. However, if they occur, it supports the crew by guiding the attention, offering a possible solution, or directly implementing a solution. The following section explains which stage of intervention (which is the “amount of help”) is appropriate.

3.2 Phase 2: Deriving Adaptive Interventions

The process of deriving adaptive interventions aims to find the appropriate level of assistance for a given problem by taking the workload, attention and criticality of the problem into account. Basis for this is the set of rules, mentioned in the introduction. To implement the restrained behavior, the associate system follows the decision process in Fig. 9: First, it traces back each trigger (grey boxes on the left side) to its causal task(s). After comparing this task with the current activity, the associate system infers if the crew works on the solution (Question A in Fig. 9). If the crew is currently not solving the problem, the associate system simulates, whether the crew can handle the current task situation including all other tasks, which are necessary to solve the problem (Question B in Fig. 9). Therefore, it combines the demand vectors of the hypothetical activity using the same method as for the resource assessment, described in Sect. 2.4. If the crew can handle the entire task situation, the associate system guides the attention to the problem (Stage 1 on the right side in Fig. 9), but if not, more assistance is necessary. If the crew is already overtaxed, or any additional task would overtax them, the associate system simplifies the task situation by adapting the human-machine interface (Stage 2a in Fig. 9) or changing the level of automation (Stage 2b in Fig. 9). Only if the problem poses an extreme or high risk (Question C in Fig. 9) and the crew cannot handle it, the associate system is allowed to adopt the problem-solving task(s) (Stage 3 in Fig. 9). We implemented this decision process with the cognitive framework Soar [18]. For detailed information about the decision process and its implementation see [17].

Fig. 9.
figure 9

Decision process of determining the appropriate stage of intervention (bold boxes on the right) based on the related trigger (grey boxes on the left) and the mental state of the crew.

Full size image

3.3 Phase 3: Stages of Adaptive Interventions

After identifying the appropriate stage of intervention, the associate system implements the different stages as follows. For guiding the attention to a problem, it uses the human-dialog interface. Therefore, it overlays a dialogs box on the multi-function displays (MFD) of the cockpit (see left image in Fig. 10) and highlights all related objects on the MFD. In addition, it can simplify the task situation by adding a “short-cut button” to the dialog box, which implements the proposed solution with one button click (second image from left in Fig. 10). This “short-cut button” is also available via the helicopter control stick. Another possibility to simplify the task situation is to change the level of automation, e.g. to increase the level of automation of the mixed-initiative planner to simplify a planning related problem. For critical situations, the associate system can adopt task(s) like a forgotten landing check (right lower image in Fig. 10), if the human is not capable to handle it. The MFD displays the information on a “history list of adopted tasks” (right upper image in Fig. 10). In addition, a speech synthesizer announces the adopted task (e.g. Set System Configuration Landing) as a human team mate would do in this situation.

Fig. 10.
figure 10

Stages of intervention, escalating from left to right: Attention Guiding (1), Task Simplification (2a and 2b) and Task Adoption (3)

Full size image

4 Evaluation

To evaluate the associate system, we conducted a human-in-the-loop experimental campaign with trained military helicopter pilots in our helicopter mission simulator at the Institute of Flight Systems (see Fig. 11). The purpose of the experiment was, to investigate the interventions of the adaptive associate system during realistic and very complex MUM-T mission scenarios. Therefore, in Sect. 4.2 we analyze two (of many occurred) situations of “near misses”, where the associate system corrected human erroneous behavior during the missions. In addition, in Sect. 4.3 we present and analyze the overall system rating given by the pilots.

Fig. 11.
figure 11

Helicopter mission simulator at the Institute of Flight Systems, which we use to implement and evaluate the associate system for MUM-T missions.

Full size image

4.1 Experimental Design

Our aim at the Institute of Flight Systems is, to evaluate new concepts by testing them as highly integrated systems in very realistic and immersive scenarios. Therefore, the results typically have a very good external validity. But the high complexity of such systems requires domain experts as test subjects (i.e. experienced military helicopter pilots) and moreover an extensive training. We trained our pilots for about two days before conducting experimental trials for another three days. The study population comprises an overall number of seven participants (Age: M 50.4 SD 9.2, Flight Hours: M 3933 SD 1807) grouped into four crews. Every crew consists of a pilot in command (PIC), who leads the mission and guides three UAV for reconnaissance purposes, and a pilot flying (PF), who is responsible to fly the helicopter, communicate and manage the systems of the helicopter. One participant was part of two crews (one time as PIC and another time as PF). Since the associate system supports both crewmember in very different ways, this participant could rate the associate system two times.

The crews performed six different transport helicopter missions (Mission duration in minutes: M 46 SD 10), with mission elements like troops transport, Medical Evacuation (MedEvac) and Combat Search and Rescue (CSAR). The missions are very complex, since all contain many events like suddenly occurring enemies and mission goal changes. These compacted, challenging missions provoke human errors. The intention behind that: In normal workload conditions, highly trained pilots perform good and human errors are very rare. However, to evaluate the associate system, the deliberate provocation of errors is necessary. Therefore, our missions are designed to represent the most stressful parts of helicopter missions. The pilots confirmed that the missions are very stressful, but nevertheless they rated the scenarios as realistic (Scenario is realistic: M 5.6 SD 0.7 on a 7-point Likert scale).

We did not vary and compare configurations of the system, but analyze, when, why and how the associate system intervenes to help the crew and how the crew rates these interventions.

4.2 Adaptive Interventions

One intervention affects the pilot in command (PIC), who is responsible for planning the mission and operating the UAV to ensure a reconnoitered flight route for the helicopter. In this situation, the PIC forgets to assign the route reconnaissance task for the next flight leg of the helicopter route to a UAV (highlighted in magenta in Figs. 12 and 13). In addition, he is involved in an ongoing route reconnaissance task of another UAV (see green frame in Fig. 12, which shows the tasks of the detailed plan for the next minutes). The two parallel tasks PlanMission and ReconRoute, starting at the red line in Fig. 12, lead to a predicted workload peak for the immediate future (red marked area, “now” is at the left border of the red marked area). To relieve the task situation for the PIC, the associate system instructs the mixed-initiative mission planner to increase the level of automation (remember the planning automation levels in Fig. 2) and to propose the next relevant tasks on its own initiative. The planner proposes the forgotten route reconnaissance task (see dialog box in Fig. 13). Due to the urgency and criticality of this task, the associate system automatically accepts the planner proposal. The pilot rated this workload-adaptive intervention as very helpful and appropriate for this risky and time-critical situation.

Fig. 12.
figure 12

Prediction of the task situation in the near future (blue blocks are the tasks) and estimation of the future workload (orange solid line) with predicted workload peak (red marked area) and missing UAV task (PlanMission task in magenta frame). (Color figure online)

Full size image
Fig. 13.
figure 13

Screenshot of the tactical map display with the planner proposal dialog box and the missing UAV reconnaissance tasks (highlighted in magenta) which are directly in front of the helicopter symbol, and therefore very urgent. (Color figure online)

Full size image

One situation in the field of system management affects the pilot flying. Because of a mode confusion during the After-Takeoff checklist, the pilot switches on the landing lights erroneously. Within a possibly threatened area, this is an avoidable safety risk. Therefore, the associate system guides the attention to this wrong configured system state (left blue dashed line in Fig. 14). Because the activity determination enables the agent to recognize, that the pilot flying is doing a low-level flight and has therefore less free resources for doing an additional manual task (see estimated overall workload, represented by the green solid graph in Fig. 14), it decides to simplify the task situation by offering the “short-cut” button. Thereby, the pilot can accept the help with a button on his cyclic stick. The pilot accepts the help via this button (brown dashed line, see Fig. 14) and the associate system switches the lights off (right blue dashed line). As soon as the associate system solves the problem (green dashed line), it notifies the pilot by announcing the related task from the task model (i.e. SetLandingLightsOff). The speech announcement is also visible as task block “ListenAssistanceSpeech” in Fig. 14. The pilot flying rated this intervention as very supporting and helpful. In addition, he stated that the explaining text of the intervention (see Fig. 15) makes it easy to understand the problem and how this problem could arise.

Fig. 14.
figure 14

Task situation (blue blocks) of the pilot flying and overall workload estimation (green solid line reaching from “no workload” at the lower border of the image to “overload” at the upper border of the image) during the intervention (blue dashed line) regarding the landing lights. (Color figure online)

Full size image
Fig. 15.
figure 15

Screenshot of the system management page of the helicopter simulator, with the green gaze point of the PF (including the local gaze measure accuracy as Gaussian distribution), who is reading the explaining text in the dialog box of the intervention. (Color figure online)

Full size image

These situations illustrate, that the associate system adaptively supports the crew by taking the (projected) mental state and the criticality into account, and that the pilots feel appropriately supported. Beside the investigation of situations of near misses, the pilots assess the overall performance and behavior of the associate system.

4.3 Results

Most of the pilots rate the supporting interventions of the associate system as expedient and helpful (see Fig. 16). The pilots state, that the interventions are justified and the system reacts correctly in dangerous situations. One strength of the associate system is, that it keeps the overview of the whole situation, if the human is focusing on one task and guide the attention to the most urgent task, e.g. new threats, if necessary. Another major benefit is that the interventions save time by simplifying the task situation, which is very valuable during complex missions. These statements, regarding the attention guiding and task simplification, support the rating, that the stage of intervention is mostly situation-adapted and appropriate (see Fig. 16).

Fig. 16.
figure 16

Overall rating of the interventions on a 7-point Likert scale (n = 8). The dots are the ratings of the pilots.

Full size image

Although the system knows the current activity, there are rare cases, where one of the pilots and the associate system start working on the same task simultaneously. This is confusing and the pilot needs to spend time to understand this unnecessary intervention. Therefore, the associate system tries to prevent such cases, but due the delay between action planning and action implementation – which both, human and associate system, have – it was not possible to preclude these cases completely. A better situation awareness of the upcoming tasks and the human-agent task sharing, i.e. by pre-announcing of the task sharing, could solve this issue. [19] proposes a possible solution for this kind of pre-announcement.

Most of the pilots appraise the behavior of the associate system as transparent and comprehensible (see Fig. 17). Due to the fact that the associate system intervenes workload-adaptive and there is no pre-announcement of interventions, the predictability is lower. The pilots think, that more training may increase the predictability, but workload-adaptive systems are low predictable at all. As mentioned above, pre-announcing the next tasks and possible interventions increases the situation awareness of the crew [19].

Fig. 17.
figure 17

Overall rating of the behavior of the associate system on a 7-point Likert scale (n = 8). The dots are the ratings of the pilots.

Full size image

The pilots state, that the associate system is neither hasty nor restrained, but in this case the neutral position is a positive rating, because they state, that they interpret “restrained” as negative like “too restrained”.

5 Conclusion

We presented a concept of a workload-adaptive associate system, which supports the crew of a manned helicopter during military MUM-T helicopter missions. The process of inferring workload-adaptive interventions relies on a context-rich operationalization of mental workload based on tasks. With this definition of mental workload, the associate system is able to project future task situations to identify all pilot tasks which are necessary to reach the mission goal. In addition, it predicts situations of high workload in the future and eases them proactively by using higher levels of automation. After identifying trigger for supporting interventions, the associate system supports the crew by guiding the attention, offering a possible solution or directly implementing a solution. Thereby, it behaves restrainedly and helps only if necessary and as less as possible to keep the human in the loop and to prevent typical pitfalls of highly automated systems, like out-of-the-loop problems, complacency and automation bias.

Our pilot-in-the-loop experiments show, that the concept of supporting cognitive tasks workload-adaptively recovers human errors like neglected tasks and relieves time critical task situations before they occur. The pilots rate the interventions of the associate system as helpful and expedient. However, future improvements of the system should address the transparency e.g. by pre-announcing the human-agent task sharing.

We implemented this concept in the domain of military aviation. But the concept is not limited to this domain. It is transferable to other domains, where a human operator collaborates with highly automated systems, e.g. civil aviation, highly automated driving or power plant management.