1 Introduction

The Canadian Armed Forces (CAF) initiated the Joint Unmanned Surveillance and Target Acquisition System (JUSTAS) program to procure and field a mature Medium Altitude Lone Endurance (MALE) Uninhabited Aircraft System (UAS) to provide mandatory capabilities for domestic and international operations [1]. The JUSTAS UAS will complement existing reconnaissance, surveillance, and target acquisition capabilities as well as increase Canada’s maritime and artic domain awareness [1]. The UAS is also required to have precision strike capability to support Land and Special Operations forces [1].

The Royal Canadian Air Force, working on JUSTAS, needs an interim ground control station (GCS) to train Canada’s legacy Heron crews and new UAS crews to prepare for the JUSTAS UAS GCS. Defence Research and Development Canada (DRDC) has been tasked with assessing the required training needs and operator interface technologies in order to make recommendations for UAS GCS design and training. DRDC is also developing recommendations for Directorate – Technical Airworthiness and Engineering Support (DTAES) on the airworthiness certification process for the UAS GCS. DTAES currently does not have any airworthiness standards for UAS GCSs. DRDC is developing an experimental UAS GCS to complete these tasks.

UAS crews can face high-workload conditions by having multiple sensor views, large amounts of data for analysis, and the teleoperation nature of UAS missions. UAS crews have indicated that new GCS HMIs would allow for substantial improvements to system effectiveness and performance, including better UAS control, more efficient data management and the dissemination of that information [2]. DRDC is performing trials on the experimental JUSTAS UAS GCS to determine what components/parts of the GCS can be effectively improved by new HMIs.

This paper presents the operator performance and participant feedback results of a human factors engineering (HFE) trial on an experimental UAS GCS. Additionally, a new HMI concept designed is presented to support the UAS crew based on the results of the HFE trial. The rest of this paper is organized as follows. Section 2 describes DRDC’s UAS GCS simulator. Section 3 summarizes the HFE trail. Section 4 presents a novel intelligent HMI based on the results of the HFE trial. Concluding remarks are presented in Sect. 5.

2 Test-Bed for Integrated Ground Control Station Experimentation and Rehearsal - TIGER

DRDC and the Warfighter Readiness Research Division of the US Air Force Research Laboratory (AFRL) (711HPW/RHA) are expanding AFRL’s PRINCE UAS GCS simulator for use as an experimental UAS GCS [3]. This new GCS simulator, called Test-bed for Integrated GCS Experimentation and Rehearsal (TIGER), will be used by DRDC for the development of new UAS GCS capabilities, (i.e., HMIs, perform learning and training studies for different crew complements, and investigate manning requirements, GCS certification requirements, and new GCS workstation layouts). The current TIGER layout is shown in Fig. 1.

Fig. 1.
figure 1

TIGER workstation layout

Full size image

TIGER is comprised of six crew workstations for UAS operators. These workstations include the Air Vehicle Operator (AVO), Payload Operator (PO), Image Analyst (IMA-A), Image Reporter (IMA-R), Electronic Warfare Analyst (EW-A), and Electronic Warfare Reporter (EW-R). Although the intelligence stations, IMA and EW, could be placed in a separate location from the AVO and PO, they are included as a part of TIGER to encourage effective crew collaboration and effectiveness. Previous research has shown that, in general, distributed teams have difficulties developing and maintain cohesion [4–6] and collaborative technologies have not eliminated these issues [7, 8]. The AVO and PO stations control the flight, weapons, and sensor operation of the simulated MALE UAS using the same software as AFRL’s PRINCE. The IMA-A and IMA-R monitor, analyze, and report any important events identified in the UAS’s video sensor stream. The IMA-A and IMA-R workstations use the Multi-Intelligence Analysis and Archive System software made by General Dynamics. The EW-A and EW-R monitor, analyze and report on important radio communications intercepted via the UAS radio. The EW-A and EW-R stations use simulation radio software custom made for TIGER. All six crew stations have a standard communication software suite consisting of a chat client and a voice radio. All the workstations have Situational Awareness (SA) displays showing sensor imagery and tactical maps. Each workstation also has Command and Control (C2) planning software consisting of Portable Flight Planning Software and Microsoft Office. The hardware at each workstation includes a desk, chair, 3 monitors, keyboard and mouse, radio headset and push to talk pedal. In addition, the AVO and PO stations have joystick and throttle flight/sensor controls.

TIGER also includes five researcher and trainer workstations. These workstations include the Instructor Operating Station (IOS), Computer Generated Forces (CGF) station, a Role Player (RP) station, and two Experimenter (Expr) stations. The experimenters monitor and record crew actions and behaviors. Experimenters have access to the Live Virtual Constructive Networked Control Suite for recording and making notes on events occurring in the simulated environment and Coalition- Performance Evaluation and Tracking Software that automatically identifies significant events occurring in the simulated environment. The CGF operator utilizes Modern Air Combat Environment (MACE) software by Battlespace Simulations Inc. to create and control any virtual entities in the simulated scenario. The RP uses a stealth viewer client of the image generator software that allows him/her to view the simulated environment from any perspective during the scenario. The RP also uses the radio and chat communication tools on TIGER to communicate with the crew as any external team member or person that has been incorporated into the training scenario, e.g., mission manager or Joint Terminal Attack Controller (JTAC). The IOS operator oversees the crew training, deciding on what actions the RP and CGF operator should take based on the performance of the crew during the training scenario.

3 TIGER Trial

A Human Factors Engineering (HFE) trial was performed at AFRL in January 2015. The trial had 2 UAS crews complete a series of tasks on TIGER during a simulated composite mission scenario. A simulated mission environment was selected to ensure an event-based approach with relevant and complex experiences that would result in rich individual and team observations [9, 10]. The composite mission scenario was also designed to be representative of a JUSTAS mission. This trial was performed to establish baseline individual and team performance levels for TIGER crews completing a simulated composite mission scenario, get UAS crew feedback, and provide specific guidance for the development of customized TIGER HMIs.

3.1 Participants

The participants included 10 CAF personnel experienced in the UAS crew roles included in TIGER. The participants’ backgrounds included time as a part of the Canadian Heron UAS Detachment, experience on the CP-140 Maritime Patrol Aircraft, and ground based analysis and reporting roles.

The participants included two personnel familiar with the AVO role, two familiar with the PO role, three familiar with image analysis and reporting, and three familiar with electronic warfare analysis and reporting. These 10 participants were divided into 2 TIGER crews, crews A and B. One of the personnel familiar with image analysis and reporting filled the role of IMA-A on crew A and IMA-R on crew B. One of the personnel familiar with electronic warfare analysis and reporting filled the role of EW-A for both crews A and B. All participants had a year or more operational experience with their corresponding UAS crew roles.

3.2 Methods

Training.

All the participants received one day of training on TIGER before performing the evaluation mission. Training involved familiarization with the hardware and software components, system functions and HMI operation for each participant at his/her corresponding TIGER workstation. This was followed by each crew participating in a training scenario so that the participants could practice mission tasks on TIGER. An example of one of the TIGER crews operating TIGER is shown in Fig. 2.

Fig. 2.
figure 2

TIGER during trial at AFRL

Full size image

As part of the training, a general project briefing was given to the participants so that they understood the limitations of TIGER and the scenario in order to manage their expectations. This was done to minimize participant feedback on known issues and encourage the participants to be involved in the evaluation of TIGER.

Mission Scenario.

A simulated composite scenario, called “Objective Titan,” was used for the TIGER evaluation. The goal of this scenario was to have a realistic series of vignettes (simulation events) linked into a narrative that stimulated all crew members.

The Objective Titan composite scenario takes place in a coastal city. Human and Signals Intelligence confirmed that enemy forces are threating a Canadian consulate and are coordinating an attack from a nearby residential area. If the UAS crew can locate and identify the leader of the insurgents, a ground-based task force, including a JTAC, will deploy by small boat and armored SUV convey to kill or capture the insurgent leader. The TIGER crew is tasked with observing the convoy route and surrounding area for threats. If threats are observed eyes should be maintained on the contacts and the task force must be notified. The JTAC may use the UAS crew for strike plans and/or in coordination with other assets. Given the urban location, collateral damage estimate is high.

Prior to performing the Objective Titan composite scenario, participants were given a briefing on the scenario including materials that would be expected for a real mission, including geographic locations, times, weather, rules of engagement, and intelligence regarding the threats and contacts of interest.

To successfully complete the scenario, the TIGER crew must complete 4 vignettes: (i) pattern of life – the TIGER crew must collect intelligence, via radio frequency transmissions and UAS sensor imagery, over areas of interest, and determine that the ground-based task force should be deployed; (ii) convoy over-watch – TIGER crew must monitor the convoy route for signs of threats, in this case potential improvised explosive devices; (iii) call for direct fire - the TIGER crew must respond to a direct fire request from the JTAC on board the convoy; (iv) call for in-direct fire – the convoy is ambushed and the JTAC calls for the UAS to provide a targeting solution for a CF-18.

Observations and Metrics.

A variety of measures and observations were taken during the TIGER evaluation mission, including: operator performance, SA, workload, and participant feedback. This paper focuses on the results of monitoring operator performance and participant feedback. It should be noted that during the pre-mission briefing, it was also explained to the participants that the experimenters would periodically administer questionnaires to them while they performed the Objective Titan composite scenario.

Operator Performance.

TIGER crew performance was measured by a UAS subject matter expert (SME) using a behavioral marker checklist with 10 items. The checklist was created as a list of appropriate behavioral responses to preprogrammed simulation events during the composite scenario. The SME rated the participants on 5-point scales (1 being “very poor” to 5 being “very good”) of competence for each behavioral marker in the checklist. Additionally the same 5-point scale was used to measure 5 team behavioral markers, based on [11], to determine the effectiveness of essential teamwork processes: (i) communication, (ii) monitoring, (iii) coordination and prioritisation, (iv) cross-checking, (v) conflict resolution. The SME also took observational notes on crew performance.

Participant Feedback.

TIGER crew feedback on TIGER and the scenario was collected throughout the evaluation and afterwards during the de-briefing. The de-briefing was also used by the researchers to examine questionnaire results on topics such as: which periods of the scenario were crew members over- or underloaded, periods of teamwork breakdown and recovery, losses of individual and shared SA, poor usability and utility ratings, poor operator performance measures.

3.3 Results

Operator Performance.

The 10 items of the behavioral markers checklist followed by the 5 team behavioral markers, and the average results for each marker are shown in Fig. 3. Crew A was rated at “fair” (3) or “good” (4) levels by the SME for all the task specific behavioral markers except in “use of EW tools to provide useful targeting information (IMA)” and “accuracy of reporting combat assessment and collateral effects,” for which they were rated as “poor” (2). Crew A struggled with respect to the team behavioral markers, receiving a “very poor” or “poor” rating in “crew monitoring,” “crew’s coordination and prioritization,” and “crew’s conflict resolution.”

Fig. 3.
figure 3

Average operator performance ratings for both crews

Full size image

Crew B was rated at “fair” or “good” levels by the SME for all the markers except in “accuracy of targeting solution to JTAC,” and “crew’s conflict resolution” for which they received “very good” ratings and “UAS positioning to optimize weapon performance” and “timeliness of providing targeting solution to JTAC” for which they received “poor” ratings.

Overall both crews performed reasonably well on the individual task markers, and Crew B has good team behaviors. However Crew A suffered from poor teamwork behaviors, which likely resulted in difficulties performing the other tasks rated on the checklist. The SME observer recommended that TIGER training include UAS doctrine, tactics techniques and procedures (TTPs), and crew concepts be developed for future TIGER trials.

SME Observations.

The SME made detailed accounts of various aspects of the crews performance, including doctrinal knowledge, leadership and crew coordination, and situational awareness.

Doctrinal Knowledge.

The various TIGER crew members did not have the same level of UAS doctrinal knowledge. There were also gaps in basic doctrinal knowledge, for example Crew A did not consider Rules of Engagement (ROEs) and the Law of Armed Conflict, which resulted striking a target without following standard operating procedures (e.g., not checking for collateral damage and not performing battle damage assessment). These gaps had a direct influence on crew SA and performance during the evaluation mission.

Leadership and Crew Coordination.

The AVO role is assigned as the crew commander for UAS operations. However, since the AVO participants came from a non-combat unit that did not require crew command experience, they did not take lead of the crew during the mission and were left out of important crew member discussions, missing important mission information. This resulted in poor SA and decision making.

Overall crew coordination was also limited, especially for Crew A, with the majority of coordination happening in each of the TIGER sub-teams (AVO and PO, IMA-A and IMA-R, and EW-A and EW-R). This also resulted in poor crew SA, with certain crew members missing important mission information that was critical for making decisions during the mission. Crew B had better overall crew coordination and had better overall mission success. The SME noted that the training did not include standardized tactical communication which led the crew into confusing situations and to act on artificially perceived priorities.

Situational Awareness.

The SME observed awareness for the mission and the system. The results showed that various team members would lose mission SA, for example Crew A maintained eyes on a CF-18 target, while the convoy they were supposed to be watching was attacked at another location. Due to limited time for crew familiarization with TIGER and the outcomes of their missions, it was assessed that the participants had a poor level of awareness of the TIGER system. For example, since the AVO was not familiar with the UAS being simulated on TIGER, they could not think ahead of the aircraft, leading to the UAS not being positioned properly for mission tasks.

Participant Feedback.

Participant feedback from the evaluation has been summarized into two themes: (i) scenario issues, and (ii) workstation usability issues. The results of the feedback were organized in to the crew roles the feedback came from, the issue observed, and the recommended changes. Table 1 summarizes the feedback on scenario issues and Table 2 on workstation usability issues.

Table 1. Feedback on scenario issues
Full size table
Table 2. Feedback on workstation usability issues
Full size table

4 Authority Pathway Concept

One of the main issues the SME observer noted was that both crews had gaps in doctrinal knowledge, particularly with respect to the ROEs and Law of Armed Conflict. One of the crews did not follow standard operating procedures for striking a target. This issue was echoed in the participant feedback that recommended a shared screen GUI concept be developed to represent the required and currently obtained permissions in order to engage a target. An intelligent adaptive interface, called the Authority Pathway concept, has been developed to ensure TIGER operators follow ROEs, the Law of Armed Conflict, and standard operating procedures for engaging a target. An intelligent adaptive interface dynamically changes the interface display of a human-machine system to adaptively react to external events [12]. For TIGER, the Authority Pathway concept intelligently changes a GUI based on the input from the TIGER crew, UAS controlling authority, and software agents.

The Authority Pathway concept supports the UAS crew by displaying and updating the status of the steps required to release a weapon on the basis of a Positive Identification (PID). Discussions with the UAS SME led to the development of a decision tree in which the steps for firing a weapon are outlined, Fig. 4. The figure describes the sequences of activities required by the crew (dashed outlines) and the permissions required from the UAS’ controlling authority (solid lines) in order to engage a target following a PID.

Fig. 4.
figure 4

Steps required to release a weapon on the basis of PID. Steps requiring authorisation external to TIGER have a solid border, steps performed by TIGER crew have a dashed border.

Full size image

The Authority Pathway concept was developed from this decision tree so that the steps for firing a weapon are visualized and the confirmation for each step presented, Fig. 5. Within such a concept, the system will not be able to fire unless the necessary confirmations are in place.

Fig. 5.
figure 5

Authority Pathway concept

Full size image

In this concept, each confirmation step is presented as either green (ready or completed) or red (awaiting action or permission). In addition, the last step completed or authorized is presented in green/blue and ‘pulsed’ so that observers can quickly ascertain which step needs to be completed next. Finally, when completed, each step in the pathway time is stamped so that it can be cross-referenced with chat-based reporting and other mission data (e.g., captured imagery of EW intercepts received).

The advantage of using a display concept comprising large blocks of colour is that the current status of the pathway (and in particular its completion) is easily ascertained from a distance (if presented on large screen wall-mounted displays) or from different viewpoints (if presented on table-top displays).

5 Conclusions

TIGER and the results of a HFE trial with two operating crews were presented. The trial had each of the crews perform a mission to collect and analyze radio and UAS sensor imagery intelligence, monitor friendlies and enemies on the ground and conduct direct and indirect strikes. The results of the trial showed that both crews performed reasonably well on individual tasks, but one crew suffered from poor teamwork behaviors. SME observations determined that doctrinal knowledge, leadership, crew coordination, and situational awareness of both crews were lacking. In particular, with respect to missing doctrinal knowledge, one crew did not follow the standard operating procedures for engaging a target. Participant feedback suggested the development of a GUI to support the crew in following the proper steps for engaging an enemy. These issues prompted the development of a new intelligent HMI concept called the Authority Pathway. The Authority Pathway concept visualizes the steps required, and the progress being made, for TIGER operators to follow the ROEs and Law of Armed Conflict to legally engage a target. Future work will continue the development and testing of Authority Pathway on TIGER as well as investigate new HMIs and crew training concepts to address other operator performance and participant feedback issues.