Keywords

1 Introduction

1.1 Purpose

There is an increase in pilot demand at current commercial training system facilities. Consequently, training providers are looking for new economical training technologies to increase pilot training and provide an alternative to Full Flight Simulators (FFSs) and Fixed Training Devices (FTDs). Pilot training conducted in these devices focuses on the mechanics of flying as well as the interaction between the Captain and First Officer, and their interaction with external personnel (e.g., ground support, cabin crew). With technology advances in head-worn augmented reality and virtual reality (AR/VR) interfaces, adaptive training technologies and human behavior representation, the potential exists to leverage this technology to offload or download training from the FFS or FTD to VR training devices. Accordingly, Boeing has been applying these capabilities to the development of VR training prototypes designed to address different aspects of a pilot training curriculum [1].

The first prototype implemented is a Ground Procedures Trainer that focuses on both taskwork and teamwork competencies, as defined in Sottilare et al. [2], prior to the taxi phase. It is an adaptive team training prototype designed for individual student training. While there have been team intelligent tutoring systems (ITSs) implemented previously [3], this prototype differs in that it employs a synthetic role-player to fill out the team. The Ground Procedures Trainer provides a lesson to an individual student pilot that covers four procedures that a Captain and First Officer perform while the aircraft is at the gate, prior to taxi. The student assumes the role of the First Officer, while the VR Pilot provides the Captain role. For the purposes of this prototype, taskwork refers to the performance of checklists and interaction with the flight deck to ensure the aircraft is ready for engine start and taxi. Teamwork in the context of the work described herein refers to the interaction between the Captain and First Officer, which is typically referred to as Crew Resource Management (CRM) [4]. The focus on this paper is to describe the methodology used to implement adaptive training that supports the development of the taskwork and teamwork skills required for ground procedures in a commercial pilot flight training scenario.

1.2 Crew Resource Management

An accepted definition of team is “a distinguishable set of two or more people who interact dynamically, interdependently, and adaptively toward a common and valued goal/objective/mission, who have each been assigned specific roles or functions to perform, and who have a limited life-span of membership” [5]. Team performance research has led to the development of a wide range of models and theories [5] to describe the interactions within a team and provide a set of competencies to train teamwork. With respect to flight crew operations and in particular, commercial aviation, crew resource management (CRM) is the term primarily used to refer to the interactions between the flight deck crew and other human roles involved with a flight, such as cabin crew in the commercial sector and maintenance personnel. A related term, Cockpit Resource Management, refers solely to the interactions between the flight deck crew. With that said, there are differing definitions of Crew/Cockpit Resource Management.

Federal Aviation Authority Crew Resource Management (CRM)

The FAA [4] defines CRM as the “effective use of all available resources: human resources, hardware and information…to operate a flight safely” (p. 2). While CRM initially referred to Cockpit Resource Management and focused on the interactions between the Captain and First Officer (flight deck crew) only, it was expanded to Crew Resource Management and the combined activities of the flight deck crew, cabin crews, aircraft dispatchers, maintenance personnel and air traffic controllers. CRM involves the following activities “team building and maintenance, information transfer, problem solving, decision making, maintaining situational awareness and dealing with automated systems” (p. 2).

The FAA has defined three high level performance clusters to describe the categories of CRM activities. The clusters are: 1-Communications Processes and Decision Behavior, 2-Teambuilding and Maintenance and 3-Workload Management and Situational Awareness. Each cluster is categorized further into a set of behaviors, and then a set of behavioral markers, which are demonstrable examples that support the behavior. “Crewmembers acknowledge their understanding of decisions” is an example of a behavior marker, and it is associated with the Communications Processes and Decision Behavior cluster and the Communications/Decisions behavior. The CRM clusters and behaviors are shown in Table 1. The FAA [4] recommends three critical components of CRM training: indoctrination/awareness, recurrent feedback and practice and continuing reinforcement.

Table 1. CRM clusters and behaviors
Full size table

CRM Reference in Evidence Based Training

Most airline companies in and outside of the United States participate in International Civil Aviation Organization (ICAO). In 2013, ICAO introduced publication 9995, entitled “Manual of Evidence-Based Training (EBT)” [6], which focuses on the core competencies of piloting skills. Five of the eight competencies defined in the EBT method focus on CRM, which underscores the importance of CRM to the role of a commercial airline pilot. While these competencies are similar to the clusters and behaviors defined by the FAA, they are distinctly different.

The CRM related competencies identified in the EBT manual are provided in Table 2. Similar to the FAA documentation, the EBT manual defines a set of performance indicators, another term for the behavioral markers used in the FAA model, to provide examples of behaviors that demonstrate performance of the CRM competencies. The EBT manual stresses that the CRM competencies be addressed throughout various training stages and regimes, as recommended by the FAA. However, the methods for developing the training regimes and defining competencies differs between the two approaches, in how the competencies were derived.

Table 2. CRM competencies defined by ICAO EBT manual.
Full size table

CRM in the Military

The Air Force refers to CRM as Cockpit/Crew Resource Management and refers to the effective use of all available resources by individuals or crews to safely and efficiently accomplish an assigned mission or task [7], or more simply, “things aircrews do” (p. 4) [8]. Air Force Instruction (AFI) 11-290 [8] defines CRM and identifies six knowledge and skill sets, which are provided in Table 3. CRM training doctrine outlined in AFI 11-290 encompasses both human crewed flight as well as remotely piloted flight activities.

Table 3. CRM knowledge and skill sets per AFI 11-290.
Full size table

1.3 Adaptive Instruction and Team Training Issues

The term “Adaptive Instructional Systems” (AIS) was first coined by Brawner and Sottilare [9] and refers to a family of systems that supports the provision of tailored instruction to a student. Intelligent tutoring systems (ITSs) fall under the AIS family. The general components of an ITS include an expert model, instructional model, student model and problem solving environment. The expert model encapsulates the expert’s problem-solving skill in a machine-executable form. The student model provides a current and historical record of the student’s mastery of the expert’s problem-solving skills. The instructor model provides the logic and implementation of an instructional strategy based on comparison of expert and student model. The problem-solving environment is where the student can demonstrate and practice his/her skills issues with team training involving monitoring task and teamwork skills.

With respect to the prototype described in this paper, our ITS architecture (Fig. 1) involves three primary components: a Student Model, an Instructional Model, and an Expert Model. The student model implements a profile of dynamically maintained variables, each corresponding to one learning objective (LO). These variables are evaluated over a number of observations. As a result, changes due to learning are reflected across exercises, as the score increases due to correct performance (based on the expert model), or decreases as errors are made. The amount that scores are changed can be weighted according to the degree to which the action reflects mastery of the LO. Amount of change is also adjusted according to the degree of support (e.g., hints) the ITS provided to the student during the exercises.

Fig. 1.
figure 1

The Boeing ITS architecture.

Full size image

This initial capability, the Boeing ITS, uses discrete event scenarios in which the student’s actions bring them through a particular path (using the instructional model) that is more or less correct. An evolution of the ITS was the Virtual Instructor (VI), which supports dynamic simulation-based training environments, in which the problem solving scenarios can be a range of simulation/gaming environments. The Ground Procedures Trainer extends the VI to support training of taskwork and teamwork skills for an individual student.

While there has been tremendous progress in the development of ITSs to train individual tasks, or taskwork - the technical performance of a role or function, ITS designed to train teamwork, such as CRM, are still in the research phases due to additional complexities [2, 3]. The implementation of an AIS capability in a team training environment requires the training audience (students) to simultaneously perform taskwork and teamwork. Further, both the individuals on the team, and the team itself, have sets of taskwork and teamwork goals, which requires the assessment of these areas at both the individual and team level as well as decision on whether to provide an instructional intervention to the entire team or just an individual and how to adapt the scenario if the issue as at the individual level rather than the team level (Fig. 2).

Fig. 2.
figure 2

Assessment and instructional intervention model for team training.

Full size image

Specifically, team training AIS requires multi-level automated human performance assessment of individual and team taskwork and teamwork-related competencies. The next issue is to use these multi-level assessments to update student models at both the individual and team level, and to diagnose the learning needs at these two levels. Learner needs are identified and used to guide the implementation of feedback and other instructional interventions. Providing interventions at a team and collective level is complex, as some individuals may require feedback irrelevant to other team members. This makes the decision of what interventions to apply when most difficult. The development of the VR pilot training tools discussed in this paper addresses the issue of assessing taskwork and teamwork simultaneously. However, since the training tool is designed for an individual student, using a synthetic team member to fill out the flight crew, the issue of whether to provide feedback to the individual or team did not need to be addressed.

2 Ground Procedures Trainer Overview

The Ground Procedures Trainer integrates three primarily technologies: (1) VR Flight Deck, (2) VR Pilot, and (3) VR Instructor. The VR Flight Deck used as the training environment in this lesson is a commercial aircraft virtual reality flight deck. The VR Pilot is a synthetic representation of the Captain (Fig. 3) to support CRM training to enable an individual student to train on procedures that require interaction between the Captain and First Officer without requiring a second student, or instructor role-player. While the student performs the procedures, the VR Pilot will perform the expected duties of a Captain, as well as respond to the student in response to the student’s actions. For example, if the student says something out of context or incorrect, the VR Pilot will ask for the student to repeat or remind the student of the procedure they are currently performing.

Fig. 3.
figure 3

The VR pilot.

Full size image

Evaluation of the student’s actions is supported by the VR Instructor using data from the VR Flight Deck and the student’s interactions with the VR Flight Deck (e.g., head movement, hand/arm movement) and the VR Pilot (e.g., speech). Additionally, the VR Instructor will provide feedback to the student in response to his or her actions, through a variety of means. For example, if the VR Instructor determines that the student is looking at the incorrect instrument, the correct instrument will be highlighted, or the VR Instructor may verbally inform the student what instrument to use. The student is also able to prompt the VR Instructor for help. In some cases, the VR Instructor will prompt the VR Pilot to provide specific feedback, such as directing the student’s attention to a different part of the procedure. The VR Pilot will interact with the student and the VR Flight Deck to complete the flight tasks that are expected of the Captain (taskwork) as well as support CRM related interactions with the student.

2.1 Lesson Learning Objectives

In order to provide adaptive training to develop both taskwork and teamwork competencies, student performance assessment is linked to either a set of task related behavioral indicators or CRM related behavioral indicators. As discussed previously, there are a variety of competencies and behavioral indicator sets that have been developed to support CRM training in a fixed wing aircrew environment. Since the VR pilot training prototypes are focused on commercial pilot type rating training, the Ground Procedures Trainer is using multiple sets of learning objectives taken from a type rating syllabus as well as the FAA and ICAO competencies. The learning objectives are identified below in Table 4.

Table 4. Learning objectives for the ground procedures trainer prototype.
Full size table

2.2 Scenario Event Linkage to Learning Objectives

After the learning objectives were specified, the ground procedures that comprise the lesson were decomposed into behaviors and expected actions, with the expected actions providing the basis for assessment. The expected actions were linked to the learning objectives identified in Table 4. In some cases, an expected action was related to more than one learning objective and therefore linked to multiple learning objectives. The linkage to learning objectives is what enables the student model to be updated as a result of an assessment of the student’s actions. The following example is provided to illustrate the process.

One of the steps in a procedure addressed in the lesson is for the student to visually check the flap setting and verbally confirm the flap setting to the Captain. This step involves two primary actions. First, the student must look at the correct instrument, in this case, the flaps. This action addresses the following learning objectives from above list:

  • T1 - Locate all airplane systems (in the scenario) – the student must be look at the flaps

  • T4 - Demonstrate proficiency in the use of the associated checklists – knowing that checking the flaps is the next step and demonstrating knowledge of the proper order for the checklist items

  • I1 - Application of Procedures: Identifies and follows all operating instructions in a timely manner – the student is given a time constraint for receiving credit for identifying the flaps

Next, the student must correctly state the current flap setting so that the Captain is able to acknowledge. This second behavior is related to the following learning objectives:

  • T1 - Locate all airplane systems (in the scenario) – the student must say the correct name for the flaps

  • T4 - Demonstrate proficiency in the use of the associated checklists – the student is verbally confirming the flaps setting, which is a required part of the procedure

  • I4 - Communication: Conveys messages clearly, accurately and concisely – in order to receive credit, the confirmation of the flaps must be understood by the VR Instructor speech recognition system

2.3 Assessment of Learning Objective Scores

Linking the learning objectives to the expected actions and related metrics enables the learning objectives to be updated in real time based on the assessment of the student’s actions. The learning objectives are scored on a scale of 1–5, adapted from commercial pilot training assessment standards with a score of 5 demonstrating perfect performance and a score of 1 indicating that performance was not at all demonstrated. Scores are decremented if the student required a hint in order to perform the task correctly. Figure 4 shows a graph of learning objectives scores by nodes over the course of a lesson.

Fig. 4.
figure 4

Learning objective scores graphed overtime throughout the lesson.

Full size image

The tabs across the top indicate that graphs are provided for the three sets of learning objectives, and the hint tab displays the number of hints used for each node. The ICAO competencies tab is selected in Fig. 4. The graph shown is for the selected learning objective, I1, and it shows the nodes in which learning objective I1 was evaluated as well as the score at each node. The average score for the node is shown at the end of the learning objective description. In this case, the average score for learning objective I1 is 4.4. The scores for the other ICAO learning objectives are shown at the end of their textual descriptions. The graphs for these learning objectives are viewable by clicking on the learning objective text.

2.4 Feedback Strategy

Instructional feedback is provided in several forms - through highlighting the instrument/equipment that the student should be viewing, providing verbal feedback from an “instructor” or the VR Pilot and auto-completion of a step in the procedure. Examples of feedback that the VR Instructor may provide to the student include statements to let the student know that the action is not part of the procedure, statements that say the action is part of the procedure but not the in the proper sequence; or even telling the student what action they need to perform. Feedback used by the VR Pilot when the student is speaking unclear include statements such as “come again” or “I didn’t get that”. The student receives feedback either at their request, though the use of the hint function, or if they fail to perform a step within a certain time period.

The student is able to request hints if he or she is having difficulty by verbally requesting a hint. There are three levels of hints. The first level hint will highlight the instrument that the student is supposed to interact. The second level hint will have the VR Instructor tell the student what task to perform. The third level hint results in the VR Instructor completing the task automatically and informing the student to move onto the next step in the procedure.

The final form of feedback provided to the student is a review of the lesson when they have completed it. The review provides the overall scores for each of the learning objectives so they are able to determine which learning objectives they were most successful at, and which learning objectives will require further work. The student, or an instructor, will be able to drill down in the graph to determine on which nodes they scored better or worse. Figure 5 provides a screenshot of the learning objective summary.

Fig. 5.
figure 5

Learning objective summary.

Full size image

3 Conclusion

3.1 Training Effectiveness Evaluation

The Ground Procedures Trainer will undergo an evaluation to compare its training effectiveness to the current means of learning flight procedures, which is in a Fixed Training Device (FTD). When the student receives training in a FTD, there are typically two students, one in the role of the Captain and one in the role of the First Officer, and the students will switch roles to experience both roles. In some cases, an instructor, in addition to the instructor teaching the lesson, plays one of the roles if two students are not available.

The lessons in the FTD walk through the procedures for an entire flight, with the goal of making the procedure more natural and fluid. The FTD typically is not flown for the entire flight, but jumped to various stages of flights to focus on the completion of procedures. The FTD uses a mix of real aircraft instrumentation and flat panel displays representing the aircraft system. Typically, a realistic seat from an aircraft cockpit is used, although some FTDs use office style chairs.

For the Ground Procedures Trainer, the VR cockpit provides the flight deck environment as described previously. The student wears a VR headset to view the cockpit with earphones and microphone for audio feedback and verbal interaction. The student uses a VR hand controller or haptic glove to manipulate instruments in the cockpit and will be seated in an office style chair. For this lesson, this student role is the First Officer, and therefore, seated in the right seat. The Captain is role-played with the VR Pilot as discussed previously and located in the right seat.

The training effectiveness evaluation is designed as an experiment to compare the FTD lesson with the VR Ground Procedures Trainer. The training effectiveness will be evaluated through the performance of the ground procedures in a Full Flight Simulator (FFS), which provides an environment that closely replicates an operational flight deck. The participants will begin by reviewing and signing the consent form. Next, they will complete a demographic survey to obtain information regarding their prior flight experience as well as gaming and VR experience. They will then be provided an overview of the ground procedures trainer and a practice period to review the ground procedures using a poster of the flight deck, which is what the students typically do to prepare for their FTD lessons. Next, the participants will receive instruction on the use of VR prototype or current training device. They will then complete the lesson three times in either the VR prototype or current training device. For the current training device condition, the FTD, one instructor pilot will serve as the instructor, and a second instructor pilot will role-play the Captain. After the participant completes the lesson three times in either condition, they will complete a post-training survey to obtain feedback on their experience. Finally, the participants will complete an assessment in a FFS with the participant assuming the First Officer role and another pilot role-playing the Captain.

3.2 Summary

The VR ground procedures trainer is the first step in the creation of a variety of VR based training capabilities to reduce time spent in large, traditional simulators. The use of adaptive training capabilities provides the student with more opportunities to receive training at their convenience, and the use of the VR Pilot enables the student to train without the need of a second student or role player.

There are multiple theories and frameworks for training and assessing teamwork competencies. In order for the VR pilot training applications to be used under various regulatory authorities that prescribe to different CRM models and training best practices, the VR adaptive training capability is designed to work with multiple competency frameworks.