Keywords

1 Introduction

1.1 Joint Terminal Attack Controllers

The U.S. Marine Corps (USMC) Vision and Strategy 2025 describes the critical path for maintaining the Corps’ dominance as the expeditionary force of choice in a future of increasingly complex and volatile environments. Marines must be prepared to operate in a decentralized manner, placing unprecedented demands on them to make difficult decisions in high stress and high stakes situations. Close Air Support (CAS) is an example of such an environment. CAS involves aircraft attacking ground-based hostile targets in close proximity to friendly forces, which requires a clear understanding of the battlespace, including the locations of targets relative to friendlies, the weapons’ capabilities of the aircraft, and the possible timing and effects of the weapons (e.g., to minimize collateral damage and fratricide). Joint Terminal Attack Controllers (JTACs) are the qualified service members who direct aircraft engaged in these highly dynamic, time-sensitive CAS missions [2]. CAS is a complex 12-step process that involves constructing plans for the mission and acquiring the necessary data for the attack (e.g., assessing attack geometry, determining the location of friendly forces) and then communicating those plans to the aircraft and other members of the ground units before, during, and after the mission’s execution.

Individuals training to become JTACs come to the schoolhouse with a variety of military occupational specialties, such as infantry, artillery, and pilots. Many enter the course with hundreds of hours of experience participating in CAS missions, such as pilots who may have executed the CAS attacks against targets in theater, whereas other trainees may have never participated in CAS at all. The marked differences in students’ experience levels in the course leads to challenges for the less experienced students, who have a very steep learning curve relative to experienced students, and for instructors who train these less experienced students up to speed. To address the challenges of training individuals at various skill levels, we developed an adaptive training testbed to provide JTAC trainees with additional opportunities to practice CAS decision-making skills called the Adaptive Trainer for Joint Terminal Attack Controllers (ATTAC). Adaptive training (AT) is training that is customized to an individual’s strengths and weaknesses, such that each student receives a tailored training experience [3].

In this paper, we briefly review the AT literature, discuss how cognitive theory drove design decisions for ATTAC, and present initial results from pilot testing on users’ perceived usability of and overall satisfaction with ATTAC. We conclude with a discussion of future research plans with the ATTAC testbed.

1.2 Adaptive Training Overview

One-on-one tutoring is often considered a best practice for training and education. In a seminal paper, Bloom [4] found in his analysis of the literature that students who received one-on-one tutoring performed two standard deviations higher than students in a traditional classroom setting [c.f. 5]. Of course, one-on-one tutoring is not a tenable solution for training in the real world, as this would be prohibitively expensive in terms of cost, manpower, and time. Therefore, it has been the goal of instructional designers to try to approximate this experience through technology-based training solutions. One such solution that has gained traction recently is AT. In a review of AT systems used in military settings, McCarthy [6] described how AT has been utilized in a vast number of domains, from troubleshooting issues in electronic systems to learning procedures for operating radar systems. ATs have been implemented to teach diverse knowledge and skills, including both conceptual information and procedural tasks [7]. Durlach and Ray [8] performed an extensive review of the AT research literature and generally found that AT methods were effective and led to better learning outcomes. However, the authors noted that there were relatively few examples of carefully controlled experiments that included comparisons between an adaptive group and a non-adaptive group or two adaptive groups, which limited their ability to draw strong conclusions about what adaptive techniques work best under which conditions. More recent research has found support for adapting feedback and difficulty of a task based on the trainee’s performance [1], while some others have not [9,10,11]; therefore, more research is needed to determine when and how to implement AT techniques successfully.

AT systems rely on three core components to effectively tailor a student’s instruction, which is referred to as the Observe-Assess-Respond (OAR) AT model [12]. First, the AT system must be able to observe characteristics about a student within the context of the learning environment. These observations could be based on a student’s behavior within the environment or some trait variable (e.g., spatial ability or prior knowledge). Next, the system must assess what these observations mean about the student. Finally, once the system has made an accurate assessment of the student, the system must then respond with some instructional intervention based on learning theory to efficiently guide the student to meet his/her learning objectives. One example of an instructional intervention is feedback provided in response to a student’s input. When developing an AT system, one must consider each component of the Observe-Assess-Respond model during the design process. In this research effort, we were particularly interested in examining the latter case; that is, we investigated what instructional interventions were most effective in a given task domain.

1.3 Research Approach

The overall approach of this effort was to build upon Landsberg and colleagues’ [1] research in which they developed a scenario-based adaptive trainer for making periscope calls. Specifically, students were trained using a periscope simulation to determine and report a contact’s angle on the bow (AOB), or the orientation the contact is presenting relative to the periscope operator’s eye. Trainees received feedback after each scenario that adapted based on how they performed, and after a series of scenarios, the difficulty of the next block of scenarios was also adapted. This research demonstrated that adapting the difficulty of scenarios and the type of feedback trainees received based on an assessment of their performance led to more effective and efficient training when compared to traditional non-adaptive training approaches of similar length or longer. Although these results were very promising, more research is needed to determine the generalizability of this approach to other types of tasks.

Therefore, for the current effort, we sought to extend Landsberg and colleagues’ [1] research in a number of important ways. First, we conducted this research to determine whether using the same approach of adapting the feedback and difficulty generalizes to a different type of task. Determining AOB is primarily a visuo-spatial task, but do these adaptive approaches apply in a complex decision-making task, such as CAS? That is – perhaps the underlying cognitive mechanisms that support learning a spatial task (e.g., calling AOB) differ from more complex conceptual tasks (e.g., CAS decision-making), because spatial information is processed differently than conceptual information. Second, determining a “level of correctness” is often less straightforward in a decision-making task. In the periscope training study, trainees responded with an orientation and angle (e.g., Starboard 160°), which has a clear correct answer and can easily be scored (e.g., an answer of Starboard 100° is off by 60°). However, in a decision-making task, there may be more than one correct answer with correct answers falling along a continuum (i.e., correct, incorrect, and partially correct) rather than a binary assessment of correct or incorrect, making assessment less straightforward. Third, in the present research, trainees were expected to make 5–7 decisions in each scenario, while in previous research adaptations were based on a single answer [1]. Therefore, determining how to handle adaptations with more decision points within a single scenario posed a unique challenge. To explore these research questions, we developed the ATTAC testbed.

2 Developing ATTAC

2.1 ATTAC Overview

Because CAS is such an involved task with 12 individual steps, we chose to focus our effort on one particularly challenging step, the critical planning process called “game plan” development. The JTAC’s game plan sets the stage for the execution of the entire CAS mission and is a difficult topic for JTAC trainees to master. The game plan consists of four interdependent decisions: Type, Method, Ordnance, and Interval. “Type” refers to the “Type of control” the JTAC employs over the CAS mission, and the decision is based on factors such as whether the JTAC can meet the criteria for controlling the mission or prefers the CAS pilots to assume some level of control. “Method” is the “Method of attack” the attacking aircraft will utilize, and the JTAC must consider how the target will be correlated with the aircraft (e.g., the JTAC provides precise grid coordinates or the aircraft uses a sensor to find the target). To determine which Ordnance to employ, the JTAC must decide which weapon to use against the target to achieve the desired effects. For Interval, the JTAC determines how much time separation is needed between attacking aircraft. In a typical CAS mission, a JTAC controls two or more aircraft, and the JTAC must decide whether each aircraft will follow the same game plan or each aircraft will have a different game plan. When designing ATTAC, we intended for trainees to practice developing game plans for a variety of situations, receive feedback about their responses, and experience different levels of scenario difficulty.

ATTAC is an adaptive scenario-based trainer that presents a series of CAS scenarios and trainees develop a game plan(s) for each scenario that will meet the commander’s intent. As depicted in Fig. 1, for each scenario, the trainee is presented with the information a JTAC has on hand while conducting a CAS mission. This information includes: a brief on the situation and description of the target(s), the capabilities of the JTAC (e.g., map and compass, laser-target designator, range finder, etc.), the type of aircraft and weapons available to conduct the mission (e.g., two F/A-18Es, each with two Mk-83 s), the distance of the target from the JTAC and nearest friendlies, the presence of any surface-to-air threats, and current weather conditions. Given this information, the trainee must choose the best Type, Method, Ordnance, and Interval combination that most efficiently and effectively prosecutes the targets and meets the scenario’s objective (i.e., commander’s intent). Based on the trainee’s response, the trainee receives tailored feedback and the difficulty of subsequent scenarios is also adjusted. The overall adaptation design decisions for ATTAC were based on popular science of learning principles and theory.

Fig. 1.
figure 1

Example ATTAC scenario.

Full size image

2.2 Theoretical Approach to Designing ATTAC

Cognitive Theory of Multimedia Learning.

During the development of ATTAC, we utilized the Cognitive Theory of Multimedia Learning [CTML; 13, 14], a popular framework in educational psychology for understanding how people learn. Using this approach allowed us to make predictions about the effectiveness of different instructional techniques and drove our overall testbed design. A main assumption of CTML is that learners have a limited working memory capacity; therefore, instruction should be designed to limit the amount of unproductive cognitive processing imposed on the learner and foster productive cognitive processing. According to CTML, individuals engage in three different types of cognitive processing while learning: unproductive processing due to poor instructional design (i.e., extraneous), processing related to the complexity of the material (i.e., essential), and processing to make sense of the material (i.e., generative). Extraneous cognitive processing arises from poor instructional design, such as using an interface that is cumbersome or awkward. Essential cognitive processing stems from the complexity of instructional content itself. For example, learning how to repair an engine would require more essential processing for novices who know very little about car repair, because they have to build these schemas about how an engine works as they learn; but repairing an engine would require less essential processing for master mechanics, because they already have schemas in place and their retrieval is an automated process, requiring fewer cognitive resources to process the incoming information. Therefore, the amount of essential cognitive processing is highly dependent upon a learner’s prior knowledge and experience. Generative processing refers to the level of mental effort learners must expend in order to make sense of the material they are learning. Generative processing is considered productive cognitive processing, because the learner is relating the learning content to his prior knowledge. These three types of cognitive processing are traditionally thought of as additive, and it is possible for individuals to reach their capacity, which is called cognitive overload. In the event of cognitive overload, learning and task performance suffer. Therefore, the goal of instructional designers should be to minimize extraneous processing, manage essential processing, and foster generative processing.

Consistent with CTML, the expertise reversal effect (ERE) is a well-documented finding that certain instructional interventions that may be effective for novice learners may actually be detrimental for more knowledgeable learners, and it has been demonstrated in a number of different domains [15]. For example, in one experiment researchers found that providing more detailed instruction was beneficial for novice learners by giving them useful information about the task. However, as learners gained more expertise about the domain, providing the detailed instruction hurt their performance; in fact, providing less structured instruction was more beneficial for the experts [16]. The authors argued that the detailed instruction led to extraneous cognitive processing for the experts, as the additional detail was unnecessary and distracting, using up limited cognitive resources that could have otherwise been devoted toward meaningful cognitive processing. On the other hand, the additional detail was necessary for the novices in order to manage their essential processing demands. The ERE is just one example of why tailoring training to the needs of an individual learner is beneficial; it is important to consider trainees’ prior knowledge of the domain as some instructional strategies may be more or less effective depending upon their level of expertise.

Adapting Feedback and Scenario Difficulty.

Grounded in our review of the literature, we determined that adapting feedback and scenario difficulty based on trainee performance were both promising techniques for ATTAC. Feedback is considered by many researchers as one of the most effective instructional strategies [e.g., 5, 17], but there are many remaining questions on how to apply feedback in complex training situations, such as AT and scenario-based training [18,19,20]. In the current research, our goal was to test predictions from CTML and ERE that providing detailed feedback may be more effective for low-performing students, while less detailed feedback may be more helpful for high-performing students.

Regarding scenario difficulty, a recent meta-analysis [21] reported that training with adaptively increased scenario difficulty was more effective than training that increased difficulty at fixed intervals or training that remained at a constant difficulty throughout. This finding is consistent with CTML. If a trainee is performing in a scenario that is too difficult, essential cognitive processing demands will be very high, and that trainee may not have enough cognitive resources available for meaningful learning to take place. Likewise, if a trainee performs in a scenario that is too easy, cognitive processing demands will be low, and the trainee may become bored or distracted, increasing extraneous processing. That is, these theories predict that an optimal strategy is to keep trainees in a “sweet spot” during training, in which the scenario is not too difficult to overwhelm or too easy to bore the learner.

2.3 How ATTAC Works

ATTAC provided individualized training by following each of the components of the OAR model, and each are discussed in turn. For each game plan scenario, ATTAC first observed a trainee’s game plan responses that were input via the drop-down menus for each decision (e.g., Type, Method, Ordnance, Interval). Once a selection was made for each drop-down, the trainee submitted the game plan for assessment.

Next, ATTAC assessed the trainee’s game plan by comparing responses to a database of possible game plan combinations. Due to the nature of game plan development, determining assessment criteria posed two distinct challenges: (1) how to assess 5–7 interdependent decisions into a single performance score from which to make adaptation decisions, and (2) how to manage that there are often many correct approaches to prosecuting a CAS mission (or as JTAC instructors like to say, “there is more than one way to skin a cat”). To address these issues, the individual decisions (e.g., the Type, Method, Ordnance, and Interval selection for each aircraft) were considered holistically because it is the combination of factors that determines whether or not the game plan will be effective (e.g., one method of attack may be appropriate if used with a certain ordnance, but not with another). Therefore, the entire game plan was assigned a score based on how likely it would meet commander’s intent. There were three possible assessment outcomes: ideal, acceptable or unacceptable. A game plan was considered ideal if it met mission requirements as efficiently as possible, while also considering the safety of the attacking aircraft, friendly forces, non-combatants, and the JTAC. An acceptable game plan also met mission requirements but may not be the most efficient answer. Finally, an unacceptable game plan was potentially unsafe, inconsistent with CAS doctrinal requirements, and/or unlikely to meet commander’s intent. Based on these criteria, it was possible for a scenario to have several different game plans that could be scored as ideal, acceptable, or unacceptable, which in turn, allowed us to handle the multiple ways the trainee could approach a scenario.

Lastly, based on the assessment of the trainee’s game plan, ATTAC responded by adapting the feedback the trainee receives and the difficulty of subsequent scenarios. As previously discussed, feedback and difficulty adaptations were selected as instructional interventions in order to support productive cognitive processing consistent with CTML. With regard to adapting feedback, we considered the ERE literature [15], such that the type of feedback trainees received was based on the correctness of their game plan. Examples of feedback for each type of game plan assessment are provided in Table 1. For example, if the trainee’s answer was assessed as an ideal game plan, the trainee received positive outcome feedback and their answer was displayed (Table 1, top row). When trainees provided an ideal game plan, we reasoned that providing elaborative process feedback would serve as an extraneous cognitive processing demand, because presumably the trainee performed the correct decision-making process to arrive at the correct answer. If the trainee’s answer was an acceptable game plan, the trainee received outcome feedback with elaborative process feedback specific to the response to help the trainee understand why his/her answer was not ideal (Table 1, middle row). In this case, the trainee’s answers were mostly correct, so we provided error-specific feedback designed to minimize the amount of cognitive processing required to understand how to arrive at the ideal answer. Finally, if the trainee’s game plan was unacceptable, outcome feedback was provided in addition to fully elaborative process feedback that described the correct decision-making process for Type, Method, Ordnance, and Interval decisions for an ideal game plan (Table 1, bottom row). For unacceptable game plans, we managed essential processing demands by modeling the decision-making steps an expert takes to arrive at an ideal solution. For both the acceptable and unacceptable game plan responses, the feedback screen also displayed the trainee’s answer and an ideal answer so that the trainee could compare their answer to an ideal solution. In all cases, trainees could toggle between the feedback screen and the scenario to review them before moving on to the next scenario.

Table 1. Example feedback by game plan assessment.
Full size table

Similar to how the feedback adaptations worked, the assessment of a trainee’s game plan also drove the difficulty adaptation of subsequent scenarios. In an attempt to manage the cognitive processing demands of the trainee, the difficulty of the scenarios (i.e., basic, intermediate, and advanced) was adapted to match to the trainee’s level of performance. To illustrate, those who were performing poorly may have been experiencing high essential processing demands, which could have prevented meaningful learning from taking place. Reducing scenario difficulty lowers the essential processing demands for the trainee and allows more cognitive resources to be directed to more productive cognitive processing. Likewise, scenario difficulty increased for trainees who performed well to ensure they continued to be challenged and not become bored which could lead to underutilization of cognitive resources. However, unlike feedback which adapted after every scenario, scenario difficulty was adjusted after a set of two scenarios. This was done to prevent difficulty level from cycling too reactively.

Once ATTAC was fully developed, we conducted a pilot study to gauge participants’ initial impressions of ATTAC and test the adaptive algorithms to ensure they were working appropriately. Our ultimate goal is to perform a training effectiveness evaluation of ATTAC, so this pilot study was a necessary first step to determine that the system was working as planned and our experimental measures were adequate.

3 Pilot Study

3.1 Design

In this study, U.S. Marine Corps personnel evaluated an adaptive version or a non-adaptive version of ATTAC in a between-subjects design. Because one of our main goals was to assess how well the adaptive algorithms were working, more participants were assigned to the adaptive condition than the non-adaptive condition.

3.2 Participants

Pilot data were collected from a total of 22 male participants. Participants ranged in age from 20 to 28 years (M = 24.15, SD = 2.99) and had been in the Marine Corps for 1 to 11 years (M = 4.23, SD = 2.46). Of the 22 participants, 10 indicated they had prior CAS experience, primarily as forward observers (i.e., qualified service members who assist the ground unit and the JTAC by providing targeting information) during live and/or simulated training scenarios.

Eighteen participants were in the adaptive condition, and four participants were in the non-adaptive condition.

3.3 ATTAC Testbeds

There were two ATTAC testbeds used during this study, the adaptive version and a non-adaptive version. There were two main differences between these versions of ATTAC. The first difference was the level of detail provided in the feedback. The adaptive version provided tailored feedback based on the trainee’s performance (as described above in Sect. 2.3; see Table 1), whereas the non-adaptive version only displayed the trainee’s submitted answer and the ideal answer. The second difference involved scenario difficulty. The adaptive version of ATTAC adapted the scenario difficulty as described above in Sect. 2.3, whereas participants in the non-adaptive version only completed intermediate difficulty scenarios regardless of how well they performed during training.

3.4 Materials and Procedure

Participants first completed a demographic questionnaire that contained biographical items about their background, military experience, and previous CAS experience. Participants then completed a 4-item scenario-based pre-test, in which they did not receive feedback on their performance. Next, during the training phase, participants briefly reviewed a PowerPoint tutorial on how to use ATTAC and then completed training scenarios in ATTAC for 45 min. After the training phase, participants completed the System Usability Scale [SUS; 22] and an Instruction Reaction Questionnaire (IRQ). The SUS is a 10-item survey that asks participants to indicate their agreement on a 1 (strongly disagree) to 5 (strongly agree) scale regarding the usability of the training environment (e.g., “I thought the system was easy to use.”). The IRQ is a 14-item survey we developed that asks participants to rate their agreement regarding their perceptions of the training on a 1 (strongly agree) to 6 (strongly disagree) scale (e.g., “Overall, the training was useful to me.”). Finally, participants completed a 4-item scenario-based post-test with no feedback and were debriefed and dismissed.

3.5 Results

We examined descriptive data from the SUS and IRQ to ascertain users’ perceptions of ATTAC. Due to the small number of participants in the non-adaptive condition, we chose not to submit these data for further statistical analysis and make direct comparisons of the conditions.

The SUS data were transformed according to the author of the scale’s [22] instructions so that the SUS scores range from 0 to 100. The average SUS score for those in the adaptive condition was 76.91 (SD = 13.59), and the non-adaptive condition had an average score of 70.63 (SD = 12.31). A higher score indicates better usability, and an average score above 68 is considered “above average” [22, 23]. Therefore, both versions of ATTAC were rated as above average usability.

On the IRQ, lower numbers indicate agreement with the statement. Means and standard deviations for the most pertinent items by condition are presented in Table 2. In general, participants in the adaptive condition rated their experience using ATTAC lower than the middle of the scale. Those in the adaptive condition reported that they liked the training, wanted to complete more training like ATTAC in the future, and believed it was useful overall. In addition, the adaptive condition participants believed the difficulty of scenarios was appropriate for their skill level and that the feedback they received helped them develop strategies to improve their performance. However, the results indicated that there was room for improvement regarding the feedback in ATTAC given the relatively neutral responses to some of the feedback-specific items. Participants’ ratings and informal discussions with them during the debrief suggested that they believed the feedback could have been more useful and easier to understand. Those in the non-adaptive condition had generally less favorable attitudes with scores generally above the midpoint of the scale. However, strong conclusions cannot be made due to the small sample size of participants in the non-adaptive condition.

Table 2. Selected results from Instruction Reaction Questionnaire. Lower scores indicate a higher level of agreement with the statement.
Full size table

4 Discussion

The initial results of the pilot study were promising in terms of usability and favorable impressions of the ATTAC system. The usability data suggest that both versions of ATTAC were easy to use and that extraneous processing due to poor interface design was minimized. Importantly, participants in the adaptive condition reported favorable attitudes toward the instruction. The adaptive training may have fostered engagement due to the variation in scenario difficulty and feedback. Furthermore, the non-adaptive training condition may have also experienced frustration and suppressed generative processing because participants were only presented with their answer and the ideal answer with no additional explanation or feedback.

Since the pilot study, we have made improvements to the ATTAC testbed and have begun experimentation for a training effectiveness evaluation. Participants in the adaptive condition indicated that the feedback could be easier to understand and more useful during the pilot study. Previously, the feedback was relative to an arbitrarily selected ideal game plan (recall that there are more than one possible correct game plans in most cases). Now, we developed an algorithm to match a student’s game plan to the closest ideal game plan, such that the feedback that is displayed will be most similar to the student’s approach to the scenario. In addition to making improvements to the feedback statements, a training effectiveness evaluation is currently underway to establish whether adapting feedback and scenario difficulty is an effective approach in this domain. In this new experiment, we are comparing learning outcomes from three between-subjects training conditions, using the adaptive and non-adaptive versions of ATTAC, along with a traditional training condition. The traditional training condition acts as a control condition, such that participants do not interact with ATTAC and it simulates JTAC students’ current training experience with no access to a game plan development trainer. We will examine performance across the three conditions to assess if learning gains are greatest after performing the training scenarios with adaptive features, compared to non-adaptive features and traditional training. In a future experiment, we plan to tease out the effects of adaptive feedback and scenario difficulty to determine whether one approach is more effective than another or whether they have an additive effect on learning gains.

The research and development of ATTAC provides an important contribution to the training literature. Previous research [8] has discussed the need for more research to determine which AT techniques are effective for which domains. Therefore, the goal of our research with ATTAC is to apply techniques that have been effective in one domain to a new domain to determine whether these effects still hold. Game plan development is a rich decision-making task, and it represents a domain that has not been explored extensively in the AT literature.