Keywords

1 Introduction

In a complex world, infantry squads must be prepared to recognize patterns, make accurate predictions, select workable courses of action (COAs), and adapt to uncertainty [6]. Squad behaviors are context-driven and shaped by the experience of decision making, using pattern recognition and predictions to select courses of action. Conditions that produce problems that reward adaptive thinkers with success based on accurate predictions and timely decision making are a challenge to simulate for training. In dynamically complex contexts like the battlefield, effective performance embedded in force-on-force actions are unscripted and required unpacking to understand and use as performance feedback.

While military training developers have adopted many adult learning strategies and instructional technologies, virtual training simulations and live training have not filled the measurement gap. Traditional, easy to produce measures like killer-victim scoreboards fall far short of explaining how cognitive complexity was mastered by the most skilled practitioners of infantry tactics. The challenge for small unit trainers is making sense of human performance and then measuring the collective performance as training takes place in both virtual- and live training systems. In other words, understanding performance of high performing teams requires more than checking the box that Warrior Leader-, collective- or individual tasks have been performed [5]. Of greater importance to the trainee in the context of the infantry squad is the ability to benchmark team performance in order to focus and adapt training for performance improvements and developing accurate shared mental models. Direct measures of cognitive performance are inconsistent because outcome measures alone cannot be traced directly to decision skills.

The purpose of this paper is to describe the development and application of a prototype observation tool and observer training rubric that was developed to address this problem. The tool development was a key effort in a series of Squad Overmatch (SOvM) studies focusing on whether an integrated training approach (ITA) of instruction, simulation, and live training, could effectively demonstrate methods, tools, and strategies for improving team performance and creating resilience among squad members [1]. An observation rubric was developed as a set of guidelines and procedures for providing observable evidence of the type of behaviors and thinking skills that expert teams use to continually improve; and it was tested during the most recent SOvM 2015 study that focused on Tactical Combat Casualty Care (TC3).

2 Approach

We adopted the Simulation-Based Training (SBT) method as an organizing framework for designing and constructing the SOvM TC3 observation tool. SBT was developed and validated to create an instructionally sound, organizing framework for designing and delivering effective team training [2, 3]. It is effective because it provides an adaptive training approach - through seven linked elements - that use performance results from training exercises to tailor future exercises that accelerate development of skills.

The cycle begins with identifying competencies (knowledge and skills) and the associated learning objectives (LOs) based on the military mission essential task lists. Then, specific instructional strategies are derived from the LOs so that skill development is optimized. Training strategies enable defining the training simulation scripts and scenario events. Events are scripted into a scenario that will allow for performing the targeted skills. Diagnostic performance measures are developed and used to determine if the LOs have been mastered. Once diagnoses are defined, a structured after action review (AAR) can be constructed and delivered so there is a basis on which to improve in subsequent scenarios. To close the loop, performance information must be incorporated into future training sessions to ensure new training objectives build on what has already been learned.

SBT will be effective if the measures used during training exercises can be employed to diagnose learning and performance for the AARs, and are used for determining future training objectives in order to adapt training to learning requirements. Next, we describe the development of the prototype team performance assessment rubric and lessons learned following the SOVM TC3 demonstration conducted in October and November 2015.

3 TC3 Mission Tasks

We leveraged the existing research products from the SOvM 2014 demonstration to focus on the TC3 mission task components for 2015 [1, 5]. Working with TC3 Subject Matter Experts (SMEs) we focused on the combined challenges of handling both the tactical and casualty care responsibilities. The key TC3 mission tasks were determined to be:

  • Integrate Medical Planning with Tactical Plans

  • Provide Care Under Fire (while in contact or kill zone)

  • Perform Tactical Field Care (once area has been secured)

  • Manage Casualty Evacuation (priority of care/treatment)

Use cases were then developed for each of the major mission task areas; examples for Care Under Fire (CUF) and Tactical Field Care (TFC) are detailed below.

3.1 Care Under Fire

The Squad is conducting combat operations and has taken casualties. During the direct fire engagement, the casualty is reported. The wounded individual informs the Squad Leader of who is injured, the nature of the injury, whether the wounded individual is capable of Shooting, Moving, Communicating (staying in the fight), what treatment is required, and who must perform the treatment. The casualty is moved to safety or covered position where treatment is performed. When any of these elements of casualty information are missing, medical decisions are delayed. Tasks common to CUF are:

  • Squad Leader receives a report of a casualty

  • Squad achieves fire superiority and continues the mission

  • Squad Leader decides how to act based on information in the casualty report.

  • Treatment is based on information displayed on the Combat Casualty Card

  • Treatment is begun in a covered (safe) area until the area is secure

  • Treatment priorities are followed in accordance with SOP

  • Treatment is appropriate for the wound type

  • Combat casualty card is prepared and maintained as treatment progresses

  • Squad Leader submits contact/casualty report to Platoon

  • Squad identifies and secures a Casualty Collection Point based on situation

3.2 Tactical Field Care

The Squad is conducting combat operations and has taken casualties. They have achieved control over the situation. A Casualty Collection Point has been established in a secure area, where the Medic/Corpsman can manage the treatment. He determines the treatment needs and allocates resources to provide the necessary care. He organizes the CCP based on the casualty markings and wound type, to facilitate effective treatment and efficient transport to an evacuation point. He maintains contact with the Squad Leader and keeps him informed on the status of injuries. He recommends evacuation actions based on his awareness of the medical situation at the CCP. The Squad Leader allocates resources and prioritizes support of the CCP. If evacuation is necessary, he prepares/ submits the 9-line report. He keeps the Platoon aware of the situation in his area. Tasks common to TFC are:

  • Squad Leader receives a report of a casualty

  • Squad achieves fire superiority and continues the mission

  • Squad Leader decides how to act based on information in the casualty report

  • Treatment is based on information displayed on the Combat Casualty Card

  • Treatment is begun in a covered area until the area is secure

  • Treatment priorities are followed in accordance with TC3 standard operating procedures

  • Treatment is appropriate for the wound type

  • A combat casualty card is prepared and maintained as treatment progresses

  • Squad Leader submits contact/casualty report to Platoon

  • Squad identifies and secures a Casualty Collection Point based on situation

4 Competencies and Learning Objectives

Listed below are the key TC3 competencies (knowledge and skills) and learning objectives that were developed and verified with SMEs based on the use case analyses. Advanced Situation Awareness (ASA) and Resilience and Performance Enhancement (RPE) had been identified under the SOvM 2014 study; and we then added Team Development and Learning to the competency requirements for SOvM TC3.

Tactical Combat Casualty Care - while maintaining a tactical mission focus, manage combat casualties and provide the right treatment, by the right person, at the right time, to reduce the death by wound rate and facilitate evacuation to the appropriate medical treatment facility.

Advanced Situational Awareness - gain time or stand-off for disrupting or preventing an attack by detecting hostile indicators and anomalies and comparing them to a baseline of normality consisting of patterns of human behavior and environmental factors; this includes accurate assessment of enemy intentions or risks by using perceptual actions (orienting on target, observing patterns, interpreting patterns), selecting courses of action, and reporting actionable information.

Resilience and Performance Enhancement - monitor and maintain individual resilience by self-monitoring and regulating physical and cognitive resources in order to balance individual energy and attention resources when encountering operational stressors.

Team Development and Learning - exchange information, communicate clearly, support and backup team members, use initiative, provide guidance and priorities, and employ guided team self-correction during AARs to improve learning effective team performance behaviors.

5 Event-Based Scenarios and Measures

Based on SOvM 2014, the SOvM TC3 curriculum was implemented in three stages and modalities: knowledge acquisition in the classroom, skill acquisition and practice in virtual simulation-based training, and skill application in live, simulation enhanced training exercises. In each modality, a standard approach was used to frame the instruction based on learning objectives, guided learning, and checks on learning. This instructional design created a foundation for learning, scaffolding for skill development, and coaching for continual improvement. As the participants progressed through each stage, the aim was to produce near-term transfer that would support achievement at the next stage.

Event-based scenarios and the performance observation rubrics were developed from the use cases. A task sequence was constructed within each use case that was aligned with the doctrinal norm and based on SME inputs. In this manner we created a set of performance indicators that we reasonably expected would occur. The same story narrative and critical events were used in both the simulation and live training scenarios.

Using one of the live scenarios as an example, Table 1 shows how each of the competencies was linked to key chronological events and tasks in the scenario. Our goal was to present sufficient opportunities for squads to demonstrate performance in each of the four competency areas over the course of the scenario, which lasted about 45 min. In this scenario we generated a total of 97 instances for the 5 key events.

Table 1. Chronological scenario events, tasks and instances of key competencies
Full size table

Next we developed the observation measurement rubric based on the complete lists of expected behaviors derived for each of the events in the two simulation and two live training scenarios. The intent was to determine what task behaviors could be observed and whether they could be reliably assessed by multiple observers.

For each scenario, a short description of each event was presented on a single page. For example, Event 4 – Respond to Sniper Attack And Care Under Fire – requires the squad to react to sniper fire and respond to a civilian casualty while on patrol. Two women (roleplayers) plead with the squad for medical assistance which should cause the Squad to stop. A sniper (shots are heard) begins engaging the squad with effective direct fire, initiating “Respond to Sniper Attack.” One of the females is shot and falls to the ground, and this should cause the squad to attempt to render aid, initiating “Care Under Fire.” Then a squad member receives a gunshot wound to the arm, which requires treatment. The Squad should take cover, returning fire and maneuvering against the sniper as its immediate response. There is no follow-on attack by the Sniper.

Then, the behavioral indicators were listed in columns for raters to look for during the event. For example, above are listed the 18 instances of expected behaviors that should be triggered in the squad from Event 4 in Table 2. TC3 had five indicators, ASA had five indicators, RPE had three indicators, and TD had five indicators.

Table 2. Instances of expected squad behaviors
Full size table

The rubric in total consisted of 40 pages of data collection sheets. Table 3 presents an example of how the Team Development behaviors for Event 4 were presented as an observer checklist. A checkmark is placed next to the behavior if it was observed during the scenario.

Table 3. Observer checklist for team development behaviors
Full size table

6 Lessons Learned

Three criteria for an effective rubric were prescribed by the SOvM research team: the content and criteria must be valid; the rubric must be usable; and the rubric must result in reliable data collection. If these criteria were met, the study results would show that critical incidents were observable and the team performance could be assessed by trained data collectors.

Therefore, several weeks before the SOvM TC3 demonstration, a team of six experienced researchers were assembled for one-day rubric training. They possessed knowledge of the curriculum, had participated in observer training, and understood the training design. The training session was conducted to develop an understanding of how the tool would be used. Then the tool was tested during the SOvM TC3 demonstration over a 5 week period in October and November 2015 with four USA and three USMC squads.

During the two virtual simulation-based training scenarios, observers listened to squad member communications over headsets and in the simulation room, and were able to observe the squad member movements in the virtual world on a “god’s eye” view computer monitor. During the two live training scenarios, some observers listened to the squad member communications in a control room facility and observed a live feed of video and audio of the environment as the squad moved through it. Other observers were co-located with the squad and walked with the squad through the exercise at a safe distance. Observers used the checklist when they heard or saw squad members performing the indicated behaviors for each scenario event.

6.1 Content

Responses from observers led us to conclude that the use cases were sufficiently clear and complete to support the data collection requirement. Each use case was based on a tactical event. However, depending on the experience level of the Squad and their interpretations of various tactical cues, it was never possible to completely track performance as it was defined in the use case. For future studies we will update the content and structure of the rubric as we improve the prototype. Use cases will be adapted to a checklist sheet similar to the TARGET methodology [4] developed to increase the reliability of observing and counting instances of the behaviors during training and as part of the team AARs that follow.

6.2 Usability

Usability issues interfered with the ability to pinpoint squad performance and assign an accurate or consistent checkmark. We found data collectors experienced difficulty keeping up with the pace of the scenarios. The layout and form of the rubric use cases tended to interfere with the direct observation and recording of reliable data. This was an inevitable outcome based on the complexity of the scenarios, size of the squad, and size and structure of the rubric. Each Observer adapted and found alternative ways to capture the performance indicators that suggested learning had taken place. This included concept maps, field notes, transcripts of voice communications, and checklists of key events. To improve its usability and the functionality necessary for efficiently navigating through the tool and reducing workload when it is critical for the observer to attend to the squad performance, we will implement the rubric on a tablet. The tablet will initially store data and accumulate results for off-line data analysis. We intend to develop a set of Behaviorally Anchored Rating Scales (BARS) that are aligned with the key performance areas. These BARS include a feature for “tagging” weighted performance indicators within each BARS and generating a rating for use by analysts.

6.3 Reliability

We also learned that the researcher’s qualifications were not sufficient for using the rubric to observe squad tactical performance during virtual or live training. Data collection was hampered by a lack of understanding of the tactical behaviors being demonstrated by the squads. A more rigorous training and preparation period for data collectors/non-participant observers is needed. This would include interactive multi-media training solutions that would provide researchers and data collectors an opportunity to test the rubric using videos and observations of live and virtual training. Aligning data collectors with specific teams that make up the squad, i.e., the Squad Leader and Medic/Corpsman and individual the infantry fire teams is expected to reduce workload and increase reliability.

7 Conclusion

Infantry squad training is a “wicked” problem. Small unit training has assumed greater importance because operations in a complex world demand agile, adaptive thinkers who must make critical decisions under pressure. Training that focuses on tactical outcomes misses the point of developing the underlying critical cognitive skills. The SOvM TC3 research exposed a gap in current training that affects the development of high performing teams. The SBT approach enabled us to systematically develop and demonstrate linkages between mission tasks, competencies, learning objectives, instructional strategies, scenarios, and a prototype team performance observation tool. The rapid development and testing of the prototype observation rubric enabled researchers to determine how to link task performance with cognitive skills such as decision making and problem solving. Lessons learned enabled us to chart a path forward for a more valid and reliable automated, transition-ready data collection tool.