Augmented Reality for Tactical Combat Casualty Care Training

Abstract

Combat Life Savers, Combat Medics, Flight Medics, and Medical Corpsman are the first responders of the battlefield, and their training and skill maintenance is of preeminent importance to the military. While the instructors that train these groups are exceptional, the simulations of battlefield wounds are extremely simple and static, typically consisting of limited moulage with sprayed-on fake blood. These simple presentations often require the imagination of the trainee and the hard work of the instructor to convey a compelling scenario to the trainee. Augmented Reality (AR) tools offer a new and potentially valuable tool for portraying dynamic, high-fidelity visual representation of wounds to a trainee who is still able to see and operate in their real environment. To enhance medical training with more realistic hands-on experiences, we are working to develop the Combat Casualty Care Augmented Reality Intelligent Training System (C3ARESYS). C3ARESYS is our concept for an AR-based training system that aims to provide more realistic multi-sensory depictions of wounds that evolve over time and adapt to the trainee interventions. This paper describes our work to date in identifying requirements for such a training system, current state of the art and limitations in commercial augmented reality tools, and our technical approach in developing a portable training system for medical trainees.

1 Problem and Motivation

Combat Life Savers, Combat Medics, Flight Medics, and Medical Corpsman are the first responders of the battlefield, and their training and skill maintenance is of preeminent importance to the military. While the instructors that train these groups are highly rated medics, most simulations of battlefield wounds are typically very simple and static. These might range from simple moulage to show some characteristics of the wound (essentially rubber overlays with fake blood painted on) to a piece of tape inscribed with the type of wound, with no physical representation of the wound itself. In many field-training exercises, each soldier carries a “casualty card” that, if they are nominated to be a casualty, tells the soldier/actor how to portray a wound named on the card. The card also tells the trainee what wound to treat.

While casualty cards themselves are relatively simple to use, the simplicity of the presentation often requires the instructor to describe the wound or remind the trainee during an exercise about the qualities of the wound that are not portrayed, including how the wound is responding to treatment. To simulate arterial bleeding, an instructor may spray fake blood on the moulage. This effort by the instructors is there to compensate for the low-fidelity simulation, and takes away from time that could be spent providing instruction. While relatively simple, even these simulations take time and effort to create, set up, and manage, before and during the training exercise. The preparation before each exercise and the overall compressed training schedule of a training course means that trainees get limited hands-on practice in realistic settings.

Augmented Reality (AR), especially the recent boom in wearable AR headsets, has the potential to revolutionize how Tactical Combat Casualty Care (TC3) training happens today. Augmented Reality can provide a unique mix of immersive simulation with the real environment. In a field exercise, a trainee could approach a casualty role-player or mannequin and see a simulated wound projected on the casualty. The hands-on, tactile experience combined with the simulated, dynamic wounds and casualty response has the potential to drastically increase the realism of medical training. To enhance Army medical training with more realistic hands-on training, we are working to develop what we call the Combat Casualty Care Augmented Reality Intelligent Training System (C3ARESYS). This paper outlines our work to date in identifying how AR tools could fit into, and augment, current US Army medical training. We first briefly cover the types of training that occur in the standard 68 W (Army Medic) course, and the types of injuries on which they are trained. We also briefly describe the task analyses we conducted related to medical training. Together these serve as a basis for identifying elements of training including some requirements that an AR-based training system would need to meet. We then describe our C3ARESYS concept, our anticipated approach, and challenges to developing and evaluating the system. In this work, we have evaluated current AR technologies on the market relative to the requirements we identified. While there are significant limitations to current AR systems, our approach works within the current limitations of current AR technologies, while anticipating future advances that we could leverage.

2 Background: Augmented Reality

AR typically refers to technology that allows a user to see a real environment while digital information is overlaid on that view. Heads-Up Displays (HUDs) such as in cockpits or fighter pilot helmets represent early work in AR, though typically these overlays do not register with objects in the environment. Later work includes registering information with the environment for tasks ranging from surgery, to machine maintenance, to entertainment such as the addition of AR scrimmage lines in NFL football games, or the highlighting the hockey puck in NHL games. See [1, 2] for thorough surveys of augmented reality. As mobile devices (phones, tablets) have become more capable, augmented reality has become more mobile, with game examples such as Pokemon Go™, which provides an “AR view” option to show 3D renderings of game characters overlaid on top of camera views. More recently, wearable AR hardware has tended to focus on see-through glasses, visors, or individual lenses that allow for computer-generated imagery to be projected hands-free, while allowing the user to see the surrounding environment directly. Additionally, more sophisticated AR projections are registered with the real environment, where digital objects can be placed on real tables or seem to interact with real obstacles. It is these latter wearable, spatially aware technologies we focus on.

While the technology continues to improve, there are several limitations with current AR systems that have real implications in training, including limited computer processing power and limited field of view. We will cover these limitations, and their impact on training, throughout this paper in the context of a medic training application.

3 Related Work

The main method of hands-on medic training is through simulation. This often focuses on hands-on physical simulants, such as moulage overlaid on a simulated human casualty, either a mannequin or a human playing the role. Some training facilities use instrumented mannequins that can bleed, exhibit a pulse, and even talk. However, these systems, including the computers that enable them, are expensive, not very portable for field training and are not at every training site. There are also physical part-task training simulators, such as tools to teach proper tourniquet application that require purpose-built hardware. Examples include a computerized portion of a fake leg with fake blood (e.g., TeamST’s T3 Tourniquet Task Trainer [3]), or instances with metaphoric cues – lights that go out when the tourniquet is properly tightened (CHI Systems’ HapMed Tourniquet Trainer [4]).

There are also examples of digital simulations for training medics. For example, ARA’s virtual reality medical simulation (“HumanSim: Combat Medic” [5]) provides game-like ways to view wounds and apply treatments. Rather than the trainee physically performing a treatment, this environment focuses on the procedures. The trainee in uses the mouse or keyboard to select some treatment; the game visuals then show that treatment happening, along with the effect of treatment. Instead of naturalistic cues about the wound or the casualty (e.g., such as feeling a pulse by putting fingers on a wrist), the game provides metaphoric cues (such as displaying the pulse on the screen). With more portable and more capable technology, Augmented Reality is starting to be used in medical training, including Case Western Reserve University using Microsoft’s Hololens™ for anatomy training [6], and CAE’s VimedixAR ultrasound training system [7].

4 Domain and Requirements Analysis

Wounds and Procedures.

To help define the scope of the system, we surveyed current training recommendations, manuals, and other TC3-related publications, and also interviewed instructors to get a broad view of medic training. Findings from recent conflicts identify particular distribution and mechanisms of wounds [8, 9], which are summarized in Table 1 below. More specifically, the Army Medical Department (AMEDD) Approved Task List (2016) gives the assessments and treatments that a trainee must know to become a medic. The TC3 handbook [10] also provides details of the types of injuries seen in recent conflicts, along with treatment procedures.

Table 1. Injuries in recent conflicts (from [8])

Along with identifying injuries, we worked to identify and document treatment procedures for these injuries using task analysis methods. We focused on three main sources for our task analysis: published documents (e.g., field manuals and related publications [9, 10]), interviews with SMEs, and observations of medic training. We conducted interviews with subject matter experts on our team, with instructors at the Pennsylvania National Guard Medical Battalion Training Site (MBTS), and with a medic at Fort Bragg, and also observed training at MBTS. These interactions helped us understand the spectrum of tactical combat casualty care, including the types of training that occurs in Army medical training, and details on particular treatments.

Along with scoping, the goal of our analysis was to identify specific wounds and related procedures that medics train for, so we could identify how an AR system could contribute to training. We looked broadly at medic training, and then looked more narrowly at selective examples to assess the level of detail required for an AR system. The Army’s Tactical Combat Casualty Care training manual [10] includes step-by-step instructions about procedures. There are also previously published task analyses of treatments such as cricothyroidotomy [11, 12] and hemorrhage control [11].

For our purposes, we needed to identify not just the treatment procedures that a medic would perform, but also what the medic would perceive about the casualty and the wound to be able to perform some procedure. For this reason, our analysis was in the style of Goal-Directed Task Analysis (GDTA) [13], which captures the hierarchical nature of goals and tasks, along with decisions that must be made to perform the tasks, and the situational awareness requirements needed to make those decisions. Figure 1 shows an example of GDTA applied to a medical task. The uppermost goal is to perform an airway/breathing/circulation assessment, and a sub-goal is to perform a breathing assessment. Rectangular boxes connected by lines are the medic’s goals and sub-goals. The rounded nodes beneath the task nodes contain decisions that must be made in order to perform the tasks. The rectangle beneath the decision identifies the situation awareness requirements needed to make those decisions. Per Endsley’s approach to situation awareness (SA) [14], the three levels include: Level 1: immediate perception; Level 2: relating those perceptions to goals; and Level 3: projecting the current state into some future state.

Fig. 1.

Example Goal-Directed Task Analysis for assessing casualty breathing.

While many of these procedures are documented, not all of the documents or prior analyses included all of the elements that we needed for a GDTA. Thus, our effort included combining data from different sources to construct a more comprehensive task model with the level of detail needed to build a training system. For example, our task analysis for the process of controlling bleeding is a consolidation of the Cannon-Bowers, et al., task analysis of Hemorrhage Control [11] and the task Apply a Hemostatic Dressing task from the Soldier’s Manual [10], supplemented with other related treatments from the Soldier’s Manual and interviews with SMEs. The medical paper provided a rough outline of the task, along with some decisions to be made and SA requirements to perform the task; the Soldier’s Manual provided a more detailed breakdown of the subtasks involved, but both needed additional detail for our design purposes.

This analysis has served a few purposes toward defining the requirements for a building an AR-based training system. First, the analysis captures the steps necessary to perform a treatment task, which can serve as the basis for an expert model to compare against trainee actions in an assessment process. Second, this same model can be used as the basis for automatically recognizing trainee actions, based on the atomic actions identified as the sub-tasks in the GDTA. Third, the Level 1 Situation Awareness Requirements define the cues that need to be present in a training environment to help the trainee identify the injury and make decisions about treatment. (Levels 2 and 3 are products of the trainee’s cognition but could also be used as part of assessing the trainee’s skills or to provide additional feedback to the trainee.)

Types of Training.

A good deal of training occurs in classrooms, but our focus was on hands-on, scenario-based medic training. Sometimes called “lane training,” this type of training aims to cover different conditions and settings that medics will have to work in. At MBTS, the scenario-based training included dismounted patrols where the trainees had to care for wounded soldiers while under fire; indoor trauma aid stations where trainees had to triage, treat, and evacuate casualties; and mobile care where the trainees had to perform care while in casualty evacuation (CASEVAC) vehicles. In addition to the stress of treating casualties with life-threatening wounds, most of the scenarios included external stressors such as tight time schedules, extreme noise, or enemy fire to make the scenario more realistic to the trainee.

Role of Instructors.

In addition to the wounds and procedures for treating them, a critical part of Army medic training today is the vital role of the instructors. Their presence, instruction, and participation during scenario-based training are especially important for a number of reasons. Because the baseline presentation of wounds is extremely simple and static (e.g., painted moulage or in some cases even less detail, such as a piece of tape with “amputation” written on it), the instructor must also provide to the trainee information about the wound and overall condition of the casualty – what it is, how it starts out, and how it changes over time. This may include giving verbal descriptions of the wound (“this is an amputation below the knee”), supplying vital signs that are not present in the casualty simulation, and describing the behavior of the casualty (“the patient is moaning in pain”). The instructor may also squirt fake blood on the wound to simulate arterial blood flow. Instructors are of course observing the trainee’s treatments and other behavior as a way to assess trainee mastery of the tasks and performance under pressure. Instructors also inject dynamics into the training scenario, changing the difficulty in response to the trainee’s behavior. They also provide instruction and direction during the scenario and lead after-action review sessions.

Technical Requirements.

Based on the requirements given by the customer and our own analysis, we developed a list of stated and derived technical requirements that would help us define an AR-based training system to fit how medic training is currently done. These requirements cover a variety of categories such as wound portrayal, hardware requirements, trainee interface, and instructor interface. Table 2 below provides a subset of the roughly 40 high-level requirements we identified. These requirements guided our design of the system overall, which we cover in the next section.

Table 2. Requirements for outdoor lane training use (subset)

5 Technical Approach

The C3ARESYS concept focuses largely on the question of training fidelity. The centerpiece is the use of AR technology to enhance the visual aspects of training – portraying wounds in ways that not only look more accurate but also exhibit the dynamics of real wounds, including their progression over time and their responses to treatment. Because training is a multi-sensory experience, our approach leverages the moulage that is used today to provide the haptic sensations of wounds, while also exploring how it might be extended to provide richer training experiences. Figure 2 illustrates our C3ARESYS concept.

Fig. 2.

Concept (adapted from US Army photo).

Combat casualty care augmented reality intelligent training system (C3ARESYS)

Given the complexity of potential models, the broad range of wounds, and the broad array of treatments performed by trainees, we chose to focus the design and development on the core AR modeling elements. This includes the visual display of wounds (and their dynamics), effective registration of the wound models on moving casualties, as well as the tactile portrayal of wounds and other casualty information. Other future extensions could include automated treatment recognition and intelligent tutoring. In making this design choice, we must include an instructor in the loop to track the trainee’s actions and provide feedback, but we aim to give the instructor tools to help him or her perform these tasks.

5.1 System Design

C3ARESYS is composed of a number of technologies focused on enhancing the multi-sensory training experience. A high-level system view is given in Fig. 3. The main software component of C3ARESYS focuses on Dynamic AR Modeling. This component deals with producing a multi-modal rendering of a wound with appropriate cues relevant to the trainee. The Casualty/Wound Tracker determines where the wound (and related visual cues such as blood flowing from the wound) should be placed based on sensing the position of the casualty, moulage, and other cues. The Multi-Modal Rendering Engine renders visual and other wound effects such as the wound changing visually over time (e.g., based on treatments), audible and tactile cues associated with the wound (e.g., breathing sounds, pulse) based on parameters stored in the Multi-Modal Wound Models database. The Physiology Modeling module determines how the wound and the physiology of the casualty generally would evolve based on interventions by the trainee (or lack of intervention). We expect that the Physiology Modeling module will leverage current tools available, such as BioGears [15] or the Pulse physiology engine [16]. The input to the Physiology Modeling engine is a specification casualty’s condition and of specific treatment (e.g., saline drip at 50 ml), which would then result in changes to physiological parameters of the casualty model (e.g., increased radial pulse). These inputs would come from an instructor who is observing the trainee’s actions and entering the actions into an instructor interface (see below). The outputs of this engine (i.e., the collective set of parameters of the casualty model), combined with the Wound Models database, tell the rendering engine what to portray.

Fig. 3.

High-level view of the C3ARESYS architecture.

The outputs of the Dynamic AR Modeling component will be rendered in a few ways: (a) visual and audio output through the AR systems worn by the trainee(s) and the instructor(s); (b) commands sent to the instrumented moulage to produce tactile cues; (c) instructions for the casualty. If it’s a human volunteer, he or she might be told how to behave or what to say to portray the wound effects accurately (e.g., moaning in pain, being non-responsive, etc.). If the casualty is a mannequin, these instructions could go to a system that plays back audio recordings or generates speech from text. The system could also project AR overlays on instruments the trainees use, such as overlaying an animation on top of the blood pressure gauge to show the representative blood pressure of the casualty rather than whatever the blood pressure cuff would render from a live casualty or even a mannequin. Additionally, the Instructor’s view through the AR glasses could include ground truth data that the trainee doesn’t see, to help the instructor keep track of the condition of the casualties, for example.

The Instrumented Moulage component is standard moulage that we plan to augment in a few ways. The use of moulage by itself serves a few purposes. First, from an AR registration perspective, it provides the visual anchor to tell the AR system where to draw the wound. Without having some reference point, the AR visualization would float around independent of the position of the casualty. Second, it provides a reference point to the trainee when using AR, both to tell the trainee where to look and also to give them a low-fidelity representation of the wound even when AR system is not tracking it. Third, it provides the tactile experience of the wound that AR by itself cannot provide. Typical user interactions with pure AR are, at this point, not rich enough to provide a haptic experience, and technologies like haptic gloves are still quite nascent in their development (not to mention that trainees typically wear surgical gloves during training). At least with today’s training using typical moulage, the trainee gets some simulated version of how the wound feels.

In developing the Instrumented Moulage, we plan to explore the use of actuators (small motors) and sensors to provide an enhanced experience for trainees. We expect that the system could activate the moulage with specific patterns that simulate, for example, the casualty’s pulse at the wrist or the feel of blood flowing. Sensors in the moulage could be used to identify treatments the trainee applies. The Instrumented Moulage system could be connected wirelessly (e.g., via Bluetooth) to the rest of the system.

Lastly, the Instructor User Interface provides a way for the instructor to participate in the training session. We envision that this interface could include an AR viewer to get views of the casualty, including the trainee perspective and an instructor-only, ground-truth perspective. This could be supplemented with a hand-held tablet-like device for making changes to the scenario, tracking trainee actions, or taking notes on trainee progress. Such a system would also help the instructor manage multiple training sessions simultaneously. These tools in concert could also be used to facilitate after-action reviews.

6 Challenges with Augmented Reality

There are several challenges with using augmented reality for practical applications, including medical training. We break down these challenges into four categories: field of view, visual tracking/processing power, form and fit, and user interaction.

Field of View (FOV).

One of the most apparent when putting on wearable AR technology is the limited field of view. Most wearable technologies average around a 35° diagonal field of view. Besides taking away from an immersive experience, users often have to search around to find any AR objects placed in a scene, and large objects often get cut off by the FOV restriction. Some applications will guide the user with arrows or other indicators for where to look, but these can also distract from the user experience. Our use of moulage as a visual marker is in some ways an accommodation to this limitation. If the trainee looks away from the moulage, to outside of the core projection FOV, the digital wound model will disappear from the trainee’s view. However, the moulage will remind the trainee where the wound is, and provides at least a lower-fidelity version of the wound.

Processing Power and Tracking.

For AR applications where objects need to be registered with a location in space, those objects need to stay in place reliably while the user moves around. This is especially true in medic training, where the trainee is constantly moving around the casualty, and may even move the casualty around to perform assessments and treatments. Reliable tracking is a function of the system sensing and processing the environment fast enough as the user moves relative to the target to keep the digital object locked in place. Vision-based tracking systems also require good lighting to be able to track the environment effectively.

In our first phase of work, we implemented some simple versions of marker-based tracking as a feasibility assessment of our design as well as a way to get hands-on experience with existing AR tools. Our initial testing used Microsoft HoloLens. Because there are several limitations to what the HoloLens provides to developers (in particular, no explicit object tracking), we had to add some extensions to be able to track these markers. We explored using different 3^rd-party tools including OpenCV and Vuforia™ to recognize and track visual markers. Our first pass used OpenCV implemented on the HoloLens, using QR-style markers for tracking. The system was able to track the marker as the user moved around, while keeping the marker in view and while moving the casualty’s arm side-to-side. However, movement induced noticeable lag when tracking the markers and trying to keep imagery in place. Figure 4 shows a version of the system using Vuforia running with the HoloLens. This was faster than OpenCV, but still had some lag issues. We have also done some hands-on testing with Osterhout Design Group’s R7 glasses, with similar results with moving targets.

Fig. 4.

Visual marker (top), and wound overlaid on the marker (bottom).

Form and Fit.

The recent boom in AR wearables has opened many doors for how AR technologies might be used. However, the form that these systems take is often a bulky headset made of seemingly delicate components for the price. Many designers choose to put all the sensors, computing power, and power sources on board, which results in more weight carried on the user’s head. For example, the current $3000 Microsoft HoloLens seems too fragile for military use and is too bulky to fit under a standard Kevlar helmet. Other designers go the route of having a separate connected device to provide battery and processing power (e.g., Meta2, Epson Moverio BT-300), thereby allowing the headset to be lighter.

Ruggedness is also a question. Medic trainees operate in many environmental conditions with a bunch of other gear. They might be treating casualty in heavy rain, or diving for cover to avoid (simulated) enemy fire. These uses risk damaging or breaking what is (so far) quite expensive equipment. Instructors may be unwilling or unable to spend money on such fragile equipment. This may limit many of the training use cases to those where the conditions are more suitable to the device. Some manufacturers are starting to address the issue of being tolerant to different environmental conditions (e.g., ODG R-7HL), but this is not a universal concern among hardware providers.

User Interaction.

Perhaps more fundamental than the above is the lack of compelling interactions with AR objects. Processing power continues to increase year after year, as does battery size and efficiency, which will also contribute to more efficient, more compact devices. However, current user interaction tends to use traditional computing metaphors. The current state of the art for AR systems lies three main areas: speech interaction, head tracking to draw a cursor where the user is looking, and limited gesture recognition to capture simple interactions such as pinching or grasping objects. Speech recognition can be useful in the right conditions, but its utility is limited in our C3ARESYS application. Using one’s head as a pointing device can become tiring, especially if the objects to interact with are small and require precision. Gesture recognition is often in the form of making selections or dragging objects around (Microsoft’s “air tap”) or giving commands (Augmenta’s iconic hand shapes). Hand tracking and gesture recognition could be compelling and useful if related to objects themselves such as grasping and manipulating them naturally, but recognition of these inputs needs to be highly accurate, otherwise the user is left frustrated at the poor interaction. Some systems use hand-held controllers to manage user input, but these add additional gear that the user has to hold to operate, which takes away from the hands-free nature of wearable AR and does not fit with this medic training domain. None of these typical types of interactions are especially compelling to medic training; instead, we need ways for the trainee to interact directly with the AR wounds, which could include domain-specific interactions such as filling a cavity with gauze or putting pressure on the wound to stop bleeding. We will continue to explore interaction features such as hand tracking as the technology continues to improve.

Feedback to the user is also another area in which AR technology is lacking. Visual and audio feedback is typically the norm, as expected. However, as mentioned earlier, this hands-on medic domain relies on tactile sensations and haptic feedback to be realistic. Medics will feel for a pulse and will palpate a wound to assess its condition. Haptic gloves could be a solution, but current technology is fairly rudimentary, and they require their own power sources and computing. As mentioned, this is another reason we have chosen to stay with moulage: to provide the tactile sensation that AR currently lacks.

7 Summary

We have described the motivation, requirements, and design of a system we call the Combat Casualty Care Augmented Reality Intelligent Training System (C3ARESYS). The motto “Train as you fight” that is ubiquitous in the military is a main driver – working to improve the fidelity of hands-on medic training. Whereas today’s trainees at best experience static moulage as a representation of a wound (and very often they are presented with much less than this), AR has the potential to provide a more representative multi-modal training experience. However, as we describe above, there are many limitations in current AR technology that have forced our hand in designing a system for near-term use. Field of view, processing power, fit, form, lack of ruggedness, and limited user interaction all have very tangible effects on our system design and how readily such AR technology can be used in the field. We have tried to make design decisions that will enable us to build a prototype system today, while also being able to take advantage of AR technology as it improves in what is currently a very dynamic marketplace.

We have presented only a design here. Our next step in this work is to develop a working prototype that can be used for a limited set of treatment procedures. This will include the trainee’s experience and tools for the instructor so that we can mirror the current instructor-in-the-loop training paradigm. Once we have developed a prototype system, we aim to conduct hands-on evaluations with medic instructors and trainees to get their feedback.