Medutainment-Based AR Rally: Disaster Medical Learning Tool for Citizens

Abstract

Various attempts have been made to teach first aid to the general public on how to deal with disaster medical situations, such as sending firemen to schools to give talks and hand out instructional pamphlets. But it is hard to determine whether the message is really getting through, or whether the students attend just because they have to. Now we have come up with a novel IT-based interactive educational tool based on a medutainment (medical edutainment) model for teaching basic first aid techniques to the public. Our initial assessment shows that this new medutainment-based approach is very effective for teaching first aid and disaster medicine to large numbers of people in situations where cost-effective experience-based training is impractical or not available.

1 Introduction

Here we have developed an IT-based interactive educational tool tailored for young children and students based on medutainment (medical edutainment) [1] that entertains while teaching emergency medicine [2] (see Fig. 1). The application is called AR Rally and runs on a tablet computer or a smartphone. Players advance by solving quests (questions) presented on the tablet computer. Posters relating to the questions are put up the classroom or space where AR Rally is played. Key features of the prototype system include recognition of illustrations on the posters by the tablet’s built-in camera, overlay representation using augmented reality (AR) functions, and game playing using touchscreen functions.

Fig. 1.

AR Rally game

In the event of a mass casualty incident, the city or local community sets up special medical aid station and conducts triage, the process of determining priority of patients’ treatments based on the severity of their wounds. Minor injuries may be treated onsite, while a hospital or other medical facility is designated as the destination to transport victims with moderate or more severe injuries. There may be a good number of locals or bystanders who can lend a hand if they have a basic understanding of first aid and disaster medical care, but unfortunately not many people have this basic knowledge. There is thus a critical need to improve people’s knowledge of basic emergency medical care and first aid procedures. It was this insight that motivated the present project to build a basic disaster medical education tool to help spread first aid knowledge and awareness among Japan’s citizens.

The core content focuses on how to staunch bleeding (hemostasis), body position management, and other first aid procedures. It’s hard to learn these techniques from a textbook, and the procedures themselves are not inherently interesting to most people. First aid is not the kind of thing that people flock to and are eager to learn. So one of our primary objectives in developing this tool was to provide people with basic disaster medical care knowledge in a light, digestible way that reduces any onerous sense of obligation.

As a model, we took the edutainment type handheld device used to guide people through museums [3, 4]. By interacting with the touchscreen display on these devices, people learn about the exhibits through content that is both educational and entertaining. This is very effective for two reasons. First, these devices effectively convert the passive learning process of simply viewing exhibits in a museum into a dynamic experience. Learning first aid is similarly a passive process. Lessons tend to be shrugged off by those who are really not that interested, so harnessing this edutainment approach could bring great benefits. Second, the handheld devices offer a way of building an interactive relationship between exhibits and viewers while moving through three-dimensional space. In emergency medical care, the procedures and the transport of injured cannot be separated from 3D motion and movement. This dimension is also critically important to enhance the edutainment value for younger elementary and middle school students. Real-time scoring and evaluation, and log functions are also important elements of medutainment applications.

The educational content of AR Rally primarily focuses on disaster medical care and medical aid stations in the form of a medutainment application tailored for the higher grades of elementary school and middle school. More specifically, AR Rally consists of a multiple-choice quiz (including use of an AR marker) regarding disaster medical care as well as a procedural game. The student’s score and an explanation are provided immediately after the student gives her answers. Total results are tallied at the end to motivate the student to try harder. Note that while the story line is identical, two versions of AR Rally were developed—one for elementary students and the other for middle school students—with somewhat more advanced sentence structure and review quiz for the middle-school students.

We initially planned to implement AR Rally to run on iOS, but the programming was done in Unity [5] so it runs on both iOS and on Android. The application can thus be downloaded and run on tablet computers as well as on a variety of different smartphones, a capability that we think should help event organizers disseminate the software.

2 System Structure

Figure 2 shows an overview of all the chapters in AR Rally. Basic content is covered in Chapter 5. Note that two issues are addressed in each chapter. So, for example, Chapter 1 covers the game explaining the basics of disaster medical care and clothing-related considerations. Chapter 2 deals with hemostasis techniques (i.e., ways to staunch bleeding). Chapter 3 is concerned with body position management. Chapter 4 deals with transport-related issues. And finally, Chapter 5 considers triage and includes a final review quiz. All of the content can be covered in about 20 min, and results are displayed for each question.

Fig. 2.

Overview of AR Rally chapters

Let us next consider the features of AR Rally that make the content interesting even for students who have never shown much interest in disaster medicine. First, the students obtain information and explanations not by reading text, but by playing a game, so naturally this is much more interesting. Second, where teaching is involved, a quiz format is used which again engages and holds the attention of the students. The quiz excites interest by putting up posters to represent question options, having the students choose a question as an AR marker, and by displaying results as interactive annotations in camera images. The hand skill games fully exploit the tablet computer’s capabilities and help the student retain memory by manipulating and moving figures on the tablet using a finger tip for input. Finally, the review quiz at the end of the course helps students retain memory of the content.

Let us next take a closer look at three key features of AR Rally that help sustain interest in learning and remembering disaster medical care.

2.1 Opening: AR Triage Game

Writing a textbook about the procedures and significance of disaster medical care is easy, but presenting the content in a way that is retained in the mind of the reader is extremely difficult. First we applied gamification—game-design elements and game principles—to explain the concept of triage, which is at the very heart of emergency medical care. Triage involves the assignment of degrees of urgency to wounds or injuries to decide the order of treatment when there is a large number of casualties. At the site of an actual emergency, a paramedic or emergency medical technician writes triage information on a special tag called a triage tag, which is attached to the victims’ arm. Victims are then sorted and moved to different areas as indicated by a color-coding scheme on the tags: green for minor injuries, yellow for moderate injuries, and red to severe injuries.

In the game version, first the floor plan of the medical aid station is captured by the tablet camera, AR is used to render the station in simple 3D which is overlaid with actual images, and then the game begins. As one can see from the screenshots in Fig. 3, the victims are brought into the medical aid station and color triaged by the system. Players merely select the victims one by one and transports them to the appropriate color-coded destination. Rather than launching into a detailed description of the procedure, a game is used to teach students the nature of triage through a game experience. A basic objective of the game is to teach children that in the event of a mass casualty incident when resources are stretched thin, even assistance from ordinary bystanders can make the difference between life and death.

Fig. 3.

Opening triage game (Color figure online)

2.2 AR Quiz

Here we will consider the AR-based interactive quiz scheme for teaching emergency medicine. The quiz is played on a tablet computer. First, several posters are put up on the wall at the event site highlighting the questions that will be asked (see Fig. 4). Or several questions could be combined on a single poster as illustrated in Fig. 5. A question is presented by the tablet, and the student has to choose the most appropriate poster image dealing with the question. The key here is that, if the student picks the wrong image, then augmented reality superimposes the question over the correct image. This sustains the student’s interest while at the same time teaching the correct answer. If the student chooses the correct image, then the question returns to the tablet, and the rally continues. The opening game is similar, but the rally proceeds while going back and forth between real space and cyber space, and this too sustains the younger children’s interest.

Fig. 4.

AR Quiz

Fig. 5.

AR Quiz posters

2.3 Hand Skill Games

Trying to teach first aid techniques using a textbook is exceedingly difficult. By fully exploiting the functionality of the tablet computer, we made this material much more transparent and easy to remember by using hand and finger gestures to manipulate 3D model figures on the screen. The 3D manikin-like models mimic the actual movements of people. We used the models to illustrate hemostasis (how to staunch bleeding) and body position management. Figure 6 shows gamified results for the body position management.

Fig. 6.

Body position management game

Figure 6 illustrates how to manipulate a figure lying on her back into the recovery position using finger gestures through a sequence of four steps. The actual maneuver is done by following the trajectories of the arrows shown in the figure with one’s fingertip, and this causes the 3D model to actually change position and move in that direction. If at first the student does not succeed he can try again, and in the process learns how to put an injured person in the recovery position. Basically this is what we want the students to learn: the step-by-step procedure for putting someone into the correct position. While it’s easy enough to glace skim over the procedure in a book, we think that our approach—exploiting the benefits of the touchscreen tablet and using one’s finger to repeatedly perform the maneuver—is far better retained in the memory and also entertaining.

Emergency transport is covered in Chapter 4. In Chapter 4-2, we incorporated actual transport experience into the course by having the forth graders carry a 30-kg manikin on a stretcher. The idea was for the students to learn from experience what it is actually like to move a victim—a manikin that weighs about the same as a human child—without causing further injuries to the victim. For this exercise, a tablet computer is placed on the stretcher where its acceleration sensor measures up and down vibrations and jolts. This was used to simulate the shaking felt by the victim. High scores were given to students who could convey the stretcher with the least amount lateral shaking based on the assumption that this would result in less jostling and jolting of and injured victim. Obviously, this approach is not very accurate and does not guarantee no further injury to the victim. Rather than striving for accuracy, our primary goal was to promote learning while sustaining a feeling of tension and entertainment by incorporating real-life experience and a mechanical real-time scoring system (see figure on the right in Fig. 7).

Fig. 7.

Event scenes

3 Events and PDCA Cycle

3.1 Events

Up to now, about 780 students have participated in the stretcher-carrying exercise: roughly 600 students over four days at a shopping center, and 180 students during a one-day open house at AIST labs. Here’s a brief overview of the event held at AIST labs.

Our assumption in developing this program is that it will be used as a training exercise at community medical aid stations—that is, to introduce people in the community to the local medical aid station. This prototype system is intended to simulate what one might experience in walking around a typical disaster site. One of the advantages of implementing the system on mobile devices is that the course can be tailored to a variety of different situations: people can learn disaster training procedures in their spare time, or the course can be used for training at actual medical aid stations, and so on. Figure 7 shows photos from one of the demonstration events. The participants at this event were about equally divided between children who came on their own and children who came with a parent.

Figure 8 shows the grade distribution of the student participants. A little more than half of the participants were third graders and older, so there were more younger children than we anticipated. The younger children worked through the questions with help from mom or dad.

Fig. 8.

Breakdown of participants

3.2 Evaluation and PDCA Cycle

Further refinement the course content of the prototype system will require qualitative assessment of the effectiveness of the hands-on experiential learning, comprehension and memory retention, and usability. Subjective surveys are normally used to gather this kind of information, but in our system the basic metrics are automatically compiled by the interaction of the subjects with their tablet computers. In other words, the tablets can be programmed to record correct/wrong answers on quizzes, time required to answer, the time it takes to read commentary, the time it takes to perform hand skill games, and so on. Moreover, by simply entering a student’s grade in school, the program automatically determines the approximate difficulty of each question, the degree of interest the student will show in the commentary, and the ability of the child to understand and participate in the learning games. We are thus able to obtain useful feedback for all of the quiz questions based on quantitative data. Figure 9 illustrates the type of feedback that can be generated.

Fig. 9.

Analysis of last quiz

The figure shows analytical results for the final review quiz in Chapter 5-2. Note that this is the last chapter in the course, and thus is a review quiz of all the questions covered by the course in order to consolidate the information in memory. The figure reveals answer times and accuracy rates for Questions 3 and 5 broken out by grade in school. The vertical axes show answer times (in seconds) for graphs on the left and accuracy rates (in percentages) for graphs on the right. The horizontal axes show grades in school of the subjects. One can see that there was considerable variation in the answer times and rather poor accuracy rates for Question 3 shown at the top. On the other hand, there was much less variation in the answer times and better accuracy for Question 5 shown at the bottom. We think that Question 5 was an appropriate review question, but since log functions were taken for all questions asked, we should be able to enhance the quality of the question over the course of repeated events.

4 Conclusions and Future Work

This work describes the development of AR Rally, a disaster medicine educational application tool that runs on tablet computers. Use of the tablet permits students to learn by doing as they move around, and also exploits a number of features that create interest in students and also helps them remember emergency medicine procedures. Some of these key features include explanation through games, quizzes based on augmented reality (AR) functions, and memory retention through repeated finger movements and gestures on the screen.

Up to now, the prototype system has been demonstrated at two public events—one event held over four days at a shopping center and the other event conducted during a one-day open house at AIST Labs—but more events are planned. The course content will be further enhanced and refined during three forthcoming events (a total of ten additional hands-on trial days) at science and technology museums and other venues.

It is not so much the end users as the sponsors and promoters of the demonstration events that will promote the widespread use of AR Rally. We will certainly do our part by making the software widely available for download so it can be used to disseminate disaster medical knowledge in science museums and in population centers from villages to major cities.