Abstract
Adaptive training (AT) can be an efficient option for providing individualized instruction tailored to trainees’ needs. Given promising research findings involving AT, we were challenged with developing an AT solution for Submarine Electronic Warfare (EW). Submarine EW is a complex task that involves classifying contacts, recognizing changes in the environment, and interpreting real-time displays. To train this task, we designed and developed the Submarine Electronic Warfare Adaptive Trainer (SEW-AT). We drew from multiple theoretical perspectives to drive our design decisions, including Multiple Resource Theory (MRT) and the Zone of Proximal Development (ZPD). Following the trainer’s development, we conducted a training effectiveness evaluation (TEE) to gauge initial performance improvements from SEW-AT. Using Submariners (n = 34) from 4 different Submarine Learning Centers across the United States, we found a 46% reduction in missed reports and a 49% improvement in report accuracy while using SEW-AT. As a next step, we plan to explore how the frequency of adaptation, or adaptation schedules, affect training performance and efficiency to determine if finer-grained adaptations produce greater learning gains.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
1.1 Background
Submarine Electronic Warfare (EW) is a critical task for submarine safety and a highly perishable skill. It is a dynamic and complex assignment that requires operators to monitor multiple inputs in order to classify contacts, recognize changes in the environment, and submit reports. The sheer amount of information that operators must monitor and assess across multiple modalities while under tight time constraints presents significant workload issues. The US Navy is investigating ways to improve operator performance in this domain with a particular focus on training. For example, the Submarine Tactical Requirements Group has recommended using Adaptive Training (AT) to fulfill critical training gaps in EW. Adaptive training can be conceptualized as consisting of content that can be adjusted based on an individual’s aptitude, learning preference, or learning style [1]. Training can be adjusted, either during training or following a training session, based on a trainee’s performance. The adjustments (adaptations) can be applied to different aspects of the task (e.g. instructional material, feedback, or difficulty) depending on training requirements [2]. AT is a practical option for instructors who otherwise may be unable to tailor instruction to an individual or group. Previous research has shown the benefits of this technique when compared against non-adaptive training [3, 4]. For example, Tennyson and Rothen [5] found benefits for adapting based on in-task responses compared to adapting based on pre-task data or not adapting. Furthermore, these benefits of AT have been extended across many applied fields, including medical and military training [6, 7]. Given these promising findings, we were challenged with developing an AT solution for Submarine Electronic Warfare (EW) Operators. To meet the Fleet’s training needs, we developed the Submarine Electronic Warfare Adaptive Trainer (SEW-AT), which allows trainees to practice this task in a realistic environment.
In order to develop SEW-AT, we started with an approach used by Landsberg et al. [2] in which they adapted both the scenario difficulty and feedback content based on performance in a training system for submarine periscope operators. The approach was applied to a submarine periscope task that was visual-spatial in nature and included a temporal demand. Landsberg et al. [2] provided evidence for performance benefits on both timeliness and accuracy. Considering the similarities in task demands between the periscope task and the EW task, we chose to utilize this approach in SEW-AT. However, the multifaceted nature of Submarine EW required extending and modifying the approach used by Landsberg et al. In order to make these changes, we consulted several cognitive theories to develop SEW-AT.
1.2 Theory
In order to understand the theory-based design choices we made, it is important to understand some context of the EW task. Operators must identify task events by simultaneously listening for radio frequency signals and navigating cluttered real-time displays. Moreover, operators must remain vigilant for counter-detection and threat recognition. EW Operators are also required to submit reports based on their current contact picture to the Officer of the Deck (OOD) during specific time windows. Therefore, SEW-AT was developed to allow trainees to perform the role of an EW operator in a series of 10 to 15 min scenarios and complete tasking and reports as required. To aid with trainees’ reports submissions in SEW-AT (which are verbal reports in the operational environment), we developed a report GUI that generates their verbal reports from operator selections and typed entries.
Due to the complex nature of SEW-AT and its components, our team employed several theoretical frameworks during the design process. These theoretical frameworks contributed to the design and development of feedback delivery and difficulty adaptation. Regarding feedback delivery, previous research advocates for detailed process feedback during training for military tasks [8]. Process feedback provides trainees information about their underlying performance to encourage strategy development for the future (versus outcome feedback which provides performance accuracy). Moreover, this method of feedback better approximates individualized instruction [9]. Unfortunately, workload and temporal demands imposed by the task would not allow for delivery of process feedback in real time as we were concerned that it would overload the operator. Our solution to this issue was two-fold: we provided real-time feedback in the form of audio cues or hints to alert trainees to important events they may have missed; additionally, we provided end-of-scenario feedback (a combination of process and detailed outcome feedback) to offer insights into areas of improvement. When it came to implementation of this feedback approach, we used Multiple Resource Theory and literature on the timing of training feedback to guide our specific design decisions.
Multiple Resource Theory (MRT) and resulting box model [10] provided a theoretical framework for several key decisions in the development process. MRT assumes there are independent working memory sub-systems (i.e., verbal, spatial), which have a limited capacity. MRT emerged based on evidence that differences in performance on concurrent tasks emanated from differences in the qualitative demands of the separate tasks [11, 12]. Wickens [13] expanded upon prior research by proposing a model that linked supposed resource dimensions to underlying neurological mechanisms [14]. Specifically, the MRT framework identifies perceptual modalities (e.g. visual, auditory) through which sensory information is processed (e.g., verbal or spatial). Therefore, overloading the visual or auditory channels individually can add difficulty to information processing compared to providing information both visually and audibly. Additionally, the verbal and spatial processing codes represent the manner in which information displayed through the visual or auditory channels is encoded and then processed in working memory. Much like the separate perceptual modalities, processing codes are also subject to overuse. For example, Goodman, Tijerina, Bents, and Wierwille [15] provide evidence showing the benefits of voiced cell phone dialing while driving due to the various spatial and manual demands of controlling the vehicle. Voiced dialing avoids overloading the visual channel, which is highly taxed with driving. Further, it also utilizes verbal component versus spatial (locating numbers on the number pad) in order to avoid taxing the spatial processing used during driving. From this example, we also see how the spatial and verbal processing codes can combine with perceptual modality to increase difficulty.
Using MRT as an inspiration, we took a two phased approach to providing feedback in SEW-AT. The first was to provide immediate feedback for critical events by using audio cues. Although the EW task primarily requires auditory recognition of RF frequencies, we felt that brief audio verbal cues during the task would not overly tax the auditory channel. This decision was made with the high taxation of visual verbal channel (e.g. real-time displays, verbal report GUI, etc.) in mind. The audio verbal cues would quickly alert trainees to critical task events without inhibiting their ability to perceive and process frequencies. Moreover, the audio verbal cues would not interfere with trainees’ ability to manually respond to these events through the report GUI.
The second phase to feedback delivery was to provide a combination of process and detailed outcome feedback after each scenario. The EW task also requires trainees to utilize both visual verbal and visual spatial processing codes to recognize, analyze, and interpret stimuli. Due to the degree in which the task taxes the visual channel, we did not feel that any real-time, detailed, feedback was appropriate for SEW-AT. Furthermore, several researchers [16, 17] have argued against the use of the immediate feedback suggesting that the processing of feedback in real-time competes for limited cognitive resources that are being used to perform the task and learning suffers as a result. Given the temporal demand and modality processing issues imposed by the SEW-AT task, we chose to delay providing detailed feedback until completion of the task in hopes of encouraging long-term learning and retention.
In addition to feedback delivery, the process by which SEW-AT adapted trainees’ scenario difficulty also required theoretical consideration. Given our desire to both train and challenge EW Operators, we found that Vygotsky’s [18] Zone of Proximal Development (ZPD) most appropriately guided our design of SEW-AT’s difficulty adaptations. The ZPD represents the theoretical “zone” in which task difficulty challenges trainees during learning without overwhelming them and discouraging learning. Therefore, we developed an algorithm that adapted task difficulty between scenarios to be more challenging when trainees performed well, but less challenging when they performed poorly in order to keep them in this desired zone. For example, trainees who performed well on a given scenario would receive a more difficult scenario next; similarly, trainees who performed poorly on a given scenario receive an easier scenario next. Trainees remained at the same level of difficulty for their next scenario when their scores did not exceed either threshold for adapting. Additionally, the concept of the ZPD helped to guide our decisions for scenario length and adaptation frequency.
Although our goal was to create algorithms that kept trainees in their ZPD, this did not come without concerns. Chief among these concerns was the question of how often to schedule difficulty adaptations to more precisely position trainees in their ZPDs. To address this, we consulted Van Lehn’s [19] concept of granularity for guidance. Granularity in this context refers to the number of opportunities trainees have to interact with the system when completing a task. In SEW-AT, users interact with the system by submitting multiple reports within each scenario. The problem at hand is what amount of interaction (i.e., what granularity of measurement) should be used to inform difficulty adaptations. Should each report trigger adaptation? Or, should a set of reports trigger adaptation? Though we could have adapted scenario difficulty after every report, we felt this would have adapted trainees too often, potentially forcing them from their respective ZPDs. As a result, we chose to implement 10–15 min scenarios and to adapt following each scenario, as trainees would then have the opportunity to submit several reports prior to an adaptation decision. In doing so, we believed that our adaptation decisions would be triggered by a more complete assessment of a trainee’s performance, appropriately positioning them into their ZPD.
In sum, our design decisions for SEW-AT were informed by theoretical perspectives of human information processing, adaptive training, and empirical evidence from prior training research. In the following sections, we provide an overview of the EW operator’s task demands and detail our approach to evaluating SEW-AT’s training effectiveness. We completed our evaluations with usage data collected from trainees at several submarine learning centers across the United States.
2 Training Effectiveness Evaluation
2.1 Submarine Electronic Warfare Adaptive Trainer
SEW-AT simulates a trip to periscope depth (PD) to train Electronic Warfare Operators on three main training objectives: (1) maintaining safety of ship, (2) recognizing parameter changes, and (3) making accurate and timely reports. Prior to the PD trip, a detailed Pre-PD brief is presented for that scenario which provides the context and mission for that PD trip. Once the pre-PD brief is reviewed, the scenario starts and the EW Operator is presented with opportunities to provide reports to Control during the first 10–15 min at PD. Reports to Control are entered using a report GUI which allows text-based input of EW verbal report litany, which is passed off to algorithms to assess the accuracy and timeliness of the report. As discussed above, audio cues are presented during the scenario based on the near real-time assessment of performance. After completion of a scenario, performance feedback is provided and focuses on the training objectives listed above. SEW-AT then adapts the difficulty of the next scenario based on the operator’s performance. There are 3 levels of difficulty based on submarine doctrine - basic, intermediate, and, advanced. If a trainee performs well, they are moved up a level; if a trainee, performs poorly, they move down a level of difficulty; if their performance is fair, they stay at the same level of difficulty. All trainees start SEW-AT on an intermediate level scenario after that their path through SEW-AT is based on performance. Further, trainees create unique user log-ins so that their progress over time is tracked and to ensure scenarios are not repeated.
We delivered SEW-AT systems to several Submarine Learning Centers (SLC) across the country. The goal of providing these systems was threefold: (1) to obtain operator feedback on an initial version of the system, as well as collect site usage data that would (2) help us tune our adaptive algorithms, and (3) provide insight on initial training effectiveness. Although we did not have control over how the system was used [e.g., how many submariners used it, how many times they used the system, or if the sites incorporated the system into their curriculum (versus independent study), etc.], we were able to collect and analyze usage data periodically from the sites during software upgrades. From the first software update to the second, we received back 8 to 10 months of usage data across the sites from 66 EW Operators. These data were broken down into single-touch and multiple-touch users. Single touch users (n = 32) were defined as trainees who actively played through only one scenario in SEW-AT. Multiple touch users (n = 34) were defined as trainees who actively played more than one scenario in SEW-AT over the 8 to 10-month period. The multiple touch data allow us to assess performance improvements over time and are presented below.
2.2 Usage Results
In order to assess performance improvements, we derived a “pre-test” and a “post-test” score for both timeliness and accuracy for each operator. The pre-test data were extracted from the first scenario the operators played through and the post-test data were extracted from the last scenario they played through. Accuracy measures were determined by assessing the percent accuracy of the information that was provided for each report that was required in the scenario. A paired samples t-test was performed to assess accuracy improvements from pre to post. This showed a statistically significant improvement, Mpre = 16.06% (15%), Mpost = 33.47% (31%), t(33) = 4.37, p < .001. Timeliness scores were broken down into an analysis of the percentage of late calls (i.e., calls that were provided but were reported past the time they were due) and missed calls (i.e., reports that were never given). A paired samples t-test was performed to assess improvements from pre to post. Note: smaller numbers indicate better performance (e.g., a reduction in the amount of late and/or missed calls). The percentage of late calls was not statistically significantly different from pre to post, Mpre = 15.98% (15%), Mpost = 19.96% (21%), t(33) = –0.41, ns. However, the percentage of missed calls did show a statistically significant improvement from pre to post, Mpre = 70.10% (24%), Mpost = 51.30% (32%), t(33) = –2.52, p = .008.
Table 1 shows the number of multiple touch users by SLC site. As the number of users per site varied, we were interested in looking at how performance improvements differed by site as this was of particular interest for our Fleet stakeholders. Figures 1 and 2 show percentage performance improvements per site and overall for the accuracy and timeliness metrics. As can be seen in Fig. 1, pre to post percent accuracy improvement ranged from 28%–67% by site. Regarding timeliness, pre to post percent improvement in missed calls ranged from 31%–57% by site and pre to post percent improvements in late calls ranged from 4%–44% by site. As can be seen in Fig. 2, SLC Site 2 showed a large improvement in late calls and, interestingly, that improvement is larger than missed calls for that site. While pre to post improvement in late calls was not statistically significant overall, it appears Site 2 may have been struggling with that aspect of the task and had more opportunities to show improvements using SEW-AT.
In addition to the usage data, we also asked the operators to complete questionnaires and provide us feedback on SEW-AT. These questions can be seen in Table 2. Each question was followed up with an open-ended comments section in order for us to identify and address issues for future software upgrades. Additionally, we asked for opinions on positive aspects of the system, areas for improvement, and details on software bugs (if any occurred). We received responses from 14–17 EW operators depending on the question which was equivalent to a 21–26% response rate. As can be seen from the data below, the overall responses were promising. The system seemed to be well-received and the training materials and stimuli appeared to be face valid. Of one concern was the 3.5 rating on the difficulty of using the verbal report GUI. We found a positive correlation (r = 0.37) between the number of scenarios played and how the GUI was rated. Specifically, the more scenarios an operator played, the more highly operators rated the GUI as easy to use which indicated there may be a slight learning (1–2 scenarios) curve for operators to become comfortable with the verbal report GUI.
3 Discussion
3.1 Review of TEE Results
As mentioned previously, we had several goals when providing initial versions of SEW-AT software to the SLC sites. The first goal was to obtain operator perceptions and feedback on SEW-AT. In general, EW Operators rated the system favorably and indicated they would be willing to use the system in the future. The open ended comments they provided helped us identify software bugs and additional system capabilities that we intend to include in future versions of the system.
The usage data we received also allowed us to gain insight on initial training effectiveness. Specifically, we wanted to determine if the previous approach and the theory-based decisions we made were impacting performance in a positive way. Overall, the performance improvements are highly encouraging and suggest this is the case. As alluded to earlier, the data should be viewed tentatively. The number of EW operators, days between sessions, and the number of scenarios that were completed varied by site and operator. Because we did not have experimental control, we cannot attribute performance improvements to SEW-AT versus practice alone. However, these initial data on the system are helpful to us for making improvements and tuning our adaptive and assessment algorithms. Additionally, we have positive results on Kirkpatrick’s Level 1 (Reactions) and Level 2 (Learning) training evaluation criteria (see Kirkpatrick [20] for a full description of the training evaluation criteria). As this research program continues, we will be able to also gain insight on Levels 3 (Behavior) and 4 (Results) as we will have data on training transfer and be able to assess effects on operational performance. In addition to TEE data, we plan to take a more experimental approach to assess performance and learning gains in our future research.
3.2 Future Directions
Our results support that using SEW-AT improved trainee performance, but we are also interested in refining SEW-AT’s adaptive algorithms in efforts to continue to improve learning gains. Our future research will involve further exploring difficulty adaptation to examine questions about the ZPD and adaptive training methodologies. Specifically, we aim to investigate difficulty adaptation frequency empirically. Theoretically, adaptation schedules that are more frequent should allow for finer-tuned adjustments that more accurately align to a trainee’s ZPD [18]. This adaptation approach assesses performance to adapt training within scenarios in real-time. Conversely, it is possible that a less-frequent adaptation schedule (as implemented in the present version of SEW-AT) could be better suited to aligning difficulty to the trainee’s ZPD. Such an adaptation would be based on a more comprehensive assessment of performance at the conclusion of a scenario. These opposing schedules for difficulty adaptation are two approaches to Vygotsky’s [18] characterization of the ZPD as the zone where a task is challenging while still being achievable. For future research, we seek to understand which type of adaptation schedule is optimal for learning gains.
In addition to adapting training difficulty, we also plan to investigate new approaches to adapting feedback in the Submarine EW task. Currently, SEW-AT provides end-of-scenario feedback to avoid disrupting trainees during this complex and temporally-demanding task. It may be the case, however, that providing feedback in real time can guide trainees to more efficient means of excelling during training and retaining their skills long-term. This is especially true in tasks spanning multiple modalities, as providing immediate feedback in one modality (e.g. the auditory channel) may help to alleviate overwhelming demand on another modality (e.g. the visual channel) [21]. Potential avenues for implementing real-time feedback in SEW-AT include directing trainees toward real-time displays, as well as providing visual cues for new events occurring in the environment.
An additional area we aim to explore is the individual differences that impact learning gains in adaptive training. For instance, previous research has identified that certain individuals can become distressed when task difficulty changes (e.g., those who employ emotion-focused coping strategies; [22]). If this is true, it is possible that a highly frequent adaptation schedule could be distressing for some trainees, limiting their ability to learn from training. This characteristic will be taken into consideration during our future investigations with these different adaptation schedules, as adaptive training solutions may need to account for trainee characteristics.
References
Landsberg, C.R., Van Buskirk, W.L., Astwood, R.S., Mercado, A.D., Aakre, A.J.: Adaptive training considerations for simulation-based training. (Special report 2010-001). Naval Air Warfare Center Training Systems Division, Orlando (2011)
Landsberg, C.R., Astwood Jr., R.S., Van Buskirk, W.L., Townsend, L.N., Steinhauser, N.B., Mercado, A.D.: Review of adaptive training system techniques. Mil. Psychol. 24(2), 96–113 (2012)
Cook, D.A., Beckman, T.J., Thomas, K.G., Thompson, W.G.: Adapting web-based instruction to residents’ knowledge improves learning efficiency. J. Gen. Intern. Med. 23(7), 985–990 (2008)
Landsberg, C.R., Mercado, A., Van Buskirk, W.L., Lineberry, M., Steinhauser, N.: Evaluation of an adaptive training system for submarine periscope operations. In: Proceedings of the Human Factors and Ergonomics Society 56th Annual Meeting, pp. 2422–2426. SAGE Publications, Los Angeles, CA (2012). (CD ROM)
Tennyson, R.D., Rothen, W.: Pretask and on-task adaptive design strategies for selecting number of instances in concept acquisition. J. Educ. Psychol. 69(5), 586–592 (1977)
Romero, C., Ventura, S., Gibaja, E.L., Hervas, C., Romera, F.: Web-based adaptive training simulator system for cardiac support. Artif. Intell. Med. 38, 67–78 (2006)
Bauer, K.N., Brusso, R.C., Orvis, K.A.: Using adaptive difficulty to optimize videogame-based training performance: the moderating role of personality. Mil. Psychol. 24(2), 148–165 (2012)
Buff, W.L., Campbell, G.E.: What to do or what not to do? Identifying the content of effective feedback. In: Proceedings of the 46th Annual Meeting of the Human Factors and Ergonomics Society, pp. 2074–2078. SAGE Publications, Santa Monica, CA (2002)
Park, O.C., Lee, J.: Adaptive instructional systems. In: Jonassen, D. (ed.). Handbook of Research for Educational Communications and technology, pp. 651–684. MacMillan Publishers, New York (1996)
Wickens, C.D., Hollands, J.: Engineering Psychology and Human Performance, 3rd edn. Prentice Hall, Upper Saddle River (2000)
Wickens, C.D.: The effects of divided attention on information processing in tracking. J. Exp. Psychol. Hum. Percept. Perform. 2, 1–13 (1976)
Wickens, C.D.: Multiple resources and performance prediction. Theoret. Issues Ergon. Sci. 3(2), 159–177 (2002)
Wickens, C.D.: The structure of attentional resources. In: Nickerson, R. (ed.) Attention and Performance VIII, pp. 239–257. Lawrence Erlbaum, Hillsdale (1980)
Kinsbourne, M., Hicks, R.E.: Functional cerebral space: a model for overflow, transfer and interference effects in human performance. In: Attention and Performance VII, pp. 345–362 (1978)
Goodman, M.J., Tijerina, L., Bents, F.D., Wierwille, W.W.: Using cellular telephones in vehicles: safe or unsafe? Transport. Hum. Factors 1, 3–42 (1999)
Schmidt, R.A., Wulf, G.: Continuous concurrent feedback degrades skill learning: implications for training and simulation. Hum. Factors 39, 509–525 (1997)
Schooler, L.J., Anderson, J.R.: The disruptive potential of immediate feedback. In: Proceedings of the Twelfth Annual Conference of the Cognitive Science Society, Cambridge, MA, pp. 702–708 (1990)
Vygotsky, L.S.: Interaction between learning and development. In: Gauvain, M., Cole, M. (eds.) Readings on the Development of Children, 4th edn., pp. 34–40. Worth, New York, NY (2005). Reprinted from: Cole, M., John-Steiner, V., Scribner, S., Souberman, E. (eds.) Mind in Society: The Development of Higher Psychological Processes, pp. 71–91. Harvard University Press, Cambridge (1978)
VanLehn, K.: The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ. Psychol. 46(4), 197–221 (2011)
Kirkpatrick, D.: Great ideas revisited: Revisiting Kirkpatrick’s four-level model. Training Dev. 50, 54–59 (1996)
Moreno, R., Mayer, R.E.: A learner-centered approach to multimedia explanations: deriving instructional design principles from cognitive theory. Inter. Multimedia Electron. J. Comput. Enhanc. Learn. 2(2) (2000). http://imej.wfu.edu/articles/2000/2/05/index.asp
Matthews, G.: Extraversion, emotion and performance: a cognitive-adaptive model. Adv. Psychol. 124, 399–442 (1997)
Acknowledgements
We gratefully acknowledge Dr. Kip Krebs and the Office of Naval Research who sponsored this work (Funding Doc# N0001418WX00447). We would also like to thank Marc Prince, Bryan Pittard, and Derek Tolley for their development of the SEW-AT system. Presentation of this material does not constitute or imply its endorsement, recommendation, or favoring by the U.S. Navy or Department of Defense (DoD). The opinions of the authors expressed herein do not necessarily state or reflect those of the U.S. Navy or DoD.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 This is a U.S. government work and not under copyright protection in the United States; foreign copyright protection may apply
About this paper
Cite this paper
Van Buskirk, W.L., Fraulini, N.W., Schroeder, B.L., Johnson, C.I., Marraffino, M.D. (2019). Application of Theory to the Development of an Adaptive Training System for a Submarine Electronic Warfare Task. In: Sottilare, R., Schwarz, J. (eds) Adaptive Instructional Systems. HCII 2019. Lecture Notes in Computer Science(), vol 11597. Springer, Cham. https://doi.org/10.1007/978-3-030-22341-0_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-22341-0_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-22340-3
Online ISBN: 978-3-030-22341-0
eBook Packages: Computer ScienceComputer Science (R0)