Abstract
Team training in online, simulated environments can improve teamwork skills and task performance skills in a team setting. Teamwork assessment often relies on human observers. Instructors, team leaders, or other observers typically assess complex team competencies using checklists of observed behavior markers to infer performance. Automation can reduce training bottlenecks, provide evidence for objective assessment, and increase the impact of team training. A software capability is being developed to automate team assessments in dynamic online simulations. The simulations are dynamic to the extent that team actions and performance can change the progression of simulation events, assessment context, and the expected behavior of individuals contributing to team performance. A goal of automation design is to enhance usability for non-technical personnel to select, configure, reuse, and interpret team assessments in dynamic simulations. As a result of the reusable design, the assessments can generalize across different simulation software, settings, and scenarios. This paper describes work in progress on the research and development of an automated team assessment capability for the US Army’s Generalized Intelligent Framework for Tutoring (GIFT), an open source adaptive instructional architecture.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sottilare, R.A., Brawner, K.W., Sinatra, A.M., Johnston, J.H.: An updated concept for a Generalized Intelligent Framework for Tutoring (GIFT). Retrieved from GIFTtutoring.org on 1–9 May 2017
Sottilare, R., Brawner, K.: Component interaction within the Generalized Intelligent Framework for Tutoring (GIFT) as a model for adaptive instructional system standards. In: The Adaptive Instructional System (AIS) Standards Workshop of the 14th International Conference of the Intelligent Tutoring Systems (ITS) Conference, Montreal, Quebec, Canada, June 2018
Folsom-Kovarik, J.T., Sinatra, A.M.: Automating assessment and feedback for teamwork to operationalize team functional resilience. In: Proceedings of the 8th Annual Generalized Intelligent Framework for Tutoring (GIFT) Users Symposium (GIFTSym8) 28 May 2020, p. 126. US Army Combat Capabilities Development Command–Soldier Center (2020)
Goldberg, B., Hoffman, M., Graesser, A.C.: Adding a human to the adaptive instructional system loop: integrating GIFT and battlespace visualization. In: Design Recommendations for Intelligent Tutoring Systems Volume 8 – Data Visualization, 191 (2020)
Dillenbourg, P.: What do you mean by collaborative learning?
Ososky, S.J., Sottilare, R.A.: A Heuristic Evaluation of the Generalized Intelligent Framework for Tutoring (GIFT) Authoring Tools. US Army Research Laboratory Aberdeen Proving Ground United States, 1 March 2016
Sottilare, R., DeFalco, J.: Fundamentals of adaptive instructional systems tutorial. In: Interservice/Industry Simulation, Training & Education Conference (IITSEC), December 2020
Sottilare, R.A., Burke, C.S., Salas, E., Sinatra, A.M., Johnston, J.H., Gilbert, S.B.: Designing adaptive instruction for teams: a meta-analysis. Int. J. Artif. Intell. Educ. 28(2), 225–264 (2018)
Smith-Jentsch, K.A., Cannon-Bowers, J.A., Tannenbaum, S.I., Salas, E.: Guided team self-correction: Impacts on team mental models, processes, and effectiveness. Small Group Res. 39(3), 303–327 (2008)
Smith-Jentsch, K.A., Zeisig, R.L., Acton, B., McPherson, J.A.: Team dimensional training: a strategy for guided team self-correction. In: Cannon-Bowers, J.A., Salas, E. (eds.), Making Decisions Under Stress: Implications for Individual and Team Training (1998)
Salas, E., et al.: Does team training improve team performance? A meta-analysis. Hum. Fact. 50(6), 903–933 (2008)
Meneghel, I., Salanova, M., Martínez, I.M.: Feeling good makes us stronger: How team resilience mediates the effect of positive emotions on team performance. J. Happiness Stud. 17(1), 239–255 (2016)
Fowlkes, J.E., Lane, N.E., Salas, E., Franz, T., Oser, R.: Improving the measurement of team performance: the TARGETS methodology. Milit. Psychol. 6(1), 47–61 (1994)
Fowlkes, J.E., Shawn Burke, C.: Targeted acceptable responses to generated events or tasks (TARGETs). Handbook of Human Factors and Ergonomics Methods, 29 (2005)
Allen, W.C.: Overview and evolution of the ADDIE training system. Adv. Dev. Hum. Resour. 8(4), 430–441 (2006)
Peterson, C.: Bringing ADDIE to life: Instructional design at its best. J. Educ. Multimedia Hypermedia 12(3), 227–241 (2003)
Wray, R.E., Folsom-Kovarik, J.T., Woods, A., Jones, R.M.: Motivating narrative representation for training cross-cultural interaction. Procedia Manuf. 3, 4121–4128 (2015)
Graffeo, C., Benoit, T.S., Wray, R.E., Folsom-Kovarik, J.T.: Creating a scenario design workflow for dynamically tailored training in socio-cultural perception. Procedia Manuf. 3, 1486–1493 (2015)
McCormack, R.K., Kilcullen, T., Sinatra, A.M., Brown, T., Beaubien, J.M.: Scenarios for training teamwork skills in virtual environments with GIFT. In: Proceedings of the Sixth Annual GIFT Users Symposium, vol. 6, p. 189. US Army Research Laboratory, May 2018
McCormack, R., Kilcullen, T., Sinatra, A. M., Case, A., Howard, D.: Teamwork training architecture, scenarios, and measures in GIFT. In: Proceedings of the 7th Annual GIFT Users Symposium, p. 131. US Army Combat Capabilities Development Command–Soldier Center, May 2019
Gilbert, S.B., et al.: Creating a team tutor using GIFT. Int. J. Artif. Intell. Educ. 28(2), 286–313 (2018)
Ostrander, A., et al.: Evaluation of an intelligent team tutoring system for a collaborative two-person problem: Surveillance. Comput. Hum. Behav. 104, 105873 (2020)
Laird, J.E.: The Soar Cognitive Architecture. MIT Press, 13Apr 2012
Acknowledgments
The research reported in this paper was performed under a US Army Contracting Command - Aberdeen Proving Ground (ACC-APG) contract (W912CG-19-C-0009). The views and conclusions contained in this paper are those of the authors and should not be interpreted as presenting the official policies or position, either expressed or implied, of ACC-APG, DEVCOM-SC, STTC or the U.S. Government unless so designated by other authorized documents. Citation of manufacturer's or trade names does not constitute an official endorsement or approval of the use thereof. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Folsom-Kovarik, J., Sinatra, A.M., Sottilare, R.A. (2021). Automating Team Competency Assessment in Support of Adaptive Dynamic Simulations. In: Sottilare, R.A., Schwarz, J. (eds) Adaptive Instructional Systems. Design and Evaluation. HCII 2021. Lecture Notes in Computer Science(), vol 12792. Springer, Cham. https://doi.org/10.1007/978-3-030-77857-6_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-77857-6_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77856-9
Online ISBN: 978-3-030-77857-6
eBook Packages: Computer ScienceComputer Science (R0)