Abstract
In this paper, we present a framework for implementing Team Performance Measures based on a temporal accuracy measure called the relative accuracy index (RAI) to evaluate and compare team performance in a command-and-control human-in-the-loop simulation. The framework allows researchers to collect and analyze team outcomes in an unbiased objective manner based on a temporal performance measure known as a Time Window. Our framework provides experimenters and subject matter experts the necessary tools to evaluate performance in terms of task demands. We also provide a sample analysis of individual and team performance using the RAI.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bhandarkar D, Thiruvengada H, Rothrock L, Campbell GE, Bolton AE (2004) TASP: a toolkit to analyze team performance in a complex task environment. In: Paper presented at the annual conference of the Institute of Industrial Engineers, Houston
Booch G, Rumbaugh J, Jacobson I (1999) The unified modeling language user guide. Addison-Wesley, Reading
Bolton AE, Campbell GE, Buff WL, Rothrock L, Bhandarkar D, Thiruvengada H, Kukreja U (2004) TASP: A flexible alternative for team training and performance research. In: Paper presented at the interservice/industry training, simulation and education conference (I/ITSEC), Orlando
Brannick M, Prince C (1997) An overview of team performance measurement. In: Brannick MT, Salas E, Prince C (eds). Team performance assessment and measurement: theory, methods, and applications. Lawrence Erlbaum, Mahwah, pp 3–16
Brannick MT, Salas E, Prince C (eds) (1997) Team performance assessment and measurement: theory, methods, and applications. Lawrence Erlbaum, Mahwah
Brehmer B (1990) Strategies in real-time dynamic decision making. In: Hogarth RM (ed) Insights in decision making: a tribute to Hillel J. Einhorn. University of Chicago Press, Chicago, pp 262–279
Brehmer B, Allard R (1991) Dynamic decision making: the effects of task complexity and feedback delays. In: Rasmussen J, Brehmer B, Leplat J (eds). Distributed decision making: cognitive models for cooperative work. Wiley, New York, pp 319–334
Briggs GE, Johnston WA (1967) Team training (NTDC technical report No. 1327–4). Naval Training Device Center, Orlando
Cannon-Bowers JA, Salas E (1997a) Teamwork competencies: the intersection of team member knowledge, skills, and attitudes. In: O’Neil HF (ed) Workforce readiness: competencies and assessment. Lawrence Erlbaum, Hillsdale, pp 151–174
Cannon-Bowers JA, Salas E (1997b) A framework for developing team performance measures in training. In: Brannick MT, Salas E, Prince C (eds). Team performance assessment and measurement: theory, methods, and applications. Lawrence Erlbaum, Mahwah, pp 45–62
Cannon-Bowers JA, Tannenbaum SI, Salas E, Volpe CE (1995) Defining competencies and establishing team training requirements. In: Guzzo RA, Salas E (eds). Team effectiveness and decision making in organizations. Jossey-Bass publishers, San Francisco, pp 333–380
Dickinson TL, McIntyre RM (1998) A conceptual framework for teamwork measurement. In: Brannick MT, Salas E, Prince C (eds). Team performance measurement and assessment: theory, methods and applications. Lawrence Erlbaum, Mahwah, pp 19–43
Green DM, Swets JA (1988) Signal detection theory and psychophysics. Peninsula publishing, Los Altos
Hobson CJ, Mendel RW, Gibson FW (1981) Clarifying performance appraisal criteria. Organ Behav Hum Perform 28(2):164–188
Kirlik A, Fisk AD, Walker N, Rothrock L (1998) Feedback augmentation and part-task practice in training dynamic decision-making skills. In: Cannon-Bowers JA, Salas E (eds). Making decisions under stress: Implications for individual and team training. American Psychological Association, Washington, pp 91–113
McEvoy G, Beatty R, Bernardin J (1987) Unanswered questions in assessment center research. J Bus Psychol 2:97–111
McIntyre RM, Salas E (1995) Measuring and managing for team performance: emerging principles from complex environments. In: Guzzo RA, Salas E (eds). Team effectiveness and decision making in organizations. Jossey-Bass publishers, San Francisco, pp 149–203
Mosier KL (1997) Myths of expert decision making and automated decision making aids. In: Zsambok C, Klein G (eds). Naturalistic Decision Making. Lawrence Erlbaum, Hillsdale, pp 319–330
Oviatt S (1999) Ten myths of multimodal interaction. Commun ACM 42(11):74–81
Rothrock L (2001) Using time windows to evaluate operator performance. Int J Cogn Ergon, 5(1):1–21
Rothrock L, Harvey C, Burns J (2005) A theoretical framework and quantitative architecture to assess team task complexity in dynamic environments. Theoret Issue Ergon Sci 6(2):157–171
Sackett PR, Dreher GF (1982) Constructs and assessment center dimensions: Some troubling empirical findings. J Appl Psychol 67:401–410
Smith-Jentsch KA, Johnston JH, Payne SC (1998a) Measuring team-related expertise in complex environments. In: Cannon-Bowers JA, Salas E (eds). Making decisions under stress: implications for individual and team training. American Psychological Association, Washington, pp 61–87
Smith-Jentsch KA, Zeisig RL, Acton B, McPherson JA (1998b) Team dimensional training: a strategy for guided team self-correction. In: Cannon-Bowers JA, Salas E (eds). Making decisions under stress: implications for individual and team training. American Psychological Association, Washington, pp 271–297
Swets JA (1996) Signal detection theory and ROC analysis in psychology and diagnostics: collected papers. Lawrence Erlbaum, Mahwah
Thiruvengada H, Bhandarkar DN, Kukreja U, Rothrock L (2004) TASP: a toolkit for human experimentation in a synthetic environment with a hybrid human-agent team. In: Paper presented at behavior research in modeling and simulation (BRIMS) Conference, Crystal City
Thiruvengada H, Bhandarkar DN, Rothrock L (2006) Time windows-based team performance measures: design and implementation. In: Paper to be presented at 2006 IEEE international conference on systems, man, and cybernetics, Taipei
Thiruvengada H, Rothrock L (2006) Team training. In: Nembhard D (ed) Workforce cross training handbook. CRC, Boca Raton (in press)
Acknowledgments
This research was supported by the Office of Naval Research under Contract No. N61339-05–C–0031. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the Office of Naval Research. We wish to thank Gwendolyn Campbell, Amy Bolton, Wendi Vanbuskirk and Damodar Bhandarkar for their valuable insights, which enabled the successful development of our framework.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Thiruvengada, H., Rothrock, L. Time windows-based team performance measures: a framework to measure team performance in dynamic environments. Cogn Tech Work 9, 99–108 (2007). https://doi.org/10.1007/s10111-006-0046-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10111-006-0046-8