Skip to main content
SpringerLink
Log in

Stealth Assessment of Teams in a Digital Game Environment

  • Conference paper
  • First Online:
Games and Learning Alliance (GALA 2013)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 8605))

Included in the following conference series:

Abstract

There is not much research on team collaboration in digital entertainment games, nor is there much evidence for the efficacy of game-based team training or the validity of game-based team assessment. This is a shortcoming because of an increasing pervasiveness of serious games in organizational life, e.g. for operational training, management and leadership. Is it possible to establish marked relationships between psychometric constructs that measure ‘team composition and performance’ and ‘analytics’ that unobtrusively measure gameplay performance? If so, what are the implications for game-based team research and assessment? The authors conducted explorative, quasi-experimental (field) experiments with the multiplayer serious game TeamUp. One field experiment was conducted with 150 police officers as part of task-specific two-day team training. Research data were gathered through pre-game and post-game questionnaires on team constructs such as ‘psychological safety’ and ‘team cohesion’. A large quantity of in-game data was logged to construct indicators like ‘time needed to complete the task’, ‘speak time’ and ‘avoidable mistakes’ to measure team performance. The conclusion of the analysis is that ‘team cohesion’ and ‘psychological safety’ correlate moderately and significantly with in-game performance indicators. Teams with an unequal individual game performance speak the most, while teams with an equally low or equally high individual performance spend significantly less time speaking. The indicative findings support the need to further develop validated analytics and game-based environments for team research and assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Pearce, C.L., Conger, J.A.: Shared Leadership: Reframing the Hows and Whys of Leadership, vol. 57(3), p. 330. Sage Publications, Thousand Oaks (2003)

    Google Scholar 

  2. Gressick, J., Derry, S.J.: Distributed leadership in online groups. Int. J. Comput. Support. Collaborative Learn. 5(2), 211–236 (2010)

    Article  Google Scholar 

  3. Malone, T.W., Cadwell, T., Scarborough, J., Roy, S.: Leadership in Games and at Work: Implications for the Enterprise of Massively Multiplayer Online Roleplaying Games. Most, pp. 1–34 (2007)

    Google Scholar 

  4. Siitonen, M.: Exploring the experiences concerning leadership communication in online gaming groups. In: Proceedings of the 13th International MindTrek Conference Everyday Life in the Ubiquitous Era on MindTrek 09, p. 90 (2009)

    Google Scholar 

  5. Yee, N.: The labor of fun: how video games blur the boundaries of work and play. Games Culture 1(1), 68–71 (2006)

    Article  Google Scholar 

  6. Bettenhausen, K.L.: Five years of groups research: what we have learned and what needs to be addressed. J. Manag. 17(2), 345–381 (1991)

    Google Scholar 

  7. Michael, D.R., Chen, S.: Serious Games: Games that Educate, Train, and Inform. Thomson Course Technology PTR, Boston (2005)

    Google Scholar 

  8. Counterstrike: Global Offensive, (Webpage). http://blog.counter-strike.net/index.php/about/. Accessed 14 Feb 2013

  9. Call of Duty, (Webpage). http://www.callofduty.com/blackops2. Accessed 14 Feb 2013

  10. America’s Army, (Webpage). http://www.americasarmy.com

  11. Virtual Battle Space 2 (VBS2), (Webpage). http://products.bisimulations.com/products/vbs2/overview. Accessed 13 Feb 2013

  12. Andrews, A.: Using advanced gaming technology for teaching critical thinking, problem solving, and leadership skills. In: Interactive Technologies Conference Proceedings (2007)

    Google Scholar 

  13. van der Hulst, A., Muller, T., Besselink, S.: Bloody serious gaming – experiences with job oriented training. In: Industry Training, no. 8169, pp. 1–11 (2008)

    Google Scholar 

  14. Stern, P.A., Harz, C.R.: Serious games for first responders: improving design and usage with social learning theory (Volume B). ProQuest, p. 218 (2008)

    Google Scholar 

  15. Harz, C.R., Stern, P.A.: Serious games for first responders: improving design and usage with social learning theory (Volume A). ProQuest, p. 212 (2008)

    Google Scholar 

  16. LeRoy Heinrichs, W., Youngblood, P., Harter, P.M., Dev, P.: Simulation for team training and assessment: case studies of online training with virtual worlds. World J. Surg. 32(2), 161–170 (2008)

    Article  Google Scholar 

  17. Østergaard, H.T., Østergaard, D., Lippert, A.: Implementation of team training in medical education in Denmark. Qual. Saf. Health Care 13(996), i91–i95 (2004)

    Article  Google Scholar 

  18. Virtual Leader, (Webpage). http://www.simulearn.net/site_flash/index.html. Accessed 12 Feb 2013

  19. Csikszentmihalyi, M.: Flow: The Psychology of Optimal Experience. Harper Perennial, New York (1991)

    Google Scholar 

  20. Fligby, (Webpage). http://www.slideshare.net/zadvecsey/presenting-fligby-01102012v41#btnNext. Accessed 12 Feb 2013

  21. The Gaining Leadership Program, (Webpage). http://www.ranj.com/content/werk/the-gaining-leadership-program. Accessed 12 Feb 2013

  22. The Leadership Game, (Webpage). http://www.pixelearning.com/services-the-leadership-game.shtml. Accessed 12 Feb 2013

  23. Houthoff Buruma The Game, (Webpage). http://www.ranj.com/nl/content/werk/houthoff-buruma-the-game. Accessed 20 Feb 2013

  24. NoviCraft, (Webpage). http://www.teamingstream.com/. Accessed 12 Feb 2013

  25. Manninen, T.: Interaction Forms and Communicative Actions in Multiplayer Games. Game Studies 3(1) (2003)

    Google Scholar 

  26. Manninen, T.: Virtual team interactions in networked multimedia games – Case: ‘Counterstrike’ – multi-player 3d action game. In: Presence (2001)

    Google Scholar 

  27. Lewis, S., Ellis, J.B., Kellogg, W.A.: Using virtual interactions to explore leadership and collaboration in globally distributed teams. In: Human Factors, no. Cmc, pp. 9–18 (2010)

    Google Scholar 

  28. Jakobsson, M.: Questing for knowledge - virtual worlds as dynamic processes of social interaction. In: Schroeder, R., Axelsson, A.-S. (eds.) Avatars at Work and Play: Collaboration and Interaction in Shared Virtual Environments, pp. 209–225. Springer, Dordrecht (2006)

    Chapter  Google Scholar 

  29. Brown, B., Bell, M.: Play and sociability in there: some lessons from online games for collaborative virtual environments. In: Schroeder, R., Axelsson, A.-S. (eds.) Avatars at Work and Play: Collaboration and Interaction in Shared Virtual Environments, pp. 227–245. Springer, Dordrecht (2006)

    Chapter  Google Scholar 

  30. Hamalainen, R.: Designing and evaluating collaboration in a virtual game environment for vocational learning. Comput. Educ. 50(1), 98–109 (2008)

    Article  Google Scholar 

  31. Yee, N.: The psychology of massively multi-user online role-playing games: motivations, emotional investment, relationships and problematic usage. In: Schroeder, R., Axelsson, A.-S. (eds.) Avatars at Work and Play: Collaboration and Interaction in Shared Virtual Environments, pp. 187–207. Springer, Dordrecht (2006)

    Chapter  Google Scholar 

  32. Priest, H.A., Stagl, K.C., Klein, C., Salas, E.: Virtual Teams: Creating Context for Distributed Teamwork. American Psychological Association, Washington, DC (2006)

    Google Scholar 

  33. Sivunen, A., Siitonen, M.: Comparing experiences on leadership in virtual teams and online multiplayer gaming clans. In: ICA 2010 Singapore, vol. 35, June 2010

    Google Scholar 

  34. Boyle, E.A., Hancock, F., Seeney, M., Allen, L.: The implementation of team based assessment in serious games. In: 2009 Conference in Games and Virtual Worlds for Serious Applications, pp. 28–35 (2009)

    Google Scholar 

  35. Bezuijen, A.: Teamplay The further development of TeamUp, a teamwork focused serious game, TU Delft (2012)

    Google Scholar 

  36. Sweeney, L.B., Meadows, D.L.: The Systems Thinking Playbook. Pegasus Communication, Durham (2001)

    Google Scholar 

  37. Thiagarajan, S.: Design your Own Games and Activities. Pfeiffer, San Francisco (2003)

    Google Scholar 

  38. Competence example items

    Google Scholar 

  39. Edmondson, A.C.: Psychological safety and learning behavior in work teams. Adm. Sci. Q. 44(2), 350 (1999)

    Article  Google Scholar 

  40. Seers, A.: Team-Member Exchange Quality: A New Construct for Role-making Research (1989)

    Google Scholar 

Download references

Acknowledgements

The authors wish to thank several teams of TU Delft students who participated in our Serious Game Design Project course for conceiving the TeamUp game. We thank Arne Bezuijen and Bas van Nuland for merging original ideas with their own, for turning prototypes into a high-quality product and for having enough courage to start up their company, The Barn, around it. We are grateful to the players who tested TeamUp and to the Dutch police for their trust in the game.

Author information

Authors and Affiliations

  1. TU Delft, Jaffalaan 5, 2628 BX, Delft, The Netherlands

    Igor Mayer, Theodore van Ruijven & Ivo Wenzler

  2. Rotterdam School of Management, Erasmus University, Rotterdam, The Netherlands

    Dirk van Dierendonck

  3. Accenture, Amsterdam, The Netherlands

    Ivo Wenzler

Authors
  1. Igor Mayer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dirk van Dierendonck
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Theodore van Ruijven
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ivo Wenzler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Igor Mayer .

Editor information

Editors and Affiliations

  1. University of Genova, Genova, Italy

    Alessandro De Gloria

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this paper

Cite this paper

Mayer, I., van Dierendonck, D., van Ruijven, T., Wenzler, I. (2014). Stealth Assessment of Teams in a Digital Game Environment. In: De Gloria, A. (eds) Games and Learning Alliance. GALA 2013. Lecture Notes in Computer Science(), vol 8605. Springer, Cham. https://doi.org/10.1007/978-3-319-12157-4_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-12157-4_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-12156-7

  • Online ISBN: 978-3-319-12157-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation