Skip to main content

How to Compare Usability of Techniques for the Specification of Virtual Agents’ Behavior? An Experimental Pilot Study with Human Subjects

  • Conference paper
Agents for Educational Games and Simulations (AEGS 2011)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 7471))

Included in the following conference series:

  • 736 Accesses

Abstract

Reactive or dynamic planning is currently the dominant paradigm for controlling virtual agents in 3D videogames. Various reactive planning techniques are employed in the videogame industry while many reactive planning systems and languages are being developed in the academia. Claims about benefits of different approaches are supported by the experience of videogame programmers and the arguments of researchers, but rigorous empirical data corroborating alleged advantages of different methods are lacking. Here, we present results of a pilot study in which we compare the usability of an academic technique designed for programming intelligent agents’ behavior with the usability of an unaltered classical programming language. Our study seeks to replicate the situation of professional game programmers considering using an unfamiliar academic system for programming in-game agents. We engaged 30 computer science students attending a university course on virtual agents in two programming assignments. For each, the students had to code high-level behavior of a 3D virtual agent solving a game-like task in the Unreal Tournament 2004 environment. Each student had to use Java for one task and the POSH reactive planner with a graphical editor for the other. We collected quantitative and qualitative usability data. The results indicate that POSH outperforms Java in terms of usability for one of the assigned tasks but not the other. This implies that the suitability of an AI systems-engineering approach is task sensitive. We also discuss lessons learnt about the evaluation process itself, proposing possible improvements in the experimental design. We conclude that comparative studies are a useful method for analyzing benefits of different approaches to controlling virtual agents.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Fu, D., Houlette, R.: The Ultimate Guide to FSMs in Games. In: AI Game Programming Wisdom II, pp. 283–302. Charles River Media (2004)

    Google Scholar 

  2. Champandard, A.J.: Behavior Trees for Next-Gen Game AI. Internet presentation (January 18, 2011), http://aigamedev.com/insider/presentations/behavior-trees

  3. Schuytema, P.: Game Development with Lua. Charles River Media (2005)

    Google Scholar 

  4. UnrealScript programming language (January 18, 2011), http://unreal.epicgames.com/UnrealScript.htm

  5. Schwab, B.: AI Game Engine Programming, 2nd edn. Charles River Media (2008)

    Google Scholar 

  6. AiGameDev community (January 18, 2011), http://aigamedev.com/

  7. Rabin S.: AI Game Programming Wisdom series (January 18, 2011), http://www.aiwisdom.com/

  8. Gamasutra webpage (January 18, 2011), http://www.gamasutra.com/

  9. Magerko, B., Laird, J.E., Assanie, M., Kerfoot, A., Stokes, D.: AI Characters and Directors for Interactive Computer Games. In: Proceedings of the 2004 Innovative Applications of Artificial Intelligence Conference, San Jose, CA. AAAI Press (July 2004)

    Google Scholar 

  10. Best, B.J., Lebiere, C.: Cognitive agents interacting in real and virtual worlds. In: Sun, R. (ed.) Cognition and Multi-Agent Interaction: From Cognitive Modeling to Social Simulation. Cambridge University Press, NY, NY (2006)

    Google Scholar 

  11. Hindriks, K.V., van Riemsdijk, B., Behrens, T., Korstanje, R., Kraayenbrink, N., Pasman, W., de Rijk, L.: Unreal Goal Bots: Conceptual Design of a Reusable Interface. In: Dignum, F. (ed.) Agents for Games and Simulations II. LNCS (LNAI), vol. 6525, pp. 1–18. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  12. Bryson, J.J.: Inteligence by design: Principles of Modularity and Coordination for Engineering Complex Adaptive Agent. PhD Thesis, MIT, Department of EECS, Cambridge, MA (2001)

    Google Scholar 

  13. Partington, S.J., Bryson, J.J.: The Behavior Oriented Design of an Unreal Tournament Character. In: Panayiotopoulos, T., Gratch, J., Aylett, R.S., Ballin, D., Olivier, P., Rist, T. (eds.) IVA 2005. LNCS (LNAI), vol. 3661, pp. 466–477. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  14. Köster, M., Novák, P., Mainzer, D., Fuhrmann, B.: Two Case Studies for Jazzyk BSM. In: Dignum, F., Bradshaw, J., Silverman, B., van Doesburg, W. (eds.) Agents for Games and Simulations. LNCS (LNAI), vol. 5920, pp. 33–47. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  15. Dignum, F., Bradshaw, J., Silverman, B., van Doesburg, W. (eds.): Agents for Games and Simulations. LNCS, vol. 5920. Springer, Heidelberg (2009)

    Google Scholar 

  16. Tyrrell, T.: Computational Mechanisms for Action Selection. Ph.D. Dissertation. Centre for Cognitive Science, University of Edinburgh (1993)

    Google Scholar 

  17. Bryson, J.J.: Hierarchy and Sequence vs. Full Parallelism in Action Selection. In: Simulation of Adaptive Behavior 6, Paris, pp. 147–156 (2000)

    Google Scholar 

  18. Bryson, J.J.: Action Selection and Individuation in Agent Based Modelling. In: Proceedings of Agent 2003: Challenges of Social Simulation, Argonne National Laboratory, pp. 317–330 (2003)

    Google Scholar 

  19. Hindriks, K.V., van Riemsdijk, M.B., Jonker, C.M.: An Empirical Study of Patterns in Agent Programs. In: Desai, N., Liu, A., Winikoff, M. (eds.) PRIMA 2010. LNCS, vol. 7057, pp. 196–211. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  20. Brom, C.: Curricula of the course on modelling behaviour of human and animal-like agents. In: Proceedings of the Frontiers in Science Education Research Conference, Famagusta, North, Cyprus (2009)

    Google Scholar 

  21. Gemrot, J., Brom, C., Kadlec, R., Bída, M., Burkert, O., Zemčák, M., Píbil, R., Plch, T.: Pogamut 3 – Virtual Humans Made Simple. In: Gray, J. (ed.) Advances in Cognitive Science, pp. 211–243. The Institution Of Engineering And Technology (2010)

    Google Scholar 

  22. Brom, C., Gemrot, J., Burkert, O., Kadlec, R., Bída, M.: 3D Immersion in Virtual Agents Education. In: Spierling, U., Szilas, N. (eds.) ICIDS 2008. LNCS, vol. 5334, pp. 59–70. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  23. Artifical beings course, practical lessons slides (January 18, 2011), http://diana.ms.mff.cuni.cz/pogamut-devel/doku.php?id=lectures

  24. Pogamut 3 platform documentation (January 25, 2011), http://diana.ms.mff.cuni.cz/main/tiki-index.php?page=Documentation

  25. Artificial beings course, final exam package (January 18, 2011), http://diana.ms.mff.cuni.cz/pogamut-devel/doku.php?id=human-like_artifical_agents_2009-10_summer_semester_exam_info

  26. Bordini, R.H., Hübner, J.F., Wooldridge, M.: Programming Multi-Agent Systems in AgentSpeak Using Jason. John Wiley & Sons, Ltd. (2007)

    Google Scholar 

  27. Brooks, R.A.: Intelligence Without Representation. Artificial Intelligence 47(1-3), 139–159 (1991)

    Article  Google Scholar 

  28. Bozada, T.A., Perkins, T.K., North, M.J., Kathy, K.L., Simunich, L., Tatara, E.: An Applied Approach to Representing Human Behavior in Military Logistics Operations. In: Fall Simulation Interoperability Workshop, Simulation Standards Interoperability Organization, Orlando, FL USA (September 2006)

    Google Scholar 

  29. Desai, N.: Using Describers To Simplify ScriptEase. Master Thesis. Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Gemrot, J., Brom, C., Bryson, J., Bída, M. (2012). How to Compare Usability of Techniques for the Specification of Virtual Agents’ Behavior? An Experimental Pilot Study with Human Subjects. In: Beer, M., Brom, C., Dignum, F., Soo, VW. (eds) Agents for Educational Games and Simulations. AEGS 2011. Lecture Notes in Computer Science(), vol 7471. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-32326-3_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-32326-3_3

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-32325-6

  • Online ISBN: 978-3-642-32326-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics