Skip to main content
SpringerLink
Log in

The Temporal Attentive Observation (TAO) Scale: Development of an Instrument to Assess Attentive Behavior Sequences During Serious Gameplay

  • Original research
  • Published:
Technology, Knowledge and Learning Aims and scope Submit manuscript

Abstract

The engaging nature of video games has intrigued learning professionals attempting to capture and retain learners’ attention. Designing learning interventions that not only capture the learner’s attention, but also are designed around the natural cycle of attention will be vital for learning. This paper introduces the temporal attentive observation (TAO) instrument, an instrument developed to assess attentive behavior sequences during serious gameplay. We use an established three-step process for developing observational systems that includes identifying the construct, determining validity, and demonstrating practicality criteria. We conclude that the TAO instrument reliably measures attention behaviors where participants’ faces can be recorded during an experiment. Furthermore, we suggest that TAO should be considered as a part of an attention measurement package.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Attfield, S., Kazai, G., Lalmas, M., & Piwowarski, B. (February 2011). Towards a science of user engagement (position paper). In WSDM workshop on user modeling for web applications.

  • Bailey, B. P., & Konstan, J. A. (2006). On the need for attention-aware systems: Measuring effects of interruption on task performance, error rate, and affective state. Computers in Human Behavior, 22(4), 685–708.

    Article  Google Scholar 

  • Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Boyle, E. A., Connolly, T. M., Hainey, T., & Boyle, J. M. (2011). Engagement in digital entertainment games: A systematic review. Computers in Human Behavior, 28(3), 771–780.

    Article  Google Scholar 

  • Castel, A. D., Farb, N., & Craik, F. I. M. (2007). Memory for general and specific value information in younger and older adults: Measuring the limits of strategic control. Memory and Cognition, 35, 689–700.

    Article  Google Scholar 

  • Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. (2014). Digital games, design, and learning: A systematic review and meta-analysis. Menlo Park, CA: SRI International.

    Google Scholar 

  • Clark, R. C. (2008). Building expertise: Cognitive methods for training and performance improvement. Munich: Pfeiffer.

    Google Scholar 

  • Clegg, B., Martey, R., Folkestad, J. E., Stromer-Galley, J., Kenski, K., Saulnier, T., et al. (2014). Game-based training to mitigate three forms of cognitive bias. In Interservice/industry training, simulation and education conference.

  • Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model. Simulation and Gaming, 33(4), 441–467.

    Article  Google Scholar 

  • Gee, J. P. (2003). What video games have to teach us about learning and literacy. Computers in Entertainment (CIE), 1(1), 20.

    Article  Google Scholar 

  • Godwin, K. E., Almeda, M. V., Petroccia, M., Baker, R. S., & Fisher, A. V. (2013). Classroom activities and off-task behavior in elementary school children. Cognitive Science Society.

  • Hays, R. T. (2005). The effectiveness of instructional games: A literature review and discussion (no. NAWCTSD-TR-2005-004). Naval Air Warfare Center Training Systems Division.

  • Herbert, J., & Attridge, C. (1975). A guide for developers and users of observation systems and manuals. American Educational Research Journal, 12(1), 1–20.

    Article  Google Scholar 

  • Herrington, J., Oliver, R., & Reeves, T. C. (2003). Patterns of engagement in authentic online learning environments. Australian Journal of Educational Technology, 19(1), 59–71.

    Google Scholar 

  • Holmqvist, K., Nyström, M., & Mulvey, F. (March 2012). Eye tracker data quality: What it is and how to measure it. In Proceedings of the symposium on eye tracking research and applications (pp. 45–52). ACM.

  • Hornof, A. J., & Halverson, T. (2002). Cleaning up systematic error in eye-tracking data by using required fixation locations. Behavior Research Methods, Instruments, and Computers, 34(4), 592–604.

    Article  Google Scholar 

  • Ismail, E., Sidek, S. N., Khan, M. R., Jalaludin, N. A. (July 2012). Analysis of engagement factor in trajectory tracking-based experiment. In 2012 international conference on computer and communication engineering (ICCCE). Kuala Lumpur.

  • Kahneman, D. (2011). Thinking, fast and slow. New York: Macmillan.

    Google Scholar 

  • Karimi, A., & Lim, Y. P. (December 2010). Children, engagement, and enjoyment in digital narrative. In Proceedings of ASCILITE 2010. Sydney. Retrieved June 28, 2013 from http://www.ascilite.org.au/conferences/sydney10/procs/Karimi-full.pdf.

  • Larrick, R. P. (2004). Debiasing. In D. J. Koehler & N. Harvey (Eds.), Blackwell handbook of judgment and decision making (pp. 316–338). Malden, MA: Blackwell.

    Chapter  Google Scholar 

  • Lim, C. P. (2008). Spirit of the game: Empowering students as designers in schools? British Journal of Educational Technology, 39(6), 996–1003.

    Article  Google Scholar 

  • López, F., Menez, M., & Hernández-Guzmán, L. (2005). Sustained attention during learning activities: An observational study with pre-school children. Early Child Development and Care, 175(2), 131–138.

    Article  Google Scholar 

  • Martey, R. M., Kenski, K., Folkestad, J., Feldman, L., Gordis, E., Shaw, A., et al. (2014). Measuring game engagement multiple methods and construct complexity. Simulation and Gaming, 45, 528–547.

    Article  Google Scholar 

  • McGonigal, J. (2011). Reality is broken: Why games make us better and how they can change the world. New York, NY: Penguin.

    Google Scholar 

  • Medina, J. (2008). Brain rules: 12 principles for surviving and thriving at work, home, and school. Edmonds: Pear Press.

    Google Scholar 

  • Medina, J. (2014). Brain rules for baby, updated and expanded: How to raise a smart and happy child from zero to five. Edmonds: Pear Press.

  • Mezirow, J. (1990). Fostering critical reflection in adulthood. San Francisco: Jossey-Bass.

    Google Scholar 

  • Middlebrooks, C. D., McGillivray, S., Murayama, K., & Castel, A. D. (2015). Memory for allergies and health foods: How younger and older adults strategically remember critical health information. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 71(3), 389–399.

  • Mullinix, G., Gray, O., Colado, J., Veinott, E., Leonard, J., Papautsky, E. L., & Todd, P. M. (September 2013). Heuristica: Designing a serious game for improving decision making. In 2013 IEEE international games innovation conference (IGIC) (pp. 250–255). IEEE.

  • O’Brien, H. L., & Toms, E. G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59(6), 938–955.

    Article  Google Scholar 

  • O’Brien, H. L., Toms, E. G., Kelloway, E. K., & Kelley, E. (2008). Developing and evaluating a reliable measure of user engagement. Proceedings of the American Society for Information Science and Technology, 45(1), 1–10.

    Article  Google Scholar 

  • O’Neil, H. F., Wainess, R., & Baker, E. L. (2005). Classification of learning outcomes: Evidence from the computer games literature. The Cirriculum Journal, 16(4), 455–474.

    Article  Google Scholar 

  • Packard, R. G. (1970). The control of “classroom attention”: A group contingency for complex behavior 1. Journal of Applied Behavior Analysis, 3(1), 13–28.

    Article  Google Scholar 

  • Prensky, M. (2003). Digital game-based learning. Computers in Entertainment (CIE), 1(1), 21.

    Article  Google Scholar 

  • Ravaja, N., Saari, T., Salminen, M., Laarni, J., & Kallinen, K. (2006). Phasic emotional reactions to video game events: A psychophysiological investigation. Media Psychology, 8, 343–367.

    Article  Google Scholar 

  • Read, J. C., MacFarlane, S. J., & Casey, C. (2002). Endurability,  engagement and expectations: Measuring children’s fun. Interaction Design and Children, 2, 1–23.

  • Reese, D. D. (2015). CyGaMEs selene player log dataset: Gameplay assessment, flow dimensions and non-gameplay assessments. British Journal of Educational Technology, 46(5), 1005–1014.

    Article  Google Scholar 

  • Renshaw, T., Stevens, R., & Denton, P. D. (2009). Towards understanding engagement in games: An eye-tracking study. On the Horizon, 17, 408–420.

    Article  Google Scholar 

  • Reynolds, R. E., & Anderson, R. C. (1982). Influence of questions on the allocation of attention during reading. Journal of Educational Psychology, 74(5), 623.

    Article  Google Scholar 

  • Ricci, K., Salas, E., & Cannon-Bowers, J. A. (1996). Do computer-based games facilitate knowledge acquisition and retention? Military Psychology, 8(4), 295–307.

    Article  Google Scholar 

  • Squire, K. (2011). Video games and learcning: Teaching and participatory culture in the digital age. Technology, Education--Connections (the TEC Series). New York: Teachers College Press.

  • Stevens, C., & Bavelier, D. (2012). The role of selective attention on academic foundations: A cognitive neuroscience perspective. Developmental Cognitive Neuroscience, 2, S30–S48.

    Article  Google Scholar 

  • Symborski, C., Barton, M., Quinn, M., Morewedge, C., Kassam, K., Korris, J., & Hollywood, C. A. (2014). Missing: A serious game for the mitigation of cognitive biases. In Proceedings of the interservice/industry training, simulation, and education conference (I/ITSEC) (pp. 1–13).

  • Teixeira, T., Wedel, M., & Pieters, R. (2012). Emotion-induced engagement in Internet video advertisements. Journal of Marketing Research, 49, 144–159.

    Article  Google Scholar 

  • Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131.

    Article  Google Scholar 

  • Vogel, J. F., Vogel, D. S., Cannon-Bowers, J., Bowers, C. A., Muse, K., & Wright, M. (2006). Computer gaming and interactive simulations for learning: A meta-analysis. Journal of Educational Computing Research, 34, 229–243.

    Article  Google Scholar 

  • Wozniak, R. H. (1999). Introduction to memory: Hermann Ebbinghaus (1885/1913). In Classics in the history of psychology. Bristol: Thoemmes Press.

Download references

Acknowledgements

This work was supported by the Intelligence Advanced Research Projects Activity (IARPA) via the Air Force Research Laboratory contract number FA8650-11-C-7176. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, AFRL, or the US Government.

Author information

Authors and Affiliations

  1. School of Education, Colorado State University, Fort Collins, CO, 80523, USA

    James E. Folkestad

  2. Department of Interdisciplinary Studies, The Sage Colleges, Albany, NY, 12208, USA

    Brian McKernan

  3. Journalism and Technical Communication, Colorado State University, Fort Collins, CO, 80523, USA

    Stephanie Train & Rosa Mikeal Martey

  4. Department of Psychology, Colorado State University, Fort Collins, CO, 80523, USA

    Matthew G. Rhodes & Benjamin A. Clegg

  5. Department of Communication, University of Arizona, Tucson, AZ, 85721, USA

    Kate Kenski

  6. Temple University, Philadelphia, PA, 19122, USA

    Adrienne Shaw

  7. Syracuse University, Syracuse, NY, 13244, USA

    Jennifer Stromer-Galley

  8. The Institute for Informatics, Logics, and Security Studies, University of Albany, SUNY, Albany, NY, 12222, USA

    Tomek Strzalkowski

Authors
  1. James E. Folkestad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian McKernan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephanie Train
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rosa Mikeal Martey
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Matthew G. Rhodes
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kate Kenski
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Adrienne Shaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jennifer Stromer-Galley
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Benjamin A. Clegg
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Tomek Strzalkowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James E. Folkestad.

Appendix

Appendix

See Table 2.

Table 2 TAOS coding guide sheet
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Folkestad, J.E., McKernan, B., Train, S. et al. The Temporal Attentive Observation (TAO) Scale: Development of an Instrument to Assess Attentive Behavior Sequences During Serious Gameplay. Tech Know Learn 23, 65–81 (2018). https://doi.org/10.1007/s10758-017-9302-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10758-017-9302-7

Keywords

Search

Navigation