Abstract
A key untapped feature of game-based learning environments is their capacity to generate a rich stream of fine-grained learning interaction data. The learning behaviors captured in these data provide a wealth of information on student learning, which stealth assessment can utilize to unobtrusively draw inferences about student knowledge to provide tailored problem-solving support. In this paper, we present a long short-term memory network (LSTM)-based stealth assessment framework that takes as input an observed sequence of raw game-based learning environment interaction data along with external pre-learning measures to infer students’ post-competencies. The framework is evaluated using data collected from 191 middle school students interacting with a game-based learning environment for middle grade computational thinking. Results indicate that LSTM-based stealth assessors induced from student game-based learning interaction data outperform comparable models that required labor-intensive hand-engineering of input features. The findings suggest that the LSTM-based approach holds significant promise for evidence modeling in stealth assessment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Within the game, students must pair their virtual in-game computer with devices before they can manipulate or view a device’s programs.
References
Lester, J., Ha, E., Lee, S., Mott, B., Rowe, J., Sabourin, J.: Serious games get smart: intelligent game-based learning environments. AI Mag. 34(4), 31–45 (2013)
Jackson, T., McNamara, D.: Motivation and performance in a game-based intelligent tutoring system. J. Educ. Psychol. 105(4), 1036–1049 (2013)
Johnson, L.: Serious use of a serious game for language learning. Int. J. Artif. Intell. Educ. 20(2), 175–195 (2010)
Shute, V.J., Ventura, M.: Measuring and supporting learning in games: stealth assessment. In: Computer Games, Simulations & Education. The MIT Press, Cambridge (2013)
Min, W., Mott, B., Rowe, J., Liu, B., Lester, J.: Player goal recognition in open-world digital games with long short-term memory networks. In: International Joint Conference on Artificial Intelligence, pp. 2590–2596 (2016)
Brusilovsky, P., Millán, E.: User models for adaptive hypermedia and adaptive educational systems. In: The Adaptive Web, pp. 3–53 (2007)
Mislevy, R., Steinberg, L., Almond, R.: On the structure of educational assessment. Meas. Interdiscip. Res. Perspect. 1(1), 3–62 (2003)
Rosenheck, L., Lin, C.Y., Klopfer, E., Cheng, M.T.: Analyzing gameplay data to inform feedback loops in The Radix Endeavor. Comput. Educ. 111, 60–73 (2017)
Kim, Y.J., Almond, R.G., Shute, V.J.: Applying evidence-centered design for the development of game-based assessments in physics playground. Int. J. Test. 16(2), 142–163 (2016)
Smith, A., Aksit, O., Min, W., Wiebe, E., Mott, B.W., Lester, J.C.: Integrating real-time drawing and writing diagnostic models: an evidence-centered design framework for multimodal science assessment. In: Micarelli, A., Stamper, J., Panourgia, K. (eds.) ITS 2016. LNCS, vol. 9684, pp. 165–175. Springer, Cham (2016). doi:10.1007/978-3-319-39583-8_16
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1–32 (1997)
Kebritchi, M., Hirumi, A., Bai, H.: The effects of modern mathematics computer games on mathematics achievement and class motivation. Comput. Educ. 55(2), 427–443 (2010)
Min, W., Frankosky, M.H., Mott, B.W., Rowe, J.P., Wiebe, E., Boyer, K.E., Lester, J.C.: DeepStealth: leveraging deep learning models for stealth assessment in game-based learning environments. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (eds.) AIED 2015. LNCS, vol. 9112, pp. 277–286. Springer, Cham (2015). doi:10.1007/978-3-319-19773-9_28
Vannini, N., Enz, S., Sapouna, M., Wolke, D., Watson, S., Woods, S., Dautenhahn, K., Hall, L., Paiva, A., André, E., Aylett, R.: “FearNot!”: a computer-based anti-bullying-programme designed to foster peer intervention. Eur. J. Psychol. Educ. 26(1), 21–44 (2011)
Nelson, B.C., Kim, Y., Foshee, C., Slack, K.: Visual signaling in virtual world-based assessments: the SAVE Science project. Inf. Sci. 264, 32–40 (2014)
Falakmasir, M.H., Gonzalez-Brenes, J.P., Gordon, G.J., DiCerbo, K.E.: A data-driven approach for inferring student proficiency from game activity logs. In: ACM Conference on Learning at Scale, pp. 341–349 (2016)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
AP® Computer Science Principles Draft Curriculum Framework. http://www.csprinciples.org/. Accessed 05 Feb 2017
K–12 Computer Science Framework. http://www.k12cs.org/. Accessed 05 Feb 2017
Wiebe, E., Williams, L., Yang, K., Miller, C.: Computer science attitude survey. Comput. Sci. 14(25), 1–86 (2003)
Chen, G., Gully, S.M., Eden, D.: Validation of a new general self-efficacy scale. Organ. Res. Methods 4(1), 62–83 (2001)
Graves, A.: Supervised Sequence Labelling with Recurrent Neural Networks. Studies in Computational Intelligence, vol. 385. Springer, Heidelberg (2012)
Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2014)
Bengio, Y.: Practical recommendations for gradient-based training of deep architectures. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 437–478. Springer, Heidelberg (2012). doi:10.1007/978-3-642-35289-8_26
Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2015)
Chollet, F.: Keras. https://github.com/fchollet/keras. GitHub Repository. Accessed 05 Feb 2017
Keerthi, S.S., Lin, C.-J.: Asymptotic behaviors of support vector machines with Gaussian kernel. Neural Comput. 15(7), 1667–1689 (2003)
Acknowledgments
This research was supported by the National Science Foundation under Grant CNS-1138497 and Grant DRL-1640141. Any opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Min, W., Frankosky, M.H., Mott, B.W., Wiebe, E.N., Boyer, K.E., Lester, J.C. (2017). Inducing Stealth Assessors from Game Interaction Data. In: André, E., Baker, R., Hu, X., Rodrigo, M., du Boulay, B. (eds) Artificial Intelligence in Education. AIED 2017. Lecture Notes in Computer Science(), vol 10331. Springer, Cham. https://doi.org/10.1007/978-3-319-61425-0_18
Download citation
DOI: https://doi.org/10.1007/978-3-319-61425-0_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-61424-3
Online ISBN: 978-3-319-61425-0
eBook Packages: Computer ScienceComputer Science (R0)