Skip to main content

Advertisement

SpringerLink
Log in

An analysis of students’ gaming behaviors in an intelligent tutoring system: predictors and impacts

  • Original Paper
  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

Students who exploit properties of an instructional system to make progress while avoiding learning are said to be “gaming” the system. In order to investigate what causes gaming and how it impacts students, we analyzed log data from two Intelligent Tutoring Systems (ITS). The primary analyses focused on six college physics classes using the Andes ITS for homework and test preparation, starting with the research question: What is a better predictor of gaming, problem or student? To address this question, we developed a computational gaming detector for automatically labeling the Andes data, and applied several data mining techniques, including machine learning of Bayesian network parameters. Contrary to some prior findings, the analyses indicated that student was a better predictor of gaming than problem. This result was surprising, so we tested and confirmed it with log data from a second ITS (the Algebra Cognitive Tutor) and population (high school students). Given that student was more predictive of gaming than problem, subsequent analyses focused on how students gamed and in turn benefited (or not) from instructional features of the environment, as well as how gaming in general influenced problem solving and learning outcomes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aleven, V.: Helping students to become better help seekers: towards supporting metacognition in a cognitive tutor. In: Paper Presented at German-USA Early Career Research Exchange Program: Research on Learning Technologies and Technology-Supported Education (2001)

  • Aleven V., McLaren B., Roll I., Koedinger K.: Toward meta-cognitive tutoring: a model of help seeking with a cognitive tutor. Int. J. Artif. Intell. Educ. 16, 101–128 (2006)

    Google Scholar 

  • Arroyo I., Woolf  B.: Inferring learning and attitudes from a Bayesian network of log file data. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED’05), Amsterdam, Netherlands, pp. 33–40 (2005)

  • Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., Barto, A., Mahadevan, S., Woolf, B.: Repairing disengagement with non-invasive interventions. In: Proceedings of the 13th International Conference on Artificial Intelligence in Education (AIED’07), Los Angeles, United States, pp. 195–202 (2007)

  • Baker, R.: Is gaming the system state-or-trait? educational data mining through the multi-contextual application of a validated behavioral model. In: Proceedings of the Workshop on Data Mining for User Modeling, Corfu, Greece, pp. 76–80 (2007)

  • Baker, R., Corbett, A., Koedinger, K.: Detecting student misuse of intelligent tutoring systems. In: Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS’04), Maceio, Brazil, pp. 531–540 (2004a)

  • Baker, R., Corbett, A., Koedinger, K., Wagner, A.: Off-task behavior in the cognitive tutor classroom: when students “game the system”. In: Proceedings of the ACM CHI 2004: Computer-Human Interaction (CHI’04), Vienna, Austria, pp. 383–390 (2004b)

  • Baker, R.S., Roll, I., Corbett, A., Koedinger, K.: Do performance goals lead students to game the system. In: Proceedings of the 12th International Conference on Artificial Intelligence and Education (AIED2005), Amsterdam, Netherlands, pp. 57–64 (2005)

  • Baker, R., Corbett, A., Koedinger, K., Evenson, E., Roll, I., Wagner, A., Naim, M., Raspat, J., Baker, D., Beck, J.: Adapting to when students game an intelligent tutoring system. In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 392–401 (2006a)

  • Baker, R.S.J.d., Corbett, A.T., Koedinger, K., Roll, I.: Generalizing detection of gaming the system across a tutoring curriculum. In: Proceedings of the 11th International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 402–411 (2006b)

  • Baker R., Corbett A., Roll I., Koedinger K.: Developing a generalizable detector of when students game the system. User Model. User-Adap. Inter. 18(3), 287–314 (2008a)

    Article  Google Scholar 

  • Baker R., Walonoski J., Heffernan N., Roll I., Corbett A., Koedinger K.: Why students engage in “gaming the system”. Behavior in interactive learning environments. J. Interact. Learn. Res. 19(2), 185–224 (2008b)

    Google Scholar 

  • Baker, R., Corbett, A., Koedinger, K.: Educational software features that encourage and discourage “gaming the system”. In: Proceedings of the 14th international conference on artificial intelligence in education (AIED’09), Brighton, UK, pp. 475–482 (2009)

  • Baker R.: Data mining for education. In: McGaw, B., Peterson, P., Baker, E., (eds) International Encyclopedia of Education, vol. 7, 3rd edn, pp. 112–118. Elsevier Oxford, UK (2010)

    Chapter  Google Scholar 

  • Beck J.E., Jia P., Mostow J.: Using automated questions to assess reading comprehension, vocabulary, and effects of tutorial interventions. Technol. Instr. Cogn. Learn. (TICL) 2, 97–134 (2004)

    Google Scholar 

  • Beck, J., Chang, K., Mostow, J., Corbett, A.: Does help? Introducing the Bayesian evaluation and assessment methodology. In: Proceedings of the 9th International Conference on Intelligent Tutoring Systems (ITS’08), Montreal, Canada, pp. 383–394 (2008)

  • Bunt, A., Conati, C., Muldner, K.: Scaffolding self-explanation to improve learning in exploratory learning environments. In: Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS’04), Maceio, Brazil, pp. 656–667 (2004)

  • Carnegie Learning Inc.: Retrieved April 1, 2010, from http://www.carnegielearning.com/ (2010)

  • Chang, K., Beck, J., Mostow, J., Corbett, A.: A Bayes net toolkit for student modeling in intelligent tutoring systems. In: Proceedings of the 11th International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 104–113 (2006)

  • Chi, M.T.H.: How adaptive is an expert human tutor? In: Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS’10), Pittsburgh, United States, pp. 401–113 (2010)

  • Chi, M., VanLehn, K.: The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In: Proceedings of the 29th annual conference of the cognitive science society, Nashville, Tennessee, pp. 167–172 (2007a)

  • Chi, M., VanLehn, K.: Accelerated future learning via explicit instruction of a problem solving strategy. In: Proceedings of the 13th international conference on artificial intelligence in education (AIED’07), Los Angeles, United States, pp. 409–416 (2007b)

  • Chi M.T.H., Bassok M., Lewis M., Reimann P., Glaser R.: Self-explanations: how students study and use examples in learning to solve problems. Cogn. Sci. 15, 145–182 (1989)

    Google Scholar 

  • Chi M.T.H., Siler S.A., Jeong H., Yamauchi T., Hausmann R.G.: Learning from human tutoring. Cogn. Sci. 25, 471–533 (2001)

    Article  Google Scholar 

  • Chi M.T.H., Roy M., Hausmann R.: Observing tutoring collaboratively: insights about tutoring effectiveness from vicarious learning. Cogn. Sci. 32(2), 301–341 (2008)

    Article  Google Scholar 

  • Chi, M., VanLehn, K., Litman, D.: Do micro-level tutorial decisions matter: applying reinforcement learning to induce pedagogical tutorial tactics. In: Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS’10), Pittsburgh, United States, pp. 184–193 (2010a)

  • Chi, M., VanLehn, K., Litman, D., Jordan, P.: Inducing effective pedagogical strategies using learning context features. In: Proceedings of the 18th International Conference on User Modeling, Adaptation and Personalization (UMAP’10), Kona, Hawaii, pp. 147–158 (2010b)

  • Chi, M., VanLehn, K., Litman, D., Jordan P.: Empirically evaluating the Application of reinforcement learning to the induction of effective and adaptive pedagogical strategies. User model User-Adap. Inter. (to appear). Special issue on educational data mining for personalized educational systems

  • Cohen P., Beal C.: Temporal data mining for educational applications. Int. J. Softw. Inform. 3(1), 31–46 (2009)

    Google Scholar 

  • Conati C., Gertner A., VanLehn K.: Using Bayesian networks to manage uncertainty in student modeling. User Model. User-Adap. Inter. 12(4), 371–417 (2002)

    Article  MATH  Google Scholar 

  • Conati C., Maclaren H.: Empirically building and evaluating a probabilistic model of user affect. User Model. User-Adap. Inter. 19(3), 267–303 (2009)

    Article  Google Scholar 

  • Corbett A.T., Anderson J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User-Adap. Inter. 4(4), 253–278 (1995)

    Article  Google Scholar 

  • D’Mello S., Craig S., Sullins J., Graesser A.: Predicting affective states expressed through an emote-aloud procedure from autotutor’s mixed-initiative dialogue. Int. J. Artif. Intell. Educ. 16(1), 3–28 (2006)

    Google Scholar 

  • Davis W., Carson C., Ammeter A., Treadway D.: The interactive effects of goal orientation and feedback specificity on task performance. Human Perform. 18, 409–426 (2005)

    Article  Google Scholar 

  • Dean T., Kanazawa K.: A model for reasoning about persistence and causation. Comput. Intell. 5(3), 142–150 (1989)

    Article  Google Scholar 

  • Dempster L., Laird N., Rubin D.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. 39(1), 1–38 (1977)

    MATH  MathSciNet  Google Scholar 

  • Feng, M., Beck, J., Heffernan, N.T.: Using learning decomposition and bootstrapping with randomization to compare the impact of different educational interventions on learning. In: Proceedings of the 2nd International Conference on Educational Data Mining (EDM’09), Cordoba, Spain, pp. 51–60 (2009)

  • Getoor L., Rhee J., Koller D., Small P.: Understanding tuberculosis epidemiology using probabilistic relational models. J. Artif. Intell. Med. 30, 233–256 (2004)

    Article  Google Scholar 

  • Hays, M., Lane, C., Auerbach, D., Core, M., Gomboc, D., Rosenberg, M.: Feedback specificity and the learning of intercultural communication skills. In: Proceedings of the 14th International Conference on Artificial Intelligence in Education (AIED’09), Brighton, UK, pp. 391–398 (2009)

  • Heckerman D.: Bayesian networks for data mining. Data Min. Knowl. Discov. 1, 79–119 (1997)

    Article  Google Scholar 

  • Koedinger, K.R., Anderson, Hadley, W.H., Mark, M.A.: Intelligent tutoring goes to school in the big city. In: Proceedings of the 3rd International Conference on Artificial Intelligence and Education (AIED’95), Washington, DC, United States, pp. 421–428 (1995)

  • Mayo M., Mitrovic A.: Optimizing ITS behaviour with Bayesian networks and decision theory. Int. J. Artif. Intell. Educ. 12, 124–153 (2001)

    Google Scholar 

  • Muldner, K., Conati, C.: Using similarity to infer meta-cognitive behaviors during analogical problem solving. In: Proceedings of the 10th International Conference on User Modeling (UM’05), Edinborough, Scottland, pp. 134–143 (2005)

  • Muldner, K., Burleson, W., de Sande, B., VanLehn, K.: An analysis of gaming behaviors in an intelligent tutoring system. In: Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS’10), Pittsburgh, United States, pp. 195–205 (2010)

  • Muldner K., Conati C.: Scaffolding meta-cognitive skills for effective analogical problem solving via tailored example selection. Int. J. Artif. Intell. Educ. 20(2), 99–136 (2010)

    Google Scholar 

  • Murphy, K.: Dynamic Bayesian Networks: Representation, Inference and Learning. Ph.D. Thesis, UC, Berkeley (2002)

  • Murphy, K.: Bayes Net Toolbox for Matlab. http://bnt.sourceforge.net. Accessed 2010 April 1 (2004)

  • Murray, R.C., VanLehn, K.: Effects of dissuading unnecessary help requests while providing proactive help. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED’05), Amsterdam, Netherlands, pp. 887–889 (2005)

  • Nelson-Le Gall S.: Help-seeking behavior in learning review of research in education. Rev. Res. Educ. 12, 55–90 (1985)

    Google Scholar 

  • Netica Reference Manual. Retrieved March 1, 2010, from www.norsys.com/netica-j/docs/NeticaJ_Man.pdf (2010)

  • Phye G.D., Sanders C.: Advice and feedback: elements of practice for problem solving. Contemp. Educ. Psychol. 19(3), 286–301 (1994)

    Article  Google Scholar 

  • Rai, D., Gong, Y., Beck, J.: Using Dirichlet priors to improve model parameter plausibility. In: Proceedings of the international conference on educational data mining (EDM’09), pp. 141–150 (2009)

  • Razzaq, L., Heffernan, N.: To tutor or not to tutor: that is the question. In: Proceedings of th 2nd International Conference on Artificial Intelligence in Education (AIED’09), Cordoba, Spain, pp. 457–464 (2009)

  • Renkl A.: Learning from worked-examples: a study on individual differences. Cogn. Sci. 21(1), 1–30 (1997)

    Article  Google Scholar 

  • Reye J.: Student modeling based on belief networks. Int. J. Artif. Intell. Educ. 14, 1–33 (2004)

    Google Scholar 

  • Ritter, S., Harris, T., Nixon, T., Dickison, D., Murray, R.C., Towle, B.: Reducing the knowledge tracing space. In: Proceedings of the 1st International Conference On Educational Data Mining (EDM’08), Montreal, Canada, pp. 151–160 (2008)

  • Rodin A., Mosley T.H., Clark A.G., Sing C.F., Boerwinkle E.: Mining genetic epidemiology data with Bayesian networks application to APOE gene variation and plasma lipid levels. J. Comput. Biol. 12(1), 1–11 (2005)

    Article  Google Scholar 

  • Rodrigo, M., Baker, R., d’Mello, S., Gonzalez, M.C.T., Lagud, M., Lim, S., Macapanpan, A., Pascua, S., Santillano, J., Sugay, J., Tep, S., Viehland, N.: Comparing learners affect while using an intelligent tutoring systems and a simulation problem solving game. In: Proceedings of the 9th International Conference on Intelligent Tutoring Systems (ITS’08), Montreal, Canada, pp. 40–49 (2008)

  • Roll, I., Aleven, V., McLaren, B., Ryu, E., Baker, R.S.J.d., Koedinger, K.: The help tutor: does metacognitive feedback improve students’ help-seeking actions, skills and learning? In: Proceedings of the Eight International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 360–369 (2006)

  • Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach. Los Altos, CA, Morgan-Kaufman (2009)

  • Shih, B., Koedinger, K., Scheines, R.: A response time model for bottom-out hints as worked examples. In: Proceedings of the 1st International Conference on Educational Data Mining (EDM’08), Montreal Canada, pp. 117–126 (2008)

  • Shute V.: Focus on formative feedback. Rev. Educ. Res. 78(1), 153–189 (2008)

    Article  Google Scholar 

  • VanLehn K.: Analogy events: how examples are used during problem solving. Cogn. Sci. 22(3), 347–388 (1998)

    Article  Google Scholar 

  • VanLehn K., Lynch C., Schulze K., Shapiro J.A., Shelby R., Taylor L., Treacy D., Weinstein A., WintersgillM.TheAndesphysics tutoring system: Lessons learned. Int. J. Artif. Intell. Educ. 15(3), 1–47 (2005)

    Google Scholar 

  • VassarStats: Website for Statistical Computation. Retrieved April 1, 2010, from http://faculty.vassar.edu/lowry/VassarStats.html

  • Walonoski, J.A., Heffernan, N.T.: Detection and analysis of off-task gaming behavior in intelligent tutoring systems. In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 382–391 (2006a)

  • Walonoski, J.A., Heffernan, N.T.: Prevention of off-task gaming behavior in intelligent tutoring systems. In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS’06), Jhongli, Taiwan, pp. 722–724 (2006b)

  • Zhang, X., Mostow, J., Beck, J.: A case study empirical comparison of three methods to evaluate tutorial behaviors. In: Proceedings of the 9th International Conference on Intelligent Tutoring Systems (ITS’08), Montral, Canada, pp. 122–131 (2008)

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Arizona State University, Tempe, AZ, USA

    Kasia Muldner

  2. School of Computing, Informatics and Decision Systems Engineering, Arizona State University, Tempe, AZ, USA

    Winslow Burleson, Brett Van de Sande & Kurt VanLehn

Authors
  1. Kasia Muldner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Winslow Burleson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Brett Van de Sande
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kurt VanLehn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kasia Muldner.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Muldner, K., Burleson, W., Van de Sande, B. et al. An analysis of students’ gaming behaviors in an intelligent tutoring system: predictors and impacts. User Model User-Adap Inter 21, 99–135 (2011). https://doi.org/10.1007/s11257-010-9086-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-010-9086-0

Keywords

Search

Navigation