Skip to main content

Student-Tutor Mixed-Initiative Decision-Making Supported by Deep Reinforcement Learning

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13355))

Included in the following conference series:

  • 4682 Accesses

Abstract

One fundamental goal of education is to enable students to act independently in the world by continuously adapting and learning. Certain learners are less sensitive to learning environments and can always perform well, while others are more sensitive to variations in learning environments and may fail to learn. We refer to the former as high performers and the latter as low performers. Previous research showed that low performers benefit more from tutor-driven Intelligent Tutoring Systems (ITSs), in which the tutor makes pedagogical decisions, while the high ones often prefer to take control of their own learning by making decisions by themselves. We propose a student-tutor mixed-initiative (ST-MI) decision-making framework which balances allowing students some control over their own learning while ensuring effective pedagogical interventions. In an empirical study, ST-MI significantly improved student learning gains than an Expert-designed, tutor-driven pedagogical policy on an ITS. Furthermore, our ST-MI framework was found to offer low performers the same benefits as the Expert policy, while that for high performers was significantly greater than the Expert policy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aleven, V., Koedinger, K.R.: Limitations of student control: do students know when they need help? In: Intelligent Tutoring Systems, pp. 292–303 (2000)

    Google Scholar 

  2. Ausin, M.S., Maniktala, M., Barnes, T., Chi, M.: Tackling the credit assignment problem in reinforcement learning-induced pedagogical policies with neural networks. In: AIED (2021)

    Google Scholar 

  3. Barto, A.G., Mahadevan, S.: Recent advances in hierarchical reinforcement learning. Discrete Event Dyn. Syst. 13(1–2), 41–77 (2003)

    Article  MathSciNet  Google Scholar 

  4. Cordova, D.I., Lepper, M.R.: Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice. J. Educ. Psychol. 88(4), 715 (1996)

    Google Scholar 

  5. Fachantidis, A., Taylor, M.E., Vlahavas, I.P.: Learning to teach reinforcement learning agents. Mach. Learn. Knowl, Extra. (2017)

    Google Scholar 

  6. Flowerday, T., Schraw, G., Stevens, J.: The role of choice and interest in reader engagement. J. Exp. Educ. 72(2), 93–114 (2004)

    Article  Google Scholar 

  7. Ju, S., Zhou, G., Abdelshiheed, M., Barnes, T., Chi: M.: Evaluating critical reinforcement learning framework in the field. In: AIED, pp. 215–227 (2021)

    Google Scholar 

  8. Ju, S., Zhou, G., Barnes, T., Chi, M.: Pick the moment: Identifying critical pedagogical decisions using long-short term rewards. In: EDM (2020)

    Google Scholar 

  9. Kinzie, M.B., Sullivan, H.J.: Continuing motivation, learner control, and CAI. Education Tech. Research Dev. 37(2), 5–14 (1989)

    Article  Google Scholar 

  10. Mandel, T., Liu, Y.E., Levine, S., Brunskill, E., Popovic, Z.: Offline policy evaluation across representations with applications to educational games. In: AAMAS, pp. 1077–1084 (2014)

    Google Scholar 

  11. Maniktala, M., Cody, C., Barnes, T., Chi, M.: Avoiding help avoidance: Using interface design changes to promote unsolicited hint usage in an intelligent tutor. Int. J. Artif. Intell. Educ. 30(4), 637–667 (2020)

    Article  Google Scholar 

  12. McClure, S.M., Laibson, D.I., Loewenstein, G., Cohen, J.D.: Separate neural systems value immediate and delayed monetary rewards. Science 306(5695), 503–507 (2004)

    Article  Google Scholar 

  13. Mitrovic, A., Martin, B.: Scaffolding and fading problem selection in sql-tutor. In: AIED, pp. 479–481 (2003)

    Google Scholar 

  14. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

    Article  Google Scholar 

  15. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., Bergman, H.: Midbrain dopamine neurons encode decisions for future action. NatureNeuro 9(8), 1057–1063 (2006)

    Google Scholar 

  16. Najar, A.S., Mitrovic, A., McLaren, B.M.: Adaptive support versus alternating worked examples and tutored problems: which leads to better learning? In: Dimitrova, V., Kuflik, T., Chin, D., Ricci, F., Dolog, P., Houben, G.-J. (eds.) UMAP 2014. LNCS, vol. 8538, pp. 171–182. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-08786-3_15

    Chapter  Google Scholar 

  17. Renkl, A., Atkinson, R.K., Maier, U.H., Staley, R.: From example study to problem solving: smooth transitions help learning. J. Exp. Educ. 70(4), 293–315 (2002)

    Article  Google Scholar 

  18. Roesch, M.R., Calu, D.J., Schoenbaum, G.: Dopamine neurons encode the better option in rats deciding between different delayed or sized rewards. Nat. Neurosci. 10(12), 1615–1624 (2007)

    Article  Google Scholar 

  19. Roll, I., Wiese, E.S., Long, Y., Aleven, V., Koedinger, K.R.: Tutoring self-and co-regulation with intelligent tutoring systems to help students acquire better learning skills. Design Recomm. Intell. Tutoring Syst. 2, 169–182 (2014)

    Google Scholar 

  20. Rowe, J.P., Lester, J.C.: Improving student problem solving in narrative-centered learning environments: a modular reinforcement Learning framework. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (eds.) AIED 2015. LNCS (LNAI), vol. 9112, pp. 419–428. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19773-9_42

    Chapter  Google Scholar 

  21. Schneider, S., Nebel, S., Beege, M., Rey, G.D.: The autonomy-enhancing effects of choice on cognitive load, motivation and learning with digital media. Learn. Instr. 58, 161–172 (2018)

    Article  Google Scholar 

  22. Shen, S., Chi, M.: Reinforcement learning: the sooner the better, or the later the better? In: UMAP, pp. 37–44. ACM (2016)

    Google Scholar 

  23. Sul, J.H., Jo, S., Lee, D., Jung, M.W.: Role of rodent secondary motor cortex in value-based action selection. Nat. Neurosci. 14(9), 1202–1208 (2011)

    Article  Google Scholar 

  24. Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press (2018)

    Google Scholar 

  25. Torrey, L., Taylor, M.E.: Teaching on a budget: agents advising agents in reinforcement learning. In: AAMAS, pp. 1053–1060 (2013)

    Google Scholar 

  26. Van Gog, T., Kester, L., Paas, F.: Effects of worked examples, example-problem, and problem-example pairs on novices’ learning. Contemp. Educ. Psychol. 36(3), 212–218 (2011)

    Article  Google Scholar 

  27. Wang, P., Rowe, J., Min, W., Mott, B., Lester, J.: Interactive narrative personalization with deep reinforcement learning. In: IJCAI (2017)

    Google Scholar 

  28. Wood, H., Wood, D.: Help seeking, learning and contingent tutoring. Comput. Educ. 33(2), 153–169 (1999)

    Article  Google Scholar 

  29. Zhou, G., Azizsoltani, H., Ausin, M.S., Barnes, T., Chi, M.: Hierarchical reinforcement learning for pedagogical policy induction. In: AIED, pp. 544–556 (2019)

    Google Scholar 

  30. Zhou, G., Chi, M.: The impact of decision agency & granularity on aptitude treatment interaction in tutoring. In: CogSci, pp. 3652–3657 (2017)

    Google Scholar 

  31. Zhou, G., Yang, X., Azizsoltani, H., Barnes, T., Chi, M.: Improving student-tutor interaction through data-driven explanation of hierarchical reinforcement induced pedagogical policies. In: UMAP. ACM (2020)

    Google Scholar 

  32. Zimmer, M., Viappiani, P., Weng, P.: Teacher-student framework: A reinforcement learning approach. In: AAMAS Workshop (2013)

    Google Scholar 

Download references

Acknowledgements

This research was supported by the NSF Grants: 1660878, 1651909, 1726550 and 2013502.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Song Ju .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ju, S., Yang, X., Barnes, T., Chi, M. (2022). Student-Tutor Mixed-Initiative Decision-Making Supported by Deep Reinforcement Learning. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2022. Lecture Notes in Computer Science, vol 13355. Springer, Cham. https://doi.org/10.1007/978-3-031-11644-5_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-11644-5_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-11643-8

  • Online ISBN: 978-3-031-11644-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics