Skip to main content
SpringerLink
Log in

Teachable Agent as an Interactive Tool for Cognitive Task Analysis: A Case Study for Authoring an Expert Model

  • Article
  • Published:
International Journal of Artificial Intelligence in Education Aims and scope Submit manuscript

Abstract

This paper demonstrates that a teachable agent (TA) can play a dual role in an online learning environment (OLE) for learning by teaching—the teachable agent working as a synthetic peer for students to learn by teaching and as an interactive tool for cognitive task analysis when authoring an OLE for learning by teaching. We have developed an OLE (called APLUS) on which students can interactively teach a TA (called SimStudent) how to solve problems. APLUS has a cognitive tutor embedded (aka meta-tutor) that provides students with adaptive feedback and help on how to solve problems. The results showed that the expert model created by SimStudent was highly accurate with more than 0.95 precision and recall scores on two different task domains (equation solving and fraction addition). The cost analysis showed that although the authors need to provide SimStudent with task dependent Java code, the amount of code written was relatively small—about 1000–2000 lines of code in 20 to 50 classes. Those findings imply that the TA has practical potential to help authors of APLUS create a domain expert model for the embedded cognitive tutor and the tutoring interface used for the OLE. It was also observed that the failure of TA to learn expected cognitive skills helps the author identify missing essential cognitive factors that must be encoded in the OLE.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. There are TAs that functions as an avatar that human operates behind the screen (aka Wizard of Oz), such as Bird Hero (Axelsson et al., 2016). However, since we are interested in a computational theory to drive a TA, a human-driven TA is excluded from the particular categorization shown here.

  2. http://ctat.pact.cs.cmu.edu

  3. https://eclipse.org/windowbuilder

  4. https://www.eclipse.org

References

  • Aleven, V., Sewall, J., McLaren, B. M., & Koedinger, K. R. (2006). Rapid authoring of intelligent tutors for real-world and experimental use. In R. K. Kinshuk, P. Kommers, P. Kirschner, D. G. Sampson, & W. Didderen (Eds.), Proceedings of the 6th IEEE international conference on advanced learning technologies (pp. 847–851). IEEE Computer Society.

  • Anderson, J. R. (1988). Acquisition of cognitive skill. In A. M. Collins & E. E. Smith (Eds.), Readings in cognitive science: A perspective from psychology and artificial intelligence (pp. 362–380). Morgan Kaufmann, Inc ix, 661 pp.SEE BOOK.

  • Anderson, J. R., Boyle, C. F., Corbett, A. T., & Lewis, M. W. (1990). Cognitive modeling and intelligent tutoring. Artificial Intelligence, 42(1), 7–49.

  • Anderson, J. R., Corbett, A. T., Fincham, J. M., Hoffman, D., & Pelletier, R. (1992). General principles for an intelligent tutoring architecture. In J. W. Regian & V. J. Shute (Eds.), Cognitive approaches to automated instruction (pp. 81–103). Lawrence Erlbaum Associates, Inc viii, 255 pp.SEE BOOK.

  • Axelsson, A., Andersson, R., & Gulz, A. (2016). Scaffolding executive function capabilities via play-& -learn software for preschoolers. Journal of Educational Psychology, 108(7), 969–981. https://doi.org/10.1037/edu0000099.

  • Baeza-Yates, R., & Ribeiro-Neto, B. (1999). Modern information retrieval. ACM Press.

  • Biswas, G., Leelawong, K., Schwartz, D., Vye, N., & Vanderbilt, T. T. A. G. A. (2005). Learning by teaching: a new agent paradigm for educational software. Journal Applied Artificial Intelligence, 19(3 & 4), 363–392.

  • Bredeweg, B., Liem, J., Beek, W., Linnebank, F., Gracia, J., Lozano, E., Wißner, M., Bühling, R., Salles, P., Noble, R., Zitek, A., Borisova, P., & Mioduser, D. (2013). DynaLearn – An intelligent learning environment for learning conceptual knowledge. AI Magazine, 34(4), 46–65.

  • Bredeweg, B., Liem, J., Beek, W., Linnebank, F., Gracia, J., Lozano, E., ... Mioduser, D. (2014). DynaLearn – An intelligent learning environment for learning conceptual knowledge. AI Magazine, 34(4), 46–65.

  • Catrambone, R. (2011). Task analysis by problem solving (TAPS): Uncovering expert knowledge to develop high-quality instructional materials and training. Paper presented at the 2011 Learning and Technology Symposium. Columbus, GA.

  • Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. Journal of the Learning Sciences, 6(3), 271–315.

  • Clark, R., Feldon, D. F., van Merriënboer, J. J. G., Yates, K. A., & Early, S. (2008). Cognitive task analysis. In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed.). Erlbaum.

  • Cohen, P. A., Kulik, J. A., & Kulik, C. L. C. (1982). Education outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19(2), 237–248.

  • Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. International Journal of Human Computer Studies, 41(6), 801–849. https://doi.org/10.1006/ijhc.1994.1083.

  • Dillenbourg, P., & Self, J. (1992). People power: A human-computer collaborative learning system. In C. Frasson, G. Gauthier, & G. McCalla (Eds.), Proceedings of the international conference on intelligent tutoring systems (pp. 651–660). Springer.

  • Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215–251.

  • Friedman-Hill, E. (2003). Jess in action: Java rule-based systems. Manning.

  • Gomoll, K. (1990). Some techniques for observing users. In B. Laurel (Ed.), The art of human-computer Interface design (pp. 85–90). Addison-Wesley.

  • Hedin, D. (1987). Students as teachers: A tool for improving school. Social Policy, 17(3), 42–47.

  • Jonassen, D. H., Tessmer, M., & Hannum, W. H. (1999). Task analysis methods for instructional design. Lawrence Erlbaum Associates, Inc., Publishers.

  • Ketamo, H. (2009). Semantic networks-based teachable agents in an educational game. WSEAS Transactions on Computers, 8(4), 641–650.

  • King, A., Staffieri, A., & Adelgais, A. (1998). Mutual peer tutoring: Effects of structuring tutorial interaction to scaffold peer learning. Journal of Educational Psychology, 90(1), 134–152. https://doi.org/10.1037/0022-0663.90.1.134.

  • Kinnebrew, J. S., Segedy, J. R., & Biswas, G. (2017). Integrating model-driven and data-driven techniques for analyzing learning behaviors in open-ended learning environments. IEEE Transactions on Learning Technologies, 10(2), 140–153. https://doi.org/10.1109/TLT.2015.2513387.

  • Koedinger, K. R., Baker, R., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J. (2010). A data repository for the EDM community: The PSLC DataShop. In: C. Romero, S. Ventura, M. Pechenizkiy & R. S. J. D. Baker (Eds.), Handbook of Educational Data Mining. CRC Press.

  • Lau, T. A., & Weld, D. S. (1998). Programming by demonstration: An inductive learning formulation. In: Proceedings of the 4th international conference on Intelligent user interfaces (pp. 145–152). ACM Press.

  • Leelawong, K., & Biswas, G. (2008). Designing learning by teaching agents: The Betty’s brain system. International Journal of Artificial Intelligence in Education, 18(3).

  • Li, N., Matsuda, N., Cohen, W. W., & Koedinger, K. R. (2015). Integrating representation learning and skill learning in a human-like intelligent agent. Artificial Intelligence, 219, 67–91. https://doi.org/10.1016/j.artint.2014.11.002.

  • Lovett, M. (1998). Cognitive task analysis in service of intelligent tutoring system design: a case study in statistics. In B. Goettl, H. Halff, C. Redfield, & V. Shute (Eds.), Intelligent Tutoring Systems (Vol. 1452, pp. 234–243). Springer.

  • Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008). Why tutored problem solving may be better than example study: Theoretical implications from a simulated-student study. In B. P. Woolf, E. Aimeur, R. Nkambou, & S. Lajoie (Eds.), Proceedings of the international conference on intelligent tutoring systems (pp. 111–121). Springer.

  • Matsuda, N., Keiser, V., Raizada, R., Tu, A., Stylianides, G. J., Cohen, W. W., & Koedinge, K. R. (2010). Learning by teaching SimStudent: Technical accomplishments and an initial use with students. In V. Aleven, J. Kay, & J. Mostow (Eds.), Proceedings of the international conference on intelligent tutoring systems (pp. 317–326). Springer.

  • Matsuda, N., Yarzebinski, E., Keiser, V., Raizada, R., Stylianides, G. J., & Koedinger, K. R. (2013a). Studying the effect of a competitive game show in a learning by teaching environment. International Journal of Artificial Intelligence in Education, 23(1-4), 1–21 [invited for the special issue on the best of ITS2012].

  • Matsuda, N., Yarzebinski, E., Keiser, V., Raizada, R., William, W. C., Stylianides, G. J., & Koedinge, K. R. (2013b). Cognitive anatomy of tutor learning: Lessons learned with SimStudent. Journal of Educational Psychology, 105(4), 1152–1163. https://doi.org/10.1037/a0031955.

  • Matsuda, N., Griger, C. L., Barbalios, N., Stylianides, G., Cohen, W. W., & Koedinger, K. R. (2014). Investigating the effect of meta-cognitive scaffolding for learning by teaching. In S. Trausen-Matu, K. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the international conference on intelligent tutoring systems (pp. 104–113). Springer.

  • Matsuda, N., Cohen, W. W., & Koedinger, K. R. (2015). Teaching the teacher: Tutoring SimStudent leads to more effective cognitive tutor authoring. International Journal of Artificial Intelligence in Education, 25, 1–34. https://doi.org/10.1007/s40593-014-0020-1.

  • Matsuda, N., Barbalios, N., Zhao, J., Ramamurthy, A., Stylianides, G., & Koedinger, K. R. (2016). Tell me how to teach, I’ll learn how to solve problems. In A. Micarelli, J. Stamper, & K. Panourgia (Eds.), Proceedings of the international conference on intelligent tutoring systems (pp. 111–121). Springer.

  • Matsuda, N., Weng, W., & Wall, N. (2020). The effect of metacognitive scaffolding for learning by teaching a teachable agent. International Journal of Artificial Intelligence in Education, 30(1), 1–37.

  • McCalla, G., & Champaign, J. (Eds.). (2013). Proceedings of the AIED workshop on simulated learners. International AIED Society.

  • Means, B., & Gott, S. P. (1988). Cognitive task analysis as a basis for tutor development: Articulating abstract knowledge representations. In J. Psotka, L. D. Massey, & S. A. Mutter (Eds.), Intelligent tutoring systems: Lessons learned (pp. 35–57). Erlbaum.

  • Michie, D., Paterson, A., & Hayes, J. E. (1989). Learning by teaching Proc. of Second Scandinavian Conference on Artificial Intelligence (pp. 413–436). IOS.

  • Muggleton, S., & de Raedt, L. (1994). Inductive logic programming: Theory and methods. Journal of Logic Programming, 19-20(supplement 1), 629–679.

  • Nathan, M. J., & Koedinger, K. R. (2000). An investigation of teachers' beliefs of students'algebra development. Cognition and Instruction, 18(2), 209–237.

  • Palthepu, S., Greer, J., & McCalla, G. (1991). Learning by teaching. Proceedings of the International Conference on Learning Sciences. Illinois.

  • Pareto, L. (2014). A teachable agent game engaging primary school children to learn arithmetic concepts and reasoning. International Journal of Artificial Intelligence in Education, 24(3), 251–283. https://doi.org/10.1007/s40593-014-0018-8.

  • Ritter, S., Anderson, J. R., Koedinger, K. R., & Corbett, A. (2007). Cognitive tutor: Applied research in mathematics education. Psychonomic Bulletin & Review, 14(2), 249–255.

  • Roscoe, R. D., & Chi, M. T. H. (2007). Understanding tutor learning: Knowledge-building and knowledge-telling in peer tutors'explanations and questions. Review of Educational Research, 77(4), 534–574.

  • Schwartz, D., Blair, K. P., Biswas, G., & Leelawong, K. (2007). Animations of thought: Interactivity in the teachable agent paradigm. In R. Lowe & W. Schnotz (Eds.), Learning with animation: Research and implications for design (pp. 114–140). Cambrige University Press.

  • Tan, J., Biswas, G., & Schwartz, D. (2006). Feedback for metacognitive support in learning by teaching environments. In: R. Sun & N. Miyake (Eds.), Proceedings of the 28th annual meeting of the cognitive science society (pp. 828–833).

  • VanLehn, K., Ohlsson, S., & Nason, R. (1994). Applications of simulated students: An exploration. Journal of Artificial Intelligence in Education, 5(2), 135–175.

  • Velmahos, G. C., Toutouzas, K. G., Sillin, L. F., Chan, L., Clark, R. E., Theodorou, D., & Maupin, F. (2004). Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. American Journal of Surgery, 18, 114–119.

  • Walker, E., Rummel, N., & Koedinger, K. (2011). Designing automated adaptive support to improve student helping behaviors in a peer tutoring activity. International Journal of Computer-Supported Collaborative Learning, 6(2), 279–306. https://doi.org/10.1007/s11412-011-9111-2.

Download references

Acknowledgements

The research reported here was partly supported by the National Science Foundation through Grant No. 1643185 to Texas A & M University, and U.S. Department of Education (the Institute of Education Sciences) through Grant R305A180319 to North Carolina State University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Author information

Authors and Affiliations

  1. Department of Computer Science, North Carolina State University, Raleigh, NC, USA

    Noboru Matsuda

Authors
  1. Noboru Matsuda
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Noboru Matsuda.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Sample Java code for task dependent background knowledge

An operator to compute a division quotient:

figure a

A feature predicate for equivalence:

figure b

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Matsuda, N. Teachable Agent as an Interactive Tool for Cognitive Task Analysis: A Case Study for Authoring an Expert Model. Int J Artif Intell Educ 32, 48–75 (2022). https://doi.org/10.1007/s40593-021-00265-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40593-021-00265-z

Keywords

Search

Navigation