Skip to main content
SpringerLink
Log in

Diagnosing and acting on student affect: the tutor’s perspective

  • Original Paper
  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

In this paper we explore human tutors’ inferences in relation to learners’ affective states and the relationship between those inferences and the actions that tutors take as their consequence. At the core of the investigations presented in this paper lie fundamental questions associated with the role of affective considerations in computer-mediated educational interactions. Theory of linguistic politeness is used as the basis for determining the contextual factors relevant to human tutors’s actions, with special attention being dedicated to learner affective states. A study was designed to determine what affective states of the learners are relevant to tutoring mathematics and to identify the mechanisms used by tutors to predict such states. Logs of tutor-student dialogues were recorded along with contextual factors taken into consideration by tutors in relation to their specific tutorial dialogue moves. The logs were annotated in order to determine the types and range of student and tutor actions. Machine learning techniques were then applied to those actions to predict the values of three factors: student confidence, interest and effort. Whilst due to limited size and sparsity of data the results are not conclusive, they are very valuable as the basis for empirically derived hypotheses to be tested in further studies. The potential implications of the hypotheses, if they were confirmed by further studies, are discussed in relation to the impact of tutor’s ability to diagnose student affect on the nature of computer-mediated tutorial interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Allanson J. and Fairclough S. (2004). A research agenda for physiological computing. Interact. Comput.: Special Issue Physiol. Comput. 16(5): 857–878

    Google Scholar 

  • Arroyo I. and Woolf B. (2005). Inferring learning and attitudes with a Bayesian Network of log files data. In: Looi, C.-K., McCalla, G., Bredeweg, B. and Breuker, J. (eds) Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology (Proceedings of AIED-2005 Conference, 18-22 July, 2005, Amsterdam), pp 88–95. IOS Press, Amsterdam

    Google Scholar 

  • Baker R., Corbett A. and Koedinger K. (2004a). Detecting student misuse of intelligent tutoring systems. In: Lester, J.C., Vicari, R.M., and Paraguacu, F. (eds) Proceedings of Seventh International Conference on Intelligent Tutoring Systems, ITS 2004, August 30–September 3, 2004, Maceio, Brasil, pp 531–540. Springer, Berlin

    Google Scholar 

  • Baker, R., Corbett, A., Koedinger, K., Wagner, A.: Off-task behavior in the cognitive tutor classroom: when students “game The system”. In: Proceedings of ACM CHI 2004 Conference on Human Factors in Computing Systems, 24–29 April 2004, Vienna, Austria, pp. 383–390. ACM Press, New York (2004b)

  • Breiman L. (1996). Bagging predictors. Mach. Lear. 24(2): 123–140

    MATH  MathSciNet  Google Scholar 

  • Brown P. and Levinson S. (1987). Politeness. Someuniversalsinlanguageusage. In: Gumperz, J.J. (eds) Studies in Interactional Sociolinguistics, Vol. 4, pp. Cambridge University Press, Cambridge

    Google Scholar 

  • Burleson, W.: Affective learning companions: strategies for empathetic agents with real-time multimodal affective sensing to foster meta-cognitive and meta-affective approaches to learning, motivation, and perseverance. Ph.D. thesis, Massachusetts Institute of Technology (2006)

  • Burleson, W., Picard, R.: Affective agents: sustaining motivation to learn through failure and a state of stuck. In: Porayska-Pomsta, K., Frasson, C. (eds.) Proceedings of Workshop on Social and Emotional Intelligence in Learning Environments, in association with Seventh International Conference on Intelligent Tutoring Systems, ITS 2004, August 30–September 3, 2004, Maceio, Brasil. Intelligent Tutoring Systems. Available (29th June 2007) at: http://www.itsconference.org/2004/w7/ (2004)

  • Chae, H.M., Kim, J.H., Glass, M.: Effective behaviors in a comparison between novice and expert algebra tutors. In: Proceedings of the Sixteenth Midwest Artificial Intelligence and Cognitive Science Conference (MAICS’2005); Dayton, Ohio; April 16–17, 2005, pp. 25–30. AAAI Press (2005)

  • Chi M., Siler S., Jeong H., Yamauchi T. and Hausmann R.G. (2001). Learning from human tutoring. Cogn. Sci. 25: 471–533

    Article  Google Scholar 

  • Chi M.T.H., Siler S. and Jeong H. (2004). Can tutors monitor students’ understanding accurately?. Cogn. Instruct. 22(3): 363–387

    Article  Google Scholar 

  • Cohen J. (1960). A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20: 37–46

    Article  Google Scholar 

  • Cohen J., Cohen P., West S. and Aiken L. (2003). Applied multiple regression/correlation analysis for the behavioral sciences, 3rd edn. Lawrence Erlbaum Associates, Hillsdale

    Google Scholar 

  • Conati C. (2002). Probabilistic assessment of user’s emotions in educational games. J. Appl. Arti. Intelligence 16(7–8): 555–575

    Article  Google Scholar 

  • Conati C. and Maclaren H. (2004). Evaluating a probabilistic model of student affect. In: Lester, J.C., Vicari, R.M. and Paraguacu, F. (eds) Proceedings of Seventh International Conference on Intelligent Tutoring Systems, ITS 2004, August 30–September 3, 2004, Maceio, Brasil, pp 55–66. Springer, Berlin

    Google Scholar 

  • Conlon T. and Pain H. (1996). Persistent collaboration: a methodology for applied AIED. Int. J. Arti. Intelligence Educ. 7(3/4): 219–252

    Google Scholar 

  • Corbett, A., Trask, H.: Instructional interventions in computer-based tutoring: differential impact on learning time and accuracy. In: Proceedings of ACM CHI’2000 Conference on Human Factors in Computing Systems; April 1–6, 2000, The Hague, The Netherlands, pp. 97–104. ACM Press (2000)

  • Cromley J.G. and Azevedo A. (2005). What do reading tutors do?: a naturalistic study of more and less experienced tutors in reading. Discourse Process. 40(2): 83–113

    Article  Google Scholar 

  • Csikszentmihalyi M. (1990). Flow: the psychology of optimal experience. Harper Row, New York

    Google Scholar 

  • de Vicente, A., Pain, H.: Eliciting motivation diagnosis knowledge. In: Lajoie, S.P., Vivet, M. (eds.) Proceedings of the 9th International Conference on Artificial Intelligence in Education, (AI-ED ’99), Le Mans, France, July 19–23, 1999, pp. 651–653. IOS Press, Amsterdam (1999)

  • de Vicente A. and Pain H. (2002). Informing the detection of students’ motivational state: an empirical study. In: Cerri, S.A., Gouarderes, G. and Paraguacu, F. (eds) Proceedings of Sixth International Conference on Intelligent Tutoring Systems, ITS 2002, June 2–7, 2002, Biarritz, France and San Sebastian, Spain, Lecture Notes in Computer Science 2363, pp 955–962. Springer, Berlin

    Google Scholar 

  • del Soldato T. and Boulay B. (1995). Implementation of motivational tactics in tutoring systems. J. Artif. Intelligence Educ. 4(6): 337–378

    Google Scholar 

  • Devillers L., Vidrascu L. and Lamel L. (2005). Challenges in real-life emotion annotation and machine learning based detection. Neural Net. 18: 407–422

    Article  Google Scholar 

  • Di Eugenio, B., Lu, X., Kershaw, T., Corrigan-Halpern, A., Ohlsson, S.: Positive and negative verbal feedback for intelligent tutoring systems. In: Looi, C.-K., McCalla, G., Bredeweg, B., Breuker, J. (eds.) Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology (Proceedings of 12th Conference on Artificial Intelligence in Education, AIED2005, 18-22 July, 2005, Amsterdam), pp. 798–800. IOS Press, Amsterdam (2005)

  • Dillon J.T. (1990). Practice of questioning. Routledge, London

    Google Scholar 

  • D’Mello, S., Craig, S., Gholson, B., Franklin, S., Graesser, A.: Integrating Affect Sensors in an Intelligent Tutoring System. In: Affective Interactions: The Computer in the Affective Loop, Workshop in International Conference on Intelligent User Interfaces, January 9–12, 2005, San Diego, Ca, pp. 7–13. ACM Press, New York (2005)

  • D’Mello S., Craig S., Sullins C. and Graesser A. (2006). Predicting affective states expressed through an emote-aloud procedure from autotutor’s mixed initiative dialogue. Int. J. Artti. Intelligence Educ. 16(1): 3–28

    Google Scholar 

  • D’Mello, S., Craig, S., Witherspoon, A., McDaniel, B., Graesser, A.: Automatic detection of learner’s affect from conversational cues. User Model. User-Adapt. Interact.: J. Personal. Res. 18(this issue): doi:10.1007/s11257-007-9037-6 (2008)

  • du Boulay B. and Luckin R. (2001). Modelling human teaching tactics and strategies for tutoring systems. Int. J. Artif. Intelligen. Edu. 12(3): 235–256

    Google Scholar 

  • Ericsson K.A. and Simon H.A. (1993). Protocol Analysis: Verbal Reports as Data. MIT Press, Cambridge

    Google Scholar 

  • Fetzer A. (2003). “No thanks”: a socio-semantic approach. Linguistik 14: 137–160

    Google Scholar 

  • Fleiss J.L. (1981). Statistical Methods for Rates and Proportions. John Wiley and Sons, New York

    MATH  Google Scholar 

  • Fox B.A. (1993). The human tutorial dialogue project: issues in the design of instructional systems. Lawrence Erlbaum Associates, Hillsdale

    Google Scholar 

  • Glass, M., Kim, J., Evens, M., Joel, M., Rovick, A.: Novice vs. expert tutors: a comparison of style. In: Priss, U. (ed.) Proceedings of the Tenth Midwest Artificial Intelligence and Cognitive Science Conference (MAICS99); Bloomington, IN; April 23–25, 1999, pp. 43–49. AAAI Press, Menlo Park (1999)

  • Goleman D. (1996). Emotional intelligence: why it can matter more than IQ. Boomsbury, London

    Google Scholar 

  • Graesser, A.: Dialogue patterns and feedback mechanisms during naturalistic tutoring. In: Proceedings of the 15th Annual Conference of the Cognitive Science Society, pp. 126–130. Lawrence Erlbaum Associates, Hillsdale (1993)

  • Graesser A. and Person N. (1994). Question asking during tutoring. Am. Educ. Res. J. 31(1): 104–137

    Article  Google Scholar 

  • Graesser A.C., Person N. and Huber J.D. (1992). Mechanisms that generate questions. In: Lauer, T., Peacock, E., and Graesser, A.C. (eds) Questions and Information Systems, pp 167–187. Lawrence Erlbaum Associates, Hillsdale

    Google Scholar 

  • Graesser A., Person N. and Magliano J. (1995). Collaborative dialogue patterns in naturalistic one-to-one tutoring. Appl. Cogn. Psychol. 9(1): 1–28

    Article  Google Scholar 

  • Haro, A., Essa, I., Flickner, M.: Detecting and tracking eyes by using their physiological properties, dynamics and appearance. In: Proceedings of IEEE Computer Vision and Pattern Recognition; June 13-15, 2000, Hilton Head, SC, pp. 163–168. IEEE Computer Society Press (2000)

  • Higgins M.C. (2001). When is helping helpful? effects of evaluation and intervention timing on individual task performance. J. Appl. Behav. Sci. 37(3): 280–298

    Article  Google Scholar 

  • Johnson, W.L., Rizzo, P.: Politeness in tutoring dialogs: “run the factory, that’s what i’D Do”. In: Lester, J.C., Vicari, R.M., Paraguacu, F. (eds.) Proceedings of Seventh International Conference on Intelligent Tutoring Systems, ITS 2004, August 30–September 3, 2004, Maceio, Brasil, pp. 67–76. Lecture Notes in Computer Science 3220, Springer, Berlin (2004)

  • Kapoor, A., Picard, R.: Multimodal affect recognition in learning environments. In: Proceedings of the 13th annual ACM international conference on Multimedia (MM’05), 6–11 November, 2005, Singapore, pp. 677–682. ACM Press (2005)

  • Kapoor, A., Picard, R., Ivanov, Y.: Probabilistic combination of multiple modalities to detect interest. In: Proceedings of the International Conference on Pattern Recognition, 23–26 August 2004, Cambridge, UK, pp. 969– 972. IEEE Computer Society Press (2004)

  • Katz S., Allbritton D. and Connelly J. (2003). Going beyond the problem given: How human tutors use post-solution discussions to support transfer. Int. J. Artif. Intell. Educ. 13: 79–116

    Google Scholar 

  • Keller J.M. (1983). Motivational design of instruction. In: Reigeluth, C.M. (eds) Instructional-design Theories and Models: an Overview of their Current Status, pp 383–434. Laurence Erlbaum Associates, Hillside

    Google Scholar 

  • Kort, B., Reilly, R., Picard, R.: An affective model of interplay between emotions and learning: reengineering educational pedagogy; Building a learning companion. In: Okamoto, T., Kinshuk, R.H., Klus, J. (eds.) International Conference on Advanced Learning Technologies, (ICALT 2001), Madison, WI, 6–8 August, 2001, pp. 43–49. IEEE Computer Society Press (2001)

  • LeAM: LeActiveMath consortium: language-enhanced, user adaptive, interactive elearning for mathematics. http://www.leactivemath.org: accessed 30th June, 2007 (2003)

  • Leech, G.N.: Language and tact. In: Explorations in Semantics and Pragmatics, Pragmatics and Beyond, pp. 79–119. John Benjamins, Amsterdam (1980)

  • Lepper, M.R., Chabay, R.W.: Socializing the intelligent tutor: Bringing empathy to computer tutors. In: Learning issues for intelligent tutoring systems, pp. 242–257. Springer-Verlag, New York (1988)

  • Lepper, M., Drake, M., O’Donnell-Johnson, T.: Scaffolding techniques of expert human tutors. In: Scaffolding Student Learning: Instructional Approaches and Issues, pp. 108–144. Brookline Books, Cambridge (1997)

  • Lepper M.R., Woolverton M., Mumme D.L. and Gurtner J. (1993). Motivational techniques of expert human tutors: lessons for the design of computer-based tutors. In: Lajoie, S. and Derry, S. (eds) Computers as Cognitive Tools, pp 75–107. Lawrence Erlbaum Associates, Hillsdale

    Google Scholar 

  • Lu, X., Di Eugenio, B., Kershaw, T., Ohlsson, S., Corrigan-Halpern, A.: Tutorial dialogue patterns: expert vs. non-expert tutors. In: The Third Midwest Computational Linguistics Colloquium (MCLC-2006), May 20-21th, 2006, Urbana IL (2006)

  • Malone T.W. and Lepper M.R. (1987). Making learning fun: a Taxonomy of Intrinsic Motivations for Learning. In: Snow, R.E. and Farr, M.J. (eds) Aptitude, Learning and Instruction: Conative and Affective Process Analyses, vol. 3, pp 223–253. Lawrence Erlbaum Associates, Hillsdale

    Google Scholar 

  • Mavrikis, M., Maciocia, A.: WaLLiS: a Web-based ILE for science and engineering students studying mathematics. In: Aleven, V., Hoppe, U., Kay, J., Mizoguchi, R., Pain, H., Verdejo, F., Yacef, K. (eds.) Supplemental Proceedings of the 11th International Conference on Artificial Intelligence in Education, AIED2003, Vol. VIII, Workshop on Advanced Technologies for Mathematics Education, pp. 505–512. School of Information Technologies, University of Sydney (2003)

  • McArthur D., Stasz C. and Zmuidzinas M. (1990). Tutoring techniques in algebra. Cogn. Instruct. 7: 197–244

    Article  Google Scholar 

  • McQuiggan, S., Mott, B.W., Lester, J.: Modeling self-efficacy in intelligent tutoring systems: an inductive approach. User Model. User-Adapt. Interact.: J. Personal. Res. 18(this issue), doi:10.1007/s11257-007-9040-y (2008)

  • Messom C., Sarrafzadeh A., Johnson M. and Fan C. (2005). Affective state estimation from facial images using neural networks and fuzzy logic. In: Wang, D. and Lee, N. (eds) Neural Networks Applications in Information Technology and Web Engineering, pp 1–8. Borneo Publishing, Sarawak

    Google Scholar 

  • Mota, S., Picard, R.: Automated posture analysis for detecting learner’s interest level. In: 2003 Conference on Computer Vision and Pattern Recognition, vol. 5. Workshop on Computer Vision and Pattern Recognition for Human-Computer Interaction; June 16–22 2003, p. 49. IEEE Computer Society Press, Madison

  • Murphy P. and Alexander P. (2000). A motivated exploration of motivation terminology. Contemp. Educ. Psychol. 25: 3–53

    Article  Google Scholar 

  • Person N.K., Kreuz R.J., Zwaan R.A. and Graesser A.C. (1995). Pragmatics and pedagogy: conversational rules and politeness strategies may inhibit effective tutoring. Cogn. and Instruct. 2(13): 161–188

    Article  Google Scholar 

  • Picard, R., Daily, S.: Evaluating affective interactions: alternatives to asking what users feel. In: Conference on Human Factors in Computing Systems. Paper presented at Workshop on Innovative Approaches to Evaluating Affective Interfaces, April 2–7, 2005, Portland, OR. New York (2005)

  • Picard R.W. (1997). Affective computing. MIT Press, Cambridge

    Google Scholar 

  • Porayska-Pomsta, K.: Influence of situational context on language production: modelling teachers’ corrective responses. Ph.D. thesis, The University of Edinburgh (2003)

  • Porayska-Pomsta, K., Mellish, C.: Modelling politeness in natural language generation. In: Proceedings of Natural Language Generation: Third International Conference (INLG 2004), 14–16 July 2004, Brockenhurst, UK, pp. 141–150. Springer, Berlin (2004)

  • Porayska-Pomsta, K., Pain, H.: Providing cognitive and affective scaffolding through teaching strategies. In: Lester, J.C., Vicari, R.M., Paraguacu, F. (eds.) Proceedings of Seventh International Conference on Intelligent Tutoring Systems, ITS 2004, August 30–September 3, 2004, Maceio, Brasil, pp. 77–86. Springer, Berlin (2004)

  • Quinlan J. (1993). C4.5: Programs for Machine Learning. Morgan Kaufmann, San Francisco

    Google Scholar 

  • Quinlan J. (1996). Improved use of continuous attributes in C4.5. J. Artif. Intell. Res. 4: 77–90

    MATH  Google Scholar 

  • Rebolledo G., du Boulay, B., Luckin, R.: Motivating the learer: an empirical evaluation. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) Proceedings of Eighth International Conference on Intelligent Tutoring Systems, ITS 2006, June 26-30, 2006, Jhongli, Taiwan, Lecture Notes in Computer Science 4053, pp. 545–554. Springer, Berlin (2006)

  • Sadock, J.: Queclaratives. In: Papers from the 7th Regional Meeting of the Chicago Linguistic Society, pp. 223–231. Chicago Linguistic Society, Chicago (1971)

  • Schank R. and Neaman A. (2001). Motivation and failure in educational simulation design. In: Forbus, K. and Feltovich, P. (eds) Smart Machines in Education: the Coming Revolution in Educational Technology, pp 37–69. AAAI Press/The MIT Press, Cambridge

    Google Scholar 

  • Shah F., Evens M., Michael J. and Rovick A. (2002). Classifying student initiatives and tutor responses in human keyboard-to-keyboard tutoring sessions. Discourse Process. 33: 23–52

    Article  Google Scholar 

  • Sinclair J.M. and Brazil D. (1982). Teacher talk. Oxford University Press, Oxford

    Google Scholar 

  • Stevens, R., Soller, A., Cooper, M., Sprang, M.: Modeling the development of problem solving skills in chemistry with a web-based tutor. In: Lester, J.C., Vicari, R.M., Paraguacu, F. (eds.) Proceedings of Seventh International Conference on Intelligent Tutoring Systems, ITS 2004, August 30–September 3, 2004, Maceio, Brasil, pp. 580–591. Springer, Berlin (2004)

  • Stevenson, J.C.: Cognitive structures for the teaching of adaptability in vocational education. In: Evans, G.T. (ed.) Learning and Teaching Cognitive Skills, pp. 144–163. Monograph Series II. Australian Council for Educational Research, Hawthorn (1991)

  • Stone, A., Bachrach, C., Jobe, J., Kurtzman, H., Howard, S. (eds) (2000). The science of self-report: implications for research and practice. Lawrence Erlbaum Associates, Mahwah

    Google Scholar 

  • Wagner, J., Kim, J., André, E.: From physiological signals to emotions: implementing and comparing selected methods for feature extraction and classification. In: Proceedings of IEEE International Conference on Multimedia and Expo (ICME 2005), 6–8 July, Amsterdam, pp. 940–943. IEEE Press (2005)

  • Wetherell, M., Taylor, S., Yates, S. (eds.): Discourse as data; a guide for analysis. Sage and The Open University, London and Milton Keynes (2001)

  • Witten I.H. and Frank E. (2005). Data mining: practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. London Knowledge Lab, University of London, 23–29 Emerald Street, London, WC1N 3QS, UK

    Kaśka Porayska-Pomsta

  2. Institute for Communicating and Collaborative Systems, The University of Edinburgh, 2 Buccleuch Place, Edinburgh, EH8 9LW, UK

    Manolis Mavrikis & Helen Pain

Authors
  1. Kaśka Porayska-Pomsta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manolis Mavrikis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Helen Pain
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kaśka Porayska-Pomsta.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Porayska-Pomsta, K., Mavrikis, M. & Pain, H. Diagnosing and acting on student affect: the tutor’s perspective. User Model User-Adap Inter 18, 125–173 (2008). https://doi.org/10.1007/s11257-007-9041-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-007-9041-x

Keywords

Search

Navigation