Skip to main content
Log in

Automatic detection of learner’s affect from conversational cues

  • Original Paper
  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

We explored the reliability of detecting a learner’s affect from conversational features extracted from interactions with AutoTutor, an intelligent tutoring system (ITS) that helps students learn by holding a conversation in natural language. Training data were collected in a learning session with AutoTutor, after which the affective states of the learner were rated by the learner, a peer, and two trained judges. Inter-rater reliability scores indicated that the classifications of the trained judges were more reliable than the novice judges. Seven data sets that temporally integrated the affective judgments with the dialogue features of each learner were constructed. The first four datasets corresponded to the judgments of the learner, a peer, and two trained judges, while the remaining three data sets combined judgments of two or more raters. Multiple regression analyses confirmed the hypothesis that dialogue features could significantly predict the affective states of boredom, confusion, flow, and frustration. Machine learning experiments indicated that standard classifiers were moderately successful in discriminating the affective states of boredom, confusion, flow, frustration, and neutral, yielding a peak accuracy of 42% with neutral (chance = 20%) and 54% without neutral (chance = 25%). Individual detections of boredom, confusion, flow, and frustration, when contrasted with neutral affect, had maximum accuracies of 69, 68, 71, and 78%, respectively (chance = 50%). The classifiers that operated on the emotion judgments of the trained judges and combined models outperformed those based on judgments of the novices (i.e., the self and peer). Follow-up classification analyses that assessed the degree to which machine-generated affect labels correlated with affect judgments provided by humans revealed that human-machine agreement was on par with novice judges (self and peer) but quantitatively lower than trained judges. We discuss the prospects of extending AutoTutor into an affect-sensing ITS.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Aist, G., Kort, B., Reilly, R., Mostow, J., Picard, R.: Adding human-provided emotional awareness to an automated reading tutor that listens. Intelligent Tutoring Systems 2002, pp. 992–993. Berlin, Germany (2002)

  • Aleven V. and Koedinger K.R. (2002). An effective metacognitive strategy: learning by doing and explaining with a computer based Cognitive Tutor. Cogn. Sci. 26: 147–179

    Article  Google Scholar 

  • Alm, C.O., Sproat, R.: Perceptions of emotions in expressive storytelling. InterSpeech 2005, pp. 533–536. Lisbon, Portugal (2005)

  • Anderson J.R., Corbett A.T., Koedinger K.R. and Pelletier R. (1995). Cognitive tutors: lessons learned. J. Learn. Sci. 4: 167–207

    Article  Google Scholar 

  • Ang, J., Dhillon, R., Krupski A., Shriberg, E., Stolcke, A.: Prosody-based automatic detection of annoyance and frustration in human-computer dialog. Proceedings of the International Conference on Spoken Language Processing (ICSLP’02), pp. 2037–2039. Denver, CO (2002)

  • Azevedo R. and Cromley J.G. (2004). Does training on self-regulated learning facilitate students’ learning with hypermedia. J. Educ. Psychol. 96: 523–535

    Article  Google Scholar 

  • Batliner A., Fischer K., Huber R., Spilker J. and Nöth E. (2003). How to find trouble in communication. Speech Commun. 40: 117–143

    Article  MATH  Google Scholar 

  • Batliner, A., Steidl, S., Hacker, C., Nöth, E.: Private emotions vs. social interaction—a data-driven approach towards analysing emotion in speech. User Model. User-Adapt. Interact. J. Pers. Res. 18 (2008). doi:10.1007/s11257-007-9039-4

  • Bianchi-Berthouze N. and Lisetti C.L. (2002). Modeling multimodal expression of users affective subjective experience. User Model. User-Adapt. Interact. 12(1): 49–84

    Article  MATH  Google Scholar 

  • Boersma, P., Weenink D.: Praat: doing phonetics by computer (Version 4.3.14) [Computer program]. Retrieved May 02, 2006, from http://www.praat.org/ (2006)

  • Bosch L.T. (2003). Emotions, speech and the ASR framework. Speech Commun. 40(1–2): 213–215

    MATH  Google Scholar 

  • Bower G.H. (1981). Mood and memory. Am. Psychol. 36(2): 129–148

    Article  Google Scholar 

  • Bruner J.S. (1961). The act of discovery. Harv. Educ. Rev. 31(1): 21–32

    Google Scholar 

  • Bull, E.P.: Posture and Gesture. Pergamon Press (1987)

  • Campbell D.T. and Fiske D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol. Bull. 56: 81–105

    Article  Google Scholar 

  • Carberry S., Lambert L. and Schroeder L. (2002). Toward recognizing and conveying an attitude of doubt via natural language. Appl. Artif. Intell. 16(7): 495–517

    Article  Google Scholar 

  • Cohn, J.F., Kanade, T.: Use of automated facial image analysis for measurement of emotion expression. In: Coan J.A., Allen J.B. (eds.) The Handbook of Emotion Elicitation and Assessment. Oxford University Press Series in Affective Science. New York, Oxford (In press)

  • Conati C. (2002). Probabilistic assessment of user’s emotions in educational games. J. Appl. Artif. Intell. 16: 555–575

    Article  Google Scholar 

  • Craig S.D., Graesser A.C., Sullins J. and Gholson B. (2004). Affect and learning: an exploratory look into the role of affect in learning. J. Educ. Media 29: 241–250

    Google Scholar 

  • Craig, S.D., D’Mello, S., Witherspoon, A., Sullins, J., Graesser, A.C.: Emotions during learning: the first step toward an affect sensitive intelligent tutoring system. Proceedings of the International Conference on eLearning, pp. 284–288. AACE, Boston, MA (2004a)

  • Csikszentmihalyi M. (1990). Flow: the Psychology of Optimal Experience. Harper-Row, New York

    Google Scholar 

  • De Vicente, A., Pain, H.: Informing the detection of students’ motivational state: an empirical study. Intelligent tutoring systems 2002, pp. 933–943. Berlin, Germany, Springer (2002)

  • D’Mello, S.K., Craig, S.D., Gholson, B., Franklin, S., Picard, R., Graesser, A.C.: Integrating affect sensors in an intelligent tutoring system. Affective Interactions: The Computer in the Affective Loop Workshop at 2005 International conference on Intelligent User Interfaces, pp. 7–13. AMC Press, New York (2005)

  • D’Mello K.S., Craig S.D., Sullins J. and Graesser A.C. (2006). Predicting affective states through an emote-aloud procedure from AutoTutor’s mixed-initiative dialogue. Int. J. Artif. Intell. Educ. 16: 3–28

    Google Scholar 

  • Ekman P. and Friesen W.V. (1978). The Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto

    Google Scholar 

  • Ericsson K.A. and Simon H.A. (1993). Protocol Analysis: Verbal Reports as Data. Revised edition. The MIT Press, Cambridge, MA

    Google Scholar 

  • Forbes-Riley, K., Litman, D.: Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources. Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pp. 201–208. Association for Computational Linguistics, Boston, MA (2004)

  • Forbes-Riley, K., Rotaru, M., Litman, D.: The relative impact of student affect on performance models in a spoken dialogue tutoring system. User Model. User-Adapt. Interact.: J. Pers. Res. 18 (2008). doi:10.1007/s11257-007-9038-5

  • Fredrickson B.L. and Branigan C. (2005). Positive emotions broaden the scope of attention and thought-action repertoires. Cogn. Emot. 19: 313–332

    Article  Google Scholar 

  • Gertner, A.S., VanLehn, K.: Andes: a coached problem solving environment for physics. Intelligent Tutoring Systems: 5th International Conference, ITS 2000, pp. 133–142. Springer, New York (2000)

  • Goleman D. (1995). Emotional Intelligence. Bantam Books, New York

    Google Scholar 

  • Gorin A.L., Riccardi G. and Wright J.H. (1997). How may I help you?. Speech Commun. 23: 113–127

    Article  Google Scholar 

  • Graesser, A.C., Wiemer-Hastings, P., Wiemer-Hastings, K., Harter, D., Person, N.K.: Tutoring Research Group: Using latent semantic analysis to evaluate the contributions of students in AutoTutor. Interac. Learn. Environ. 8:129–148 (2000)

    Google Scholar 

  • Graesser A., VanLehn K., Rosé C., Jordan P. and Harter D. (2001). Intelligent tutoring systems with conversational dialogue.. AI Mag. 22(4): 39–51

    Google Scholar 

  • Graesser A.C., Person N., Harter D. and Tutoring Research Group (2001). Teaching tactics and dialogue in AutoTutor. Int. J. Artif. Intell. Educ. 12: 257–279

    Google Scholar 

  • Graesser A.C. and Olde B. (2003). How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down. J. Educ. Psychol. 95: 524–536

    Article  Google Scholar 

  • Graesser A.C., McNamara D.S., Louwerse M.M. and Cai Z. (2004). Coh-Metrix: analysis of text on cohesion and language. Behav. Res. Meth. Instruments Comput. 36: 193–202

    Google Scholar 

  • Graesser A.C., Chipman P., Haynes B.C., Olney A.: 2005a. AutoTutor: an intelligent tutoring system with mixed-initiative dialogue. IEEE Trans. Educ. 48:612–618

  • Graesser A.C., Person N., Lu Z., Jeon M.G. and McDaniel B. (2005b). Learning while holding a conversation with a computer. In: PytlikZillig, L., Bodvarsson, M., and Bruning, R. (eds) Technology-based Education: Bringing Researchers and Practitioners together, pp 143–167. Information Age Publishing, Greenwich, CT

    Google Scholar 

  • Graesser, A.C., McDaniel, B., Chipman, P., Witherspoon, A., D’Mello, S., Gholson, B.: Detection of emotions during learning with AutoTutor. Proceedings of the 28th Annual Conference of the Cognitive Science Society, pp. 285–290. Erlbaum, Mahwah, NJ (2006)

  • Graesser A.C., Penumatsa P., Ventura M., Cai Z. and Hu X. (2007). Using LSA in AutoTutor: learning through mixed initiative dialogue in natural language. In: Landauer, T., McNamara, D., Dennis, S., and Kintsch, W. (eds) Handbook of Latent Semantic Analysis, pp 243–262. Erlbaum, Mahwah, NJ

    Google Scholar 

  • Grimm, M., Mower, E., Kroschel, K., Narayan, S.: Combining categorical and primitives-based emotion recognition. 14th European Signal Processing Conference (EUSIPCO), Florence, Italy (2006)

  • Hoque, M.E., Yeasin, M., Louwerse, M.M. (2006) Robust Recognition of Emotion from Speech. IVA 2006, LNAI 4133, pp. 42–53. Springer-Verlag, Berlin, Heidelberg (2006)

  • Hudlicka E. and McNeese D. (2002). Assessment of user affective and belief states for interface adaptation: application to an Air Force pilot task. User Model. User-Adapt. Interact. 12(1): 1–47

    Article  MATH  Google Scholar 

  • Issroff, K., del Soldato, T.: Incorporating motivation into computer-supported collaborative learning. Proceedings of the European Conference on Artificial Intelligence in Education, pp. 284–290. Lisbon, Colibri (1996)

  • Kim, Y.: Empathetic virtual peers enhanced learner interest and self-efficacy. Workshop on Motivation and Affect in Educational Software at the 12th International Conference on Artificial Intelligence in Education, pp. 9–16. Amsterdam, The Netherlands (2005)

  • Klein J., Moon Y. and Picard R. (2002). This computer responds to user frustration—Theory, design, and results. Interac. Comput. 14(2): 119–140

    Google Scholar 

  • Koedinger K.R., Anderson J.R., Hadley W.H. and Mark M.A. (1997). Intelligent tutoring goes to school in the big city. Int. J. Artif. Intell. Educ. 8: 30–43

    Google Scholar 

  • Kort, B., Reilly, R., Picard, R.: An affective model of interplay between emotions and learning: Reengineering educational pedagogy—building a learning companion. Proceedings IEEE International Conference on Advanced Learning Technology: Issues, Achievements and Challenges, IEEE Computer Society, pp. 43–48. Madison, Wisconsin (2001)

  • Kozma R. and Freeman W.J. (2001). Chaotic resonance: methods and applications for robust classification of noisy and variable patterns. Int. J. Bifurcat. Chaos 11: 1607–1629

    Article  Google Scholar 

  • Landauer T.K. and Dumais S.T. (1997). A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction and representation of knowledge. Psychol. Rev. 104: 211–240

    Article  Google Scholar 

  • Lee C.M. and Narayanan S. (2004). Towards detecting emotions in spoken dialogs. IEEE Trans. Speech Audio Process. 13(2): 293–303

    Article  Google Scholar 

  • Lepper M.R. and Chabay R.W. (1988). Socializing the intelligent tutor: bringing empathy to computer tutors. In: Mandl, H. and Lesgold, A. (eds) Learning Issues for Intelligent Tutoring Systems, pp 242–257. Erlbaum, Hillsdale, NJ

    Google Scholar 

  • Lepper M.R. and Woolverton M. (2002). The wisdom of practice: Lessons learned from the study of highly effective tutors. In: Aronson, J. (eds) Improving Academic Achievement: Impact of Psychological Factors on Education, pp 135–158. Academic Press, Orlando, FL

    Google Scholar 

  • Lesgold A., Lajoie S., Bunzo M. and Eggan G. (1992). SHERLOCK: a coached practice environment for an electronics troubleshooting job. In: Larkin, J.H. and Chabay, R.W. (eds) Computer-Assisted Instruction and Intelligent Tutoring Systems, pp 201–238. Erlbaum, Hillsdale, NJ

    Google Scholar 

  • Linnenbrink E. and Pintrich P. (2004). Role of affect in cognitive processing in academic contexts. In: Dai, D. and Sternberg, R. (eds) Motivation, Emotion and Cognition: Integrative Perspectives on Intellectual Functioning and Development, pp 57–87. Lawrence Erlbaum, Mahwah, NJ

    Google Scholar 

  • Linnenbrink E.A. and Pintrich P. (2002). The role of motivational beliefs in conceptual change. In: Limon, M. and Mason, L. (eds) Reconsidering Conceptual Change: Issues in Theory and Practice, pp 115–135. Kluwer Academic Publishers, Dordretch, The Netherlands

    Chapter  Google Scholar 

  • Liscombe, J., Riccardi, G., Hakkani-Tür D.: Using context to improve emotion detection in spoken dialog systems. EUROSPEECH’05, 9th European Conference on Speech Commun. and Technology, pp. 1845–1848. Lisbon, Portugal (2005)

  • Litman, D.J., Forbes-Riley, K.: Predicting student emotions in computer-human tutoring dialogues. Proceedings of the 42nd annual meeting of the association for computational linguistics. Association for Computational Linguistics, pp. 352–359. East Stroudsburg, PA (2004)

  • Litman, D.J., Rose, C.P., Forbes-Riley, K., VanLehn, K., Bhemhe, D., Silliman, S.: Spoken versus typed human and computer dialogue tutoring. Proceedings of the Seventh International Conference on Intelligent Tutoring Systems, pp. 368–379. Springer Verlag, Berlin (2004)

  • Litman, D.J., Silliman, S.: ITSPOKE: an intelligent tutoring spoken dialogue system. Proceedings of the Human language technology conference: 3rd meeting of the North American chapter of the association of computational linguistics, pp. 52–54. ACL, Edmonton, Canada (2004)

  • Mandler G. (1984). Mind and Body: Psychology of Emotion and Stress. Norton, New York

    Google Scholar 

  • Matsubara, Y., Nagamachi, M.: Motivation systems and human models for intelligent tutoring. Proceedings of the Third International Conference in Intelligent Tutoring Systems, pp. 139–147. Springer-Verlag, London, England, (1996)

  • McQuiggan, S., Mott, B., Lester, J.: Modeling self-efficacy in intelligent tutoring systems: an inductive approach. User Model. User-Adapt. Interact. J. Pers. Res. 18 (2008). doi:10.1007/s11257-007-9040-y

  • Miserandino M. (1996). Children who do well in school: Individual differences in perceived competence and autonomy in above-average children. J. Educ. Psychol. 88: 203–214

    Article  Google Scholar 

  • Morimoto, C., Koons, D., Amir, A., Flickner, M.: Pupil detection and tracking using multiple light sources. Technical report, IBM Almaden Research Center (1998)

  • Mota, S., Picard, R.W.: Automated posture analysis for detecting learner’s interest level. Workshop on Computer Vision and Pattern Recognition for Human-Computer Interaction, CVPR HCI (2003)

  • Oliver, N., Pentland, A., Berand, F.: LAFTER: a real-time lips and face tracker with facial expression recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 123–129. IEEE, San Juan, Puerto Rico, (1997)

  • Olney, A., Louwerse, M., Mathews, E., Marineau, J., Hite-Mitchell, H., Graesser, A.: Utterance classification in AutoTutor. Proceedings of the HLT-NAACL 03 Workshop on Building Educational Applications using Natural Language Processing, pp. 1–8 (2003)

  • Ortony A., Clore G.L. and Collins A. (1988). The Cognitive Structure of Emotions. Cambridge University Press, Cambridge, UK

    Google Scholar 

  • Pantic, M., Rothkrantz, L.J.M.: Towards an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, Special Issue on Multimodal Human-Computer Interaction (HCI), vol. 91(9), pp. 1370–1390 (2003)

  • Patrick B., Skinner E. and Connell J. (1993). What motivates children’s behavior and emotion? Joint effects of perceived control and autonomy in the academic domain. J. Pers. Soc. Psychol. 65: 781–791

    Article  Google Scholar 

  • Person N.K., Graesser A.C., Tutoring Research Group Human or computer? AutoTutor in a bystander Turing test. In: Cerri S.A., Gouarderes G., Paraguacu F. (eds.) Intelligent Tutoring Systems. pp. 821–830. Springer, Berlin, Germany (2002)

  • Picard R.W. (1997). Affective Computing. MIT Press, Boston, MA

    Google Scholar 

  • Picard R.W., Vyzas E. and Healey J. (2001). Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23(10): 1175–1191

    Article  Google Scholar 

  • Prendinger H. and Ishizuka M. (2005). The empathic companion: a character-based interface that addresses users’ affective states. Int. J. Appl. Artif. Intell. 19(3,4): 267–285

    Article  Google Scholar 

  • Porayska-Pomsta, K., Mavrikis, M., Pain, H.: Diagnosing and acting on student affect: the tutor’s perspective. User Model. User-Adapt. Interact. J. Pers Res. 18 (2008) doi:10.1007/s11257-007-9041-x

  • Rani, P., Sarkar, N., Smith, C.A.: An affect-sensitive human-robot cooperation—theory and experiments. Proceedings of the IEEE Conference on Robotics and Automation, pp. 2382–2387. IEEE, Taipei, Taiwan (2003)

  • Robson C. (1993). Real World Research: A Resource for Social Scientist and Practitioner Researchers. Blackwell, Oxford

    Google Scholar 

  • Rus, V., Graesser, A.C.: Deeper natural language processing for evaluating student answers in intelligent tutoring systems. Proceedings of the American Association of Artificial Intelligence, pp. 1495–1600. AAAI, Menlo Park, CA (2006)

  • Russell J.A. (2003). Core affect and the psychological construction of emotion. Psychol. Rev. 110: 145–172

    Article  Google Scholar 

  • Scheirer J., Fernandez R., Klein J. and Picard R. (2002). Frustrating the user on purpose: a step toward building an affective computer. Interact. Comput. 14(2): 93–118

    Google Scholar 

  • Schutzwohl A and Borgstedt K. (2005). The processing of affectively valenced stimuli: the role of surprise. Cogn. Emot. 19: 583–600

    Article  Google Scholar 

  • Selfridge, O.G.: Pandemonium: A Paradigm for learning. Symposium on the Mechanization of Thought Processes, pp. 511–531. Her Majesty’s Stationary Office, London (1959)

  • Shafran, I., Mohri, M.: A comparison of classifiers for detecting emotion from speech. Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, pp. 341–344. IEEE, Philadelphia, PA (2005)

  • Shafran, I., Riley, M., Mohri, M.: Voice signatures. Proc. IEEE Automatic Speech Recognition and Understanding Workshop, pp. 31–36. IEEE, Piscataway, NJ, (2003)

  • Silvia P. and Abele A. (2002). Can positive affect induce self-focused attention? Methodological and measurement issues. Cogn. Emot. 16: 845–853

    Article  Google Scholar 

  • Sleeman D. and Brown J. (1982). Intelligent Tutoring Systems. Academic Press, New York

    Google Scholar 

  • Stein N. and Levine L. (1991). Making sense out of emotion: the representation and use of goal-structured knowledge. In: Kessen, W., Ortony, A. and Craik, F. (eds) Memories, Thoughts and Emotions: Essays in Honor of George Mandler, pp 295–322. Laurence Erlbaum Associates, Hillsdale, NJ

    Google Scholar 

  • Tekscan (1997) Tekscan Body Pressure Measurement System User’s Manual. Tekscan Inc., South Boston, MA

  • VanLehn K. (1990). Mind Bugs: The Origins of Procedural Misconceptions. MIT Press, Cambridge, MA

    Google Scholar 

  • VanLehn, K., Jordan, P., Rosé, C.P., Bhembe, D., Bottner, M., Gaydos, A., et~al.: The architecture of Why2-Atlas: A coach for qualitative physics essay writing. Proceedings of the Sixth International Conference on Intelligent Tutoring, pp. 158–167. Springer-Verlag, Berlin (2002)

  • VanLehn, K., Graesser, A.C., Jackson, G.T. Jordan, P., Olney, A., Rose, C.P.: When are tutorial dialogues more effective than reading? Cogn. Sci. 30, 1–60 (2006)

    Google Scholar 

  • Vavik L. (1993). Facilitating discovery learning in computer-based simulation learning environments. In: Tennyson, R.D. and Baron, A.E. (eds) Automating Instructional Design: Computer-Based Development and Delivery Tools, pp 403–449. Springer-Verlag, Berlin, Germany

    Google Scholar 

  • Walker M.A., Langkilde -Geary I., Hastie H.W., Wright J. and Gorin A. (2002). Automatically training a problematic dialogue predictor for a spoken dialogue system. J. Artif. Intell. Res. 16: 293–319

    MATH  Google Scholar 

  • Whang M.C., Lim J.S. and Boucsein W. (2003). Preparing computers for affective communication: a psychophysiological concept and preliminary results. Hum. Factors 45(4): 623–634

    Article  Google Scholar 

  • Wiemer-Hastings P., Wiemer-Hastings K. and Graesser A.C. (1999). Improving an intelligent tutor’s comprehension of students with latent semantic analysis. In: Lajoie, S.P. and Vivet, M. (eds) Artificial Intelligence in Education, pp 535–542. IOS Press, Amsterdam

    Google Scholar 

  • Witten, I.H., Frank, E.: (2005) Data mining: practical machine learning tools and techniques, 2nd ed. Morgan Kaufmann, San Francisco (2005)

  • Yannakakis, G., Hallam, J., Lund, H.: Entertainment capture through heart rate activity in physical interactive playgrounds. User Model. User-Adapt. Interact. J. Pers. Res. 18 (2008). doi:10.1007/s11257-007-9036-7

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sidney K. D’Mello.

Rights and permissions

Reprints and permissions

About this article

Cite this article

D’Mello, S.K., Craig, S.D., Witherspoon, A. et al. Automatic detection of learner’s affect from conversational cues. User Model User-Adap Inter 18, 45–80 (2008). https://doi.org/10.1007/s11257-007-9037-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-007-9037-6

Keywords

Navigation