Skip to main content
SpringerLink
Log in

Predicting Co-occurring Emotions from Eye-Tracking and Interaction Data in MetaTutor

  • Conference paper
  • First Online:
Book cover Artificial Intelligence in Education (AIED 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12748))

Included in the following conference series:

Abstract

Emotions in Intelligent Tutoring Systems (ITS) are often modeled as single affective states, however there is evidence that emotions co-occur during learning, with implications for affect-aware ITS that need to have a comprehensive understanding of a student’s affective state to react accordingly. In this paper we broaden the evidence that emotions co-occur in an educational context, and present a first attempt to predict these co-occurrences from data, using the MetaTutor ITS as a test-bed. We show that boredom+frustration, as well as curiosity+anxiety, frequently co-occur in MetaTutor, and that we can predict when these emotions co-occur significantly better than a baseline using eye-tracking and interaction data. These findings provide a first step toward building affect-aware ITS that can adapt to these complex co-occurring affective states.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Wortha, F., Azevedo, R., Taub, M., Narciss, S.: Multiple negative emotions during learning with digital learning environments–Evidence on their detrimental effect on learning from two methodological approaches. Front. Psychol. 10, 2678:1–2678:19 (2019). https://doi.org/10.3389/fpsyg.2019.02678

  2. Baker, R., D’Mello, S., Rodrigo, M.M., Graesser, A.C.: Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. Int. J. Hum.-Comput. Stud. 68, 223–241 (2010). https://doi.org/10.1016/j.ijhcs.2009.12.003

    Article  Google Scholar 

  3. Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., Picard, R.: Affect-aware tutors: recognising and responding to student affect. Int. J. Learn. Technol. 4, 129–164 (2009). https://doi.org/10.1504/ijlt.2009.028804

    Article  Google Scholar 

  4. Grawemeyer, B., Mavrikis, M., Holmes, W., Gutierrez-Santos, S., Wiedmann, M., Rummel, N.: Affecting off-task behaviour: how affect-aware feedback can improve student learning. In: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. pp. 104–113. ACM, Edinburgh (2016). https://doi.org/10.1145/2883851

  5. Lallé, S., Conati, C., Azevedo, R.: Prediction of student achievement goals and emotion valence during interaction with pedagogical agents. In: Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, pp. 1222–1231. IFAAMAS, Stockholm (2018). https://doi.org/10.1145/2883851

  6. Salmeron-Majadas, S., Santos, O.C., Boticario, J.G.: An evaluation of mouse and keyboard interaction indicators towards non-intrusive and low cost affective modeling in an educational context. Procedia Comput. Sci. 35, 691–700 (2014)

    Article  Google Scholar 

  7. Jaques, N., Conati, C., Harley, J.M., Azevedo, R.: Predicting affect from gaze data during interaction with an intelligent tutoring system. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 29–38. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07221-0_4

    Chapter  Google Scholar 

  8. Paquette, L., et al.: Sensor-free affect detection for a simulation-based science inquiry learning environment. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 1–10. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07221-0_1

    Chapter  Google Scholar 

  9. Sabourin, J., Mott, B., Lester, J.C.: Modeling learner affect with theoretically grounded dynamic bayesian networks. In: D’Mello, S., Graesser, A., Schuller, B., Martin, J.-C. (eds.) ACII 2011. LNCS, vol. 6974, pp. 286–295. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24600-5_32

    Chapter  Google Scholar 

  10. Baker, R.S., et al.: Towards sensor-free affect detection in cognitive tutor algebra. In: Proceedings of the 5th International Conference on Educational Data Mining, pp. 126–133. IEDMS, Montréal (2012)

    Google Scholar 

  11. Wixon, M., Arroyo, I., Muldner, K., Burleson, W., Rai, D., Woolf, B.: The opportunities and limitations of scaling up sensor-free affect detection. In: Proceedings of the International Conference on Educational Data Mining, pp. 145–152. IEDMS, London (2014)

    Google Scholar 

  12. Litman, D.J., Forbes-Riley, K.: Predicting student emotions in computer-human tutoring dialogues. In: Proceedings of the Annual Meeting on Association for Computational Linguistics, pp. 351–358, Barcelona, Spain (2004). https://doi.org/10.3115/1218955.1219000

  13. Bosch, N., D’Mello, S.: Co-occurring affective states in automated computer programming education. In: Proceedings of the Workshop on AI-supported Education for Computer Science (AIEDCS) at the 12th International Conference on Intelligent Tutoring Systems, pp. 21–30 (2014)

    Google Scholar 

  14. Dillon, J., et al.: Student emotion, co-occurrence, and dropout in a MOOC context. In: Proceedings of the 9th International Conference on Educational Data Mining, pp. 353–357. IEDMS, Raleigh (2016)

    Google Scholar 

  15. Harley, J.M., Bouchet, F., Azevedo, R.: Measuring learners’ co-occurring emotional responses during their interaction with a pedagogical agent in MetaTutor. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 40–45. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30950-2_5

    Chapter  Google Scholar 

  16. Gutica, M., Conati, C.: Student emotions with an edu-game: a detailed analysis. In: Proceedings of the Humaine Association Conference on Affective Computing and Intelligent Interaction. pp. 534–539. IEEE, Geneva (2013). https://doi.org/10.1109/acii.2013.94

  17. Sinclair, J., Jang, E.E., Azevedo, R., Lau, C., Taub, M., Mudrick, N.V.: Changes in emotion and their relationship with learning gains in the context of MetaTutor. In: Nkambou, R., Azevedo, R., Vassileva, J. (eds.) ITS 2018. LNCS, vol. 10858, pp. 202–211. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91464-0_20

    Chapter  Google Scholar 

  18. Azevedo, R., et al.: Using trace data to examine the complex roles of cognitive, metacognitive, and emotional self-regulatory processes during learning with multi-agent systems. In: Azevedo, R., Aleven, V. (eds.) International Handbook of Metacognition and Learning Technologies. SIHE, vol. 28, pp. 427–449. Springer, New York (2013). https://doi.org/10.1007/978-1-4419-5546-3_28

    Chapter  Google Scholar 

  19. Petrovica, S., Anohina-Naumeca, A., Ekenel, H.K.: Emotion recognition in affective tutoring systems: collection of ground-truth data. Procedia Comput. Sci. 104, 437–444 (2017). https://doi.org/10.1016/j.procs.2017.01.157

    Article  Google Scholar 

  20. Ekman, P.: Basic emotions. In: Handbook of Cognition and Emotion. pp. 45–60. Wiley (1999)

    Google Scholar 

  21. Pekrun, R., Frenzel, A.C., Goetz, T., Perry, R.P.: The control-value theory of achievement emotions: An integrative approach to emotions in education. In: Emotion in Education, pp. 13–36. Elsevier (2007)

    Google Scholar 

  22. Pekrun, R., Vogl, E., Muis, K.R., Sinatra, G.M.: Measuring emotions during epistemic activities: the epistemically-related emotion scales. Cogn. Emot. 31, 1268–1276 (2017). https://doi.org/10.1080/02699931.2016.1204989

    Article  Google Scholar 

  23. Ocumpaugh, J., Baker, R.S., Rodrigo, M.M.: Baker rodrigo ocumpaugh monitoring protocol (BROMP) 2.0 technical and training manual. Technical Report. Teachers College, Columbia University, New York. Ateneo Laboratory for the Learning Sciences, Manila (2015)

    Google Scholar 

  24. Malekzadeh, M., Mustafa, M., Lahsasna, A.: A review of emotion regulation in intelligent tutoring systems. Educ. Technol. Soc. 18, 435–445. https://www.jstor.org/stable/10.2307/jeductechsoci.18.4.435

  25. Jarrell, A., Harley, J.M., Lajoie, S., Naismith, L.: Success, failure and emotions: examining the relationship between performance feedback and emotions in diagnostic reasoning. Educ. Tech. Res Dev. 65(5), 1263–1284 (2017). https://doi.org/10.1007/s11423-017-9521-6

    Article  Google Scholar 

  26. Paquette, L., et al.: Sensor-free or sensor-full: a comparison of data modalities in multi-channel affect detection. In: Proceedings of the 8th International Conference on Educational Data Mining, pp. 93–100. IEDMS, Madrid (2016)

    Google Scholar 

  27. Kim, J., Seo, J., Laine, T.H.: Detecting boredom from eye gaze and EEG. Biomed. Sig. Process. Control 46, 302–313 (2018). https://doi.org/10.1016/j.bspc.2018.05.034

    Article  Google Scholar 

  28. Lallé, S., Conati, C., Carenini, G.: Predicting confusion in information visualization from eye tracking and interaction data. In: Proceedings on the 25th International Joint Conference on Artificial Intelligence, pp. 2529–2535. AAAI Press, New York (2016)

    Google Scholar 

  29. Henderson, N., Emerson, A., Rowe, J., Lester, J.: Improving sensor-based affect detection with multimodal data imputation. In: Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction, pp. 669–675. IEEE, Cambridge (2019)

    Google Scholar 

  30. Hutt, S., Mills, C., White, S., Donnelly, P.J., D’Mello, S.K.: The eyes have it: gaze-based detection of mind wandering during learning with an intelligent tutoring system. In: Proceedings of the 9th International Conference on Educational Data Mining, pp. 86–93. IEDMS, Raleigh (2016)

    Google Scholar 

  31. Kardan, S., Conati, C.: Comparing and combining eye gaze and interface actions for determining user learning with an interactive simulation. In: Carberry, S., Weibelzahl, S., Micarelli, A., Semeraro, G. (eds.) UMAP 2013. LNCS, vol. 7899, pp. 215–227. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38844-6_18

    Chapter  Google Scholar 

  32. Pekrun, R., Bühner, M.: Self-report measures of academic emotions. In: International Handbook of Emotions in Education. Routledge, London (2014)

    Google Scholar 

  33. Tan, P.-N., Kumar, V., Srivastava, J.: Selecting the right interestingness measure for association patterns. In: Proceedings of the 8th ACM International Conference on Knowledge Discovery and Data Mining, pp. 32–41. ACM, Edmonton (2002). https://doi.org/10.1145/775047.775053

  34. Villarica, R., Richards, D.: Intelligent and empathic agent to support student learning in virtual worlds. In: Proceedings of the Conference on Interactive Entertainment, pp. 1–9. ACM, Newcastle (2014). https://doi.org/10.1145/2677758.2677761

  35. Moridis, C.N., Economides, A.A.: Affective learning: empathetic agents with emotional facial and tone of voice expressions. IEEE Trans. Affect. Comput. 3, 260–272 (2012). https://doi.org/10.1109/t-affc.2012.6

    Article  Google Scholar 

  36. Craig, S., Graesser, A., Sullins, J., Gholson, B.: Affect and learning: an exploratory look into the role of affect in learning with AutoTutor. J. Educ. Media. 29, 241–250 (2004). https://doi.org/10.1080/1358165042000283101

    Article  Google Scholar 

  37. Liu, Z., Pataranutaporn, V., Ocumpaugh, J., Baker, R.: Sequences of frustration and confusion, and learning. In: Proceedings of the International Conference on Educational Data Mining, pp. 114–120. IEDMS, Memphis (2013)

    Google Scholar 

  38. D’Mello, S., Graesser, A.: The half-life of cognitive-affective states during complex learning. Cogn. Emot. 25, 1299–1308 (2011). https://doi.org/10.1080/02699931.2011.613668

    Article  Google Scholar 

  39. Huang, X., Mayer, R.E.: Benefits of adding anxiety-reducing features to a computer-based multimedia lesson on statistics. Comput. Hum. Behav. 63, 293–303 (2016). https://doi.org/10.1016/j.chb.2016.05.034

    Article  Google Scholar 

  40. Meyer, D.K.: Emotion regulation in K–12 classrooms. In: Handbook of Social Influences in School Contexts: Social-Emotional, Motivation, and Cognitive Outcomes. Routledge (2016)

    Google Scholar 

  41. Kardan, S., Lallé, S., Toker, D., Conati, C.: EMDAT: eye movement data analysis toolkit (1.x). The University of British Columbia (2021). https://doi.org/10.5281/zenodo.4699774

  42. Bouchet, F., Harley, J.M., Trevors, G.J., Azevedo, R.: Clustering and profiling students according to their interactions with an intelligent tutoring system fostering self-regulated learning. J. Educ. Data Min. 5, 104–146 (2013). https://doi.org/10.5281/zenodo.3554613

    Article  Google Scholar 

  43. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  44. Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39–58 (2009). https://doi.org/10.1109/TPAMI.2008.52

    Article  Google Scholar 

  45. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002). https://doi.org/10.1613/jair.953

    Article  MATH  Google Scholar 

  46. Holm, S.: A simple sequentially rejective multiple test procedure. Scand. J. Stat. 65–70 (1979)

    Google Scholar 

Download references

Acknowledgements

This paper is based upon work funded by the National Science Foundation (#DRL-1431552) and the Natural Sciences and Engineering Research Council (#22R01881).

Author information

Authors and Affiliations

  1. The University of British Columbia, Vancouver, BC, Canada

    Sébastien Lallé, Rohit Murali & Cristina Conati

  2. University of Central Florida, Orlando, FL, USA

    Roger Azevedo

Authors
  1. Sébastien Lallé
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rohit Murali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cristina Conati
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Roger Azevedo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sébastien Lallé .

Editor information

Editors and Affiliations

  1. Technion – Israel Institute of Technology, Haifa, Israel

    Ido Roll

  2. Arizona State University, Tempe, AZ, USA

    Danielle McNamara

  3. Utrecht University, Utrecht, The Netherlands

    Sergey Sosnovsky

  4. London Knowledge Lab, London, UK

    Rose Luckin

  5. University of Leeds, Leeds, UK

    Vania Dimitrova

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lallé, S., Murali, R., Conati, C., Azevedo, R. (2021). Predicting Co-occurring Emotions from Eye-Tracking and Interaction Data in MetaTutor. In: Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2021. Lecture Notes in Computer Science(), vol 12748. Springer, Cham. https://doi.org/10.1007/978-3-030-78292-4_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-78292-4_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-78291-7

  • Online ISBN: 978-3-030-78292-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation