Skip to main content
SpringerLink
Log in

Measuring and Integrating Facial Expressions and Head Pose as Indicators of Engagement and Affect in Tutoring Systems

  • Conference paper
  • First Online:
Adaptive Instructional Systems. Adaptation Strategies and Methods (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12793))

Included in the following conference series:

Abstract

While using online learning software, students demonstrate many reactions, various levels of engagement, and emotions (e.g. confusion, boredom, excitement). Having such information automatically accessible to teachers (or digital tutors) can aid in understanding how students are progressing, and suggest who and when needs further assistance. As part of this work, we conducted two studies using computer vision techniques to measure students’ engagement and affective states from their head pose and facial expressions, as they use an online tutoring system, MathSpring.org, designed to aid students’ practice of mathematics problem-solving. We present a Head Pose Tutor, which estimates the real-time head direction of students and responds to potential disengagement, and a Facial Expression-Augmented Teacher Dashboard, that identifies students’ affective states and provides this information to teachers. We collected video data of undergraduate students interacting with MathSpring. Preliminary results on MathSpring videos were encouraging indicating accuracy in detecting head orientation. A usability study was conducted with actual teachers to start to evaluate the possible impact of the proposed Teacher Dashboard software.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Arroyo, I., Woolf, B.P., Burelson, W., Muldner, K., Rai, D., Tai, M.: A multimedia adaptive tutoring system for mathematics that addresses cognition, metacognition and affect. Int. J. Artif. Intell. Educ. 24(4), 387–426 (2014)

    Article  Google Scholar 

  2. Baker, R.S., D’Mello, S.K., Rodrigo, M.M.T., Graesser, A.C.: Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. Int. J. Hum.-Comput. Stud. 68(4), 223–241 (2010)

    Article  Google Scholar 

  3. Bosch, N., D’mello, S.K., Ocumpaugh, J., Baker, R.S., Shute, V.: Using video to automatically detect learner affect in computer-enabled classrooms. ACM Trans. Inter. Intell. Syst. (TiiS) 6(2), 1–26 (2016)

    Article  Google Scholar 

  4. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3D facial landmarks). In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1021–1030 (2017)

    Google Scholar 

  5. Chang, F.J., Tuan Tran, A., Hassner, T., Masi, I., Nevatia, R., Medioni, G.: Faceposenet: making a case for landmark-free face alignment. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1599–1608 (2017)

    Google Scholar 

  6. Corrigan, S., Barkley, T., Pardos, Z.: Dynamic approaches to modeling student affect and its changing role in learning and performance. In: Ricci, F., Bontcheva, K., Conlan, O., Lawless, S. (eds.) UMAP 2015. LNCS, vol. 9146, pp. 92–103. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20267-9_8

    Chapter  Google Scholar 

  7. D’Mello, S., Dieterle, E., Duckworth, A.: Advanced, analytic, automated (AAA) measurement of engagement during learning. Educ. Psychol. 52(2), 104–123 (2017)

    Article  Google Scholar 

  8. D’Mello, S., Olney, A., Williams, C., Hays, P.: Gaze tutor: a gaze-reactive intelligent tutoring system. Int. J. Hum.-Comput. Stud. 70(5), 377–398 (2012)

    Article  Google Scholar 

  9. D’Mello, S., Lehman, B., Pekrun, R., Graesser, A.: Confusion can be beneficial for learning. Learn. Instr. 29, 153–170 (2014)

    Article  Google Scholar 

  10. D’Mello, S.K.: Gaze-based attention-aware cyberlearning technologies. In: Parsons, T.D., Lin, L., Cockerham, D. (eds.) Mind, Brain and Technology. ECTII, pp. 87–105. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-02631-8_6

    Chapter  Google Scholar 

  11. Ekman, P., Friesen, W.V., Hager, J.C.: Facial action coding system. Research Nexus, Salt Lake City (2002)

    Google Scholar 

  12. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124 (1971)

    Article  Google Scholar 

  13. Fanelli, G., Weise, T., Gall, J., Van Gool, L.: Real time head pose estimation from consumer depth cameras. In: Mester, R., Felsberg, M. (eds.) DAGM 2011. LNCS, vol. 6835, pp. 101–110. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23123-0_11

    Chapter  Google Scholar 

  14. Gou, C., Wu, Y., Wang, F.Y., Ji, Q.: Coupled cascade regression for simultaneous facial landmark detection and head pose estimation. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 2906–2910. IEEE (2017)

    Google Scholar 

  15. Grafsgaard, J.F., Wiggins, J.B., Vail, A.K., Boyer, K.E., Wiebe, E.N., Lester, J.C.: The additive value of multimodal features for predicting engagement, frustration, and learning during tutoring. In: Proceedings of the 16th International Conference on Multimodal Interaction, pp. 42–49 (2014)

    Google Scholar 

  16. Hoffman, J.E., Subramaniam, B.: The role of visual attention in saccadic eye movements. Percept. Psychophysics. 57(6), 787–795 (1995)

    Article  Google Scholar 

  17. Hu, Y., Chen, L., Zhou, Y., Zhang, H.: Estimating face pose by facial asymmetry and geometry. In: Proceedings of Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004, pp. 651–656. IEEE (2004)

    Google Scholar 

  18. Hutt, S., Mills, C., Bosch, N., Krasich, K., Brockmole, J., D’mello, S.: Out of the fr-eye-ing pan towards gaze-based models of attention during learning with technology in the classroom. In: Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization, pp. 94–103 (2017)

    Google Scholar 

  19. Khan, A.Z., Blohm, G., McPeek, R.M., Lefevre, P.: Differential influence of attention on gaze and head movements. J. Neurophysiol. 101(1), 198–206 (2009)

    Article  Google Scholar 

  20. Khorrami, P., Paine, T., Huang, T.: Do deep neural networks learn facial action units when doing expression recognition? In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 19–27 (2015)

    Google Scholar 

  21. Kumar, A., Alavi, A., Chellappa, R.: Kepler: keypoint and pose estimation of unconstrained faces by learning efficient H-CNN regressors. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pp. 258–265. IEEE (2017)

    Google Scholar 

  22. Martins, P., Batista, J.: Accurate single view model-based head pose estimation. In: 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 1–6. IEEE (2008)

    Google Scholar 

  23. Meng, Z., Liu, P., Cai, J., Han, S., Tong, Y.: Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pp. 558–565. IEEE (2017)

    Google Scholar 

  24. Mukherjee, S.S., Robertson, N.M.: Deep head pose: Gaze-direction estimation in multimodal video. IEEE Trans. Multimedia. 17(11), 2094–2107 (2015)

    Article  Google Scholar 

  25. Ranjan, R., Patel, V.M., Chellappa, R.: Hyperface: a deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans. Pattern Anal. Mach. Intell. 41(1), 121–135 (2017)

    Article  Google Scholar 

  26. Rayner, K.: Eye movements in reading and information processing: 20 years of research. Psychol. Bull. 124(3), 372 (1998)

    Article  Google Scholar 

  27. Ruiz, N., Chong, E., Rehg, J.M.: Fine-grained head pose estimation without keypoints. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2074–2083 (2018)

    Google Scholar 

  28. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: A comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  29. Sharma, K., Alavi, H.S., Jermann, P., Dillenbourg, P.: A gaze-based learning analytics model: in-video visual feedback to improve learner’s attention in MOOCs. In: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, pp. 417–421 (2016)

    Google Scholar 

  30. Whitehill, J., Serpell, Z., Lin, Y.C., Foster, A., Movellan, J.R.: The faces of engagement: automatic recognition of student engagement from facial expressions. IEEE Trans. Affect. Comput. 5(1), 86–98 (2014)

    Article  Google Scholar 

  31. Wixon, M., Arroyo, I.: When the question is part of the answer: examining the impact of emotion self-reports on student emotion. In: Dimitrova, V., Kuflik, T., Chin, D., Ricci, F., Dolog, P., Houben, G.-J. (eds.) UMAP 2014. LNCS, vol. 8538, pp. 471–477. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-08786-3_42

    Chapter  Google Scholar 

  32. Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., Picard, R.: Affect-aware tutors: recognising and responding to student affect. Int. J. Learn. Technol. 4(3–4), 129–164 (2009)

    Article  Google Scholar 

  33. Yang, T.Y., Chen, Y.T., Lin, Y.Y., Chuang, Y.Y.: FSA-net: learning fine-grained structure aggregation for head pose estimation from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1087–1096 (2019)

    Google Scholar 

  34. Zatarain-Cabada, R., Barrón-Estrada, M.L., Camacho, J.L.O., Reyes-García, C.A.: Affective tutoring system for android mobiles. In: Huang, D.-S., Jo, K.-H., Wang, L. (eds.) ICIC 2014. LNCS (LNAI), vol. 8589, pp. 1–10. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-09339-0_1

    Chapter  Google Scholar 

  35. Zhang, F., Zhang, T., Mao, Q., Xu, C.: Joint pose and expression modeling for facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3359–3368 (2018)

    Google Scholar 

  36. Zhi, R., Flierl, M., Ruan, Q., Kleijn, W.B.: Graph-preserving sparse nonnegative matrix factorization with application to facial expression recognition. IEEE Trans. Syst. Man Cybern. B Cybern. 41(1), 38–52 (2010)

    Google Scholar 

  37. Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J., Metaxas, D.N.: Learning active facial patches for expression analysis. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2562–2569. IEEE (2012)

    Google Scholar 

  38. Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: a 3D solution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 146–155 (2016)

    Google Scholar 

  39. Zhu, X., Liu, X., Lei, Z., Li, S.Z.: Face alignment in full pose range: a 3D total solution. IEEE Trans. Pattern Anal. Mach. Intell. 41(1), 78–92 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Boston University, Boston, MA, 02215, USA

    Hao Yu & Margrit Betke

  2. Worcester Polytechnic Institute, Worcester, MA, 01609, USA

    Ankit Gupta

  3. University of Massachusetts-Amherst, Amherst, MA, 01003, USA

    Will Lee, Ivon Arroyo, Danielle Allesio, Tom Murray & Beverly P. Woolf

  4. Clark University, Worcester, MA, 01610, USA

    John Magee

Authors
  1. Hao Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ankit Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Will Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ivon Arroyo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Margrit Betke
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Danielle Allesio
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Tom Murray
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. John Magee
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Beverly P. Woolf
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ivon Arroyo .

Editor information

Editors and Affiliations

  1. Soar Technology, Inc., Orlando, FL, USA

    Robert A. Sottilare

  2. Fraunhofer FKIE, Wachtberg, Germany

    Jessica Schwarz

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yu, H. et al. (2021). Measuring and Integrating Facial Expressions and Head Pose as Indicators of Engagement and Affect in Tutoring Systems. In: Sottilare, R.A., Schwarz, J. (eds) Adaptive Instructional Systems. Adaptation Strategies and Methods. HCII 2021. Lecture Notes in Computer Science(), vol 12793. Springer, Cham. https://doi.org/10.1007/978-3-030-77873-6_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77873-6_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77872-9

  • Online ISBN: 978-3-030-77873-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation