Skip to main content
SpringerLink
Log in

Student engagement study based on multi-cue detection and recognition in an intelligent learning environment

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Student engagement has great impact on learning performance. It’s necessary to investigate student engagement objectively from learning behavior. In this paper, we propose a student engagement study approach in an intelligent learning environment, which automatically detects and analyses multiple learning behavioral cues based on five modules, i.e., attendance management, teacher-student (T&S) communication, visual focus of attention (VFOA) recognition, smile detection and engagement analysis. Attendance management matches the student’s identity and locates his/her profile using face recognition. T&S communication provides an additional channel of Question and Answer (Q&A) between a teacher and students for students’ behavioral engagement analysis via their cell phones. VFOA recognition is used to recognize students’ cognitive engagement through capturing students’ attention based on the estimated head poses, visual environment cues and prior states in class. Smile detection achieves students’ affective engagement through spontaneous smile expression classification. Finally, a tree-structural engagement model is proposed to decide student engagement based on multi-cues of one’s behavioral, cognitive and affective engagement. We thoroughly evaluated each module for engagement study on some public available datasets and practical video sequences in class applications. The experimental results suggest that the proposed approach can automatically detect and analyze student class engagement objectively and effectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Agarwal M, Agrawal H, Jain N, Kumar M (2010) Face recognition using principle component analysis, eigenface and neural network. In: International Conference on Signal Acquisition and Processing, pp 310–314

  2. An L, Yang S, Bir B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363

    Article  Google Scholar 

  3. Ba SO, Odobez JM (2011) Multiperson visual focus of attention from head pose and meeting contextual cues. IEEE Trans Pattern Anal Mach Intell 33(1):101–116

    Article  Google Scholar 

  4. Boyle JT, Nicol DJ (2003) Using classroom communication systems to support interaction and discussion in large class settings. Research in Learning Technology 11 (3):43–57

    Article  Google Scholar 

  5. Bradsky GR (1998) Computer vision face tracking for use in a perceptual user interface. Intel Technol J 2:1–15

    Google Scholar 

  6. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  Google Scholar 

  7. Chen D, Hu Y, Wang L, Zomaya AY, Li X (2017) H-PARAFAC: hierarchical parallel factor analysis of multidimensional big data. IEEE Trans Parallel Distrib Syst 28(4):1091–1104

    Article  Google Scholar 

  8. Chen J, Chen D, Li X (2014) Toward improving social communication skills upon multimodal sensory information. IEEE Trans Ind Inf 10(1):323–330

    Article  Google Scholar 

  9. Chen J, Luo N, Liu Y, Liu L, Zhang K, Kolodziej J (2016) A hybrid intelligence-aided approach to affect-sensitive e-learning. Computing 98(1-2):215–233

    Article  MathSciNet  Google Scholar 

  10. Cohen M, Shimshoniand I, Rivlin E (2012) Detecting mutual awareness events. IEEE Trans Pattern Anal Mach Intell 34(12):2327–2340

    Article  Google Scholar 

  11. Dantone M, Gall J, Fanelli G, Van Gool L (2012) Real time facial feature detection using conditional regression forests. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 2578–2585

  12. Gall J, Lempitsky V (2013) Class specific hough forests for object detection. In: Decision Forests for Computer Vision and Medical Image Analysis, pp 143–157

    Chapter  Google Scholar 

  13. Gourier N, Hall D, Crowley J (2004) Estimating face orientation from robust detection of salient facial features in pointing. In: International conference on pattern recognition Workshop on Visual Observation of Deictic Gestures, pp 1379–1382

  14. Graesser A, Chipman P, King B (2007) Emotions and learning with auto tutor. Frontiers in Artificaial Intelligence and Applications 158:569

    Google Scholar 

  15. Guerra J, Hosseini R, Somyurek S, Brusilovsk P (2016) An intelligent interface for learning content combining an open learner model and social comparison to support self-regulated learning and engagement. In: Proceedings of the 21st International Conference on Intelligent User Interfaces. ACM, pp 152–163

  16. Gui J, Liu T, Tao D, Sun Z, Tan T (2016) Representative vector a unified framework for classical classifiers. IEEE Trans Cybernet 46(8):1877–1888

    Article  Google Scholar 

  17. Guo P, Kim J, Rubin R (2014) How video production affects student an empirical study of mooc videos. In: Proceedings of the first ACM conference on Learning@ scale conference. ACM, pp 41–50

  18. Huang GB, Ramesh M, Berg T, Learned-Mill E (2007) Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical report, Technical Report 07-49, University of Massachusetts, Amherst

  19. Ito A, Wang X, Suzuki M, Makino S (2005) Smile and laughter recognition using speech processing and face recognition from conversation video. In: IEEE International Conference on Cyberworlds, pp 437–444

  20. Kahou SE, Froumenty P, Pal C (2014) Facial expression analysis based on high dimensional binary features. In: European Conference on Computer Vision, pp 135–147

    Google Scholar 

  21. Koji Y, Atsushi I, Shinji T, Hiroharu K (2016) Analysis of computer event logs to assess student engagement in classroom: a case study in the united states. In: 12th International Conference on Natural Computation Fuzzy Systems and Knowledge Discovery, pp 2098–2101

  22. Liu Y, Chen J, Shan C (2014) Dirichlet-tree distribution enhanced random forests for facial feature detection. In: IEEE Conference on Image Processing, pp 235–238

  23. Liu Y, Chen J, Gong Y, Zhang K, Liu L, Luo N (2015) Robust head pose estimation using dirichlet-tree dis-tribution enhanced random forests. Neurocomputing 127:42–53

    Google Scholar 

  24. Liu Y, Xie Z, Chen J (2016) An intelligent learning system for supporting interactive learning through student engagement study. IEEE, pp 618–623

  25. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews (2010) The extended cohn-kanade dataset (ck+) a complete dataset for action unit and emotion specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp 94–101

  26. Luo C, Wang Z, Wang S (2015) Locating facial landmarks using probabilistic random forest. IEEE Sign Process Lett 22(12):2324–2328

    Article  Google Scholar 

  27. Mi J, Lei D, Gui J (2013) A novel method for recognizing face with partial occlusion via sparse representation. Optik-International Journal for Light and Electron Optics 124(24):6786–6789

    Article  Google Scholar 

  28. Odobez JM, Ba SO (2007) A cognition and unsupervised map adaptation approach to the recognition of focus of attention from head pose. In: IEEE International Conference on Multimedia and Expo, pp 183–191

  29. Podder P, Paul M, Debnath T, Murshed M (2015) An analysis of human engagement behaviour using descriptors from human feedback, eye tracking,and saliency modelling. In: Digital Image Computing: Techniques and Applications, pp 1–8

  30. Rodgers T (2008) Student engagement in the e-learning process and the impact on their grades. International Journal of Cyber Society and Education 1(2):143–156

    Google Scholar 

  31. Scornavacca E, Huff S, Marshall S (2007) Developing a sms-based classroom interaction system. In: Proceedings of Mobile Learning Technologies and Applications, pp 47–54

  32. Senechal T, Turcot J, Kaliouby R (2013) Smile or smirk? automatic detection of spontaneous asymmetric smiles to understand viewer experience. In: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp 1–8

  33. Shan C (2012) Smile detection by boosting pixel differences. IEEE Trans Image Process 21(1):431–436

    Article  MathSciNet  Google Scholar 

  34. Siau K, Sheng H, Nah FF (2006) Use of a classroom response system to enhance classroom interactivity. IEEE Transactions on Education 49(3):398–403

    Article  Google Scholar 

  35. Ting C, Cheah W, Ho C (2013) Student engagement modeling using bayesian networks. In: IEEE International Conference on Systems, Man, and Cybernetics, pp 2939–2944

  36. Vazquez Rodriguez CA, Mejia Lavalle M, Pinto Elias R (2015) Modeling student engagement by means of nonverbal of nonverbal behavior and decision trees. In: International Conference on Mechatronics, Electronics and Automotive Engineering, pp 81–85

  37. Viola M, Jones MJ (2003) Fast multi-view face detection. Mitsubishi Electric Research Lab TR-20003-96 3:14

    Google Scholar 

  38. Woolf B, Burleson W, Arroyo I (2009) Affect-aware tutors: recognising and responding to student affect. Int J Learn Technol 4(3-4):129–164

    Article  Google Scholar 

  39. Yang C, Duraiswami R, Davis L (2005) Efficient mean-shift tracking via a new similarity measure. In: IEEE conference on Computer Vision and Pattern Recognition, pp 176–183

  40. Zhang T, Zheng W, Cui Z, Zhong Y, Yan J, Yan K (2016) A deep neural network driven feature learning method for multi-view facial expression recognition. IEEE Trans Multimedia 18(12):2528–2536

    Article  Google Scholar 

  41. Zhao X, Kim T, Luo W (2014) Unified face analysis by iterative multi-output random forests. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1–8

Download references

Acknowledgements

This work was supported by the National Social Science Foundation of China (Grant no. 16BSH107).

Author information

Authors and Affiliations

  1. Faculty of Information Engineering, China University of Geosciences, Wuhan, China

    Yuanyuan Liu

  2. National Engineering Research Center for E-Learning, Central China Normal University, Wuhan, China

    Jingying Chen, Mulan Zhang & Chuan Rao

Authors
  1. Yuanyuan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingying Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mulan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chuan Rao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jingying Chen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Chen, J., Zhang, M. et al. Student engagement study based on multi-cue detection and recognition in an intelligent learning environment. Multimed Tools Appl 77, 28749–28775 (2018). https://doi.org/10.1007/s11042-018-6017-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6017-2

Keywords

Search

Navigation