Skip to main content

Advertisement

SpringerLink
Log in

3DLIM: Intelligent analysis of students’ learning interest by using multimodal fusion technology

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

Learning interest affects the way of learning and its process, which is an important factor that affects the learning effect. At present, students' learning interest in a teaching environment is mainly based on a traditional questionnaire or case analysis, which is not conducive for teachers to promptly access students' interest in class to effectively improve the teaching behavior. To intelligently analyze students' learning interest, a Three-Dimensional Learning Interest Model (3DLIM) is proposed based on educational psychology angle, which includes cognitive attention (Attention), learning emotion (Emotion), and thinking activities (Thinking). The proposed approach consisted of multimodal information recognition and fusion on head pose as well as facial expression and class interaction analysis, so as to comprehensively analyze students' interest in a teaching environment. To verify the validity and feasibility of the model, full-time graduate students participated in the experiments in real classroom scenarios. The Super Star interactive platform was used to collect interactive information from students in a class. Surveillance video tracked the learning process of the whole professional theoretical course. Experimental results show that the proposed approach can reflect the students' learning interest in learning as well as the differences among individuals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data availability

Data used to support the findings of this study are included within the article.

References

  • Ainley, M., Hidi, S., & Berndorff, D. (2019). Interest learning and the psychological processes that mediate their relationship. Journal of Educational Psychology, 94(3), 545–561. https://doi.org/10.1037//0022-0663.94.3.545

    Article  Google Scholar 

  • Ainley, M., & Hidi, S. (2014). Interest and enjoyment. In E. G. Carayannis (Ed.) Encyclopedia of Creativity Invention, Innovation and Entrepreneurship, (pp. 31–58). Springer.

    Google Scholar 

  • Abelardo, P., Dragan, G., Jovanovic, J. M., Shane, D., & Negin, M. (2019). Exploring student interactions with preparation activities in a flipped classroom experience. IEEE Transactions on Learning Technology, 12, 333–346. https://doi.org/10.1109/TLT.2018.2858790

    Article  Google Scholar 

  • Ashwin, T. S., & Guddeti, R. M. R. (2019). Unobtrusive Behavioral Analysis of Students in Classroom Environment Using Non-Verbal Cues. IEEE Access, 7, 150693–150709. https://doi.org/10.1109/ACCESS.2019.2947519

    Article  Google Scholar 

  • Cédric, R., Vanessa, L. K., & Denis, P. (2020). Students’ individual interest in physical education: Development and validation of a questionnaire. Journal of Psychology, 62(1), 64–73. https://doi.org/10.1111/sjop.12669

    Article  Google Scholar 

  • Chen, J., Luo, N., Liu, Y., Liu, L., Zhang, K., & Kolodziej, J. (2016). A hybrid intelligence-aided approach to affect-sensitive e-learning. Computing, 98, 215–233. https://doi.org/10.1007/s00607-014-0430-9

    Article  MathSciNet  Google Scholar 

  • Dantone, M., Gall, J., Fanelli, G., & Van Gool, L. (2012). Real-time facial feature detection using conditional regression forests. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, Rhode Island, 2578–2585. https://doi.org/10.1109/CVPR.2012.6247976.

  • Errico, D., Paciello, M., & Cerniglia, L. (2016). When emotions enhance students' engagement in e-learning processes. Journal of e-Learning and Knowledge Society, 12 (4). https://doi.org/10.20368/1971-8829/1144.

  • Fanelli, G., Gall, J., & Van Gool, L. (2011). Real time head pose estimation with random regression forests. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 617–624,. https://doi.org/10.1109/CVPR.2011.5995458

  • Fujisawa, K., & Aihara, K. K. (2009). Estimation of user interest from face approaches captured by webcam. International Conference on Virtual and Mixed Reality, 5622, 51–59. https://doi.org/10.1007/978-3-642-02771-0_6

    Article  Google Scholar 

  • Gupta, S. K., Ashwin, T. S., & Guddeti, R. M. R. (2019). Students’ affective content analysis in smart classroom environment using deep learning techniques. Multimedia Tools Applications, 78, 25321–25348. https://doi.org/10.1007/s11042-019-7651-z

    Article  Google Scholar 

  • Hidi, S. (2006). Interest: A motivational variable with a difference. Educational Research Review, 7, 323–350. https://doi.org/10.1016/j.edurev.2006.09.001

    Article  Google Scholar 

  • Huang, G.B., Mattar, Mattar, M., Berg, T., & Learned-Miller, E. (2008). Labeled faces in the wild:a database for studying face recognition in unconstrained environments. Workshop on Faces in 'Real-Life' Images: Detection, Alignment, and Recognition, Erik Learned-Miller and Andras Ferencz and Frédéric Jurie, Marseille, France.

  • Iran-Nejad, A., & Chissom, B. S. (1992). Contributions of active and dynamic self-regulation to learning. Innovative Higher Education, 17, 125–136. https://doi.org/10.1007/BF00917134

    Article  Google Scholar 

  • Kang, M. J., Hsu, M., Krajbich, I. M., Loewenstein, G. F., McClure, S. M., Wang, J. T., & Camerer, C. F. (2009). The wick in the candle of learning: Epistemic curiosity activates reward circuitry and enhances memory. Psychological Science, 20, 963–973. https://doi.org/10.1111/j.1467-9280.2009.02402.x

    Article  Google Scholar 

  • Kipf., T.N & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR). https://doi.org/10.48550/arXiv.1609.02907.

  • Krapp, A. (2005). Basic needs and the development of interest and intrinsic motivational orientations. Learning and Instruction, 12, 383–409. https://doi.org/10.1016/j.learninstruc.2005.07.007.58790

    Article  Google Scholar 

  • Luo, Z. Z., Chen, J. Y., Wang, G. S., & Liao, M. Y. (2020). A three-dimensional model of student interest during learning using multimodal fusion with natural sensing technology. Interactive Learning Environments (ILE), 30(6), 1117–1130. https://doi.org/10.1080/10494820.2019.1710852

    Article  Google Scholar 

  • Liu, Y., Chen, J., Su, Z., Luo, Z., Luo, N., Liu, L., & Zhang, K. (2016a). Robust head pose estimation using Dirichlet-tree distribution enhanced random forests. Neurocomputing, 173, 42–53. https://doi.org/10.1016/j.neucom.2015.03.096

    Article  Google Scholar 

  • Liu, Y., Liu, L., Chen, J., Su, C., & Zhang, K. (2016b). Multi-person visual focus of attention from head pose on a natural classroom. International Conference on Pattern Recognition Applications and Methods (ICPRAM), 205–213.

  • Luo, Z., Liu, L., Chen, J., Liu, Y., & Su, Z. (2016). Spontaneous smile recognition for interest detection. In T. Tan, X. Li, X. Chen, J. Zhou, J. Yang, & H. Cheng (Eds.), Pattern Recognition. CCPR 2016. Communications in Computer and Information Science (vol. 662, pp. 119–130). Springer. https://doi.org/10.1007/978-981-10-3002-4_10

  • Mills, C. M., Danovitch, J. H., Rowles, S. P., & Campbell, I. L. (2017). Children’s success at detecting circular explanations and their interest in future learning. Psychonomic Bulletin & Review, 24, 1465–1477. https://doi.org/10.3758/s13423-016-1195-2

    Article  Google Scholar 

  • Mohammad, N. M., Sara, F., Zahra, T., & Hesari, M. (2013). The study of the teacher's role and student interaction in e-leaming process. In The 4th International Conference on e-Learning and e-Teaching (ICELET), Shiraz, Iran (pp. 130–134). IEEE. https://doi.org/10.1109/ICELET.2013.6681659

  • Mota, S., & Picard, R. W. (2008). Automated posture analysis for detecting learner's interest level. In 2033 IEEE International conference on Computer Vision and Pattern Recognition Workshop (CVPRW), Madison, Wisconsin (vol. 5, pp. 49–55). IEEE. https://doi.org/10.1109/CVPRW.2003.10047

  • Nakamura, K., Kakusho, K., Shoji, T., & Minoh, M. (2012). Investigation of a method to estimate learners’ interest level for agent-based conversational e-learning. Advanced in Computational Intelligence, IPMU, 2012(298), 425–433. https://doi.org/10.1007/978-3-642-31715-6_45

    Article  Google Scholar 

  • Jain, A., Sah, H. R., & Kothari, A. (2021). Study for emotion recognition of different age groups students during online class. In 2021 8th International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India. (pp. 621–625). IEEE

  • Jamil, N., Belkacem, A. N., Ouhbi, S., & Guger, C. (2021). Cognitive and Affective Brain-Computer Interfaces for Improving Learning Strategies and Enhancing Student Capabilities: A Systematic Literature Review. IEEE Access, 9, 134122–134147. https://doi.org/10.1109/ACCESS.2021.3115263

    Article  Google Scholar 

  • Orji, F., & Vassileva, J. (2020). Using machine learning to explore the relation between student engagement and student performance. In 2020 24th International Conference Information Visualisation (IV), Melbourne, Australia (pp. 480–485). IEEE. https://doi.org/10.1109/IV51561.2020.00083

  • Park, B., Lisa, Knorzer, Plass, J. L., & Roland, Bruenken. (2015). Emotional design and positive emotions in multimedia learning: An eye tracking study on the use of anthropomorphisms. Computer and Education, 86, 30–42. https://doi.org/10.1016/j.compedu.2015.02.016

    Article  Google Scholar 

  • Ramakrishnan, A., Zylich, B., Ottmar, E., Locasale, C. J., & Whitehill, J. (2021). Toward automated classroom observation: Multimodal machine learning to estimate CLASS positive climate and negative climate. In IEEE Transactions on Affective Computing (pp. 1–16). IEEE. Advance online publication. https://doi.org/10.1109/TAFFC.2021.3059209

  • Renninger, K. A., & Hidi, S. (2011). Revisiting the Conceptualization, Measurement, and Generation of Interest. Educational Psychologist, 46(3), 168–184. https://doi.org/10.1080/00461520.2011.587723

    Article  Google Scholar 

  • Rodríguez, M. A., Antino, M., Ruiz, Z. P., & Ortega, E. (2021). Positive emotions, engagement, and objective academic performance: A weekly diary study. Learning and Individual Difference, 92, 102087. https://doi.org/10.1016/j.lindif.2021.102087

    Article  Google Scholar 

  • Rotgans, J. I., & Schmidt, H. G. (2017). Interest development: Arousing situational interest affects the growth trajectory of individual interest. Contemporary Educational Psychology, 49, 175–184. https://doi.org/10.1016/j.cedpsych.2017.02.003

    Article  Google Scholar 

  • Sabourin, J. L., & Lester, J. C. (2013). Affect and engagement in game-based learning environments. IEEE Transactions on Affective Computing, 5(1), 45–56. https://doi.org/10.1109/T-AFFC.2013.27

    Article  Google Scholar 

  • Santos, P.B., Wahle, C.V., & Gurevych, T. (2018). Using Facial Expressions of Students for Detecting Levels of Intrinsic Motivation. 2018 IEEE 14th International Conference on e-Science (e-Science), 323–324. https://doi.org/10.1109/eScience.2018.00075.

  • Schukajlow, S., & Rakoczy, K. (2016). The power of emotions: Can enjoyment and boredom explain the impact of individual preconditions and teaching methods on interest and performance in mathematics? Learning and Instruction, 44, 117–127. https://doi.org/10.1016/j.learninstruc.2016.05.001

    Article  Google Scholar 

  • Singh, M., James, P. S., Paulc, H., & Bolard, K. (2022). Impact of cognitive-behavioral motivation on student engagement. Heliyon, 8(7), e09843. https://doi.org/10.1016/j.heliyon.2022.e09843

    Article  Google Scholar 

  • Sümer, Ö., Goldberg, P., Mello, S. D., Gerjets, P., Trautwein, U., & Kasneci, E. (2021). Multimodal engagement analysis from facial videos in the classroom. In IEEE Transactions on Affective Computing (pp. 1–16). IEEE. Advance online publication. https://doi.org/10.1109/TAFFC.2021.3127692

  • Terraza, A. D. F., Amaya, M., Piedrahita, C. A., Rodriguez, M. P. A., Duque, M. L., & Martinez, V. J. D. (2022). Students’ Attention Monitoring System in Learning Environments based on Artificial Intelligence. IEEE Latin America Transactions, 20(1), 126–132. https://doi.org/10.1109/TLA.2022.9662181

    Article  Google Scholar 

  • Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., & Picard, R. (2009). Affect-aware tutors: Recognizing and responding to student affect. International Journal of Learning Technology, 4(4), 129–164. https://doi.org/10.1504/IJLT.2009.028804

    Article  Google Scholar 

  • Zhang, S., Li, L., & Zhao, Z. (2012). Audio-visual emotion recognition based on facial expression and affective speech. Multimedia and Signal Processing, 346, 46–52. https://doi.org/10.1016/j.brainres.2008.04.023

    Article  Google Scholar 

  • Zhu, Y., Tian, D., & Yan, F. (2020). Effectiveness of Entropy Weight Method in Decision-Making. Mathematical Problems in Engineering, 7, 1–5. https://doi.org/10.1155/2020/3564835

    Article  Google Scholar 

  • Zhu, Z., Ober, S., & Jafari, R. (2017). Modeling and detecting student attention and interest level using wearable computers. In 2017 IEEE 14th International Conference on Wearable and Implantable Body Sensor Networks (BSN), Eindhoven, Netherlands (pp. 13–18). IEEE. https://doi.org/10.1109/BSN.2017.7935996

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (No. 61967010, 62067003), Natural Science Foundation of Jiangxi Province (No.20212BAB212007), Key Project of Science and Technology Research of Education Department of Jiangxi Province(No.GJJ210309), Teaching Reform Project of Colleges and Universities of Jiangxi Province (No.JXJG-19-2-24).

Author information

Authors and Affiliations

  1. School of Software, Jiangxi Normal University, Nanchang, Jiangxi, China

    Zhenzhen Luo, Chaoyu Zheng, Jun Gong, Shaolong Chen, Yong Luo & Yugen Yi

Authors
  1. Zhenzhen Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chaoyu Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jun Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shaolong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yong Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yugen Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yugen Yi.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luo, Z., Zheng, C., Gong, J. et al. 3DLIM: Intelligent analysis of students’ learning interest by using multimodal fusion technology. Educ Inf Technol 28, 7975–7995 (2023). https://doi.org/10.1007/s10639-022-11485-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-022-11485-8

Keywords

Search

Navigation