Abstract
Recognition of affective states to enhance e-learning platforms has been a topic of machine learning research. Compared to other input modalities, facial expressions have the potential to reveal nonverbal cues about a learner’s learning affect. However, most studies were limited in their analysis of learning affects exhibited by a learner with the possibility of providing appropriate feedback to teachers and learners. This work proposes an adaptive reasoning mechanism that considers the estimated affective states and learning affect in generating feedback with reasoning incorporated. This work utilizes a Convolutional Neural Network- Bidirectional Long-Short Term Memory (CNN-BiLSTM) cascade framework for affective states analysis through processing a live/stored observation of a learner in the form of a temporal signal. Using the proposed ensemble, four affective states were estimated, namely boredom, confusion, frustration, and engagement. Dataset for Affective States in E-Environment (DAiSEE) was used to train, validate, and test the baseline model, which reported an accuracy of 86% on 4305 test samples. In the next stage, mappings between estimated affective states and learning affects (i.e. positive, negative and neutral) were established based on an adaptive mapping mechanism, to consolidate the mapping between affective states and learning affects. Live testing and survey feedback were then used to further validate, adapt and amend the feedback process. Incorporating and interpreting the estimated affective states and learning affect is imperative in providing information to both teachers and learners, and hence potentially improve the existing e-learning platforms.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abedi, A., Khan, S.S.: Improving state-of-the-art in detecting student engagement with ResNet and TCN hybrid network. In: 2021 18th Conference on Robots and Vision (CRV), pp. 151–157. IEEE (2021)
Asaju, C., Vadapalli, H.: A temporal approach to facial emotion expression recognition. In: Jembere, E., Gerber, A.J., Viriri, S., Pillay, A. (eds.) SACAIR 2021. CCIS, vol. 1551, pp. 274–286. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-95070-5_18
Asaju, C.B., Vadapalli, H.: Affects analysis: a temporal approach to estimate students’ learning. In: 2021 3rd International Multidisciplinary Information Technology and Engineering Conference (IMITEC), pp. 1–7. IEEE (2021)
Ayvaz, U., Gürüler, H., Devrim, M.O.: Use of facial emotion recognition in e-learning systems (2017)
Butler, D.L., Winne, P.H.: Feedback and self-regulated learning: a theoretical synthesis. Rev. Educ. Res. 65(3), 245–281 (1995)
Calvo, R.A., D’Mello, S.: Frontiers of affect-aware learning technologies. IEEE Intell. Syst. 27(6), 86–89 (2012)
Chen, S., Dai, J., Yan, Y.: Classroom teaching feedback system based on emotion detection. In: 9th International Conference on Education and Social Science (ICESS 2019), pp. 940–946 (2019)
Csikszentmihalyi, M.: The contribution of flow to positive psychology (2000)
Csikszentmihalyi, M., Larson, R.: Flow and the Foundations of Positive Psychology, vol. 10. Springer, Heidelberg (2014). https://doi.org/10.1007/978-94-017-9088-8
Daschmann, E.C., Goetz, T., Stupnisky, R.H.: Testing the predictors of boredom at school: development and validation of the precursors to boredom scales. Br. J. Educ. Psychol. 81(3), 421–440 (2011)
DeFalco, J.A., et al.: Detecting and addressing frustration in a serious game for military training. Int. J. Artif. Intell. Educ. 28(2), 152–193 (2018). https://doi.org/10.1007/s40593-017-0152-1
Dewan, M.A.A., Murshed, M., Lin, F.: Engagement detection in online learning: a review. Smart Learn. Environ. 6(1), 1–20 (2019). https://doi.org/10.1186/s40561-018-0080-z
D’Mello, S.: A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. J. Educ. Psychol. 105(4), 1082 (2013)
D’Mello, S.K., Graesser, A.C.: Confusion. In: International Handbook of Emotions in Education, pp. 299–320. Routledge (2014)
Ferreira, M., Martinsone, B., Talić, S.: Promoting sustainable social emotional learning at school through relationship-centered learning environment, teaching methods and formative assessment. J. Teach. Educ. Sustain. 22(1), 21–36 (2020)
Fwa, H.L.: Fine-grained detection of academic emotions with spatial temporal graph attention networks using facial landmarks (2022)
Guo, Y., Klein, B., Ro, Y., Rossin, D.: The impact of flow on learning outcomes in a graduate-level information management course. J. Glob. Bus. Issues 1(2), 31–39 (2007)
Gupta, A., D’Cunha, A., Awasthi, K., Balasubramanian, V.: DAiSEE: towards user engagement recognition in the wild. arXiv preprint arXiv:1609.01885 (2016)
Joshi, R.: Accuracy, precision, recall & f1 score: interpretation of performance measures (2016). Accessed 1 Apr 2018
Kapoor, A., Mota, S., Picard, R.W., et al.: Towards a learning companion that recognizes affect. In: AAAI Fall Symposium, vol. 543, pp. 2–4 (2001)
Krithika, L., Lakshmi Priya, G.G.: Student emotion recognition system (SERS) for e-learning improvement based on learner concentration metric. Procedia Comput. Sci. 85, 767–776 (2016)
Lederman, N.G.: Teachers’ understanding of the nature of science and classroom practice: factors that facilitate or impede the relationship. J. Res. Sci. Teach. Off. J. Natl. Assoc. Res. Sci. Teach. 36(8), 916–929 (1999)
Li, J., et al.: Adaptive hierarchical graph reasoning with semantic coherence for video-and-language inference. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1867–1877 (2021)
Liao, J., Liang, Y., Pan, J.: Deep facial spatiotemporal network for engagement prediction in online learning. Appl. Intell. 51(10), 6609–6621 (2021). https://doi.org/10.1007/s10489-020-02139-8
Mukhopadhyay, M., Pal, S., Nayyar, A., Pramanik, P.K.D., Dasgupta, N., Choudhury, P.: Facial emotion detection to assess learner’s state of mind in an online learning system. In: Proceedings of the 2020 5th International Conference on Intelligent Information Technology, pp. 107–115 (2020)
Nakamura, J., Csikszentmihalyi, M.: The concept of flow. In: Nakamura, J., Csikszentmihalyi, M. (eds.) Flow and the Foundations of Positive Psychology, pp. 239–263. Springer, Dordrecht (2014). https://doi.org/10.1007/978-94-017-9088-8_16
Nguyen, L.D., Lin, D., Lin, Z., Cao, J.: Deep CNNs for microscopic image classification by exploiting transfer learning and feature concatenation. In: 2018 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5. IEEE (2018)
Oliveira, W.: Towards automatic flow experience identification in educational systems: a human-computer interaction approach. In: Extended Abstracts of the Annual Symposium on Computer-human Interaction in Play Companion Extended Abstracts, pp. 41–46 (2019)
Pan, M., Wang, J., Luo, Z.: Modelling study on learning affects for classroom teaching/learning auto-evaluation. Science 6(3), 81–86 (2018)
Peng, J., et al.: Residual convolutional neural network for predicting response of transarterial chemoembolization in hepatocellular carcinoma from CT imaging. Eur. Radiol. 30(1), 413–424 (2020). https://doi.org/10.1007/s00330-019-06318-1
Pise, A.A., Vadapalli, H., Sanders, I.: Estimation of learning affects experienced by learners: an approach using relational reasoning and adaptive mapping. Wirel. Commun. Mobile Comput. 2022 (2022)
Pooley, J.A., O’Connor, M.: Environmental education and attitudes: emotions and beliefs are what is needed. Environ. Behav. 32(5), 711–723 (2000)
Popham, W.J.: Assessment literacy for teachers: faddish or fundamental? Theory Into Pract. 48(1), 4–11 (2009)
Rahman, M., Watanobe, Y., Nakamura, K., et al.: A bidirectional LSTM language model for code evaluation and repair. Symmetry 13(2), 247 (2021)
Sailer, M., et al.: Adaptive feedback from artificial neural networks facilitates pre-service teachers’ diagnostic reasoning in simulation-based learning. Learn. Instruct. 101620 (2022)
Santoro, A., et al.: A simple neural network module for relational reasoning. In: Advances in Neural Information Processing Systems, p. 30 (2017)
Sathik, M.M., Sofia, G.: Identification of student comprehension using forehead wrinkles. In: 2011 International Conference on Computer, Communication and Electrical Technology (ICCCET), pp. 66–70. IEEE (2011)
Sathik, M., Jonathan, S.G.: Effect of facial expressions on student’s comprehension recognition in virtual educational environments. Springerplus 2(1), 1–9 (2013). https://doi.org/10.1186/2193-1801-2-455
Shernoff, D.J., Csikszentmihalyi, M., Schneider, B., Shernoff, E.S.: Student engagement in high school classrooms from the perspective of flow theory. In: Shernoff, D.J., Csikszentmihalyi, M., Schneider, B., Shernoff, E.S. (eds.) Applications of Flow in Human Development and Education, pp. 475–494. Springer, Dordrecht (2014). https://doi.org/10.1007/978-94-017-9094-9_24
Singh, S., Mujinga, M., Lotriet, H., Tait, B.: SAICSIT conference 2021: proceedings of the South African institute of computer scientists and information technologists (2021)
Stein, N.L., Levine, L.J.: Making sense out of emotion: the representation and use of goal-structured knowledge. In: Psychological and Biological Approaches to Emotion, pp. 63–92. Psychology Press (2013)
Zakka, B.E., Vadapalli, H.: Estimating student learning affect using facial emotions. In: 2020 2nd International Multidisciplinary Information Technology and Engineering Conference (IMITEC), pp. 1–6. IEEE (2020)
Zhang, H., et al.: Generative visual dialogue system via adaptive reasoning and weighted likelihood estimation. arXiv preprint arXiv:1902.09818 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Asaju, C., Vadapalli, H. (2022). Adaptive Reasoning: An Affect Related Feedback Approach for Enhanced E-Learning. In: Pillay, A., Jembere, E., Gerber, A. (eds) Artificial Intelligence Research. SACAIR 2022. Communications in Computer and Information Science, vol 1734. Springer, Cham. https://doi.org/10.1007/978-3-031-22321-1_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-22321-1_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22320-4
Online ISBN: 978-3-031-22321-1
eBook Packages: Computer ScienceComputer Science (R0)