Skip to main content
Springer Nature Link
Log in

Adaptive Reasoning: An Affect Related Feedback Approach for Enhanced E-Learning

  • Conference paper
  • First Online:
Artificial Intelligence Research (SACAIR 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1734))

Included in the following conference series:

  • 464 Accesses

Abstract

Recognition of affective states to enhance e-learning platforms has been a topic of machine learning research. Compared to other input modalities, facial expressions have the potential to reveal nonverbal cues about a learner’s learning affect. However, most studies were limited in their analysis of learning affects exhibited by a learner with the possibility of providing appropriate feedback to teachers and learners. This work proposes an adaptive reasoning mechanism that considers the estimated affective states and learning affect in generating feedback with reasoning incorporated. This work utilizes a Convolutional Neural Network- Bidirectional Long-Short Term Memory (CNN-BiLSTM) cascade framework for affective states analysis through processing a live/stored observation of a learner in the form of a temporal signal. Using the proposed ensemble, four affective states were estimated, namely boredom, confusion, frustration, and engagement. Dataset for Affective States in E-Environment (DAiSEE) was used to train, validate, and test the baseline model, which reported an accuracy of 86% on 4305 test samples. In the next stage, mappings between estimated affective states and learning affects (i.e. positive, negative and neutral) were established based on an adaptive mapping mechanism, to consolidate the mapping between affective states and learning affects. Live testing and survey feedback were then used to further validate, adapt and amend the feedback process. Incorporating and interpreting the estimated affective states and learning affect is imperative in providing information to both teachers and learners, and hence potentially improve the existing e-learning platforms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abedi, A., Khan, S.S.: Improving state-of-the-art in detecting student engagement with ResNet and TCN hybrid network. In: 2021 18th Conference on Robots and Vision (CRV), pp. 151–157. IEEE (2021)

    Google Scholar 

  2. Asaju, C., Vadapalli, H.: A temporal approach to facial emotion expression recognition. In: Jembere, E., Gerber, A.J., Viriri, S., Pillay, A. (eds.) SACAIR 2021. CCIS, vol. 1551, pp. 274–286. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-95070-5_18

    Chapter  Google Scholar 

  3. Asaju, C.B., Vadapalli, H.: Affects analysis: a temporal approach to estimate students’ learning. In: 2021 3rd International Multidisciplinary Information Technology and Engineering Conference (IMITEC), pp. 1–7. IEEE (2021)

    Google Scholar 

  4. Ayvaz, U., Gürüler, H., Devrim, M.O.: Use of facial emotion recognition in e-learning systems (2017)

    Google Scholar 

  5. Butler, D.L., Winne, P.H.: Feedback and self-regulated learning: a theoretical synthesis. Rev. Educ. Res. 65(3), 245–281 (1995)

    Article  Google Scholar 

  6. Calvo, R.A., D’Mello, S.: Frontiers of affect-aware learning technologies. IEEE Intell. Syst. 27(6), 86–89 (2012)

    Article  Google Scholar 

  7. Chen, S., Dai, J., Yan, Y.: Classroom teaching feedback system based on emotion detection. In: 9th International Conference on Education and Social Science (ICESS 2019), pp. 940–946 (2019)

    Google Scholar 

  8. Csikszentmihalyi, M.: The contribution of flow to positive psychology (2000)

    Google Scholar 

  9. Csikszentmihalyi, M., Larson, R.: Flow and the Foundations of Positive Psychology, vol. 10. Springer, Heidelberg (2014). https://doi.org/10.1007/978-94-017-9088-8

    Book  Google Scholar 

  10. Daschmann, E.C., Goetz, T., Stupnisky, R.H.: Testing the predictors of boredom at school: development and validation of the precursors to boredom scales. Br. J. Educ. Psychol. 81(3), 421–440 (2011)

    Article  Google Scholar 

  11. DeFalco, J.A., et al.: Detecting and addressing frustration in a serious game for military training. Int. J. Artif. Intell. Educ. 28(2), 152–193 (2018). https://doi.org/10.1007/s40593-017-0152-1

    Article  Google Scholar 

  12. Dewan, M.A.A., Murshed, M., Lin, F.: Engagement detection in online learning: a review. Smart Learn. Environ. 6(1), 1–20 (2019). https://doi.org/10.1186/s40561-018-0080-z

    Article  Google Scholar 

  13. D’Mello, S.: A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. J. Educ. Psychol. 105(4), 1082 (2013)

    Article  Google Scholar 

  14. D’Mello, S.K., Graesser, A.C.: Confusion. In: International Handbook of Emotions in Education, pp. 299–320. Routledge (2014)

    Google Scholar 

  15. Ferreira, M., Martinsone, B., Talić, S.: Promoting sustainable social emotional learning at school through relationship-centered learning environment, teaching methods and formative assessment. J. Teach. Educ. Sustain. 22(1), 21–36 (2020)

    Article  Google Scholar 

  16. Fwa, H.L.: Fine-grained detection of academic emotions with spatial temporal graph attention networks using facial landmarks (2022)

    Google Scholar 

  17. Guo, Y., Klein, B., Ro, Y., Rossin, D.: The impact of flow on learning outcomes in a graduate-level information management course. J. Glob. Bus. Issues 1(2), 31–39 (2007)

    Google Scholar 

  18. Gupta, A., D’Cunha, A., Awasthi, K., Balasubramanian, V.: DAiSEE: towards user engagement recognition in the wild. arXiv preprint arXiv:1609.01885 (2016)

  19. Joshi, R.: Accuracy, precision, recall & f1 score: interpretation of performance measures (2016). Accessed 1 Apr 2018

    Google Scholar 

  20. Kapoor, A., Mota, S., Picard, R.W., et al.: Towards a learning companion that recognizes affect. In: AAAI Fall Symposium, vol. 543, pp. 2–4 (2001)

    Google Scholar 

  21. Krithika, L., Lakshmi Priya, G.G.: Student emotion recognition system (SERS) for e-learning improvement based on learner concentration metric. Procedia Comput. Sci. 85, 767–776 (2016)

    Article  Google Scholar 

  22. Lederman, N.G.: Teachers’ understanding of the nature of science and classroom practice: factors that facilitate or impede the relationship. J. Res. Sci. Teach. Off. J. Natl. Assoc. Res. Sci. Teach. 36(8), 916–929 (1999)

    Google Scholar 

  23. Li, J., et al.: Adaptive hierarchical graph reasoning with semantic coherence for video-and-language inference. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1867–1877 (2021)

    Google Scholar 

  24. Liao, J., Liang, Y., Pan, J.: Deep facial spatiotemporal network for engagement prediction in online learning. Appl. Intell. 51(10), 6609–6621 (2021). https://doi.org/10.1007/s10489-020-02139-8

    Article  Google Scholar 

  25. Mukhopadhyay, M., Pal, S., Nayyar, A., Pramanik, P.K.D., Dasgupta, N., Choudhury, P.: Facial emotion detection to assess learner’s state of mind in an online learning system. In: Proceedings of the 2020 5th International Conference on Intelligent Information Technology, pp. 107–115 (2020)

    Google Scholar 

  26. Nakamura, J., Csikszentmihalyi, M.: The concept of flow. In: Nakamura, J., Csikszentmihalyi, M. (eds.) Flow and the Foundations of Positive Psychology, pp. 239–263. Springer, Dordrecht (2014). https://doi.org/10.1007/978-94-017-9088-8_16

    Chapter  Google Scholar 

  27. Nguyen, L.D., Lin, D., Lin, Z., Cao, J.: Deep CNNs for microscopic image classification by exploiting transfer learning and feature concatenation. In: 2018 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5. IEEE (2018)

    Google Scholar 

  28. Oliveira, W.: Towards automatic flow experience identification in educational systems: a human-computer interaction approach. In: Extended Abstracts of the Annual Symposium on Computer-human Interaction in Play Companion Extended Abstracts, pp. 41–46 (2019)

    Google Scholar 

  29. Pan, M., Wang, J., Luo, Z.: Modelling study on learning affects for classroom teaching/learning auto-evaluation. Science 6(3), 81–86 (2018)

    Google Scholar 

  30. Peng, J., et al.: Residual convolutional neural network for predicting response of transarterial chemoembolization in hepatocellular carcinoma from CT imaging. Eur. Radiol. 30(1), 413–424 (2020). https://doi.org/10.1007/s00330-019-06318-1

    Article  Google Scholar 

  31. Pise, A.A., Vadapalli, H., Sanders, I.: Estimation of learning affects experienced by learners: an approach using relational reasoning and adaptive mapping. Wirel. Commun. Mobile Comput. 2022 (2022)

    Google Scholar 

  32. Pooley, J.A., O’Connor, M.: Environmental education and attitudes: emotions and beliefs are what is needed. Environ. Behav. 32(5), 711–723 (2000)

    Article  Google Scholar 

  33. Popham, W.J.: Assessment literacy for teachers: faddish or fundamental? Theory Into Pract. 48(1), 4–11 (2009)

    Article  Google Scholar 

  34. Rahman, M., Watanobe, Y., Nakamura, K., et al.: A bidirectional LSTM language model for code evaluation and repair. Symmetry 13(2), 247 (2021)

    Article  Google Scholar 

  35. Sailer, M., et al.: Adaptive feedback from artificial neural networks facilitates pre-service teachers’ diagnostic reasoning in simulation-based learning. Learn. Instruct. 101620 (2022)

    Google Scholar 

  36. Santoro, A., et al.: A simple neural network module for relational reasoning. In: Advances in Neural Information Processing Systems, p. 30 (2017)

    Google Scholar 

  37. Sathik, M.M., Sofia, G.: Identification of student comprehension using forehead wrinkles. In: 2011 International Conference on Computer, Communication and Electrical Technology (ICCCET), pp. 66–70. IEEE (2011)

    Google Scholar 

  38. Sathik, M., Jonathan, S.G.: Effect of facial expressions on student’s comprehension recognition in virtual educational environments. Springerplus 2(1), 1–9 (2013). https://doi.org/10.1186/2193-1801-2-455

    Article  Google Scholar 

  39. Shernoff, D.J., Csikszentmihalyi, M., Schneider, B., Shernoff, E.S.: Student engagement in high school classrooms from the perspective of flow theory. In: Shernoff, D.J., Csikszentmihalyi, M., Schneider, B., Shernoff, E.S. (eds.) Applications of Flow in Human Development and Education, pp. 475–494. Springer, Dordrecht (2014). https://doi.org/10.1007/978-94-017-9094-9_24

    Chapter  Google Scholar 

  40. Singh, S., Mujinga, M., Lotriet, H., Tait, B.: SAICSIT conference 2021: proceedings of the South African institute of computer scientists and information technologists (2021)

    Google Scholar 

  41. Stein, N.L., Levine, L.J.: Making sense out of emotion: the representation and use of goal-structured knowledge. In: Psychological and Biological Approaches to Emotion, pp. 63–92. Psychology Press (2013)

    Google Scholar 

  42. Zakka, B.E., Vadapalli, H.: Estimating student learning affect using facial emotions. In: 2020 2nd International Multidisciplinary Information Technology and Engineering Conference (IMITEC), pp. 1–6. IEEE (2020)

    Google Scholar 

  43. Zhang, H., et al.: Generative visual dialogue system via adaptive reasoning and weighted likelihood estimation. arXiv preprint arXiv:1902.09818 (2019)

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa

    Christine Asaju & Hima Vadapalli

  2. Academy of Computer Science and Software Engineering, University of Johannesburg, Johannesburg, South Africa

    Hima Vadapalli

Authors
  1. Christine Asaju
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hima Vadapalli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christine Asaju .

Editor information

Editors and Affiliations

  1. University of KwaZulu-Natal, Durban, South Africa

    Anban Pillay

  2. University of KwaZulu-Natal, Durban, South Africa

    Edgar Jembere

  3. University of Pretoria, Pretoria, South Africa

    Aurona Gerber

Appendix

Appendix

See Fig. 3.

Fig. 3.
figure 3figure 3

Consent form and questionnaire

Full size image

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Asaju, C., Vadapalli, H. (2022). Adaptive Reasoning: An Affect Related Feedback Approach for Enhanced E-Learning. In: Pillay, A., Jembere, E., Gerber, A. (eds) Artificial Intelligence Research. SACAIR 2022. Communications in Computer and Information Science, vol 1734. Springer, Cham. https://doi.org/10.1007/978-3-031-22321-1_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-22321-1_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-22320-4

  • Online ISBN: 978-3-031-22321-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation