Abstract
Considering that emotions have a great impact on motivation, reasoning, and decision making, affective computing methods, that were designed to attempt to understand and respond to human emotional states, have been used in more than one field including e-learning. Thus, a systematic literature review was conducted on 4 search engines resulting in a set of papers that were filtered in a systematic way until we obtained a corpus of 27 papers. Data were extracted to answer four research questions concerning the use and efficacy of affective computing in e-learning in recent years. We found out that the majority of studies about emotion recognition use uni-modal systems in which facial expressions emotion detection is the most present. The major research purpose is designing/building systems, approaches, methods, detectors for emotion recognition. For the e-learning environments, the most present is conversational agents. The emotions detected or used are basic emotions, non-basic emotions, learning-centered emotions, trait emotions, or a combination of two or three of them. This systematic literature review also provides the major findings, challenges, and future research.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
Not applicable.
Code availability
Not applicable.
References
Akputu, O. K., Seng, K. P., Lee, Y., & Ang, L. M. (2018). Emotion recognition using multiple kernel learning toward E-learning applications. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 14(1), 1. https://doi.org/10.1145/3131287
Al Osman, H., Dong, H., & El Saddik, A. (2016). Ubiquitous biofeedback serious game for stress management. IEEE Access, 4, 1274–1286. https://doi.org/10.1109/ACCESS.2016.2548980
Alexandra, J. M., Andres, L., Ocumpaugh, J., Baker, R. S., Slater, S., Paquette, L., ... & Moore, A. (2019, March). affect sequences and learning in Betty’s brain. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, Tempe, AZ, USA (pp. 4–8).
Bachtiar, F. A., Sulistyo, G. H., Cooper, E. W., & Katsuari, K. (2017, January). Affect, personality, and learning styles in online reading comprehension. In Proceedings of the 5th International Conference on Information and Education Technology (pp. 78–83). ACM. https://doi.org/10.1145/3029387.3029422
Bahreini, K., Nadolski, R., & Westera, W. (2016). Towards real-time speech emotion recognition for affective e-learning. Education and Information Technologies, 21(5), 1367–1386. https://doi.org/10.1007/s10639-015-9388-2
Bahreini, K., van der Vegt, W., & Westera, W. (2019). A fuzzy logic approach to reliable real-time recognition of facial emotions. Multimedia Tools and Applications, 78(14), 18943–18966. https://doi.org/10.1007/s11042-019-7250-z
Baker, R., & Ocumpaugh, J. (2015). Interaction-based affect detection in educational software (pp. 233–245). Oxford University Press.
Baldassarri, S., Hupont, I., Abadía, D., & Cerezo, E. (2015). Affective-aware tutoring platform for interactive digital television. Multimedia Tools and Applications, 74(9), 3183–3206. https://doi.org/10.1007/s11042-013-1779-z
Barbalet, J. M. (2001). Emotion, social theory, and social structure: A macrosociological approach. Cambridge University Press.
Bontchev, B., & Vassileva, D. (2016, November). Assessing engagement in an emotionally-adaptive applied game. In Proceedings of the Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality (pp. 747–754). ACM. https://doi.org/10.1145/3012430.3012602
Bosch, N., D'mello, S. K., Ocumpaugh, J., Baker, R. S., & Shute, V. (2016). Using video to automatically detect learner affect in computer-enabled classrooms. ACM Transactions on Interactive Intelligent Systems (TiiS), 6(2), 17. https://doi.org/10.1145/2946837
Bower, G. H., & Cohen, P. R. (1982). Emotional influences in memory and thinking: Data and theory. Affect and cognition, 1.
Brigham, T. J. (2017). Merging technology and emotions: Introduction to affective computing. Medical Reference Services Quarterly, 36(4), 399–407.
Buck, R. (1994). Social and emotional functions in facial expression and communication: The readout hypothesis. Biological Psychology, 38(2–3), 95–115. https://doi.org/10.1016/0301-0511(94)90032-9
Calvo, R. A., & D’Mello, S. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18–37. https://doi.org/10.1109/T-AFFC.2010.1
Chen, J., Luo, N., Liu, Y., Liu, L., Zhang, K., & Kolodziej, J. (2016). A hybrid intelligence-aided aproach to affect-sensitive e-learning. Computing, 98(1–2), 215–233. https://doi.org/10.1007/s00607-014-0430-9
Clore, G. L., & Palmer, J. (2009). Affective guidance of intelligent agents: How emotion controls cognition. Cognitive Systems Research, 10(1), 21–30. https://doi.org/10.1016/j.cogsys.2008.03.002
Dirkx, J. M. (2001). The power of feelings: Emotion, imagination, and the construction of meaning in adult learning. New Directions for Adult and Continuing Education, 2001(89), 63–72.
D'mello, S., & Graesser, A. (2012). AutoTutor and affective AutoTutor: Learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(4), 23. https://doi.org/10.1145/2395123.2395128
Eid, M., Schneider, C., & Schwenkmezger, P. (1999). Do you feel better or worse? The validity of perceived deviations of mood states from mood traits. European Journal of Personality, 13(4), 283–306. https://doi.org/10.1002/(SICI)1099-0984(199907/08)13:4%3c283::AID-PER341%3e3.0.CO;2-0
Ekman, P., Friesen, W. V., O'sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., ... & Scherer, K. (1987). Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology, 53(4), 712. https://doi.org/10.1037/0022-3514.53.4.712
Fehr, B., & Russell, J. A. (1984). Concept of emotion viewed from a prototype perspective. Journal of Experimental Psychology: General, 113(3), 464. https://doi.org/10.1037/0096-3445.113.3.464
Forsyth, C. M., Graesser, A., Olney, A. M., Millis, K., Walker, B., & Cai, Z. (2015, June). Moody agents: affect and discourse during learning in a serious game. In International Conference on Artificial Intelligence in Education (pp. 135–144). Springer, Cham. https://doi.org/10.1007/978-3-319-19773-9_14
Frijda, N. H., Manstead, A. S., & Bem, S. (2000). The influence of emotions on beliefs. Emotions and beliefs: How feelings influence thoughts, 1–9.
Fwa, H. L. (2018). An architectural design and evaluation of an affective tutoring system for novice programmers. International Journal of Educational Technology in Higher Education, 15(1), 38. https://doi.org/10.1186/s41239-018-0121-2
Graesser, A. C., & D’Mello, S. (2012). Moment-to-moment emotions during reading. The Reading Teacher, 66(3), 238–242. https://doi.org/10.1002/TRTR.01121
Grafsgaard, J. F., Wiggins, J. B., Boyer, K. E., Wiebe, E. N., & Lester, J. C. (2013, September). Automatically recognizing facial indicators of frustration: a learning-centric analysis. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (pp. 159–165). IEEE. https://doi.org/10.1109/ACII.2013.33
Grawemeyer, B., Mavrikis, M., Holmes, W., Gutierrez-Santos, S., Wiedmann, M., & Rummel, N. (2016, April). Affecting off-task behaviour: how affect-aware feedback can improve student learning. In Proceedings of the sixth international conference on learning analytics & knowledge (pp. 104–113). ACM. https://doi.org/10.1145/2883851.2883936
Harley, J. M., Carter, C. C., Papaionnou, N., Bouchet, F., Landis, R. S., Azevedo, R., & Karabachian, L. (2015, June). Examining the predictive relationship between personality and emotion traits and learners’ agent-direct emotions. In International Conference on Artificial Intelligence in Education (pp. 145–154). Springer, Cham. https://doi.org/10.1007/978-3-319-19773-9_15
Harley, J. M., Carter, C. K., Papaionnou, N., Bouchet, F., Landis, R. S., Azevedo, R., & Karabachian, L. (2016). Examining the predictive relationship between personality and emotion traits and students’ agent-directed emotions: Towards emotionally-adaptive agent-based learning environments. User Modeling and User-Adapted Interaction, 26(2), 177–219. https://doi.org/10.1007/s11257-016-9169-7
Hung, J. C., Lin, K. C., & Lai, N. X. (2019). Recognizing learning emotion based on convolutional neural networks and transfer learning. Applied Soft Computing, 84, 105724. https://doi.org/10.1016/j.asoc.2019.105724
Isen, A. M. (2004). Some perspectives on positive feelings and emotions: Positive affect facilitates thinking and problem solving. In Feelings and Emotions: The Amsterdam Symposium, Jun, 2001, Amsterdam, Netherlands. Cambridge University Press.
Kamour, M. (2012). Importance of emotional intelligence among open learning and distance learning students. In EDULEARN12 Proceedings (pp. 2895–2899). IATED.
Kitchenham, B. (2004). Procedures for performing systematic reviews. Keele, UK, Keele University, 33(2004), 1–26.
Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering.
Kleinginna, P. R., & Kleinginna, A. M. (1981). A categorized list of emotion definitions, with suggestions for a consensual definition. Motivation and Emotion, 5(4), 345–379. https://doi.org/10.1007/BF00992553
Kołakowska, A. (2013, June). A review of emotion recognition methods based on keystroke dynamics and mouse movements. In 2013 6th international conference on human system interactions (HSI) (pp. 548–555). IEEE.
Krithika, L. B., & GG, L. P. (2016). Student emotion recognition system (SERS) for e-learning improvement based on learner concentration metric. Procedia Computer Science, 85, 767-776.
Levy, Y. (2007). Comparing dropouts and persistence in e-learning courses. Computers & Education, 48(2), 185–204.
Lin, H. C. K., Wu, C. H., & Hsueh, Y. P. (2014). The influence of using affective tutoring system in accounting remedial instruction on learning performance and usability. Computers in Human Behavior, 41, 514–522. https://doi.org/10.1016/j.chb.2014.09.052
Liu, Y., Fu, Q., & Fu, X. (2009). The interaction between cognition and emotion. Chinese Science Bulletin, 54(22), 4102. https://doi.org/10.1007/s11434-009-0632-2
Matejka, M. M., Kazzer, P., Seehausen, M., Bajbouj, M., Klann-Delius, G., Margrit, G., ... & Prehn, K. (2013). Talking about emotion: prosody and skin conductance indicate emotion regulation. Frontiers in Psychology, 4, 260.
Moore, J. L., Dickson-Deane, C., & Galyen, K. (2011). e-Learning, online learning, and distance learning environments: Are they the same? The Internet and Higher Education, 14(2), 129–135. https://doi.org/10.1016/j.iheduc.2010.10.001
Pardos, Z. A., Baker, R. S., San Pedro, M. O., Gowda, S. M., & Gowda, S. M. (2013, April). Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In Proceedings of the third international conference on learning analytics and knowledge (pp. 117–124). ACM. https://doi.org/10.1145/2460296.2460320
Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist, 37(2), 91–105.
Phelps, E. A. (2006). Emotion and cognition: Insights from studies of the human amygdala. Annual Review of Psychology, 57, 27–53. https://doi.org/10.1146/annurev.psych.56.091103.070234
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98–125. https://doi.org/10.1016/j.inffus.2017.02.003
Qualter, P., Gardner, K. J., & Whiteley, H. E. (2007). Emotional intelligence: Review of research and educational implications. Pastoral Care in Education, 25(1), 11–20.
Rimé, B. (2009). Emotion elicits the social sharing of emotion: Theory and empirical review. Emotion Review, 1(1), 60–85.
Salmeron-Majadas, S., Arevalillo-Herráez, M., Santos, O. C., Saneiro, M., Cabestrero, R., Quirós, P., ... & Boticario, J. G. (2015, June). Filtering of spontaneous and low intensity emotions in educational contexts. In International Conference on Artificial Intelligence in Education (pp. 429–438). Springer, Cham. https://doi.org/10.1007/978-3-319-19773-9_43
Salovey, P., & Grewal, D. (2005). The science of emotional intelligence. Current Directions in Psychological Science, 14(6), 281–285.
Salovey, P., & Mayer, J. D. (1990). Emotional intelligence. Imagination, Cognition and Personality, 9(3), 185–211.
Santos, O. C., Rodriguez-Ascaso, A., Boticario, J. G., Salmeron-Majadas, S., Quirós, P., & Cabestrero, R. (2013, July). Challenges for inclusive affective detection in educational scenarios. In International Conference on Universal Access in Human-Computer Interaction (pp. 566–575). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39188-0_61
Schuller, B. W. (2018). Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends. Communications of the ACM, 61(5), 90–99.
Souza, N., & Perry, G. (2018). Identification of affective states in MOOCs: A systematic literature review. Int. J. Innov. Educ. Res., 6(12), 39–55. https://doi.org/10.31686/ijier.vol6.iss12.1250
Spann, C. A., Shute, V. J., Rahimi, S., & D’Mello, S. K. (2019). The productive role of cognitive reappraisal in regulating affect during game-based learning. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2019.03.002
Spaulding, S., Gordon, G., & Breazeal, C. (2016, May). Affect-aware student models for robot tutors. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems (pp. 864–872). International Foundation for Autonomous Agents and Multiagent Systems.
Tyng, C. M., Amin, H. U., Saad, M. N., & Malik, A. S. (2017). The influences of emotion on learning and memory. Frontiers in Psychology, 8, 1454. https://doi.org/10.3389/fpsyg.2017.01454
Vogel, S., & Schwabe, L. (2016). Learning and memory under stress: implications for the classroom. npj Science of Learning, 1(1), 1–10. https://doi.org/10.1038/npjscilearn.2016.11
Weidt, F., & Silva, R. (2016). Systematic Literature Review in Computer Science-A Practical Guide. Relatórios Técnicos do DCC/UFJF, 1.
Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., & Picard, R. (2009). Affect-aware tutors: Recognising and responding to student affect. International Journal of Learning Technology, 4(3–4), 129–164. https://doi.org/10.1504/IJLT.2009.028804
Wu, C. H., Huang, Y. M., & Hwang, J. P. (2016). Review of affective computing in education/learning: Trends and challenges. British Journal of Educational Technology, 47(6), 1304–1323. https://doi.org/10.1111/bjet.12324
Xing, B., Zhang, L., Gao, J., Yu, R., & Lyu, R. (2016, November). Barrier-free affective communication in MOOC study by analyzing pupil diameter variation. In SIGGRAPH ASIA 2016 Symposium on Education (pp. 1–8). https://doi.org/10.1145/2993352.2993362
Yadegaridehkordi, E., Noor, N. F. B. M., Ayub, M. N. B., Affal, H. B., & Hussin, N. B. (2019). Affective computing in education: A systematic review and future research. Computers & Education, 142, 103649. https://doi.org/10.1016/j.compedu.2019.103649
Zatarain-Cabada, R., Barrón-Estrada, M. L., & Ríos-Félix, J. M. (2016, October). Affective learning system for algorithmic logic applying gamification. In Mexican International Conference on Artificial Intelligence (pp. 536–547). Springer, Cham. https://doi.org/10.1007/978-3-319-62428-0_44
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interests
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mejbri, N., Essalmi, F., Jemni, M. et al. Trends in the use of affective computing in e-learning environments. Educ Inf Technol 27, 3867–3889 (2022). https://doi.org/10.1007/s10639-021-10769-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-021-10769-9