Modeling adaptive E-Learning environment using facial expressions and fuzzy logic

https://doi.org/10.1016/j.eswa.2020.113460Get rights and content

Highlights

  • The approach interprets the facial expressions to provide adaptive e-learning system.

  • The approach take the variances of learners’ emotional states into consideration.

  • The approach proposed a methodology to summarize the detected emotional states.

  • The approach uses fuzzy logic to create fuzzy learning flow adapted with each learner.

  • We introduce two real corpora to evaluate the performance of the proposed approach.

Abstract

The integration of two or more intelligent systems, such as deep neural networks and fuzzy techniques, results in so-called hybrid intelligent systems. These have recently attracted considerable attention owing to their broad success in several complex real-world applications. An interesting example is the development of adaptive computer-based learning environments, in which learner responses to certain questions are considered so that the next level of the learning process may be determined. However, these methods fail to capture the behavior and emotional expressions of the learner during the learning process. Capturing these could make the learning flow more adaptive, and could allow the learning environment to redirect learners to different learning paths based on their capabilities and interaction levels. In this regard, the contribution of the present study is threefold. First, it proposes an approach for modeling an intelligent adaptive e-learning environment by considering the integration of the learner responses to questions and their emotional states. In the proposed approach, a loosely coupled integration between a convolutional neural network (CNN) and a fuzzy system is adopted. The CNN is used to detect a learner’s facial expressions, and outperforms other CNN models on the same training benchmark. The fuzzy system is used to determine the next learning level based on the extracted facial expression states from the CNN and several response factors by the learner. Second, the study introduces methods whereby a group of facial expressions is aggregated into a single representative. Third, it introduces corpora for evaluating the performance of the proposed approach. The corpora of 12 learners contain 72 learning activities and 1735 data points of distinct emotional states. The experimental results using these corpora demonstrate that the proposed approach provides adaptive learning flows that match the learning capabilities of all learners in a group. Moreover, the approach allows decision makers to monitor the learning performance for each learner.

Introduction

Intelligent techniques are attracting increasing attention and producing encouraging results compared with conventional techniques. Although these techniques are useful in certain tasks, several complex real-world problems cannot be tackled by a single intelligent technique. This is because each intelligent technique has certain computational characteristics that are appropriate in specific problems. These limitations prompted the development of hybrid intelligent systems that integrate two or more systems, such as deep neural networks (Aggarwal, 2018) and fuzzy techniques (Alcantud, et al., de Silva, 2013), into a single framework. These systems are obtained either by loosely coupled integration or by fusion of individual techniques. Recently, hybrid intelligent systems have attracted considerable attention owing to their broad success in several complex real-world applications. An interesting example is the modeling of adaptive computer-based learning environments (Moos & Azevedo, 2009), which are technology-based environments, such as intelligent tutoring systems (Corbett, Koedinger, & Anderson, 1997) and web-based learning systems (Kularbphettong, Kedsiribut, & Roonrakwit, 2015). These can be used as an instructional aid for educational purposes. The modeling of such environments has received wide attention from multiple disciplines, including computer science, psychology, and education (Lajoie & Naismith, 2012). Usually, the learning flow in these environments depends on the learner’s mental responses based on solving tests and answering exam questions, so that the next level of the learning process may be determined. However, conventional methods do not consider the emotional behavior of the learner during the learning process. According to Goetz, Lüdtke, Nett, Keller, and Lipnevich (2013); Kärner and Kögler (2016), the emotional behavior is considered an important factor affecting the quality of the learning process and the achievement of the desired learning outcomes. The learner’s interaction with the learning environment can be classified into two categories. The first consists of mental responses to test questions, and the second consists of emotional responses through facial expressions. Therefore, integrating both types is critical for developing more adaptive and intelligent computer-based learning environments. The interaction with the learning environment varies among learners owing to several factors related to the learner’s responses or the learning environment itself. Likewise, the educator’s judgment about the expression of the learners’ emotions and the effectiveness of their interactions varies owing to the educator’s perception and knowledge. Thus, the evaluation of the learners’ interactions from various perspectives is a significant challenge. A large amount of research has been conducted to model the interaction of learners in computer-based learning environments from different perspectives. However, these studies avoided complex interactions and ignored critical interaction parts and features; rather, the focus was on simple modeling. For instance, several studies have proposed production rules that consider the learner’s answers to tests and exam questions to be the only means of communication with the learning environment, and hence they determine the next level of the learning flow Gross, Mokbel, Hammer, and Pinkwart (2015); Kularbphettong et al. (2015); Mosa, Albatish, and Abu-Naser (2018). However, owing to the importance of considering the learner’s emotions during the learning process, other research efforts focused on modeling these emotions by interpreting facial expressions using machine learning algorithms (Ayvaz, Gürüler, Devrim, 2017, Chickerur, Joshi, 2015, Khalfallah, Slama, 2015, Krithika, 2016, Petrovica, Anohina-Naumeca, Ekenel, 2017, Yang, Alsadoon, Prasad, Singh, Elchouemi, 2018). Some studies proposed modeling the variance in the communication of the learner with the learning environment. These studies can be categorized into two types. The first is concerned with the variance in solving tests and exams (Meenakshi, Pankaj, 2015, Yadav, Singh, 2011), whereas the second is concerned with the variance of learners’ emotions during the learning process (Zatarain-Cabada, Barrón-Estrada, García-Lizárraga, Muñoz Sandoval, & Ríos-Félix, 2015). In both categories, it is attempted to model more complex interaction behaviors by introducing fuzzy logic to handle differences in the communication with the learning environment. Although these studies proposed various fuzzy models for handling the uncertainty in the emotional responses, they attempted to abstract the problem into a simple form and did not consider the probability degree of different facial expression classes. Traditionally, each instance of a facial expression, such as happiness, sadness, or surprise, should be represented using probability degrees over multiple classes. Additionally, these studies did not consider the variation of the learner’s emotional states during the test sessions because their simplified methodologies depend on analyzing only one picture of the learner’s face at the end of the test session and ignore other important facial expressions during the test.

The main contribution of this study is threefold. First, to model complex adaptive learning environments, it proposes a hybrid approach that integrates in the same framework a convolutional neural network (CNN) and a fuzzy system. In this approach, we consider learner mental responses to the test questions as well as their emotional responses. Moreover, we consider the variance of the learners’ emotions in test and exam sessions. The CNN in the hybrid approach is responsible for modeling, analyzing, and classifying learner facial expressions that reflect their emotional states based on online video streamed during the test and exams sessions (Kim, Roh, Dong, Lee, 2016, Li, Zhang, Zhang, Zhang, Li, Xia, Yan, Xun, 2017, Lopes, de Aguiar, Oliveira-Santos, 2015, Shan, Guo, You, Lu, Bie, 2017, Shin, Kim, Kwon, 2016). The proposed CNN outperforms other CNN models on the same training benchmark of facial expressions. The fuzzy part in the proposed hybrid approach is responsible for handling the uncertainty in the learner’s interactions with the learning environment. The fuzzy system considers several uncertain features compared to other fuzzy systems applied in the same domain, including the diversity of learner emotions. In addition to the inputs received from the CNN, the fuzzy inference system takes features from the learning environment, including the answer validity ratio, elapsed test time, and current learning level. The ultimate goal of the fuzzy part is to provide the next level of the learning process based on the input features.

As the CNN provides a large number of emotion states that span several classes of facial expressions, the second contribution is two aggregated methods that summarize the detected set of emotional states and provide a single representative aggregated facial expression class at end of the test and exam sessions. This aggregated facial expression is then fed to the fuzzy part together with the features of the environment. To the best of our knowledge, no study has provided an aggregated representative facial expression for all emotional states and fed it to a fuzzy system.

The third contribution is the introduction of two real corpora of learner emotional states and learning flows. The first corpus is used to specify the different emotional states and variances during the test sessions. The second corpus is introduced to specify the learning flows across different learning activities of the group in this study. The proposed hybrid approach provides decision makers with info-graphs that explain the learning flows and emotional states of all learners who participate in the learning activities.

The rest of this paper is organized as follows: Section 2 provides a brief background on emotions, CNNs, and fuzzy logic. Section 3 surveys related work. Section 4 introduces the details of the proposed approach. Section 5 proposes two corpora that are used to evaluate the proposed approach. Section 6 discusses the experiment and the results. Finally, Section 7 concludes the paper.

Section snippets

Background

Emotion could be defined according to the Oxford dictionaries as a strong feeling deriving from one’s circumstances, mood, or relationships with others. Furthermore, it could be defined as a feeling comprising physiological and behavioral reactions to internal and external events (Sternberg, 1995). Emotion includes both physical and cognitive actions that interpret biological responses to an external stimulus. That biological response affects our handling of situations and our interaction with

Related work

This section overviews studies related to the present. These efforts can be regarded from the perspective of psychology, fuzzy logic, and automatic facial expression interpretation.

From a psychological perspective, and regarding the effect of the emotional states of the learner, the authors in Goetz et al. (2013) explored the relations between teaching features and learners’ academic emotions. They found clear relations between these two factors at the intrahuman level. Moreover, they mentioned

Proposed approach

Before presenting the proposed approach, we formalize some basic concepts. In particular, we define the concepts of emotional state, emotional behavior, and aggregated emotional state.

Definition 1 Emotional State

Let E={e1,e2,,en} be a set of n different classes of facial expressions. We define the emotional state σ=(∊1, ∊2, ∊3, ...,∊n) as a probability distribution over E such that i=P(E=ei), 0 ≤ ∊i ≤ 1 and ii=1 for 1 ≤ i ≤ n. The emotional state σ at particular instant t is denoted as σt=(1t,2t,3t,,nt) for t

Proposed corpora

This section describes the corpora that will be used to test the proposed approach as well as generate the output results. The data of the corpora were collected using an implementation of the proposed learning environment. Therefore, we begin this section by explaining this implementation and elaborate on the processing flow between the learner and the components of the learning environment.

Experimental results

In this section, we use the generated corpora to experiment with the proposed approach. We discuss the results from different perspectives. First, we analyze the effect of learner’s emotions on the learning environment. Second, we study the effect of using different aggregation functions (mean and median) on the learning flow.

Table 9 describes an example of learning activities of the five individuals 1, 2, 3, 11, and 12 based on summarizing their emotional states using the mean. Table 10

Conclusion

Adaptive computer-based learning environments are receiving increasing attention. However, these environments lack effective modeling of complex learner interactions during learning sessions. In addition to the responses to questions, learners experience emotions, as manifested through facial expressions. This is critical in an adaptive learning process. A survey of the related literature demonstrated that the majority of studies focus on modeling a simplified learning environment using domain

Declaration of Competing Interest

We wish to draw the attention of the Editor to the following facts which may be considered as potential conflicts of interest and to significant financial contributions to this work. [OR] We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.

We confirm that the manuscript has been read and approved by all named authors and that there are no other

CRediT authorship contribution statement

Mohammed Megahed: Conceptualization, Methodology, Software, Data curation, Writing - original draft, Visualization, Investigation, Software, Formal analysis. Ammar Mohammed: Supervision, Conceptualization, Writing - review & editing, Validation, Project administration.

References (56)

  • L.A. Zadeh

    Fuzzy sets

    Information and Control

    (1965)
  • C.C. Aggarwal

    Neural networks and deep learning

    (2018)
  • A. Agrawal et al.

    Using CNN for facial expression recognition: Astudy of the effects of kernel size and number of filters on accuracy

    The Visual computer

    (2019)
  • R. Al-Shabandar et al.

    Machine learning approaches to predict learning outcomes in massive open online courses

    2017 international joint conference on neural networks (IJCNN)

    (2017)
  • Alcantud, C. R. et al. (2018). Fuzzy techniques for decision making....
  • O.A.M. Ali et al.

    Comparison between the effects of different types of membership functions on fuzzy logic controller performance

    International Journal of Emerging Engineering Research and Technology

    (2015)
  • O. Arriaga et al.

    Real-time convolutional neural networks for emotion and gender classification

    arXiv preprint arXiv:1710.07557

    (2017)
  • U. Ayvaz et al.

    Use of facial emotion recognition in e-learning systems

    Information Technologies and Learning Tools

    (2017)
  • R.E. Bellman et al.

    Decision-making in a fuzzy environment

    Management science

    (1970)
  • M. Blanton et al.

    Human-computer interaction

    Encyclopedia of database systems

    (2009)
  • S. Chickerur et al.

    3D face model dataset: Automatic detection of facial expressions and emotions for educational environments

    British Journal of Educational Technology

    (2015)
  • Chollet, F. et al. (2015 (accessed February 3, 2014)). Keras....
  • P. Cingolani et al.

    jfuzzylogic: a robust and flexible fuzzy-logic inference system language implementation

    Fuzzy systems (FUZZ-IEEE), 2012 IEEE international conference on

    (2012)
  • P. Cingolani et al.

    Jfuzzylogic: A java library to design fuzzy logic controllers according to the standard for fuzzy control programming

    International Journal of Computational Intelligence Systems

    (2013)
  • H. Ding et al.

    FaceNet2expnet: Regularizing a deep face recognition net for expression recognition

    2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017)

    (2017)
  • I.J. Goodfellow et al.

    Challenges in representation learning: A report on three machine learning contests

    Neural information processing

    (2013)
  • S. Gross et al.

    Learning feedback in intelligent tutoring systems

    KI - Künstliche Intelligenz

    (2015)
  • S.L. Happy et al.

    Automatic facial expression recognition using features of salient facial patches

    IEEE transactions on affective computing

    (2015)
  • Cited by (0)

    View full text