Modeling adaptive E-Learning environment using facial expressions and fuzzy logic
Introduction
Intelligent techniques are attracting increasing attention and producing encouraging results compared with conventional techniques. Although these techniques are useful in certain tasks, several complex real-world problems cannot be tackled by a single intelligent technique. This is because each intelligent technique has certain computational characteristics that are appropriate in specific problems. These limitations prompted the development of hybrid intelligent systems that integrate two or more systems, such as deep neural networks (Aggarwal, 2018) and fuzzy techniques (Alcantud, et al., de Silva, 2013), into a single framework. These systems are obtained either by loosely coupled integration or by fusion of individual techniques. Recently, hybrid intelligent systems have attracted considerable attention owing to their broad success in several complex real-world applications. An interesting example is the modeling of adaptive computer-based learning environments (Moos & Azevedo, 2009), which are technology-based environments, such as intelligent tutoring systems (Corbett, Koedinger, & Anderson, 1997) and web-based learning systems (Kularbphettong, Kedsiribut, & Roonrakwit, 2015). These can be used as an instructional aid for educational purposes. The modeling of such environments has received wide attention from multiple disciplines, including computer science, psychology, and education (Lajoie & Naismith, 2012). Usually, the learning flow in these environments depends on the learner’s mental responses based on solving tests and answering exam questions, so that the next level of the learning process may be determined. However, conventional methods do not consider the emotional behavior of the learner during the learning process. According to Goetz, Lüdtke, Nett, Keller, and Lipnevich (2013); Kärner and Kögler (2016), the emotional behavior is considered an important factor affecting the quality of the learning process and the achievement of the desired learning outcomes. The learner’s interaction with the learning environment can be classified into two categories. The first consists of mental responses to test questions, and the second consists of emotional responses through facial expressions. Therefore, integrating both types is critical for developing more adaptive and intelligent computer-based learning environments. The interaction with the learning environment varies among learners owing to several factors related to the learner’s responses or the learning environment itself. Likewise, the educator’s judgment about the expression of the learners’ emotions and the effectiveness of their interactions varies owing to the educator’s perception and knowledge. Thus, the evaluation of the learners’ interactions from various perspectives is a significant challenge. A large amount of research has been conducted to model the interaction of learners in computer-based learning environments from different perspectives. However, these studies avoided complex interactions and ignored critical interaction parts and features; rather, the focus was on simple modeling. For instance, several studies have proposed production rules that consider the learner’s answers to tests and exam questions to be the only means of communication with the learning environment, and hence they determine the next level of the learning flow Gross, Mokbel, Hammer, and Pinkwart (2015); Kularbphettong et al. (2015); Mosa, Albatish, and Abu-Naser (2018). However, owing to the importance of considering the learner’s emotions during the learning process, other research efforts focused on modeling these emotions by interpreting facial expressions using machine learning algorithms (Ayvaz, Gürüler, Devrim, 2017, Chickerur, Joshi, 2015, Khalfallah, Slama, 2015, Krithika, 2016, Petrovica, Anohina-Naumeca, Ekenel, 2017, Yang, Alsadoon, Prasad, Singh, Elchouemi, 2018). Some studies proposed modeling the variance in the communication of the learner with the learning environment. These studies can be categorized into two types. The first is concerned with the variance in solving tests and exams (Meenakshi, Pankaj, 2015, Yadav, Singh, 2011), whereas the second is concerned with the variance of learners’ emotions during the learning process (Zatarain-Cabada, Barrón-Estrada, García-Lizárraga, Muñoz Sandoval, & Ríos-Félix, 2015). In both categories, it is attempted to model more complex interaction behaviors by introducing fuzzy logic to handle differences in the communication with the learning environment. Although these studies proposed various fuzzy models for handling the uncertainty in the emotional responses, they attempted to abstract the problem into a simple form and did not consider the probability degree of different facial expression classes. Traditionally, each instance of a facial expression, such as happiness, sadness, or surprise, should be represented using probability degrees over multiple classes. Additionally, these studies did not consider the variation of the learner’s emotional states during the test sessions because their simplified methodologies depend on analyzing only one picture of the learner’s face at the end of the test session and ignore other important facial expressions during the test.
The main contribution of this study is threefold. First, to model complex adaptive learning environments, it proposes a hybrid approach that integrates in the same framework a convolutional neural network (CNN) and a fuzzy system. In this approach, we consider learner mental responses to the test questions as well as their emotional responses. Moreover, we consider the variance of the learners’ emotions in test and exam sessions. The CNN in the hybrid approach is responsible for modeling, analyzing, and classifying learner facial expressions that reflect their emotional states based on online video streamed during the test and exams sessions (Kim, Roh, Dong, Lee, 2016, Li, Zhang, Zhang, Zhang, Li, Xia, Yan, Xun, 2017, Lopes, de Aguiar, Oliveira-Santos, 2015, Shan, Guo, You, Lu, Bie, 2017, Shin, Kim, Kwon, 2016). The proposed CNN outperforms other CNN models on the same training benchmark of facial expressions. The fuzzy part in the proposed hybrid approach is responsible for handling the uncertainty in the learner’s interactions with the learning environment. The fuzzy system considers several uncertain features compared to other fuzzy systems applied in the same domain, including the diversity of learner emotions. In addition to the inputs received from the CNN, the fuzzy inference system takes features from the learning environment, including the answer validity ratio, elapsed test time, and current learning level. The ultimate goal of the fuzzy part is to provide the next level of the learning process based on the input features.
As the CNN provides a large number of emotion states that span several classes of facial expressions, the second contribution is two aggregated methods that summarize the detected set of emotional states and provide a single representative aggregated facial expression class at end of the test and exam sessions. This aggregated facial expression is then fed to the fuzzy part together with the features of the environment. To the best of our knowledge, no study has provided an aggregated representative facial expression for all emotional states and fed it to a fuzzy system.
The third contribution is the introduction of two real corpora of learner emotional states and learning flows. The first corpus is used to specify the different emotional states and variances during the test sessions. The second corpus is introduced to specify the learning flows across different learning activities of the group in this study. The proposed hybrid approach provides decision makers with info-graphs that explain the learning flows and emotional states of all learners who participate in the learning activities.
The rest of this paper is organized as follows: Section 2 provides a brief background on emotions, CNNs, and fuzzy logic. Section 3 surveys related work. Section 4 introduces the details of the proposed approach. Section 5 proposes two corpora that are used to evaluate the proposed approach. Section 6 discusses the experiment and the results. Finally, Section 7 concludes the paper.
Section snippets
Background
Emotion could be defined according to the Oxford dictionaries as a strong feeling deriving from one’s circumstances, mood, or relationships with others. Furthermore, it could be defined as a feeling comprising physiological and behavioral reactions to internal and external events (Sternberg, 1995). Emotion includes both physical and cognitive actions that interpret biological responses to an external stimulus. That biological response affects our handling of situations and our interaction with
Related work
This section overviews studies related to the present. These efforts can be regarded from the perspective of psychology, fuzzy logic, and automatic facial expression interpretation.
From a psychological perspective, and regarding the effect of the emotional states of the learner, the authors in Goetz et al. (2013) explored the relations between teaching features and learners’ academic emotions. They found clear relations between these two factors at the intrahuman level. Moreover, they mentioned
Proposed approach
Before presenting the proposed approach, we formalize some basic concepts. In particular, we define the concepts of emotional state, emotional behavior, and aggregated emotional state. Definition 1 Emotional State Let be a set of n different classes of facial expressions. We define the emotional state σ=(∊1, ∊2, ∊3, ...,∊n) as a probability distribution over E such that 0 ≤ ∊i ≤ 1 and for 1 ≤ i ≤ n. The emotional state σ at particular instant t is denoted as for t
Proposed corpora
This section describes the corpora that will be used to test the proposed approach as well as generate the output results. The data of the corpora were collected using an implementation of the proposed learning environment. Therefore, we begin this section by explaining this implementation and elaborate on the processing flow between the learner and the components of the learning environment.
Experimental results
In this section, we use the generated corpora to experiment with the proposed approach. We discuss the results from different perspectives. First, we analyze the effect of learner’s emotions on the learning environment. Second, we study the effect of using different aggregation functions (mean and median) on the learning flow.
Table 9 describes an example of learning activities of the five individuals 1, 2, 3, 11, and 12 based on summarizing their emotional states using the mean. Table 10
Conclusion
Adaptive computer-based learning environments are receiving increasing attention. However, these environments lack effective modeling of complex learner interactions during learning sessions. In addition to the responses to questions, learners experience emotions, as manifested through facial expressions. This is critical in an adaptive learning process. A survey of the related literature demonstrated that the majority of studies focus on modeling a simplified learning environment using domain
Declaration of Competing Interest
We wish to draw the attention of the Editor to the following facts which may be considered as potential conflicts of interest and to significant financial contributions to this work. [OR] We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.
We confirm that the manuscript has been read and approved by all named authors and that there are no other
CRediT authorship contribution statement
Mohammed Megahed: Conceptualization, Methodology, Software, Data curation, Writing - original draft, Visualization, Investigation, Software, Formal analysis. Ammar Mohammed: Supervision, Conceptualization, Writing - review & editing, Validation, Project administration.
References (56)
- et al.
Intelligent tutoring systems
Handbook of human-computer interaction
(1997) - et al.
Characteristics of teaching and students’ emotions in the classroom: Investigating differences across domains
Contemporary educational psychology
(2013) - et al.
Facial expression recognition for intelligent tutoring systems in remote laboratories platform
Procedia computer science
(2015) - et al.
Developing an adaptive web-based intelligent tutoring system using mastery learning technique
Procedia-Social and Behavioral Sciences
(2015) - et al.
Facial expression recognition with faster r-CNN
Procedia computer science
(2017) - et al.
An experiment in linguistic synthesis with a fuzzy logic controller
International journal of man-machine studies
(1975) - et al.
Data on learners emotional states, mental responses and fuzzy learning flows during interaction with learning environment
Data in brief
(2019) - et al.
Emotion recognition in affective tutoring systems: collection of ground-truth data
Procedia computer science
(2017) - et al.
Fpga platform applied for facial expression recognition system using convolutional neural networks
Procedia computer science
(2019) - et al.
An emotion recognition model based on facial recognition in virtual learning environment
Procedia computer science
(2018)