Impact of inquiry interventions on students in e-learning and classroom environments using affective computing framework

Abstract

Effective teaching strategies improve the students’ learning rate within academic learning time. Inquiry-based instruction is one of the effective teaching strategies used in the classrooms. But these teaching strategies are not adapted in other learning environments like intelligent tutoring systems, including auto tutors. In this paper, we propose an automatic inquiry-based instruction teaching strategy, i.e., inquiry intervention using students’ affective states. The proposed model contains two modules: the first module consists of the proposed framework for predicting the unobtrusive multi-modal students’ affective states (teacher-centric attentive and in-attentive states) using the facial expressions, hand gestures and body postures. The second module consists of the proposed automated inquiry-based instruction teaching strategy to compare the learning outcomes with and without inquiry intervention using affective state transitions for both an individual and a group of students. The proposed system is tested on four different learning environments, namely: e-learning, flipped classroom, classroom and webinar environments. Unobtrusive recognition of students’ affective states is performed using deep learning architectures. After student-independent tenfold cross-validation, we obtained the students’ affective state classification accuracy of 77% and object localization accuracy of 81% using students’ faces, hand gestures and body postures. The overall experimental results demonstrate that there is a positive correlation with $r=0.74$ between students’ affective states and their performance. Proposed inquiry intervention improved the students’ performance as there is a decrease of 65%, 43%, 43%, and 53% in overall in-attentive affective state instances using the inquiry interventions in e-learning, flipped classroom, classroom and webinar environments, respectively.

Appendix

• 1. Affective State Definitions

  We used the definitions mentioned in (D’Mello et al. 2010; Whitehill et al. 2014; D’Mello et al. 2007).

  Boredom: uninterested in the current problem.

  Confusion: poor comprehension of material, attempts to resolve erroneous belief.

  Disgust: annoyance and/or irritation with the material and/or their abilities.

  Fear: feelings of panic and/or extreme feelings of worry.

  Frustration: difficulty with the material and an inability to fully grasp the material.

  Happiness: satisfaction with performance, feelings of pleasure about the material.

  Neutral: displays no visible affect, at a state of homeostasis.

  Sadness: feelings of melancholy, beyond negative self-efficacy.

  Surprise: genuinely does not expect an outcome or feedback.

  Delight: a high degree of satisfaction.

  Sleepy: extremely not interested and in a mental state of sleep.

  Engaged: a state of interest that results from involvement in an activity.

• 2. Inquiry Intervention Module

  These are the sample questions asked in the inquiry intervention module.

  • (a) Examples of questions to stimulate critical thinking.

    Q: What point you think the Schachter and Singer is making in their proposed third interpretation [312a]?

    This illustrates a question that asks students for clarification.

    Q: What are the notable similarities and/or differences between the items on your list and those provided by D A Norman’s Emotion and Design?

    This question challenges the students to compare and contrast ideas provided by two sources.

    Q: What are the possible underlying assumptions behind Shneiderman’s criteria?

    This question challenges the students to demonstrate the critical skill of identifying assumptions made by an author.

    Q: What alternative assumptions might an HCI Analyst make?

    This question further probes students’ understanding of assumptions by asking to identify alternative assumptions that might be made.

    Q: Do you personally agree with Shneiderman’s criteria and what types of evidence can you offer to support your position?

    These two critical thinking questions ask students to both take a personal position and to provide relevant information or evidence to support their decision.

  • (b) Examples of questions to stimulate creative thinking.

    Q: How many different heuristics can you list that are the most important and will receive attention in heuristic evaluation stated in Jakob Nielsen’s ten heuristics?

    This question challenging students to come up with as many possible different ideas as they can and elicits the type of creative thinking process that has been described in the literature as fluency.

    Q: If Shneiderman wanted to identify Eight Golden Rules of Interface Design instead of only five, what three additional possibilities would you suggest him to add to his list?

    This question, requiring students to embellish or expand upon ideas, elicits a type of creative thinking process that has been described in the literature as elaboration.

    Q: What novel, unique, or unusual types of course activities and assignments do you think they are most likely to inspire today‘s college and university students?

    This type of question illustrates one approach to stimulate the creative thinking skill of originality.

  • (c) Examples of questions to stimulate curiosity.

    Q: Who is Don Norman and why might people be interested in his perspective on this topic?

    Q: What was Jakob Nielsen like as a student?

  The questions asked are not having the similar pattern mentioned in the examples for all the subjects, but the underlying concepts for framing the questions remains the same.

• 3. Duplications removal during training phase dataset annotation

  The image frames obtained from classroom data with spontaneous expressions are collected at the rate of 60 fps. There is a chance that two successive frames may have more than 80% similarity. In order to avoid this, we considered the image frames with a gap of 5 frames. For multi-person’ posed expressions, we selected 2 to 5 image frames with peak emotion intensity where all the students are present in that single frame image. If the expression or pose in those image frames are similar (more than 80%), then the redundant image frames are discarded. Thus, we avoided the duplications. We did not remove the duplications during the real-time affective content analysis.

• 4. Affective State Classification and Localization Technique

  Inception-V3 (Szegedy et al. 2016) and YOLOv2 (Redmon et al. 2016) are the base architectures for the affective state classification and localization, respectively. We modified the existing architecture to perform better in real time for students’ affective state classification and localization. The details are mentioned as follows. The proposed architecture is divided into four major parts. The first part is stem, which contains the convolutional and pooling layers until the factorization of convolutional layers. The second part contains the factorization into smaller convolutions using filter concats. The third part contains the auxiliary classifiers, and the last part contains fully connected layers and softmax classifier. The entire proposed architecture consists of 80 layers with stride 2 and 6 pooling layers which are used for an independent evaluation over each activation maps and for size reduction. The last pooling layer is connected to two fully connected layers. Those fully connected layers are connected to the softmax layer.

  Training phase starts with the backpropagation using the recursive chain rule for computing the gradients for all inputs, parameters and intermediates. The entire process of backpropagation is stored in a graph structure. The forward pass computes the results of an operation and stores the gradient, value whereas the backward pass computes the gradient of loss function w.r.t. the inputs.

  Equation 2 shows the loss function performed using multinomial logistic regression where $y_i$ is the ith image label and $x_i$ is ith image frame, using these, the loss for the ith image $L_i$ is calculated. The overall mean of training loss of the entire training data $L_t$ is defined as the total data loss given in Eq. 2.

  $$\begin{aligned} L_{t}= & {} \frac{1}{N}\sum _{i=1}^{N}L_{i}\nonumber \\ \mathrm{where}, \; L_{i}= & {} -\log \left( \frac{e^{sy_{i}}}{\sum _{j}e^{s_{j}}}\right) \end{aligned}$$

  (2)

  Since the data are used for analyzing multiple class in a single image frame (dense labeling), we used the activation function which neither saturates for both positive and negative values nor dies for negative values. Hence, Leaky ReLU (Rectified Linear Unit) is used as an activation function (Eq. 3) where x is an is an input column vector containing all pixel data of the image.

  $$\begin{aligned} f\left( x \right) = \max (\alpha x,x) \; \mathrm{where} \; \alpha = 0.01 \end{aligned}$$

  (3)

  Xavier initialization (Eq. 4 where $n\_in$ & $n\_out$ are input neurons from current & next layer and r is to calculate zero mean) is used for weight initialization. Batch normalization is also used along with the hyperparameter optimization. Minibatch stochastic gradient descent is used in a loop structure where it performs 4 major steps; Step 1: data are divided into N samples. Step 2: forward propagate to get the loss. Step 3: calculate the gradients using backpropagation. Step 4: from the calculated gradients, update the parameters.

  $$\begin{aligned} W=random.r(n\_in,n\_out)/sqrt(n\_in) \end{aligned}$$

  (4)

  Parameters update is performed using RMSProp. Learning rate is a hyperparameter for RMSProp. Regularization is performed using Dropout. Monte Carlo approximation is used in dropout where several forward passes with different dropout masks are performed and final prediction is average of all predictions. Numeric gradient is used for gradient check.

Group Engagement Score Generally, the data collected from e-learning environment contain single student in a single image frame, but the students’ spontaneous classroom data can have different affective states for each student present in a single image frame. Hence we used feature fusion to calculate the same. The multi-modal (here, it is an intra-image multi-modality where the features of different students with their facial expressions, hand gestures and body postures present within that image frame are considered) feature fusion vector $V_f$ for any pixel $p_i$ and normalized prediction vector $N_{P_i}$ uses normalized predicted probability distribution $N_{P_i,a}$ of class a using the softmax function (Eq. 5).

$$\begin{aligned} N_{P_i,a} = \frac{e^{W^T_{a}V_f}}{\sum _{i\in classes} e^{W^T_{i}V_f}} \end{aligned}$$

(5)

where W is the temporary weight matrix used to learn the features. The training generally converges in $T=6000$ epocs. The final collective average affective state score $A_{S_i}$ is given by Eq. 6.

$$\begin{aligned} A_{S_i}=arg\, max \, N_{P_i,a} \; where \; a\in classes \end{aligned}$$

(6)

Data augmentation Data augmentation has increased training data size by tenfold. Following are the different data augmentation techniques which we performed on our datasets.

Table 12 Types of data augmentation used

• channel_shift_range: Random channel shifts of the image.

• zca_whitening: Applies ZCA whitening to the image.

• rotation_range: Random rotation of image with a degree range.

• width_shift_range: Random horizontal shifts of the image with a fraction of total width.

• height_shift_range: Random vertical shifts of the image with a fraction of total height.

• shear_range: Shear intensity of the image where the shear angle is in the counter-clockwise direction as radian.

• zoom_range: Random zoom of the image where the lower value is 1-room_range and upper value is 1+zoom_range.

• fill_mode: If any of constant, nearest, reflect or wrap are filled according to the given mode, if any points outside the boundaries of the input.

• horizontal_flip: Randomly flip the inputs horizontally. Table 12 shows the details of different data augmentations performed on our dataset.

• 5. Student Engagement Level Analysis using its Graphical Representation

  A sample snapshot of student’s engagement level for a mini-concept of 15 min duration is shown in Fig. 12. There is a drop in the engagement score (Attentive State to In-Attentive state) of the student from 10 to 12.5 min range. Initially, we observed a similar drop after 5 to 6 min of mini-concept video completion for more than 30% of students in e-learning environment. After introducing InIvs, the engagement drop is reduced from 30% to 7%.

Fig. 12

A sample snapshot of student’s engagement level for a mini-concept

Fig. 13

The distribution of attentive and in-attentive state instances for the sample mini-concept duration in all four learning environments

The distribution of attentive state and in-attentive state instances for a sample 5 min duration of a mini-concept (the same concept is taught in all four learning environments) is shown in Fig. 13. Though the number of students is significantly more in the classroom environment, we can observe that the median is almost the same.

• 6. Semi-automatic Annotation Process The semi-automatic annotation process speeds up the process. Initially, annotations are performed manually. Once sufficient annotated image frames are obtained for training, we used the semi-automatic annotation process (Sekachev et al. 2019) to make the annotation process faster. Here, we used deep learning architectures for object localization and classification, where the deep learning architecture predicts the bounding box and the affective state. The annotator manually checks and corrects if the bounding boxes and classified affective state is not correct. For example, if the bounding box predicted by the deep learning architecture is not a tight bounding box, then the annotator adjusts it to make it close instead of drawing the bounding box from scratch for all the multitude of modalities present in that image frame.