A study on the application of multidimensional feature fusion attention mechanism based on sight detection and emotion recognition in online teaching

3.1 Face feature points and human eye opening and closing detection in online teaching

The specificity of online teaching starts with the detection of whether the student is in front of the computer screen, i.e., the detection of whether the student has a human face or not. At the same time, there are currently more and more students wearing glasses, and in the detection, an algorithm that is not affected by glasses and is more robust and real time needs to be selected. In online teaching, it is necessary to check the attention level of students, so it is necessary to recognize, detect, and judge based on feature points. For example, facial contours such as eyes, mouth, and nose are a set of feature points that can depict the morphological structure of the face. Extracting feature points lays the foundation for subsequent research on gaze detection, emotion recognition, etc. [12]. Among them, the SDM is a representative regression analysis-based face feature point extraction method, which is a relatively simple calculation process, less sensitive to changes in light, has the characteristics of fast and stable, and is currently a popular face alignment method. To satisfy attention detection in online teaching, the research article adopts the SDM algorithm with many advantages to implement the detection of face feature points.

The eye is an important organ and contains a great deal of information that can be obtained from the movements and states of the eye as needed. One of the key factors affecting the extraction of visual information is the state of the human eye. In online teaching, human eye detection is an important tool to reflect students’ attention status. The detection includes several indicators such as human eye opening and closing status, blink frequency, and gaze direction. In this study, the focus will be on detecting students’ eye-opening and closing status. The study proposes an least square ellipse fitting (LSEF) to meet the requirements of human eye detection. After fitting the human eye contour accurately, if the ratio of the fitted ellipse area to the eye contour area is not within the threshold, the ellipse height can be corrected using the isometric method to obtain true eye width and height values. Then, the aspect ratio of the ellipse is used to determine whether the eye is in the open or closed state. Finally, the Canny operator in OpenCV is used to extract the contours of the eye region, and least squares are used to fit the ellipse so that it can quickly and accurately determine the relevant parameters of the human eye. When fitting the contours of the human eye region using the LSEF method, failure to match eye contours is noted as a missing detection; even if a match is completed, the resulting contours may not be the true eye, or there may even be more than one. For this reason, it is necessary to improve LSEF to fit the eye contour more accurately, and to judge the eye conditions accordingly, so that it can obtain optimal results. The flow chart of the improved human eye opening and closing-state detection algorithm is shown in Figure 1.

Figure 1

Flow chart of improved human eye opening and closing-state detection algorithm.

In Figure 2, this algorithm consists of three parts: optimal solution search, threshold judgment, and actual eye aspect ratio. This method can effectively eliminate all kinds of interference, making it unique and able to fit eye contours better. Additionally, the improved algorithm can eliminate the influence of glasses on the target, so the detection of the target will be greatly improved even when glasses are worn [13,14]. If the ratio of the fitted ellipse area to the eye contour area is not in range, eye contour height should be calculated again and actual eye height can be found by the following equation:

where and represent the width of the human eye and eye contour, respectively. When the true contour height is obtained, the open and closed states can be determined directly from the width ratio . In the case of , the eye is open, and in the opposite case, the eye is closed.

Figure 2

Sight landing situation.

3.2 Sight detection methods in online teaching

The line of sight is an imaginary straight line between the eye and an object, which reflects information such as the relative position of the eye and the object. Line of sight detection and tracking technology is a very important human–computer interaction detection technology. Incorporating a line-of-sight detection module into online teaching can effectively improve students’ emotional deficits and can facilitate the development of online teaching by adjusting the teaching mode according to the area of interest and length of stay of the learner’s line of sight. The study will make use of an ordinary personal computers (PC’s) low-resolution camera for detection and rational zoning of the sighting area. In the division of the line-of-sight landing area, it will be divided into five areas, namely, top left, bottom left, middle, top right, and bottom right. Based on the divided sight areas, the corresponding features are extracted and the students’ sight drop on the screen is then analyzed [15]. By reducing the human eye contour to an ellipse and eliminating irrelevant information, it was possible to obtain six contour feature points, which were smoothly connected to obtain the human eye contour. The next step is to construct a coordinate system from the coordinates of the feature points and their position relations and is the reference point. Combining the characteristics of online teaching and considering factors such as cost, the study selects an image gradient-based iris center localization algorithm for iris center extraction. It is assumed that in a circle, denotes the possible location of the pupil, is the true circle center, denotes the gradient vector of , is the displacement vector, and and are oriented in the same direction. The possible positions of the center of the circle are represented by and . When is the true circle center, and have the same displacement vector; if not, there will be an angle between and , and then, the center of the circle is calculated as shown in the following equation:

where represents the center of the circle found. and are calculated as shown in the following equation:

In equation (3), represents the bias to . However, when there is bright light or the influence of the eyelids, a bright pupil situation can occur, making the algorithm’s localization biased. The study combines the luminance information by assigning a weight value to by the pupil color being darker than the nearby area, and then, the final formula for the circle center is shown in the following equation:

The algorithm detects the center of the iris with good results when the test subject has a small head shift and wears glasses. However, because the algorithm is also influenced by eyebrows, for example, the iris center can be positioned outside the eye contour, and the large area of the detection area makes the calculation process more complex. Therefore, the study uses the human eye contour extracted using the SDM algorithm as the detection center region of the iris and uses to represent the iris center point. In turn, it is possible to obtain the learner’s visual dropout through the visual region division and , , as shown in Figure 2.

Figure 2 shows the five sightline situations, from left to right and from top to bottom: top left, top right, bottom left, bottom right, and center. Considering that human eye movements have the same or opposite pattern, this study will take the right eye as an example to investigate the sighting situation, and the plane distance between and d can be found by the following equation:

However, in practice, the detection accuracy decreases significantly when the detector’s line of sight looks downward, and the image gradient-based iris center detection algorithm also suffers from reduced iris centroid extraction accuracy when the line of sight is directed downward, so there is a need to find an effective method that can reduce the influence of the eyelid on the detection results. To address these issues, an improved line-of-sight detection method is proposed, which introduces an eye aspect ratio parameter based on the original line-of-sight detection method . The process of the improved line-of-sight detection method is shown in Figure 3.

Figure 3

Improved line of sight detection method.

In Figure 3, it can be seen that is mainly used to determine whether the human eye is closed; when , this implies that when the detector gazes downward, the algorithm can assess the left and right directions, distinguishing between left-down and right-down gaze orientations. Through this enhanced gaze detection method, it is possible to effectively and accurately detect the specific landing position of the gaze.

3.2.1 Fuzzy reasoning-based emotion recognition and attention detection system

The attention detection system, including indicators such as face presence detection, human eye opening and closing detection, and sight fall detection, significantly improves the human–computer interaction experience in online teaching environments. It also enables the real-time detection of learners’ attention status, making a valuable contribution to the development of online education [16,17]. However, analysis of students’ attentional status from these perspectives alone does not provide a true understanding of students’ attentional status. Based on this, the study added emotion recognition to the system, hoping to improve the attention detection system and make the most of the advantages of the online teaching model. However, emotion recognition has subjectivity, and fuzziness, making traditional emotion recognition methods difficult to effectively handle. Fuzzy theory can simulate the human mind and quantify the subjective concepts of human beings [18]. Fuzzy theory converts the problem of subjectivity and uncertainty into a problem of affiliation for quantitative study. Based on this principle, a new fuzzy reasoning method is proposed. The fuzzy reasoning method is applied to analyze students’ emotions qualitatively and quantitatively. The process of emotion identification based on fuzzy reasoning is shown in Figure 4.

Figure 4

Emotion recognition process based on fuzzy reasoning.

In Figure 4, emotions are first classified, then emotional features are selected, fed into fuzzy theory, fuzzy reasoning is performed, and the results of fuzzy reasoning are output in combination with a rule base, which, in turn, enables objective identification of emotional states. Considering the characteristics of online teaching, this study focuses on the learners’ facial emotions and classifies learning emotions into three types: pleasant, bland, and distressed. The external representation of facial emotion change is the change in features of the parts of the face associated with it, which are extracted and analyzed to identify the emotional state of the learner while learning [19]. The mouth and eye regions are the most obvious areas of emotional change, so the parameters of top and lower mouth curvature, eyebrow length, and forehead texture features will be extracted and analyzed. In the process of mouth angle curvature extraction, it is assumed that and represent the left and right mouth angle points, respectively, while and are the center points of the top and lower lips, respectively. The SDM algorithm is used to extract these four feature points, and it is assumed that , , , and represent the coordinates of , , , and , respectively, and and represent the tangent values of the top and lower curvature of the mouth angle, respectively, in the following equation:

(6)

After calculating and by equation (6), it can facilitate the recognition of emotions. The SDM algorithm can extract three learners’ eyebrow features in different emotional states , , and , which, in turn, allows the calculation of the overall length of the eyebrows , as shown in the following equation:

(7)

There is a connection between the forehead region and the eyebrows, and the skin folds shown in the forehead region can vary in different emotional states. Therefore, the texture of the forehead can be detected to reflect the changes in the forehead skin folds and thus identify the emotional state [20]. The texture information of the forehead region is detected using the Canny operator that comes with OpenCV. The fuzzy sets extend the affiliation from 0 and 1 to [0, 1], thus representing the attribution level of each element in the set that is consistent with certain requirements. Assuming that represents a set and is an element in , the fuzzy set can be represented using the affiliation function. Assuming that is a fuzzy subset of and is a representation of the affiliation function, the mapping relationship can be derived from the following equation:

(8)

The subordination of to is , which describes the degree to which the elements in belong to and takes the value range [0, 1]. When the value of is closer to 1, the element belongs to set to a greater extent; conversely, when the value of is closer to 0, it means that element belongs to the set to a lesser extent. After each module of visual attention detection, it can be fused to realize the multidimensional feature fusion attention detection based on face detection, human eye opening and closing detection, visual detection, and emotion recognition in online teaching, and the flow chart is shown in Figure 5.

Figure 5

Flow chart of attention detection method.

In the attention detection in Figure 5, the discrimination of learners’ attention status is achieved by fusing multiple face features, thus solving the problem that cannot be detected by a single feature. Specifically, the study will introduce an academic emotion recognition index to improve the attention detection method so that it can detect students’ academic emotions while they are paying attention, and thus make real-time adjustments to their learning emotions, to make judgments about their attention levels to fully exploit online teaching.

4.1 Analysis of face feature points and line of sight detection results

The study focused on capturing real-time facial images of students using a typical PC equipped with a low-resolution camera. Attention detection methods were implemented using software tools, such as Visual Studio 2013. The Intel Core i7-6700HQ had a 4-core CPU, 8GB of memory, and a camera pixel size of around 300,000. The SDM algorithm set thresholds and scales for facial detection to effectively detect facial feature points in low-resolution cameras, optimizing the detection speed and accuracy of feature points to adapt to real-time video stream processing. The least-squares ellipse-fitting algorithm-adjusted algorithm parameters to adapt to accurate detection of human eye opening and closing status in low-resolution images, including parameters such as eye size and position and optimized the algorithm to handle eye state detection under different lighting and angle conditions. Eye detection and emotion recognition required determining key features such as eye direction and pupil size for eye detection. The study involved selecting and optimizing emotion recognition algorithms, including facial expression-based emotion recognition, along with related model parameters and training datasets. The research identified key characteristics of academic emotions and designed corresponding emotion recognition indicators. Emotion recognition algorithms were carefully selected and optimized to ensure accurate recognition of academic emotions. Attention detection methods were combined with academic emotion recognition for model improvement by adjusting model weights and feature selection, enhancing the ability to judge student attention and emotions. Adjustment strategies and mechanisms were designed to achieve real-time adjustment of student learning emotions, considering the potential adjustment amplitude and methods under different emotional states.

The faces were detected and 70 key feature points were obtained using the SDM algorithm for face feature point extraction. To analyze its performance, 20 online students were selected and asked to sit in front of a computer screen and change and adjust the direction of their eyes during the experiment in the order of top left, bottom left, middle, top right, and bottom right. During this process, the experimental students needed to maintain their heads stationary or slightly rotate, and recorded a 1-minute video for each student, each recording 10 videos, including 5 groups each with and without glasses. The videos of the 20 students were examined and analyzed, and the results of the tests of three randomly selected students are shown in Figure 6.

Figure 6

Eye opening and closing status detection results of three students: (a) Volunteer 1, (b) Volunteer 2, and (c) Volunteer 3.

In Figure 6, the accuracy of detecting eye state for students with and without glasses was compared using a modified least-squares Nan’s circle-fitting algorithm. The detection accuracies for Student 1 with and without glasses were 82.8 and 91.7%, respectively; for Student 2, 86.6 and 93.1%; and for Student 3, 84.1 and 91.2%. That is, the average accuracy rates with and without glasses were 84.5 and 92.0%, respectively. This indicates that under the same conditions, the measurement accuracy without glasses was higher due to lens reflection. To further analyze the effectiveness of the eye open/closed-state detection algorithm, the study selected two common human eye state detection methods for comparison and analyzed the detection accuracy and the average detection time taken by the three algorithms with and without glasses, and the results are shown in Table 1.

In Table 1, the detection accuracies of the three algorithms are 74.8, 82.3, and 84.5% with glasses on and 88.7, 90.7, and 92.0% without glasses on, respectively; the average time consumption cases of the three algorithms are 13, 103, and 17 ms with glasses on and 12, 100 and 15 ms without glasses on, respectively. This indicates that the study method has high detection accuracy and fast real-time performance. Afterward, five students were selected from a video of 20 students, and their visual directions were analyzed and compared with the improved visual detection method, and the results are shown in Figure 7.

Figure 7

Accuracy of each algorithm in different line of sight directions: (a) accuracy of various algorithms in different line of sight directions without wearing glasses and (b) accuracy of various algorithms in different line of sight directions when wearing glasses.

In Figure 7, the accuracy of learners in both the top-left and top-right views was good, with the highest accuracy of 99.0% in both views and 97.3% in the middle view. The accuracy rates for the bottom-left and bottom-right detections were 58.3 and 60.9%, respectively, far short of the recognition requirements. In the three visual directions of top left, top right, and middle, the accuracy of recognition using the improved method was the same as before the improvement. While in the lower-left and lower-right directions, the accuracy was up to 90% in the lower-left direction and 86% in the lower-right direction, both of which were greatly improved and fully met the test requirements. When they put on the glasses, the prediction accuracy also improved significantly, with the highest accuracy of 73.9 and 72.3% in the lower-left and lower-right directions. Results showed that the improved method approach had high recognition accuracy and was more accurate compared to the existing methods.

4.2 Analysis of emotion feature extraction results and attention detection results

There is a certain connection between the forehead region and the eyebrows, and the skin folds shown in the forehead region can vary in different emotional states. Therefore, the detection of the forehead texture can reflect the changes in the forehead skin folds, and thus, the emotional state can be identified. The forehead texture was first detected using different Canny operators, and the results are shown in Figure 8.

Figure 8

Detection results of forehead texture using different Canny operators: (a) traditional canny operator and (b) adaptive canny operator.

Figure 8 shows that in the traditional Canny operator, the percentage of forehead texture for the same emotion under different lighting conditions of strong light, normal light, and low light had a large difference; while in the adaptive Canny operator, the percentage of forehead texture for the same emotion under different lighting was relatively close. Since the forehead texture ratio also changed when the distance between the learner and the screen was changed, it is assumed that the lighting conditions were normal lighting, the face area was judged, and a threshold was set to obtain the forehead texture ratio detection results under different distances, as shown in Figure 9.

Figure 9

Frontal texture proportion detection results at different distances.

In Figure 9, the normal distance is between 50 and 70 cm from the computer screen, and the far distance is between 85 and 105 cm from the computer screen. In Figure 10, after setting the lighting conditions and thresholds, it was able to provide a better improvement of the forehead texture changes caused by the distance variation. The emotional changes of the learners were then identified and analyzed using an emotional recognition method based on fuzzy inference, and the learning emotions were classified into three types: pleasant, bland, and distressed, and information such as the percentage of relevant textures was input into the fuzzy inference. The relevant data of five learners, each involving 100 data, were taken, and the abnormal data were eliminated to obtain the results and recognition rate of emotion recognition, and the results are shown in Figure 10.

Figure 10

Results and recognition rate of emotion recognition.

In Figure 10, the results show that the research method achieved an average recognition rate of 90.7% for the three different types of learning emotions, which fully meets the requirements for learning emotion recognition in online teaching. Relatively speaking, the accuracy rate was somewhat higher under the bland emotion, at 92%. Under pleasant emotions, the accuracy rate reached 91%, and for distressed emotions, there was an 89% accuracy rate. When the forehead texture percentage reached 8% or more, its detection rate reached 100%. Afterward, the facial images of the learners were captured in real time using a normal PC with its camera, and the learners were kept at a normal distance from the computer screen to capture learning videos of 15 learners, each video being 10 minutes long and detected 200 times for each segment. The attentional state during the learning process was detected and analyzed in Figure 11.

Figure 11

Comparison of detection results of attention states using different algorithms.

In Figure 11, the use of a single feature to detect a student’s attentional state was not ideal. When the eyes were open and the line of sight was away from the specified direction, the single human eye opening and closing detection did not provide an accurate judgment of the results, and a missed and false detection occurred. The use of the research algorithm, which performed attention detection on the learner, incorporated both human eye open/closed detection and line of sight deviation detection, allowing for effective improvement on these issues and a significant increase in accuracy. The detection accuracies for the single feature and the research method were 74.1 and 91.9%, respectively.