Keywords

1 Introduction

To improve the efficiency of teaching and learning, it is important to grasp learners’ mental states during their learning processes. Although human teachers have the ability to comprehend learners’ mental states and teach them in accordance with their mental states, the computer realization of this ability remains a long-term problem in the area of education systems. The field of educational technology also has provided us with much knowledge on the relationships between learners’ physiological information, such as eye motion and amount of sweat, and learners’ mental states [5]. As today’s computers are currently becoming more and more convenient in function and lower in price, they can be utilized to perform a great deal of real-time processing of human physiological data. Therefore, many researchers are striving to develop automatic inference systems for learners’ mental states. Furthermore, it is generally acknowledged that during a process of teaching and learning, the interaction between the teacher and learners, especially the teachers’ speech acts, will greatly influence the learners’ mental states. Therefore, it is also vital to clarify the relationships between teachers’ speech acts and learners’ mental states, and to integrate such knowledge into those automatic inference systems.

In this study, we sought to extract in an experimental way the relationships between learners’ mental states and learners’ physiological information complemented with teachers’ speech acts. Different from the previous study [9], which analyzed only one scene (about 60 s) in a single class (about 60 min), in this study we analyzed five scenes in a single class, each about 30 to 90 s in length. Through this experiment, we obtained several sets of association rules with high degrees of generality that describe the relationships between learners’ mental states and the measurable physiological information of teachers and learners.

2 Comprehensive Analysis of Learning-Related Information

In this study, we attempted to analyze various aspects of learning using the technology of data mining in an experimental manner. In detail, we extracted the relationships between the learners’ mental states and the learners’ physiological information (NIRS signal, EEG signal, respiration intensity, skin conductance, and pulse volume) together with the information on the teachers’ speech acts using association rule mining. Before conducting the association rule mining, because different kinds of physiological information have different data forms and recording frequencies, we changed all the recorded physiological information into categorical data. In the past, physiological data were processed mainly by mathematic methods, leading to results that are difficult to interpret. Because we divided our data into categories that have fairly simple universal meanings, we resolved the problem of data interpretation. In this section, we introduce the experiment settings as well as the methods of data processing and interpretation.

2.1 Experiment Settings

This experiment aims to collect the learners’ physiological information. Two subjects, Subject A and Subject B, participated in the experiment. They were both junior high school students taking extra classes in a private tutoring school where there was only one teacher and one student in a single class. The classroom settings in the experiment were the same as in the private tutoring school. The physiological information collected in the experiment included an EEG signal (recorded using Emotive EEG), a NIRS signal (recorded using Hitachi WOT-100), respiration intensity, skin conductance, and pulse volume (recorded using NeXus). Except for the EEG signal and the NIRS signal, these psychological data were recorded at the same time. Because it was impossible to place the EEG instrument and the NIRS instrument on a single person’s head at the same time, Subject A used the EEG instrument and Subject B used the NIRS instrument. In order to align the recording time of these instruments, we placed time markers at the beginning and end of the measurement. Three video cameras set at different places were used to record the course of the experiment. After several days, the subjects were asked to report on their mental states during the course of the experiment while watching the videos.

2.2 Data Selected for Analysis

From the course of the experiment, which lasted about 60 min, six scenes displaying rich teacher-learner interactions were selected for data analysis. Each scene includes various sorts of teacher-learner interactions. For each scene, the temporal location and the types of teacher-learner interactions included are described as follows.

Scene 1.

Temporal location: 30:50-31:50 (60 s).

Main teacher-learner interactions:

The teacher pointed out the mistakes in the learner’s calculations. The learner asked the teacher about the right calculating methods. The teacher, in a relatively light-hearted tone, told the student to be careful not to make these kinds of mistakes again, for they were rather easy to make.

Scene 2.

Temporal location: 34:30-35:30 (60 s).

Main teacher-learner interactions:

The teacher praised the learner for his/her calculating methods. Then, the teacher pointed out and explained the mistakes in the learner’s calculations, and introduced some more efficient calculating skills. (This scene is the only one that included praises, and there is relatively little chatting in this scene.)

Scene 3.

Temporal location: 36:53-37:20 (27 s).

Main teacher-learner interactions:

After the learner told the teacher that he/she had finished the previously assigned calculations, the teacher gave the learner the next calculation question. (This scene is one of the few scenes in which the teacher checked for task fulfillment.)

Scene 4.

Temporal location: 51:04-52:27 (83 s).

Main teacher-learner interactions:

The teacher explained the calculation questions. The teacher then explained the calculating methods with some examples. Finally, the teacher asked the learner what parts of the class the learner considered to be difficult to understand.

Scene 5.

Temporal location: 53:51-54:22 (31 s).

Main teacher-learner interactions:

The teacher alerted the learner to the mistakes made by the learner. The teacher found that the learner did not actually understand what the teacher had said but instead pretended to understand, and so the teacher further alerted the learner.

Scene 6.

Temporal location: 57:55-58:51 (56 s).

Main teacher-learner interactions:

The teacher alerted the learner because the learner had made some mistakes more than once. The teacher alerted the learner with some jokes, that is, in a relatively light-hearted manner.

3 Extraction of Association Rules from Scene 1

In order to extract the most basic relationships between the learners’ mental states and the learners’ physiological information together with the teachers’ speech acts, we employed the association rule mining technique. This section introduces the general procedure of the association rule mining [9] applied to Scene 1. (For details of this association rule mining method, please refer to the previous study [9].)

3.1 Data Preprocessing

In view of the fact that different kinds of physiological information collected in the experiment vary in data form and recording frequency, we transformed all of them into categorical data. Table 1 shows the different types of physiological information and their corresponding categories. Considering that respiration intensity and skin conductance are continuous data, we categorized these using a five-point scale representing the amount of variation from the preceding data point. After treating the NIRS data using the global average reference method [2, 6], considering that the recording frequency of the NIRS instrument is low (5 Hz), we categorized the data according to their magnitudes instead of their degrees of variation. We utilized the Achievement Emotions Questionnaire (AEQ) [7] to divide the mental states into nine categories—“Enjoyment,” “Hope,” “Pride,” “Anger,” “Anxiety,” “Shame,” “Hopelessness,” “Boredom,” and “Others.” We then developed a computer program that allowed the subjects to choose the categories that they considered their mental states to belong to while watching the videos recording the courses of their own experiments. Based on the categories used in some previous studies [1, 3, 8], we divided the teachers’ speech acts into nine categories—“Explaining,” “Questioning,” “Comprehension Checking,” “Repeating,” “Praising,” “Chatting,” and “Others.” With these nine categories, we labelled the teachers’ speech acts while watching the videos recording the experiment courses.

Table 1. Different types of physiological information and their corresponding categories
Full size table

Each kind of physiological information was labelled with the corresponding categories, and the labelled physiological information types were then aligned in the dimension of time, producing 2,267 recording points in total. Due to the fact that different kinds of physiological information have different recording frequencies, one or more types of physiological information are absent in some time segments. Considering that these time segments are all quite short and that the physiological information usually changes smoothly, we used interpolation to fill the missing data points, with the exception of the EEG and pulse volume. Because the EEG data and the pulse volume data differed so greatly from the other kinds of physiological information in recording frequency, the EEG and pulse volume channels contained too many missing data points. Therefore, the EEG data and the pulse volume data were excluded from the data analysis.

3.2 Results of Association Rule Extraction

Next, we performed the association rule mining to the labelled data. As a result, twelve total association rules were extracted, which are listed in Table 2 (support = 0.02, confidence = 0.89, lift = 2.2).

Table 2. The association rules extracted from Scene 1 (SA: speech acts; RI: respiration intensity; SC: skin conductance).
Full size table

Rule 1 regards the mental state category of Enjoyment (E1). This rule shows that when the learner’s brain blood volume has increased a bit (A2), the teacher is alerting (D7), and the learner’s respiration intensity has greatly risen (C1), the learner’s mental state tends to fall under the category of Enjoyment (E1).

Rules 2–4 concern the mental state category of Pride (E3). These tell us that when the learner’s brain blood volume has increased a bit (A2), the teacher is checking the learner’s comprehension (D3), the learner’s respiration intensity is low (C4), and the learner’s skin conductance does not change (B3), the learner’s mental state tends to be that of Pride (E3).

Rule 5 regards the mental state category Anxiety (E5). It demonstrates that when the learner’s brain blood volume is very low (A5) and the learner’s respiration intensity is very high (C1), the learner’s mental state tends to fall under the category of Anxiety (E5).

Rules 6–12 regard the mental state category of Shame (E6). These rules mean that when the teacher is checking the learner’s comprehension (D3) or providing explanations (D1) and the learner’s brain blood volume is a bit low (A4), the learner’s mental state tends to be that of Shame (E6). In addition, when the teacher is providing explanations (D3), the learner’s respiration intensity tends to be high (C1).

3.3 Interpretation of Association Rules

Rule 1 suggests that if teachers alert learners in a relatively light-hearted manner (e.g., with some jokes), learners may laugh, causing their brain blood volume and respiration intensity to rise, thus resulting in the mental state of Enjoyment. In fact, we have observed in the current study’s videos such scenes where the teacher alerted the learners with jokes.

From Rules 2–4, we can know that teachers’ comprehension checking may increase learners’ mental burdens, causing learners’ brain blood volumes to rise. In addition, given the fact that learners’ respiration intensity and skin conductance do not suggest any anxiety or nervousness, it is probable that the learners in our experiment were able to resolve the tasks assigned by the teacher, thus resulting in a feeling of pride.

Rule 5 implies that in the case of the mental state Anxiety, the intensity of the NIRS signal decreases. This is perhaps because the brain areas activated by the emotion of anxiety were different from the brain areas measured in our experiment, and the increases in the blood volumes at the former brain areas parallel the decreases of the blood volumes at the latter brain areas. Given the fact that this rule is the only one that includes a low-level NIRS signal (A5), and further that this rule also includes a great change in respiration intensity, we expect that the mental state of anxiety can be inferred to a certain extent from the physiological information.

With regard to Rules 6–12, we think that because the learners in our experiment failed to understand what the teacher had said and therefore could not provide the teacher with satisfactory answers, the learners began to feel shameful (E6), accompanied by the quickening of their breath (C1). On the other hand, when the teacher was giving explanations (D1), the learners breathed relatively slowly (C4). We conjecture that because the teacher did not require the learners to answer questions, the learners could listen to the teacher’s words with ease, thus showing no increase in their respiration intensity. In addition, it is possible that within these rules, the NIRS signal decreases for the same reason as in the cases of the mental state of Anxiety (E5).

4 Integration of Association Rules Across Scenes

To evaluate the generality of the association rules extracted from Scene 1, we applied the same association rule mining procedure to Scenes 2–6.

4.1 Extraction and Interpretation of Association Rules from Each Scene

Scene 2.

Eight rules were extracted from the time-aligned categorical data set (2,375 recording points) derived from Scene 2 (support = 0.02, confidence = 0.89, lift = 1.3). Six of these are concerned with the mental state of Enjoyment (E1), and the remaining two regard the mental state of Anxiety (E5). The rules concerning the mental state Enjoyment (E1) include a relatively high NIRS signal (A2), an increase in skin conductance (B2), respiration intensity that has increased a little or remains constant (C2–C3), and the speech category Explaining (D1). Although in this scene it was the teacher’s speech acts that accounted for most of the time course, from the increases of the learners’ brain blood volumes, amount of sweat, and respiration intensity, we can infer that the teacher’s explanations were a bit difficult for the learner to grasp; that is, the teacher’s explanations added some mental burdens to the learner. However, considering the fact that the learners enjoyed the learning processes, the difficulty of the performance tasks given by the teacher might be just right.

The rules regarding the mental state of Anxiety (E5) include a relatively low NIRS signal (A4), a large increase in skin conductance (B1), and the speech category Praising (D5). In other words, the teachers’ words during the time segments corresponding to these rules were praises to the learners. We conjecture that the learners felt psychological stress due to an unease with the praises, thus showing high skin conductance.

Scene 3.

Thirteen rules were extracted from the time-aligned categorical data set (1,086 recording points) derived from Scene 3 (support = 0.02, confidence = 0.89, lift = 1.4). Eight of these are concerned with the mental state Enjoyment (E1), three with the mental state Pride (E3), and the remaining two with the mental state Anxiety (E5).

The rules concerning the mental state of Enjoyment (E1) include a relatively high NIRS signal (A2), a small or large increase in skin conductance (B1–B2), and the speech categories of Explaining (D1) and Questioning (D2). In view of the teachers’ frequent explaining and questioning as well as the increases in the learners’ brain blood volumes and skin conductance, we infer that the learners experienced some mental burdens rather than being in a relaxed state. However, considering the fact that the learners’ mental states were Enjoyment (E1), we think that in the corresponding time segments the teacher instructed in ways that were desirable to the learners.

As to the rules regarding the mental state Pride (E3), in most cases the speech category is Task Fulfillment Checking (D6). It is possible that the teachers’ acknowledgements of the learners’ fulfillment of the previously assigned calculation tasks offered the learners a sense of achievement, thus causing the learners to feel pride. Nevertheless, due to the absence of the data on the NIRS signal and skin conductance, there is not enough physiological information for us to predict the existence of this mental state.

With regard to the rules about the mental state Anxiety (E5), because the support value is lower than that of the other scenes, we lowered the threshold and extracted these two rules. The rules tell us that when the teacher was engaged in comprehension checking (D3), the learners’ NIRS signals tended to decrease and their amount of sweat tended to rise. This is perhaps because the teachers’ words placed some mental burdens on the learners. In addition, it is possible that the decrease in the NIRS signal resulted from the fact that the brain areas activated by the emotion of anxiety were different from the brain areas measured in our experiment.

Scene 4.

Four rules, all of which are concerned with the mental state Enjoyment (E1), were extracted from the time-aligned categorical data set (3,278 recording points) derived from Scene 4 (support = 0.02, confidence = 0.89, lift = 1.3). These rules demonstrate that when the teacher was giving explanations (D1), the learners’ physiological information included a low NIRS signal, unchanged respiration intensity (C3), and an increase in skin conductance (B2). This scene is unique in its low NIRS signal (A5), which means that the mental state of Enjoyment may have different patterns of brain activities in different environments or under different conditions.

Scene 5.

Seventeen rules were extracted from the time-aligned categorical data set (1,249 recording points) derived from Scene 5 (support = 0.02, confidence = 0.89, lift = 2.1). Two of these regard the mental state Enjoyment (E1), thirteen regard the mental state Anxiety (E5), and the remaining two are concerned with the mental state Shame (E6).

With regard to the rules concerning the mental state Enjoyment (E1), due to the low confidence value, we lowered the threshold and extracted the two rules. These rules include a high NIRS signal (A1), unchanged skin conductance (B3), a large increase in respiration intensity (C1), and the speech category of Alerting (D7). We infer from the moderate amount of sweat and the increase in respiration intensity that in this scene, the teacher’s alerting words were rich in jokes, and the jokes made the learners laugh, leading to an increase in the learners’ brain blood volumes.

The rules regarding the mental state Anxiety (E5) include a moderate or relatively high NIRS signal (A3–A4), a large increase or decrease in respiration intensity (C1 or C5), a large increase in skin conductance (B1), and the speech categories Repeating (D4) or Alerting (D7).

As to the rules regarding the mental state Shame (E6), respiration intensity rose in some cases but fell in other cases, which means that respiration intensity can hardly be used as a predicting factor of this mental state. Besides, the relatively high brain blood volume may result from the mental burdens produced by the teacher’s words of comprehension checking.

Scene 6.

Twenty-one rules were extracted from the time-aligned categorical data set (1,085 recording points) derived from Scene 6 (support = 0.02, confidence = 0.89, lift = 1.28). Twelve of these regard the mental state of Anxiety (E5), and the remaining nine are concerned with the mental state Shame (E6).

The rules concerning the mental state Anxiety (E5) contain a large increase in skin conductance (B1), unchanged respiration intensity (C3), and the speech category Explaining (D1). In addition, the intensity of the NIRS signal varies to a certain extent, but generally shows a relatively high level. These rules tell us that when the teacher was providing explanations, the learners’ amount of sweat and brain blood volumes rose, and they experienced unease. Perhaps this implies that the learners found it difficult to understand the teacher’s explanations, which led to an excitement of the learners’ brains.

The rules regarding the mental state Shame (E6) contain a moderate NIRS signal, an increase in skin conductance (B2), an increase in respiration intensity, and the speech category of Alerting. We think that the learners’ large amount of sweat and high respiration intensity, as well as the shame that they felt, were the direct results of the teacher’s alerting words. As to the moderate NIRS signal, it is reasonable to propose that although in this scene the teacher had spent much time pointing out the mistakes made by the learners, these words were fairly easy for the learner to understand.

4.2 Association Rules with High Generality

The rules with high generality across the scenes are summarized in Table 3. These rules described the relationships between learners’ mental states and learners’ physiological information complemented with teachers’ speech acts. These rules are introduced in detail as follows.

Table 3. The association rules with high generality across the scenes
Full size table

With regard to the rules regarding the mental state Enjoyment (E1), in most cases the teacher was offering explanations or alerting the learners. When the teacher realized that the learners had made mistakes, the teacher alerted the learners to these mistakes in a friendly tone along with some jokes, making the learners feel happy about the learning processes. We can see that the mental state of Enjoyment usually accompanies a brain blood volume with a level no less than A2 (High), as well an amount of sweat and respiration intensity that remain constant or increase (≥B3 and ≥C3). Hence, when a learner’s physiological information results in a brain blood volume of ≥A2, the amount of sweat at ≥B3, and the respiration intensity of ≥C3, the learner’s mental state is most likely that of Enjoyment (E1).

There are relatively few rules concerning the mental state Pride (E3), and these rules possess no features that are common across all scenes. Moreover, only in these rules does the speech category of Task Fulfillment Checking (D6) appear. It is therefore reasonable to infer that this mental state cannot be elicited until a very limited set of conditions have been met. All in all, we think that although these rules can provide us with various idiosyncratic information regarding the mental state Pride, it is rather difficult to predict the existence of this mental state due to the lack of features that are common across the scenes.

As to the rules concerning the mental state Anxiety (E5), we found tendencies common across the scenes in brain blood volume and amount of sweat, but not in speech category. According to these rules, when brain blood volume is at or lower than the level of A4 (Low), in most cases the amount of sweat is at the level of B1 (Much Increased), and the respiration intensity level tends to be at C1 (Much Increased). As with the cases of rules regarding the mental state Enjoyment (E1), the mental state Anxiety also couples with substantial increases in respiration intensity and amount of sweat, but the mental state of Anxiety is exceptional for its low level (≤A4) of brain blood volume.

The rules regarding the mental state Shame (E6) differ so much that no characteristics common across the scenes can be found in terms of brain blood volume, amount of sweat, or respiration intensity. The only such characteristic present in the speech category is Comprehension Checking (D3). Like the mental state Pride, the rules regarding the mental state of Shame contain too much variation that it is very difficult to predict the existence of this mental state.

In conclusion, from the four groups of rules previously introduced, we can see that it is possible to infer a learner’s mental states from the learner’s physiological information together with the teacher’s speech acts if the learner’s mental states are that of Enjoyment (E1) or Anxiety (E5).

5 Summary and Future Works

In this study, we extracted four sets of association rules with high generality that describe the relationships between learners’ mental states, especially Enjoyment and Anxiety, and learners’ physiological information complemented with teachers’ speech acts. In the future, we will further promote the accuracy of the predicting rules by employing more types of data on teachers’ speeches and behaviors. We will also pay more attention to the changes of learners’ mental states over time. Furthermore, in order to improve the generality of our study results, we plan to place more teachers and learners in a single class in our future studies. These experimental paradigms will become more convenient as EEG instruments are currently becoming less expensive and more sophisticated in function.