Keywords

1 Introduction

In recent years, healthcare has been performed to record the daily exercise amount and travel distance based on physical sensor data such as six axis sensors and GPS on mobile terminals such as smartphones [1]. In addition to this, life logs have been performed to record daily health by recording not only data from physiological sensor, but also data from physiological sensors such as skin temperature sensors and pulse sensors of wearable terminals such as smart watch [2]. Based on the time-series data of these sensors, context-aware computing [3] that provides optimal content in consideration of not only the current state of the user, but also the previous and next states have been proposed. Also, emotion-aware computing [4] that provides contents suitable for the estimation result of the user’s emotions based on these sensor data has been proposed.

Numerous methods for estimating human emotions based on physical sensor data have been studied, such as image analysis [5] and speech recognition [6]. Emotion estimation using image analysis shows more than 90% accuracy by using machine learning [5], emotion estimation using speech recognition is necessary for realizing natural dialogue between people and things [6]. However, in the emotion estimation by image analysis, the cultures that are hard to express emotion by facial expression [7], the complexity of the processing the background of the subject [8], etc. are problems, and in emotion estimation by voice recognition, the environment and personality of the subject that cannot be obtained by sensor data is necessary to take in account [6].

Methods for estimating emotions using physiological sensor data have already been proposed [9,10,11]. Nasoz et al. and Kim et al. estimated the emotion from biological signal such as perspiration, pulse, skin temperature, etc. using a noninvasive simple sensor [9, 11]. In the existing study of Sakamatsu et al. [12] it has been shown that emotion estimation using biological signal such as pulse is effective for prevention and improvement of deterioration of mental health condition. In addition, Ikeda et al., Hiramatsu et al., Koelstra et al. showed that the emotion estimation result by biological signal can fully explain subjective emotional evaluation [10, 13, 14]. On the other hand, in these studies, each indicator of biological signal is analyzed as an independent or integrated indicator, and interaction between indicators and subjective evaluation are not fully taken into consideration.

In this research, in order to solve the above problem, an evaluation was done against an emotion estimation method using biometric emotion as the answer to know to what extent people can understand and express their emotions. In evaluating multiple biometric signal, emotions were defined by the emotional model by Mehrabian et al. and the authenticity of the subjective evaluation was evaluated by the EQ in order to know how well the subjective evaluation can expresses the evaluator. For emotion estimation, a general stimulus was used to avoid limiting the application environment of the emotion estimation method. In addition, emotion estimation was performed using two biological signals, which were brain wave and of pulse.

In emotion estimation experiments using biological signal conducted by Sakamatsu, Ikeda, Hiramatsu et al., it was found that self-evaluated emotion and biometric emotion against a given stimuli were correlated. However, in the experiments that they performed, the stimulus for evoking emotions was limited, the lag before biological signal shows reaction was not considered, and there is not enough discussion about the validity of the subjective evaluation in the first place. In solving these problems, we describe the details of the emotion estimation method, evaluation method and its index used in our research, and the stimulus used to evoke emotion.

2 Emotion Estimation by Biological Signal

2.1 Biological Signals Used

Means of estimating emotions based on biometric signal have been actively studied in recent years. In particular, the PAD model by Mehrabian et al. evaluate emotion by Pleasure (degree of comfortableness to a certain event), Arousal (how active and bored one feels), and Dominance (how much control you have or how obedient you are) [15]. In addition, Russell et al. proposed a circular model of emotion by Valence and Arousal (Circumplex Model of Affect) [16] using Pleasure and Arousal, which relates to the PAD model of Mehrabian.

This model suggests that emotions are plotted on a circle on a two-dimensional coordinate axis. Many methods of classifying emotions using this circular model have been proposed, and Tanaka et al. plotted the position of emotion using EEG and nose thermal image processing on the circumplex model and visualized them [17]. Here, Tanaka et al. proposed a method to associate brain waves to Arousal and nasal skin temperature to Valence, and estimated emotion using these indices [17]. In addition, Ikeda et al. proposed a method to estimate emotion by correlating the value obtained by pulse measurement rather than nose thermal image processing with Arousal [13]. In this proposal, Ikeda et al. showed that there is little error between self-evaluated emotion estimation biometric emotion. Therefore, in this study like Ikeda et al., we used brain waves and pulse to estimate emotion.

2.2 The Biometric Emotion

In this study, the values obtained from the brain waves and pulses were calculated so as to correspond to the Arousal axis and the Valence axis of Russell’s circumplex model, and the values of Arousal and Valence were plotted on the two-dimensional coordinate.

The brain wave value associated with the Arousal axis was measured using an electroencephalograph called NeuroSky’s MindWave Mobile [18]. We used the value Attention and Meditation calculated by this electroencephalograph. Attention and Meditation are each a value indicating the degree of concentration and the resting degree of the person, and are calculated at the level of 0 to 100. From this, in this study, we assumed that the difference between the value of Attention and Meditation was appropriate to express the degree of arousal of a person and corresponded to the value of Arousal axis of Russell’s circular model.

The value of the Valence axis was correlated with the pulse rate earned by the Sparkfun’s Pulse Sensor [19]. This sensor measures pulse rate by photoelectric volumetric pulse wave recording method, and pNN50 was used as a pulse value corresponding to the Valence axis. The pNN50 shows the rate at which the difference between the 30 adjacent RR intervals exceeds 50 ms. Generally, pNN50 is said to indicate the degree of tension of the vagus nerve, and the smaller the value, the more tense/uncomfortable a person is. Therefore, it can be said that when someone is normal/pleasant, the RR interval exceeds 50 ms for a fair amount. From this, pNN50 was calculated at a rate of 0 to 1.0, and the value was correlated with the Valence axis.

3 Emotion Estimation by Self-evaluation

3.1 Self-evaluated Emotion

In a study of emotion estimation using biometric signal, a comparison between the proposed method and subjective evaluation is generally done. Here, the subjective evaluation is a subjective judgment of what kind of emotion was evoked for the presented stimulus. For example, evaluating a horror image as “scary” when viewing it is a subjective evaluation for that image.

3.2 Evaluation Method

There is SAM (Self-assessment Mannequin) used by Lang et al. [20] as a method for quantitatively measuring subjective evaluation for presented stimuli. SAM is a system that evaluates continuously varying scales for three indicators of PAD model by Mehrabian et al. using images imitating human beings. Regarding the Pleasure index, it is treated as a Valence index like Russell et al. It is displayed in the range of the image from an image of a person frowning to an image of a person smiling, and the Arousal index displayed in the range of an image of a person whose eyes are opened largely with a big impulse, to a person whose eyes are closed shut. For the Dominance index, it is displayed from the image of a person in a small corner to a person who protrudes the frame and is folding the arm. Each index is scaled in 9 stages, where 9 is the highest rating (high Valence, high Arousal, high Dominance), 1 is the lowest rating (low Valence, low Arousal, low Dominance).

In this research, only Valence and Arousal were evaluated according to the biological signal used (Fig. 1).

Fig. 1.
figure 1

SAM used in the experiment.

Full size image

4 Evaluation of Emotion Estimation by EQ

4.1 The Emotional Intelligence Quotient

There are many studies showing the usefulness of biometric signal to estimate a person’s emotion by showing that there is only a small difference between the biometric emotion and the self-evaluated emotion, but the correctness of the self-evaluation has not been discussed enough in the first place. Therefore, in this study, an index called EQ (Emotional Intelligence Quotient) was used to evaluate the correctness of subjective evaluation.

The EQ index is a numerical value of the intelligence of the mind (EI, Emotional Intelligence), which is an ability to perceive the feelings of self and other people in general and to control their own emotions. In this research, we used a method [21] to evaluate each ability by dividing EQ proposed by Goleman et al. into four fields.

The four areas are “Self-Awareness”, “Self-Management”, “Social Awareness” (understanding of emotions) and “Relationship Management” (emotional adjustment). To quantitatively evaluate these, we used the questionnaire created by Takayama et al. [22].

4.2 Experiment Conducted

The purpose of the experiment was to estimate the emotion based on the brain waves and pulses, to clarify the difference from the subjective evaluation, and to clarify what effect the EQ has on it, when a generalized stimulus is presented. In an environment at room temperature of 26° and humidity of 58%, experiments were conducted with 20 subjects (10 males, 10 females, average age 22 years) according to the following procedure.

  1. 1.

    Answer to questionnaire about EQ

  2. 2.

    Attach EEG, pulse sensor and rest for 30 s

  3. 3.

    Present standby image for 5 s

  4. 4.

    Present images that evoke emotion for 6 s

  5. 5.

    Perform a subjective evaluation on the presented image for 15 s

  6. 6.

    Repeat steps 3 to 5 for each image corresponding to the four quadrants of Russell’s circular model (a total of 4 times).

Subjective evaluation needs to be done immediately after the stimulus is presented to obtain a fresh subjective assessment of the stimulus. In order to realize this, we developed a subjective evaluation system using a web browser. In addition, images registered in IAPS (International Affective Picture System) [20] were used as stimuli (Fig. 2).

Fig. 2.
figure 2

System used to present stimulus.

Full size image

4.3 Experiment Results

Figure 3 shows the pNN50 and the degree of arousal of two subjects. For the subject A, there is a tendency to decrease after an increases for both pNN50 and degree of arousal after stimulus presentation. On the other hand, for subject B, while pNN50 continues to decrease, it can be confirmed that the degree of arousal has increased after the initial decrease. While subject A had a high EQ score, subject B didn’t have such a high score. Therefore, it can be said that people with high EQ score have a tendency to self-evaluate accordingly to biometric signals, whereas people with low EQ score do not.

Fig. 3.
figure 3

Time-lapse data of Pleasure and Attention of 2 subjects.

Full size image

We also conducted a t-test on the average of the differences between biometric signal and subjective evaluation for the presented images of the group with low EQ ability and the average of the difference between the biometric signal and the subjective evaluation for the presented images of the group with high EQ ability. To this end, the difference between each biometric signal and subjective evaluation was obtained by the following formula. In applying this formula, each biological signal was similarly scaled in 9 steps in order to be comparable with SAM which is a subjective evaluation.

$$ \begin{aligned} {\text{Valence}}\;{\text{Difference}}\; = \;|{\text{Biometric}}\;{\text{Valence}}\;{\text{Evaluation}}({\text{pNN}}50)\; - \hfill \\ {\text{SelfValence}}\;{\text{Evaluation}}({\text{Valence}}\;{\text{SAM}})| \hfill \\ \end{aligned} $$
(1)
$$ \begin{aligned} {\text{Arousal}}\;{\text{Difference}}\; = \;|{\text{Biometric}}\;{\text{Arousal}}\;{\text{Evaluation}}({\text{Degree}}\;{\text{of}}\;{\text{Arousal}})\; - \hfill \\ {\text{Self}}\;{\text{Arousal}}\;{\text{Evaluation}}({\text{Arousal}}\;{\text{SAM}})| \hfill \\ \end{aligned} $$
(2)

For all four EQ abilities, there was no decisive significant difference in the Valence evaluation in the classification based on the high and low of its abilities. However, a significant tendency (p < .15) was observed for classification by “Self-Awareness” (Fig. 4).

Fig. 4.
figure 4

T-test on the difference of Valence by biometric emotion and self-evaluated emotion.

Full size image

Also, there was no significant difference in Arousal evaluation between classification by “Self-Management” and classification by “Social Awareness”. However, from Fig. 5, a significant difference was found at p < .05 in the classification based on the ability to “Self-Awareness” and classification by “Relationship Management”.

Fig. 5.
figure 5

T-test on the difference of Arousal by biometric emotion and self-evaluated emotion.

Full size image

4.4 Consideration and Discussion

Based on the experiment results, the difference in valence depends on the subjects’ “Self-Awareness”, while the difference in arousal depends on the subjects’ “Self-Awareness” and “Relationship Management”.

  1. 1.

    “Self-Awareness”

    “Self-Awareness” shows how well you can understand and accurately assess your own emotional state. “Self-Awareness” is the ability to perceive emotion, so it can be comprehended as the ability to correctly recognize the response to a stimulus as an emotion.

  2. 2.

    “Relationship Management”

    “Relationship Management” expands your awareness to others, showing how well you can emphasize to others’ emotions. It can be comprehended as the ability to manipulate one’s own emotions accordingly to the given stimuli.

From these facts, when a person has high “Self-Awareness”, it is possible to accurately ascertain the biological signal as a subjective evaluation, and when a person has high “Relationship Management”, he/she can change subjective evaluation according to biological data. Therefore, it can be considered that the higher one’s “Self-Awareness” and “Relationship Management” is, the less the difference between emotion by biometric data and emotion by subjective evaluation is.

5 Conclusion

In this study, we conducted a validation study on the emotion evaluation using biometric data by the highs and lows of EQ quadrants. As a result, it was suggested that the “Self-Awareness” and “Relationship Management” are effective as an indicator of the credibility of the subjective evaluation of arousal degree. However, as an indicator of the credibility of the subjective evaluation of valence, none of the quadrants were suggested to be effective. The reason for this could possibly the use of image as the stimuli. Images are useful as a general stimulus, but it does not move and therefore lack dynamism, which proved to be insufficient as a stimulus for evoking emotion. To solve this, it is necessary to use stimuli that sufficiently evoke emotions to the extent where generality is not lost.