Keywords

1 Introduction

Estimation of emotion is one of the major interests amongst robotics interaction development researchers. Extensive research has been carried out towards this end, especially focused on extracting the rules and features of expressions such as user’s eye-gaze direction [4], head position, facial and mouth expression [5] and behaviors [6]. These approaches use observable symbols that people can sense with senses (sight, hearing, touch, etc.) to estimate emotion. However, the limitation of these approaches is that it is difficult to apply them when facial expressions or words and emotion differ from the norm. For example, people can fake a smile when feeling angry inside. In psychology, researchers consider that emotions are sometimes not expressed, and information that is expressed - such as facial expressions, voices and emotion - can differ from the true emotion. This has been widely recognized in psychology [7]. Our research is based on this understanding and aims to understand true feelings. To estimate emotion we use biological information such as brain waves and heart rate. We intend to measure the human conditions such as being excited [8], stressed [9], concentrating [10], and relaxing [11]. In particular, for emotion, based on the Circumflex Model of Russell [12], by using biometric information, various studies have performed human emotion estimation, and reported that certain evaluations can be made [10]. However, in those studies, the engineering implementation methods and algorithms have not been sufficiently presented.

The purpose of this study is to present an algorithm for the psychology model to separate the symbols and feelings, to estimate emotion with the biological information from the proposed method, and to complement it with brain waves and heart rate. The implemented system and planned experiments are intended to provide an algorithm and evaluation method for the accuracy of emotion estimate. Since the evaluation is the result of the limited method presented in this research, it is necessary to experiment by increasing the number of subjects in the future.

Structure of this paper is as follows: In Sect. 2, we propose the emotion estimate method; in Sect. 3, we propose the experiment and evaluation, and in Sect. 4, we conclude.

2 Estimation Method with Biological Information

2.1 Issues and Objectives

As we described in the introduction, we embody the technique that estimates emotion by using biological information. Firstly, we try to separate an emotion into information that is expressed externally, and interior information, in accordance with the psychological classification [7]. The expressed information, we call symbolic information that people use in communication such as words, voice and facial expression. On the other hand, the interior information we call emotional information that is internal. Without taking into account physiological phenomena such as breathing, pulse and blood pressure, it can be difficult to understand the state of the person. These biological signals have been used in areas such polygraph testing [13]. This is something that detects a number of physiological phenomena such as electrical and physical signals like breathing, pulse, blood pressure, etc. However, there has not been enough research on estimating emotion through using direct biological information from the person.

The purpose of this research is based on the separation of the symbolic and emotional in psychology, and an object is to present an algorithm using biometric information such as brain wave and heart rate speculation emotions. With this proposed method, we will achieve an accurate estimation by integrating symbolic and biological information. As a first step, we outline the proposed method to estimate emotion with the facial expression analysis technology.

2.2 Proposed Method

Firstly, we define the symbolic information and emotional information. Symbolic information is defined as that which can be easily read by the five senses.

On the other hand, emotional information is defined as that which people can hardly read by using the five senses. (Table 1) Symbolic information is standing on the premise that language and culture are shared and you can read signs of emotion to some extent from such words and expressions. Also, with the emotional information, it does not matter whether you know the person or not.

Table 1. The definition of emotion and symbol
Full size table

We describe a method of emotional decision-making. Emotion is a vague concept, and is interpreted differently by different people. Russell showed Russell’s Circumflex Model that was structured for emotion. In this research, for the classification of the emotion, the data obtained by the brain wave and pulse, and mapped onto a two-dimensional plane, allowing for the identification of emotion. This method has already been presented in the research of Sakamatsu [14], Yamamoto [15], Hayashi et al. [16], Yamamoto et al. saw that being relaxed or tense will appear due to a change in emotional information by communicating with the autonomic nervous system, and proposed an analysis technique that measures according to the model. In this paper we show that excitement and tension increases the heart rate, and skin temperature rises due to the contraction of the blood vessels. On the other hand, it has been shown that parasympathetic acting upon relaxation reduces heart rate, and dilates blood vessels. In addition, Sakamatsu shows the state of the autonomic nervous system corresponding to the degree of concentration by Russell’s Circumflex attention and meditation by brain wave a portion corresponding to the awakening degree of the model in the vertical axis.

Furthermore, in physiology, it is known that when emotional behavior occurs, the activity of the sympathetic nerve is enhanced, so we thought it would be possible to estimate emotions such as excitement, sadness and anxiety. Therefore, we thought that it applies to the horizontal axis of Russell’s Circumflex Model. The circular ring model that associates the two values is shown in Fig. 1. The estimation of emotion using this model have already been proposed by Sakamatsu [14]. In our research, the same value as Sakamatsu’s Y-axis of Russell’s Circumflex Model was assumed as the brain wave. The value is obtained with a brain wave headband [17] that the algorithm is provided as a value of attention and meditation from zero to 100. On the other hand, the pulse is known to increase with growing activity of the sympathetic nervous system in the event of emotional behavior from a physiological point of view. It is also believed to be capable of measuring emotions such as excitement, sadness and anxiety. Skamatsu’s research also uses pulse as value of X-axis in Russell’s Circumflex Model. They use the value at the pulse over a one minute interval [14]. In this research, we used pNN50. The measurement time and interval of pNN50 is shorter than the one minutes, and it is widely used in the measure of heart rate variability (HRV) [18]. As described above, the data processing emotion estimate using biometric information in this research is organized as follows.

Fig. 1.
figure 1

Russell’s circumflex model [12]

Full size image

(1) Calculate pulse calculated from pNN50 from the sensor.

(2) Calculate the degree of awakening by the value obtained by the operation of the brain waves.

(3) The value of (1) and (2) carry out the emotion estimate.

2.3 How to Apply Heart Rate and Brain Wave for the Method

The determination of pleasant - unpleasant uses the pulse using a pNN50 [18]. Determination of pNN50 is as follows:

m0~m30 of Fig. 2 shows a pulse interval. The difference between the adjacent pulse interval is Diff. Then, ask the 30 pieces of pulse interval Diff like DDiff0 = |Hbt1 – Hbt0|, DDiff1 = |Hbt2 – Hbt1|, … DiffD30 = |Hbt30 – Hbt29|. After that, the 30 pieces of 50 ms or more of the ratio of the Diff of the Diff calculated (the number of the number /Diff of 50 ms or more of Diff), the value of the ratio and pNN50 [18]. This value is determined to be pleasant higher has become an index of Pleasant - discomfort. In order to deal with the value of the pNN50 as the value of the X-axis of Russell’s Circumflex Model, we decided on a reference point. We have a value that becomes the point of origin 0.3. Because the standard value of pNN50 is 0.3 [19]. Originally, if the focus is on real-time, the accuracy is reduced, or to guarantee accuracy, real-time is reduced. It is trade-off requirement. In this research, we use pNN50, satisfying the real-time requirement without reducing accuracy.

Fig. 2.
figure 2

pNN50, Pulse interval

Full size image

Brain wave sensors used in this research implemented an algorithm that calculates the attention and meditation at a level from zero to 100. In this research, the value of the Y-axis of the Russell’s Circumflex Model is “Attention-meditation” ([14] reference). This value is referred to as an awakening degree. Awakening degree is −100 (attention: 0, meditation: 100)~100 (attention: 100, meditation: 0) takes a value in. In Russell’s Circumflex Model, an awakening degree at the origin or zero means attention = meditation.

Based on the calculation of biological information such as heart rate and brainwave, we apply Russell’s Model to estimate the emotion. Using these parameters, we determine that the “joy emotion” in the case that awakening degree is 0 or more and pNN50 is 0.3 or more.

3 Preliminary Experiment

3.1 Compared Method

In this research, in order to know the effect of our approach, we compare the two methods that use biometric information for estimating emotion, and use only the symbolic information. In this research, we use an existing determination algorithm, because analysis of symbolic information is not the purpose of this research. Facial expressions are the symbols to be estimated on the subjects because it is easier to detect than other symbolic information, such as voice and gestures. The detection of the facial expression is done using Omron’s OKAO VISION [20] (Fig. 5). OKAO VISION recognizes the face of the person as shown in Fig. 6 in the camera of the face recognition function. Next, it reads five types of facial expressions (Surprise, Anger, Natural, Sadness, Happiness). Next, it has the ability to output in the log, as shown in Fig. 7. In this research, we quantify the current facial expression from a string that shows a log of the facial expression (Happiness → 5, Angry → 2, Sadness → 4, Natural → 3), which will be passed to the algorithm, which in turn will be described later. Facial expression and emotion in this research correspond to the decision of OKAO as shown in Table 2.

Table 2. Correspondence of the judgment in the actual emotion and the determination with the parameter
Full size table

The value of the passed facial expression from OKAO VISION determines the Emotion in the “facial expression determination algorithm”. It should be noted that this algorithm is to take only the facial expression of “joy”. This algorithm determines that the “joy facial expression” in the event that the value of the facial expression is 5.

3.2 Experimental Method and Evaluation

We carried out the proposed method to test the validity of the estimation of emotion method using the symbolic/emotional model. Note that the subject was to be treated only for the emotion of “joy” in this experiment. Brain waves were detected by a Neuro sky’s Mindwave Mobile [17]. An Arduino heartbeat sensor [21] from the Tokyo devices company detected the pulse. As an evaluation index, Mean of Absolute Error (MAE) was adopted to use the broadest value as an index to measure the accuracy. We compared the MAE f symbolic information using the pulse and heart rate and biological information using facial expression. The formula for the MAE is as follows.

$$ {\text{Joy}}_{\text{MAE}} = | {\frac{\text{Number of determinate emotion while watching the video}}{\text{Number of determinate emotion while watching the video}} - \frac{\text{Number determined that the ''joy''}}{\text{Number of determinate emotion while watching the video}}}|. $$

3.3 Procedure

The participants were two people (a 21 and a 23-year-old man). The experiment was carried out in the laboratory and, in order to realize the goals of the robot study, we implemented the reaction operation on the robot too. Robot was used to switch sciences company of Rapiro [22]. Determination of the reaction operation compares the emotion determined by expression and biological information. If they were different, the robot swung its hands and face to the side as if to deny. On the other hand, if they were the same, the robot raised one hand as to rejoice together.

The experimental procedure is as follows:

(1) In order to evoke the emotional of “joy”, the subject searches for an interesting video [16];

(2) the subject watches the video;

(3) facial expressions and biometric information is detected by the sensors;

(4) facial expression determination as to whether emotions based on the value calculated from the camera is “joy”. Biometric information determined whether the emotional of “joy” on the basis of the “emotional decision algorithm” (Fig. 3).

Fig. 3.
figure 3

Situation of the experiment

Full size image

(5) comparison of the emotional of facial expression and biological information, and the robot operation.

※ 3–5 are repeated each second between the videos.

3.4 Result of the First Experiment

When participant A watched the video, his facial expression was “joy” for the whole time during the video playback, therefore, the value of the MAE became 0. However, when using the biological information, MAE increased to 28. We considered that showed the result that the facial expression has a higher accuracy because participant A has the attribute of rich facial expression (Table 3).

Table 3. Result of the first experiment
Full size table

On the other hand, participant B had no detection of “joy of facial expression” so MAE had the very high value of 100. However, participant B in the case of biological information has the detection of “joy” a lot of time, with a MAE value of 25, and accuracy is increased significantly compared with the case of facial expression. Participant B had the attribute of not being expressive. As a result, the average of MAE of all participants is facial expression 50, and the biological information was 26. In the emotion estimation of “joy”, biological information is better than symbolic information.

3.5 Issues and Improvement

In the first experiment, the number of participants is too few for understanding its effectiveness. Therefore, to establish the significance of the proposed method, an experiment was conducted with 11 participants. In addition, the next experiment added to the determination of the “comfort of emotional”. The experimental procedure is generally the same as in the first experiment, but there are some changes. First, a video was to be searched for “relaxing.” [16] to the participants. In addition, it added the operation of resting for 2 min between video views to stabilize the value of the biological information between (1) and (2).

3.6 Result of the 2nd Experiment

The experimental results are shown in Tables 4 and 5. The participants choose to search for a comforting video or search for a pleasurable video. Three of the 11 participants chose to search for a pleasurable video and eight of the participants chose the comfort option. If the average MAE is taken using the symbolic information, the joy of the video is 95, the comfort of video was 89.7. If the average of MAE in the case of estimates the emotion in the biological information, joy of the video is 57.6, comfort of video has become 39.8. From this result, the biological information and the facial expression of both of MAE of emotion of “easy” was lower. So, the accuracy of the emotion of the “easy” was higher. In addition, there was a difference of out-friendliness of the emotion by between personalities and there are differences in accuracy by the number of participants were suggested.

Table 4. nd experiment result (the video such as drawn, interesting)
Full size table
Table 5. nd experiment result (Relaxing the video)
Full size table

However, the overall value of MAE was higher compared with the experiment 1; the emotion estimation accuracy has decreased.

4 Conclusion

In this research, we present a method of determining emotion using biological information in the emotion estimation. The results of first experiment lead to the 2nd experiment but from the result of 2nd experiment, there is a problem that further accuracy has deteriorated. In future, we will develop improvements, and we hope to continue working on the problems one by one, in order to solve them.