Keywords

1 Introduction

In recent years, the proportion of mental work in everyday life has been increasing owing to the advancement of information technology in society. The load and burden due to mental work is called mental workload (MWL) [1], and it is said that the continuation of excessive MWL has effects such as fatigue, monotonous feelings, decreased attention, mental saturation, etc. Appropriate evaluation and management of the MWL is important for reducing human error and health damage [2,3,4,5].

We have evaluated the MWL using nasal skin temperature. Nasal skin temperature is an indicator that reflects autonomic nerve activity well and has demonstrated the capacity for MWL evaluation. However, there were also regions in the mouth, in addition to the nose, where the temperature fluctuated according to the MWL. Therefore, in order to capture the areas other than the nose area where the temperature fluctuates, we expanded the range to be analyzed to the entire face, and have proposed an MWL evaluation method and study [6, 7]. By this method, we can evaluate the MWL with higher precision than in the previous research. On the other hand, it has been clarified in previous studies that at the edge of the face, where there is no angle between the thermography and the object to be photographed, there is decreased emissivity measured by reflection or the like, and, as a result, the accuracy of the temperature data decreases. In extracting facial regions from thermal images by previous methods, it was found that the estimation accuracy was degraded by the manual extraction that included edge areas.

In order to solve this problem we performed face detection using machine learning; however, since there were many occurrences of false detection of clothes and background, the automatic extraction of facial regions was not accomplished. Therefore, in this study, we examined a facial-region extraction algorithm focusing on the temperature distribution characteristics of facial thermal images and aimed at the automatic extraction of the analysis range.

2 Proposed Method

Several methods for automatically detecting a facial region from a general visible light image have been proposed [8]. However, the conventional face detection method cannot be applied to thermal images because thermal images express temperature distribution as color information. Therefore, we propose a unique face detection method. Figure 1 shows the flow of the proposed method, which is as follows:

Fig. 1.
figure 1

Outline of algorithm

Full size image
  1. 1.

    Take a thermal image, including facial region, with thermography.

  2. 2.

    Set the temperature range to extract the region of the facial skin temperature from the background temperature. In this study, we set the threshold as the skin temperature range from 31 to 36 °C because we assume from 18 to 28 °C as the room temperature in the laboratory environment.

  3. 3.

    Scan the thermal image horizontally and vertically to determine the maximum area. The maximum area is defined as the largest thermal area among areas that are determined by multiplying every vertical length by every horizontal length. The length equals the number continuous pixels within the skin temperature range.

  4. 4.

    Create a histogram of the temperature distribution within the maximum area in order to find the most frequent temperature and the maximum temperature.

  5. 5.

    Extract the pixels included in the temperature range w or w′ within the maximum area to remove noise, such as hair and background. The temperature range between the most frequent temperature and the maximum temperature is defined as w. The temperature range obtained by subtracting w from the most frequent temperature is defined as w′.

  6. 6.

    Determine the facial region based on the edge of the area extracted in step 5. The facial region includes all of the pixels inside of this edge.

  7. 7.

    Extract a facial thermal image from the original image based on the region obtained in step 6.

3 Evaluation Experiment

In order to evaluate the proposed algorithm, we use a facial thermal image. In this research, we aim to evaluate the MWL. Therefore, we evaluate the proposed method by using a series of facial thermal images in which the subject’s mental state changes. We take the facial thermal image of the subject before the experiment, every 2 min from the start of the experiment, and finally after the experiment. We use eight thermal images per subject.

3.1 Experimental Protocol

The following experiment was conducted to take facial thermal images of different mental states when a test subject is under an MWL. In our experiment, ten 22-year-old test subjects in good health were tasked with performing mental arithmetic calculations in order to impose an MWL. Specifically, they were asked to solve two-digit number addition problems displayed on a PC. They were given 3 s for each problem, after which the next problem appeared, regardless of whether or not their answer was correct. After the test subjects relaxed for a few minutes in a seated position, the measurement commenced with 3 min of the rest, followed by 10 min of mental arithmetic calculation tasks, and then 3 min of rest. At the end of this 3-min rest, the measurement was stopped. The total length of the experiment was 16 min. In the measurement, infrared thermography (XA 0350 manufactured by View Ohre Imaging Co., Ltd.) was used. The thermal image size for this apparatus was recorded as 320 × 256 pixels, and the sampling period was recorded as 1 s. The facial skin emissivity was 0.98. The apparatus was set at a distance of 0.5 m from the subject’s face. The subject was sitting at room temperature kept at 24 ± 1 °C.

3.2 Experiment Results

The results are shown in Fig. 2. We were able to extract facial regions as targets of analysis in 10 subjects. According to Fig. 2(a) and (b), we could extract the facial region clearly. On the other hand, it can be seen from the extraction result of Fig. 2(c) that the cheek and mouth could not be extracted. These are considered to be removed along with the edge because the cheek is low in temperature. In addition, Fig. 2(d) shows that the neck had a temperature distribution similar to the face, and the neck and cloth part were extracted together with the face. As a result of evaluating whether the analysis range can be automatically extracted by using 80 facial thermal images, we succeeded in an automatic extraction that can be analyzed in 77 images.

Fig. 2.
figure 2

Results of extraction of facial region

Full size image

4 Discussion

The proposed MWL evaluation method requires the temperature of the nose and mouth and the average temperature of the entire face. However, despite testing with the same protocol, the nose and mouth were not extracted with several thermal images. There are several possible reasons for this, which are outlined below:

  • In some subjects, there were individual differences in the influence of the MWL resulting from the task, and the differences in the temperature exposed on the face caused by these individual differences prevented the appropriate extraction of the facial region.

  • There were individual differences in the areas of the face where the low temperature occurred.

  • There is the possibility that the proposed method has a weakness related to the inclination of the face in the horizontal direction.

To investigate the first and second hypotheses, it will be necessary to impose a variety of experiment conditions. The third hypothesis stems from the result of Fig. 2(c). The results show that the proposed method makes it possible to estimate autonomic nerve activity. In our future studies, it will be necessary to automatically correct the direction and tilt of an input image. If the facial region from the thermal image is extracted from an input image whose tilt is corrected, it is possible to extract the facial region with relatively more stability.

5 Conclusion

In this paper, we proposed an algorithm to automatically extract facial regions from thermal images by focusing on the temperature distribution of the thermal image. We succeeded in an automatic extraction that can be analyzed in 77 images out of 80 facial thermal images. As a result, we showed that facial regions can be automatically extracted using the proposed method, and, by using this method, our study is closer to the evaluation of the MWL in real time. However, the extraction of the face from several thermal images did not go well: something prevented the extraction of the cheek and mouth from the thermal image. Therefore, we continue to aim for the stable extraction of the facial region.

In future, in order to estimate the autonomic nerve activity in a real environment, we aim for a stable facial region extraction. For that purpose, we will develop algorithms that can respond to high temperature environments, such as in summer, and the effect of the subject’s hair style and clothing. We will examine facial region extraction for facial thermal images taken in various environments.