Skip to main content
Springer Nature Link
Log in

Facial expression recognition of a speaker using front-view face judgment, vowel judgment, and thermal image processing

  • Original Article
  • Published:
Artificial Life and Robotics Aims and scope Submit manuscript

Abstract

For facial expression recognition, we selected three images: (i) just before speaking, (ii) speaking the first vowel, and (iii) speaking the last vowel in an utterance. In this study, as a pre-processing module, we added a judgment function to distinguish a front-view face for facial expression recognition. A frame of the front-view face in a dynamic image is selected by estimating the face direction. The judgment function measures four feature parameters using thermal image processing, and selects the thermal images that have all the values of the feature parameters within limited ranges which were decided on the basis of training thermal images of front-view faces. As an initial investigation, we adopted the utterance of the Japanese name “Taro,” which is semantically neutral. The mean judgment accuracy of the front-view face was 99.5% for six subjects who changed their face direction freely. Using the proposed method, the facial expressions of six subjects were distinguishable with 84.0% accuracy when they exhibited one of the intentional facial expressions of “angry,” “happy,” “neutral,” “sad,” and “surprised.” We expect the proposed method to be applicable for recognizing facial expressions in daily conversation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Yoshitomi Y, Kimura S, Hira E, et al (1996) Facial expression recognition using infrared rays image processing. Proceedings of the Annual Convention IPS Japan, Osaka, Japan, September 4–6, 1996, 2:339–340

    Google Scholar 

  2. Yoshitomi Y, Kimura S, Hira E, et al (1997) Facial expression recognition using thermal image processing. IPSJ SIG Notes, CVIM103-3, Kyoto, Japan, January 23–24, 1997, pp 17–24

  3. Yoshitomi Y, Miyawaki N, Tomita S, et al (1997) Facial expression recognition using thermal image processing and neural network. Proceedings of the 6th IEEE International Workshop on Robot and Human Communication, Sendai, Japan, September 29–October 1, 1997, pp 380–385

  4. Sugimoto Y, Yoshitomi Y, Tomita S (2000) A method for detecting transitions of emotional states using a thermal face image based on a synthesis of facial expressions. J Robotics Auton Syst 31(3): 147–160

    Article  Google Scholar 

  5. Yoshitomi Y, Kim SIll, Kawano T, et al (2000) Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. Proceedings of the 6th IEEE International Workshop on Robot and Human Interactive Communication, Osaka, Japan, September 27–29, 2000, pp 178–183

  6. Ikezoe F, Ko R, Tanijiri T, et al (2004) Facial expression recognition for speaker using thermal image processing (in Japanese). Trans Human Interface Soc 6(1):19–27

    Google Scholar 

  7. Nakano M, Ikezoe F, Tabuse M, et al (2009) A study on the efficient facial expression using thermal face image in speaking and the influence of individual variations on its performance (in Japanese). J IEEJ 38(2):156–163

    Google Scholar 

  8. Koda Y, Yoshitomi Y, Nakano M, et al (2009) Facial expression recognition for a speaker of a phoneme of vowel using thermal image processing and a speech recognition system. Proceedings of the 18th IEEE International Symposium on Robot and Human Interactive Communication, Toyama, Japan, September 29–Octber 1, 2009, pp 955–960

  9. Yoshitomi Y (2010) Facial expression recognition for speaker using thermal image processing and speech recognition system. Proceedings of the 10th WSEAS International Conference on Applied Computer Science, Appi Kogen, Iwate, Japan, October 4–6, 2010, pp 182–186

  10. Kuno H (1994) Infrared rays engineering (in Japanese). Tokyo, IEICE, pp 22

    Google Scholar 

  11. Kuno H (1994) Infrared rays engineering (in Japanese). Tokyo, IEICE, pp 45

    Google Scholar 

  12. Yoshitomi Y, Tsuchiya A, Tomita S (1998) Face recognition using dynamic thermal image processing. Proceedings of the 7th IEEE International Workshop on Robot and Human Communication, Takamatsu, Kagawa, Japan, September 30–October 2, 1998, pp 443–448

  13. Yamazaki S, Kamakura H, Tanijiri T, et al (2004) Three-dimensional CG expression of face rotation using fuzzy algorithm and thermal face image (in Japanese). Trans Human Interface Soc 6(3): 321–331

    Google Scholar 

  14. http://julius.sourceforge.jp/

  15. Yoshitomi Y, Asada T, Shimada K, et al (2011) Facial expression recognition of a speaker using vowel judgment and thermal image processing. Proceedings of the 16th International Symposium on Artificial Life and Robotics, Beppu, Oita, Japan, January 27–29, 2011, pp 225–230

Download references

Author information

Authors and Affiliations

  1. Works Applications Co., Tokyo, Japan

    Tomoko Fujimura

  2. Division of Environmental Sciences, Graduate School of Life and Environmental Sciences, Kyoto Prefectural University, Shimogamo, Sakyo-ku, Kyoto, 606-8522, Japan

    Yasunari Yoshitomi, Taro Asada & Masayoshi Tabuse

Authors
  1. Tomoko Fujimura
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yasunari Yoshitomi
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Taro Asada
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Masayoshi Tabuse
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Yasunari Yoshitomi.

Additional information

This work was presented in part at the 16th International Symposium on Artificial Life and Robotics, Oita, Japan, January 27–29, 2011

About this article

Cite this article

Fujimura, T., Yoshitomi, Y., Asada, T. et al. Facial expression recognition of a speaker using front-view face judgment, vowel judgment, and thermal image processing. Artif Life Robotics 16, 411–417 (2011). https://doi.org/10.1007/s10015-011-0967-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10015-011-0967-z

Key words