Skip to main content
SpringerLink
Log in

Spontaneous Facial Expression Recognition by Fusing Thermal Infrared and Visible Images

  • Conference paper
Intelligent Autonomous Systems 12

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 194))

Abstract

In this paper, we propose a method for spontaneous facial expression recognition by fusing features extracted from visible and thermal infrared images. First, the active appearance model parameters and head motion features are extracted from the visible images, and several thermal statistical parameters are extracted from the infrared images. Second, a multiple genetic algorithms-based fusion method is proposed for fusing these two spectrums. We use this proposed fusion method to search for the optimal combination of a similarity measurement and a feature subset. Then, a k-nearest neighbors classifier with the optimal combination is used to classify spontaneous facial expressions. Comparative experiments implemented on the Natural Visible and Infrared Facial Expression database show the effectiveness of the proposed similarity measurement and the feature selection method, and demonstrate the fusion method’s advantage over only using visible features.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE TPAMI 31(1), 39–58 (2009)

    Article  Google Scholar 

  2. Hernández, B., Olague, G., Hammoud, R., Trujillo, L., Romero, E.: Visual learning of texture descriptors for facial expression recognition in thermal imagery. CVIU 106(2-3), 258–269 (2007)

    Google Scholar 

  3. Khan, M.M.: Cluster-analytic classification of facial expressions using infrared measurements of facial thermal features. Ph.D. Thesis, Department of Computing and Engineering, University of Huddersfield (2008)

    Google Scholar 

  4. Jarlier, S., Grandjean, D., Delplanque, S., N’Diaye, K., Cayeux, I., Velazco, M., Sander, D., Vuilleumier, P., Scherer, K.: Thermal analysis of facial muscles contractions. IEEE TAC 2(1), 2–9 (2011)

    Google Scholar 

  5. Yoshitomi, Y., Kim, S.I., Kawano, T., Kilazoe, T.: Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In: Proceedings of the 9th IEEE International Workshop on Robot and Human Interactive Communication, pp. 178–183 (2000)

    Google Scholar 

  6. Tong, Y., Chen, J., Ji, Q.: A unified probabilistic framework for spontaneous facial action modeling and understanding. IEEE TPAMI 32(2), 258–273 (2010)

    Article  Google Scholar 

  7. Valstar, M.F., Gunes, H., Pantic, M.: How to distinguish posed from spontaneous smiles using geometric features. In: ICMI 2007, pp. 38–45 (November 2007)

    Google Scholar 

  8. Gunes, H., Pantic, M.: Dimensional Emotion Prediction from Spontaneous Head Gestures for Interaction with Sensitive Artificial Listeners. In: Allbeck, J., Badler, N., Bickmore, T., Pelachaud, C., Safonova, A. (eds.) IVA 2010. LNCS, vol. 6356, pp. 371–377. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  9. Wang, S., Liu, Z., Lv, S., Lv, Y., Wu, G., Peng, P., Chen, F., Wang, X.: A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE TMM 12(7), 682–691 (2010)

    Google Scholar 

  10. Yang, L.: An overview of distance metric learning. Technical report, School of Computer Science, Carnegie Mellon University (2007)

    Google Scholar 

  11. Dash, M., Liu, H.: Feature selection for classification. Intelligent Data Analysis 1(3), 131–156 (1997)

    Article  Google Scholar 

  12. Wang, S., Zhu, H.: Musical perceptual similarity estimation using interactive genetic algorithm. In: IEEE CEC 2010, pp. 1–7 (July 2010)

    Google Scholar 

  13. Tong, Y., Wang, Y., Zhu, Z., Ji, Q.: Robust facial feature tracking under varying face pose and facial expression. Pattern Recognition 40(11), 3195–3208 (2007)

    Article  MATH  Google Scholar 

  14. Lv, Y., Wang, S.: A spontaneous facial expression recognition method using head motion and aam features. In: 2010 Second World Congress on NaBIC, pp. 334–339 (December 2010)

    Google Scholar 

  15. Lucey, S., Ashraf, A.B., Cohn, J.: Investigating spontaneous facial action recognition through aam representations of the face. Face Recognition, 275–286 (2007)

    Google Scholar 

  16. Tim Cootes, am_tools, http://www.isbe.man.ac.uk/~bim/software/am_tools_doc/.

  17. Sebe, N., Lew, M.S., Sun, Y., Cohen, I., Gevers, T., Huang, T.S.: Authentic facial expression analysis. Image and Vision Computing 25(12), 1856–1863 (2007)

    Article  Google Scholar 

  18. Li, L., Darden, T.A., Weingberg, C.R., Levine, A.J., Pedersen, L.G.: Gene assessment and sample classification for gene expression data using a genetic algorithm/k-nearest neighbor method. Combinatorial Chemistry and High Throughput Screening 4(8), 727–739 (2001)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Key Lab of Computing and Communicating Software of Anhui Province School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, P.R. China, 230027

    Shangfei Wang & Shan He

Authors
  1. Shangfei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shan He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shangfei Wang .

Editor information

Editors and Affiliations

  1. , School of Information and Communication, Sungkyunkwan University, 300 Chunchun-Dong Jangan-Ku, Kyunggi-Do, 440-746, Korea, Republic of (South Korea)

    Sukhan Lee

  2. Science and Technology, Daegu Gyeongbuk Institute of, 50-1, Sang-Ri, Hyeonpung-Myeon, Dalseong-Gun, Daegu, 711-873, Korea, Republic of (South Korea)

    Hyungsuck Cho

  3. Konkuk University, Neungdong-ro, Gwangjin-gu 120, Seoul, 143-701, Korea, Republic of (South Korea)

    Kwang-Joon Yoon

  4. , Department of Electronics Engineering, Pusan National University, Geum Jeong-ku, Pusan, 609-735, Korea, Republic of (South Korea)

    Jangmyung Lee

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Wang, S., He, S. (2013). Spontaneous Facial Expression Recognition by Fusing Thermal Infrared and Visible Images. In: Lee, S., Cho, H., Yoon, KJ., Lee, J. (eds) Intelligent Autonomous Systems 12. Advances in Intelligent Systems and Computing, vol 194. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33932-5_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-33932-5_25

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-33931-8

  • Online ISBN: 978-3-642-33932-5

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation