Skip to main content
SpringerLink
Log in

Fusion of visible and thermal images for facial expression recognition

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Most present research into facial expression recognition focuses on the visible spectrum, which is sensitive to illumination change. In this paper, we focus on integrating thermal infrared data with visible spectrum images for spontaneous facial expression recognition. First, the active appearance model AAM parameters and three defined head motion features are extracted from visible spectrum images, and several thermal statistical features are extracted from infrared (IR) images. Second, feature selection is performed using the F-test statistic. Third, Bayesian networks BNs and support vector machines SVMs are proposed for both decision-level and feature-level fusion. Experiments on the natural visible and infrared facial expression (NVIE) spontaneous database show the effectiveness of the proposed methods, and demonstrate thermal IR images’ supplementary role for visible facial expression recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Zeng Z, Pantic M, Roisman G, Huang T. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(1): 39–58

    Article  Google Scholar 

  2. Wang Z, Liu F, Wang L. Survey of facial expression recognition based on computer vision. Computer Engineering, 2006, 32(11): 231–233

    Google Scholar 

  3. Liu X m, Tan H c, Zhang Y j. New research advances in facial expression recogntion. Journal of Image and Graphics, 2006, 11(10): 1359–1368

    Google Scholar 

  4. Xue Y L, Mao X, Guo Y, Lv S W. The research advance of facial expression recognition in human computer interaction. Journal of Image and Graphics, 2009, 14(5): 764–772

    Google Scholar 

  5. Bettadapura V. Face expression recognition and analysis: The state of the art. CoRR, 2012, abs/1203.6722

    Google Scholar 

  6. Wesley A, Buddharaju P, Pienta R, Pavlidis I. A comparative analysis of thermal and visual modalities for automated facial expression recognition. Advances in Visual Computing, 2012, 51-60

  7. Yoshitomi Y, Kim S, Kawano T, Kilazoe T. Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In: Proceedings of the 9th IEEE International Workshop on Robot and Human Interactive Communication. 2000, 178-183

  8. Wang Z, Wang S. Spontaneous facial expression recognition by using feature-level fusion of visible and thermal infrared images. In: Proceedings of the 2011 IEEE International Workshop on Machine Learning for Signal Processing. 2011, 1–6

    Chapter  Google Scholar 

  9. Wang S, He S. Spontaneous facial expression recognition by fusing thermal infrared and visible images. Intelligent Autonomous Systems, 2013, 194: 263–272

    Article  Google Scholar 

  10. Cootes T, Edwards G, Taylor C. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(6): 681–685

    Article  Google Scholar 

  11. Lv Y, Wang S. A spontaneous facial expression recognition method using head motion and AAM features. In: Proceedings of the 2nd World Congress on Nature and Biologically Inspired Computing. 2010, 334–339

    Google Scholar 

  12. Wang S, Liu Z, Lv S, Lv Y, Wu G, Peng P, Chen F, Wang X. A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Transactions on Multimedia, 2010, 12(7): 682–691

    Article  Google Scholar 

  13. Littlewort G, Whitehill J, Wu T, Butko N, Ruvolo P, Movellan J, Bartlett M. The motion in emotion — a cert based approach to the fera emotion challenge. In: Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops. 2011, 897–902

    Google Scholar 

  14. Cohn J, Reed L, Ambadar Z, Xiao J, Moriyama T. Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior. In: Proceedings of the 2004 IEEE International Conference on Systems, Man and Cybernetics. 2004, 610–616

    Google Scholar 

  15. Tong Y, Chen J, Ji Q. A unified probabilistic framework for spontaneous facial action modeling and understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(2): 258–273

    Article  Google Scholar 

  16. Valstar M, Gunes H, Pantic M. How to distinguish posed from spontaneous smiles using geometric features. In: Proceedings of the 9th international conference on Multimodal interfaces. 2007, 38–45

    Chapter  Google Scholar 

  17. Gunes H, Pantic M. Dimensional emotion prediction from spontaneous head gestures for interaction with sensitive artificial listeners. In: Proceedings of the 10th International Conference on Intelligent Virtual Agents. 2010, 6356: 371–377

    Google Scholar 

  18. Jarlier S, Grandjean D, Delplanque S, N’Diaye K, Cayeux I, Velazco M, Sander D, Vuilleumier P, Scherer K. Thermal analysis of facial muscles contractions. IEEE Transactions on Affective Computing, 2011, 2(1): 2–9

    Article  Google Scholar 

  19. Hernández B, Olague G, Hammoud R, Trujillo L, Romero E. Visual learning of texture descriptors for facial expression recognition in thermal imagery. Computer Vision and Image Understanding, 2007, 106(2): 258–269

    Article  Google Scholar 

  20. Khan M, Ward R, Ingleby M. Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature. ACM Transactions on Applied Perception, 2009, 6(1): Article 6

    Google Scholar 

  21. Yoshitomi Y. Facial expression recognition for speaker using thermal image processing and speech recognition system. In: Proceedings of the 10th World Scientific and Engineering Academy and Society Inter national Conference on Applied Computer Science. 2010, 182–186

    Google Scholar 

  22. Puri C, Olson L, Pavlidis I, Levine J, Starren J. Stresscam: non-contact measurement of users’ emotional states through thermal imaging. In: Proceedings of the 2005 Conference on Human Factors in Computing Systems. 2005, 1725–1728

    Google Scholar 

  23. Shen P, Wang S, Liu Z. Facial expression recognition from infrared thermal videos. Intelligent Autonomous Systems, 2013, 194: 323–333

    Article  Google Scholar 

  24. Buddharaju P, Pavlidis I, Manohar C. Face recognition beyond the visible spectrum. In: Advances in Biometrics, 157–180. Springer, 2008

    Chapter  Google Scholar 

  25. Pavlidis I, Levine J, Baukol P. Thermal image analysis for anxiety detection. In: Proceedings of the 2001 International Conference on Image Processing. 2001, 315–318

    Google Scholar 

  26. Cootes T. am_tools. http://www.isbe.man.ac.uk/?bim/software/am_tools_doc/

  27. Tong Y, Wang Y, Zhu Z, Ji Q. Robust facial feature tracking under varying face pose and facial expression. Pattern Recognition, 2007, 40(11): 3195–3208

    Article  MATH  Google Scholar 

  28. Ding C. Analysis of gene expression profiles: class discovery and leaf ordering. In: Proceedings of the 6th Annual International Conference on Computational Biology. 2002, 127–136

    Google Scholar 

  29. Wu T F, Lin C J, Weng R C. Probability estimates for multi-class classification by pairwise coupling. The Journal of Machine Learning Research, 2004, 5: 975–1005

    MATH  MathSciNet  Google Scholar 

  30. Chang C C, Lin C J. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2011, 2: 27:1–27:27. Software available at http://www.csie.ntu.edu.tw/?cjlin/libsvm

    Article  Google Scholar 

  31. Long P M, Servedio R A. Discriminative learning can succeed where generative learning fails. In: Proceedings of the 19th Annual Conference on Learning Theory. 2006, 4005: 319–334

    MathSciNet  Google Scholar 

  32. Ng A Y, Jordan A. On discriminative vs. generative classifiers: a comparison of logistic regression and naive bayes. Advances in Neural Information Processing Systems, 2002, 14: 841

    Google Scholar 

  33. Hermosilla G, Ruiz-del-Solar J, Verschae R, Correa M. A comparative study of thermal face recognition methods in unconstrained environments. Pattern Recognition, 2012, 45(7): 2445–2459

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, University of Science and Technology of China, Hefei, 230027, China

    Shangfei Wang, Shan He & Menghua He

  2. Key Lab of Computing and Communicating Software of Anhui Province, Hefei, 230027, China

    Shangfei Wang, Shan He & Menghua He

  3. Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY, 12180-3590, USA

    Yue Wu & Qiang Ji

Authors
  1. Shangfei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shan He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yue Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Menghua He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qiang Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shangfei Wang.

Additional information

Shangfei Wang received her MS in circuits and systems, and her PhD in signal and information processing from University of Science and Technology of China (USTC), China in 1999 and 2002, respectively. From 2004 to 2005, she was a postdoctoral research fellow in Kyushu University, Japan. She is currently an associate professor in the School of Computer Science and Technology, USTC.Dr. Wang is an IEEE member. Her research interests cover computer intelligence, affective computing, multimedia computing, information retrieval, and artificial environment design. She has authored or co-authored over 50 publications.

Shan He received his BS in Computer Science from Anhui Agriculture University, China in 2010. He received his MS in Computer Science from the University of Science and Technology of China, China in 2013. His research interest is affective computing.

Yue Wu is a PhD candidate in the Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, USA. Her research interest is computer vision.

Menghua He received her BS in Information and Computation Science from Anhui University, China in 2011. She is currently pursuing herMS in Computer Science at the University of Science and Technology of China, China. Her research interesting is affective computing.

Qiang Ji received his PhD in Electrical Engineering from the University of Washington, USA. He is currently a professor with the Department of Electrical, Computer, and Systems Engineering at Rensselaer Polytechnic Institute (RPI), USA. He recently served as a program director at the National Science Foundation (NSF), where he managed NSF’s computer vision and machine learning programs. He also held teaching and research positions with the Beckman Institute at University of Illinois at Urbana-Champaign, the Robotics Institute at Carnegie Mellon University, the Dept. of Computer Science at University of Nevada at Reno, and the US Air Force Research Laboratory, USA. Prof. Ji currently serves as the director of the Intelligent Systems Laboratory (ISL) at RPI, USA.

Prof. Ji’s research interests are in computer vision, probabilis tic graphical models, information fusion, and their applications in various fields. He has published over 160 papers in peer-reviewed journals and conferences. His research has been supported by major governmental agencies including NSF, NIH, DARPA, ONR, ARO, and AFOSR as well as by major companies including Honda and Boeing. Prof. Ji is an editor on several related IEEE and international journals and he has served as a general chair, program chair, technical area chair, and program committee member in numerous international conferences/workshops. Prof. Ji is a fellow of IAPR.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wang, S., He, S., Wu, Y. et al. Fusion of visible and thermal images for facial expression recognition. Front. Comput. Sci. 8, 232–242 (2014). https://doi.org/10.1007/s11704-014-2345-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-014-2345-1

Keywords

Search

Navigation