Abstract
Most studies in human emotion estimation focus on visible image-based analysis which is sensitive to illumination changes. Under uncontrolled operating conditions, estimation accuracy degrades significantly. In this paper, we integrate both visible images and thermal image sequences. First, to address limitations of thermal infrared (IR) images, such as being opaque to eyeglasses, we apply thermal Regions of Interest (t-ROIs) to sequences of thermal images. Then, wavelet transform is applied to visible images. Second, features are selected and fused from visible features and thermal features. Third, fusion decision using Principal Component Analysis (PCA), Eigen-space Method based on class-features (EMC), PCA-EMC is applied. Experiments on the Kotani Thermal Facial Emotion (KTFE) database show the effectiveness of proposed methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Zeng, Z., Pantic, M., Roisman, G.T., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009)
Jarlier, S., Grandjean, D., Delplanque, S., N’Diaye, K., Cayeux, I., Velazco, M., Sander, D., Vuilleumier, P., Schere, K.: Automatic facial expression analysis: a survey. Pattern Recognition 36, 259–275 (2003)
Khan, M.M., Ward, R.D., Ingleby, M.: Classifying pretended and evoked facial expression of positive and negative affective states using infrared measurement of skin temperature. Trans. Appl. Percept. 6(1), 1–22 (2009)
Trujillo, L., Olague, G., Hammoud, R., Hernandez, B.: Automatic feature localization in thermal images for facial expression recognition. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, CVPR Workshops, p. 14 (2005)
Hernández, B., Olague, G., Hammoud, R., Trujillo, L., Romero, E.: Visual learning of texture descriptors for facial expression recognition in thermal imagery. Computer Vision and Image Understanding 106, 258–269 (2007)
Nhan, B.R., Chau, T.: Classifying affective states using thermal infrared imaging of the human face. IEEE Transactions on Biomedical Engineering 57, 979–987 (2010)
Yoshitomi, Y., Miyawaki, N., Tomita, S., Kimura, S.: Facial expression recognition using thermal image processing and neural network. In: 6th IEEE International Workshop Robot and Human Communication, ROMAN 1997 Proceedings, pp. 380–385 (1997)
Yoshitomi, Y.: Facial expression recognition for speaker using thermal image processing and speech recognition system. In: Proceedings of the 10th WSEAS International Conference on Applied Computer Science, pp. 182–186 (2010)
Koda, Y., Yoshitomi, Y., Nakano, M., Tabuse, M.: A facial expression recognition for a speaker of a phoneme of vowel using thermal image processing and a speech recognition system. In: The 18th IEEE International Symposium on Robot and Human Interactive Communication, ROMAN 2009, pp. 955–960 (2009)
Wang, S., He, S., Wu, Y., He, M., Ji, Q.: Fusion of visible and thermal images for facial expression recognition. J Frontiers of Computer Science (2014)
Yoshitomi, Y., Kim, S., Kawano, T., Kilazoe, T.: Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In: Proceedings of the 9th IEEE International Workshop on Robot and Human Interactive Communication, pp. 178–183 (2000)
Antonini, M., Barlaud, M., Mathieu, P., Daubechies, I.: Image coding using wavelet transform. IEEE Trans. Image Processing 1, 205–220 (1992)
Lin, D.T.: Facial Expression Classification Using PCA and Hierarchical Radial Basis Function Network. Journal of Information Science and Engineering 22, 1033–1046 (2006)
Kurozumi, T., Shinza, Y., Kenmochi, Y., Kotani, K.: Facial Individuality and Expression Analysis by Eigenspace Method Based on Class Features or Multiple Discriminant Analysis. In: ICIP (1999)
Nguyen, H., Kotani, K., Chen, F., Le, B.: A thermal facial emotion database and its analysis. In: Klette, R., Rivera, M., Satoh, S. (eds.) PSIVT 2013. LNCS, vol. 8333, pp. 397–408. Springer, Heidelberg (2014)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Nguyen, H., Chen, F., Kotani, K., Le, B. (2014). Human Emotion Estimation Using Wavelet Transform and t-ROIs for Fusion of Visible Images and Thermal Image Sequences. In: Murgante, B., et al. Computational Science and Its Applications – ICCSA 2014. ICCSA 2014. Lecture Notes in Computer Science, vol 8584. Springer, Cham. https://doi.org/10.1007/978-3-319-09153-2_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-09153-2_17
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-09152-5
Online ISBN: 978-3-319-09153-2
eBook Packages: Computer ScienceComputer Science (R0)