Skip to main content
Springer Nature Link
Log in

Human facial expression recognition using curvelet feature extraction and normalized mutual information feature selection

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

To recognize expressions accurately, facial expression systems require robust feature extraction and feature selection methods. In this paper, a normalized mutual information based feature selection technique is proposed for FER systems. The technique is derived from an existing method, that is, the max-relevance and min-redundancy (mRMR) method. We, however, propose to normalize the mutual information used in this method so that the domination of the relevance or of the redundancy can be eliminated. For feature extraction, curvelet transform is used. After the feature extraction and selection the feature space is reduced by employing linear discriminant analysis (LDA). Finally, hidden Markov model (HMM) is used to recognize the expressions. The proposed FER system (CNF-FER) is validated using four publicly available standard datasets. For each dataset, 10-fold cross validation scheme is utilized. CNF-FER outperformed the existing well-known statistical and state-of-the-art methods by achieving a weighted average recognition rate of 99 % across all the datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Abd El Meguid M, Levine M (2014) Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers. IEEE Trans Affect Comput 5(2):141–154

    Article  Google Scholar 

  2. Battiti R (1994) Using mutual information for selecting features in supervised neural net learning. IEEE Trans Neural Netw 5(4):537–550

    Article  Google Scholar 

  3. Baudat G, Anouar F (2000) Generalized discriminant analysis using a kernel approach. Neural Comput 12(10):2385–2404

    Article  Google Scholar 

  4. Baum LE (1972) An equality and associated maximization technique in statistical estimation for probabilistic functions of markov processes. Inequalities 3:1–8

    Google Scholar 

  5. Belhumeur PN, Hespanha JP, Kriegman D (1997) Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Article  Google Scholar 

  6. Bettadapura V (2012) Face expression recognition and analysis: the state of the art. arXiv preprint. arXiv:12036722

  7. Candes E, Demanet L, Donoho D, Ying L (2006) Fast discrete curvelet transforms. Multiscale Modeling & Simulation 5(3):861–899

    Article  MathSciNet  MATH  Google Scholar 

  8. Candes EJ, Donoho DL (2000) Curvelets, multiresolution representation, and scaling laws. In Proc. SPIE, vol 4119, pp 1–12

  9. Dash M, Liu H (1997) Feature selection for classification. Intell Data Anal 1(3):131–156

    Article  Google Scholar 

  10. Estevez PA, Tesmer M, Perez CA, Zurada JM (2009) Normalized mutual information feature selection. IEEE Trans Neural Netw 20(2):189–201

    Article  Google Scholar 

  11. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36(1):259–275

    Article  MATH  Google Scholar 

  12. Filko D, Martinović G (2013) Emotion recognition system by a neural network based facial expression analysis. Automatika: Journal Control Meas Electron Comput Commun 54(2):263– 272

    Google Scholar 

  13. Fodor IK (2002) A survey of dimension reduction techniques

  14. Georghiades AS, Belhumeur PN, Kriegman D (2001) From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans Pattern Anal Mach Intell 23(6):643–660

    Article  Google Scholar 

  15. Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6):7714–7734

    Article  Google Scholar 

  16. Hall MA (1999) Correlation-based feature selection for machine learning. PhD thesis. The University of Waikato

  17. He S, Wang S, Lv Y (2011) Spontaneous facial expression recognition based on feature point tracking. In 2011 Sixth International Conference on Image and Graphics (ICIG). IEEE, pp 760–765

  18. Hossain MA, Sanyal G (2012) A new improved tactic to extract facial expressions based on genetic algorithm and wvdf. Int J Adv Inf Technol 2(5):37–44

    Article  Google Scholar 

  19. Kamimura R (2011) Structural enhanced information and its application to improved visualization of self-organizing maps. Appl Intell 34(1):102–115

    Article  MathSciNet  Google Scholar 

  20. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In Proceedings. Fourth IEEE International Conference onAutomatic Face and Gesture Recognition, 2000. IEEE, pp 46–53

  21. Kashyap KL, Shantaiya S (2012) Study and analysis of statistical features of face expression in noisy environment. Int J Image Process Vis Sci 1(2):29–34

    Google Scholar 

  22. Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187

    Article  MathSciNet  Google Scholar 

  23. Kwak N, Choi CH (2002) Input feature selection for classification problems. IEEE Trans Neural Netw 13(1):143–159

    Article  Google Scholar 

  24. Liu M, Shan S, Wang R, Chen X (2014a) Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2014. CVPR 2014

  25. Liu P, Han S, Meng Z, Tong Y (2014b) Facial expression recognition via a boosted deep belief network. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2014, CVPR 2014

  26. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In Proceedings. Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998. IEEE, pp 200–205

  27. Mika S (2002) Kernel fisher discriminants. PhD thesis, Universitätsbibliothek

  28. Mika S, Ratsch G, Weston J, Scholkopf B, Mullers K (1999) Fisher discriminant analysis with kernels. In Proceedings of the 1999 IEEE Signal Processing Society Workshop. Neural Networks for Signal Processing IX, 1999. IEEE, pp 41–48

  29. Pagariya RR, Bartere MM (2013) Facial emotion recognition in videos using hmm 3(4):111–118

  30. Peng H, Long F, Ding C (2005) Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 27(8):1226–1238

    Article  Google Scholar 

  31. Piatkowska E, Martyna J (147) Spontaneous facial expression recognition: automatic aggression detection. In Hybrid Artificial Intelligent Systems. Springer

  32. Rabiner L (1989) A tutorial on hidden markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286

    Article  Google Scholar 

  33. Ramirez-Gutierrez K, Sanchez-Perez D, Perez-Meana H, Fujita H, Sasaki J (2010) Face recognition and verification using histogram equalization Selected Topics in Applied Computer Science. WSEAS, pp 85–89

  34. Ramirez Rivera A, Castillo R, Chae O (2013) Local directional number pattern for face analysis: Face and expression recognition. IEEE Trans Image Process 22(5):1740–1752

    Article  MathSciNet  Google Scholar 

  35. Russell JA (2003) Core affect and the psychological construction of emotion. Psychol Rev 110(1):145

    Article  Google Scholar 

  36. Samaria FS (1994) Face recognition using hidden markov models. PhD thesis. University of Cambridge

  37. Schels M , Schwenker F (2010) A multiple classifier system approach for facial expressions in image sequences utilizing gmm supervectors. In 2010 20th International Conference on Pattern Recognition (ICPR). IEEE, pp 4251–4254

  38. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: A comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  39. Siddiqi MH, Lee S, Lee YK, Khan AM, Truc PTH (2013) Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13(12):16682–16713

    Article  Google Scholar 

  40. Siddiqi MH, Ali R, Khan AM, Kim ES, Kim GJ, Lee S (2014) Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection. Multimedia Systems

  41. Smith BM, Brandt J, Lin Z, Zhang L (2014) Nonparametric context modeling of local appearance for pose-and expression-robust facial landmark localization

  42. Tang M, Chen F (2013) Facial expression recognition and its application based on curvelet transform and pso-svm. Optik-Int J Light Electron Optics 124(22):5401–5406

    Article  MathSciNet  Google Scholar 

  43. Uddin MZ, Lee J, Kim TS (2009) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55(4):2216–2224

    Article  Google Scholar 

  44. Wang S, Liu Z, Lv S, Lv Y, Wu G, Peng P, Chen F, Wang X (2010) A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Trans Multimedia 12(7):682–691

    Article  Google Scholar 

  45. Zhang S, Zhao X, Lei B (2012) Robust facial expression recognition via compressive sensing. Sensors 12(3):3747–3761

    Article  Google Scholar 

  46. Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors 11(10):9573–9588

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. 2013-067321)).

This research was also supported by the MSIP (Ministry of Science, ICT & Future Planning), Korea, under the ITRC (Information Technology Research Center) support program supervised by the NIPA (National IT Industry Promotion Agency) (NIPA-2014-(H0301-14-1003).

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Kyung Hee University, Suwon, 446–701, Republic of Korea

    Muhammad Hameed Siddiqi, Rahman Ali, Muhammad Idris & Sungyoung Lee

  2. Division of Information and Computer Engineering, Ajou University, Suwon, 443–749, Republic of Korea

    Adil Mehmood Khan

  3. Department of Electronic Engineering, Kwangwoon University, Seoul, 139–701, Republic of Korea

    Eun Soo Kim

  4. Division of Digital Media Engineering, Sang-Myung University, Suwon, 110–809, Republic of Korea

    Min Cheol Whang

Authors
  1. Muhammad Hameed Siddiqi
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Rahman Ali
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Muhammad Idris
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Adil Mehmood Khan
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Eun Soo Kim
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Min Cheol Whang
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Sungyoung Lee
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Sungyoung Lee.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Siddiqi, M.H., Ali, R., Idris, M. et al. Human facial expression recognition using curvelet feature extraction and normalized mutual information feature selection. Multimed Tools Appl 75, 935–959 (2016). https://doi.org/10.1007/s11042-014-2333-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-2333-3

Keywords