Skip to main content
SpringerLink
Log in

Accurate and robust facial expression recognition system using real-time YouTube-based datasets

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

This paper presents an accurate and robust real-time FER system. In this system, an unsupervised technique based on active contour (AC) model is adopted in order to detect and extract the human faces automatically from the facial expression frames. In this model, the combination of two energy functions like Chan-Vese (CV) energy and Bhattacharyya distance functions were exploited that not only minimize the dissimilarities within the object (face) but also maximize the distance between the object (face) and background. Moreover, we extracted the facial features by proposing a new feature extraction method in order to solve the limitations of the previous works of the feature extraction. Similarly, in this system, we also proposed the usage of a robust non-linear feature selection method called stepwise linear discriminant analysis (SWLDA) that focuses on selecting localized features from facial expression images and discriminating their classes based on regression values (i.e., partial F-test). Finally, the system has been trained by employing hidden Markov model (HMM) to label the expressions. Unlike most of the previous works that were evaluated using a single dataset in a controlled environment, the performance of the proposed system have been assessed by employing three different spontaneous datasets that have been collected in naturalistic environments. 10–fold cross validation rule has been exploited for the whole experiments. In last, a set of experiments were also performed to assess the effectiveness of each module of the proposed approaches separately. The proposed system achieved weighted average recognition rate (95%) across three different YouTube-based datasets against the existing state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Abidin Z, Harjoko A (2012) A neural network based facial expression recognition using fisherface. Int J Comput Appl 59(3):30–34

    Google Scholar 

  2. Aguilar-Torres G, Toscano-Medina K, Sanchez-Perez G, Nakano-Miyatake M, Perez-Meana H (2009) Eigenface-gabor algorithm for feature extraction in face recognition. Intern J Comput 3(1):20–30

    Google Scholar 

  3. Ahsan T, Jabid T, Chong UP et al (2013) Facial expression recognition using local transitional pattern on gabor filtered facial images. IETE Tech Rev 30(1):47–52

    Article  Google Scholar 

  4. Al-Allaf ONA (2014) Review of face detection systems based artificial neural networks algorithms. arXiv:1404.1292

  5. Ali HB, Powers DMW, Jia X, Zhang Y (2015) Extended non-negative matrix factorization for face and facial expression recognition. Intern J Mach Learn Comput 5(2):142

    Article  Google Scholar 

  6. Azeem A, Sharif M, Raza M, Murtaza M (2013) A survey: face recognition techniques under partial occlusion. Intern Arab J Inform Technol 11(1):1–10

    Google Scholar 

  7. Lal Banchhor B, Sharma T (2014) Hybrid approach for face detection using skin color based segmentation and edge detection. Skin 3(6):7252–7257

    Google Scholar 

  8. Baudat G, Anouar F (2000) Generalized discriminant analysis using a kernel approach. Neural Comput 12(10):2385– 2404

    Article  Google Scholar 

  9. Baum LE (1972) An equality and associated maximization technique in statistical estimation for probabilistic functions of markov processes. Inequalities 3:1–8

    Google Scholar 

  10. Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Article  Google Scholar 

  11. Bettadapura V (2012) Face expression recognition and analysis: the state of the art. arXiv:1203.6722

  12. Bettadapura VK (2009) Face expression recognition and analysis: the state of the art. Emotion:1–27

  13. Bhati D, Gupta V (2015) Survey–a comparative analysis of face recognition technique. Neural Comput 3 (2):597–609

    Google Scholar 

  14. Bhutani Tarika, Arora Rashmi, Juneja Ronak (2014) Face recognition using artificial neural network. Intern J Res 1(8):1374–1377

    Google Scholar 

  15. Boutsidis C, Mahoney MW, Drineas P (2008) Unsupervised feature selection for principal components analysis. In: Proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 61–69

  16. Buciu I, Pitas I (2004) Application of non-negative and local non negative matrix factorization to facial expression recognition. In: Proceedings of the 17th international conference on pattern recognition, ICPR 2004, vol 1. IEEE, pp 288–291

  17. Cai Y Invariant local features for face detection. Department of Computer Science, University of British Columbia

  18. Rojas Castillo JA, Ramirez Rivera A, Chae O (2012) Robust facial recognition based on local gaussian structural pattern. Int J Innov Comput Appl, Inf Control 8(12):8399–8413

    Google Scholar 

  19. Chan TF, Vese LA (2001) Active contours without edges. IEEE Trans Image Process 10(2):266–277

    Article  MATH  Google Scholar 

  20. Chitra S, Balakrishnan G (2012) A survey of face recognition on feature extraction process of dimensionality reduction techniques. J Theor Appl Inform Technol 36(1):92–100

    Google Scholar 

  21. Cynthia Christabel S, Annalakshmi M, Prince Winston D et al (2013) Facial feature extraction based on local color and texture for face recognition using neural network. Intern J Sci Eng Appl 2(4):78–82

    Google Scholar 

  22. Dai B, Zhang D, Liu H, Sun S, Li K (2009) Evaluation of face recognition techniques. In: International conference on photonics and image in agriculture engineering (PIAGENG 2009). International Society for Optics and Photonics, pp 74890M–74890M

  23. Datta AK, Datta M, Banerjee PK (2015) Face detection and recognition: theory and practice. CRC Press, Boca Raton

    Book  Google Scholar 

  24. Delac K, Grgic M, Liatsis P (2005) Appearance-based statistical methods for face recognition. In: 47th international symposium ELMAR-2005, pp 151–158

  25. Deotale N, Vaikole SL, Sawarkar SD (2010) Face recognition using artificial neural networks. In: 2010 the 2nd international conference on computer automation engineering (ICCAE), vol 2. IEEE, pp 446–450

  26. Ding C, Tao D (2015) A comprehensive survey on pose-invariant face recognition. arXiv:1502.04383

  27. Gooya DPES, Gripon V (2015) Automatic face recognition using sift and networks of tagged neural cliques. In: The seventh international conference on advanced cognitive technologies and applications, pp 57–61

  28. Endeshaw S, Raimond K (2015) Face recognition using artificial neural network. Zede J 25:43–52

    Google Scholar 

  29. Erdogan H (2004) Subspace kernel discriminant analysis for speech recognition. In: COST278 and ISCA tutorial and research workshop (ITRW) on robustness issues in conversational interaction

  30. Fu Z, Huang F, Sun X, Vasilakos A, Yang C-N (2016) Enabling semantic search based on conceptual graphs over encrypted outsourced data. IEEE Trans Serv Comput

  31. Garg A, Choudhary V (2012) Facial expression recognition using principal component analysis. International Journal of Scientific Research Engineering & Technology (IJSRET) 1(3):39–42

    Google Scholar 

  32. Ghimire D, Lee J (2013) A robust face detection method based on skin color and edges. J Inf Process Syst 9(1):141–156

    Article  Google Scholar 

  33. Gou G, Huang D, Wang Y (2012) A hybrid local feature for face recognition. In: PRICAI 2012: trends in artificial intelligence. Springer, pp 64–75

  34. Hajiarbabi M, Askari J, Sadri S, Saraee M (2008) A new linear appearance-based method in face recognition. In: Advances in communication systems and electrical engineering. Springer, pp 579–587

  35. Halder A, Jati A, Singh G, Konar A, Chakraborty A, Janarthanan R (2012) Facial action point based emotion recognition by principal component analysis. In: Proceedings of the international conference on soft computing for problem solving (SocProS 2011) December 20–22, 2011. Springer, pp 721–733

  36. Happy SL, Routray A (2015) Robust facial expression classification using shape and appearance features. In: 2015 eighth international conference on advances in pattern recognition (ICAPR). IEEE, pp 1–5

  37. Hernandez-Matamoros A, Bonarini A, Escamilla-Hernandez E, Nakano-Miyatake M, Perez-Meana H (2016) Facial expression recognition with automatic segmentation of face regions using a fuzzy based classification approach. Knowl-Based Syst 110:1–14

    Article  Google Scholar 

  38. Huang Z-L, Lu Z-M, Yu F-X, Zhang Y (2014) A fast face detection scheme utilizing hsv-based skin color model with kl transform. Inf Technol J 13(1):183

    Article  Google Scholar 

  39. Jabid T, Kabir H, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI J 32(5):784–794

    Article  Google Scholar 

  40. Jang U, Lee EC (2015) Comparative analysis on performance of pixel-based face recognition methods by considering various facial poses. In: Advanced science and technology letters, vol 116. Springer, pp 278–281

  41. Jia Q, Gao X, Guo H, Luo Z, Wang Y (2015) Multi-layer sparse representation for weighted lbp-patches based facial expression recognition. Sensors 15(3):6719–6739

    Article  Google Scholar 

  42. Jia Q, Liu Y, Guo H, Luo Z, Wang Y (2011) A sparse representation approach for local feature based expression recognition. In: 2011 international conference on multimedia technology (ICMT). IEEE, pp 4788–4792

  43. Jonnalagedda MV, Doye DD (2015) Radially defined local binary patterns for facial expression recognition. Int J Comput Appl 119(21)

  44. Kabir H, Jabid T, Chae O (2010) A local directional pattern variance (ldpv) based face descriptor for human facial expression recognition. In: 2010 seventh IEEE international conference on advanced video and signal based surveillance (AVSS). IEEE, pp 526–532

  45. Kailath T (1967) The divergence and bhattacharyya distance measures in signal selection. IEEE Trans Commun Technol 15(1):52–60

    Article  MathSciNet  Google Scholar 

  46. Kalita J, Das K (2013) Recognition of facial expression using eigenvector based distributed features and euclidean distance based decision making technique. Int J Adv Comput Sci Appl (IJACSA) 4(2):196–202

    Google Scholar 

  47. Kapoor R, Gupta R (2014) Morphological mapping for non-linear dimensionality reduction. IET Comput Vis 9(2):226–233

    Article  Google Scholar 

  48. Karande KJ, Talbar SN, Inamdar SS (2012) Face recognition using oriented laplacian of gaussian (olog) and independent component analysis (ica). In: 2012 second international conference on digital information and communication technology and it’s applications (DICTAP). IEEE, pp 99–103

  49. Kass M, Witkin A, Terzopoulos D (1988) Snakes: active contour models. Int J Comput Vis 1(4):321–331

    Article  MATH  Google Scholar 

  50. Kittusamy SRV, Chakrapani V (2012) Facial expressions recognition using eigenspaces. J Comput Sci 8 (10):1674–1679

    Article  Google Scholar 

  51. Krusienski DJ, Sellers EW, McFarland DJ, Vaughan TM, Wolpaw JR (2008) Toward enhanced p300 speller performance. J Neurosci Methods 167(1):15–21

    Article  Google Scholar 

  52. Kumbhar M, Jadhav A, Patil M (2012) Facial expression recognition based on image feature. Intern J Comput Commun Eng 1(2):117–119

    Article  Google Scholar 

  53. Lajevardi SM, Wu HR (2012) Facial expression recognition in perceptual color space. IEEE Trans Image Process 21(8):3721–3733

    Article  MathSciNet  MATH  Google Scholar 

  54. Long F, Wu T, Movellan JR, Bartlett MS, Littlewort G (2012) Learning spatiotemporal features by using independent component analysis with application to facial expression recognition. Neurocomputing 2 (1):126–132

    Article  Google Scholar 

  55. Lu J, Plataniotis KN, Venetsanopoulos AN (2003) Face recognition using lda-based algorithms. IEEE Trans Neural Netw 14(1):195–200

    Article  Google Scholar 

  56. Lu X, Kong L, Liu M, Zhang X (2015) Facial expression recognition based on gabor feature and src. In: Biometric recognition. Springer, pp 416–422

  57. Ma L (2008) Facial expression recognition using 2-d dct of binarized edge images and constructive feedforward neural networks. In: IEEE international joint conference on neural networks, 2008. IJCNN 2008. (IEEE world congress on computational intelligence). IEEE, pp 4083–4088

  58. Mika S (2002) Kernel fisher discriminants. PhD thesis, Universitätsbibliothek

  59. Mika S, Ratsch G, Weston J, Scholkopf B, Mullers KR (1999) Fisher Discriminant analysis with kernels. In: Proceedings of the IEEE signal processing society workshop neural networks for signal processing IX, 1999. IEEE, pp 41–48

  60. Mistry VJ, Goyani MM (2013) A literature survey on facial expression recognition using global features. Intern J Eng Adv Technol 2(4):653–657

    Google Scholar 

  61. Nam MY, Rhee PK (2005) Human face detection using skin color context awareness and context-based bayesian classifiers. In: Knowledge-based intelligent information and engineering systems. Springer, pp 298–307

  62. Nanni L, Lumini A, Dominio F, Zanuttigh P (2014) Effective and precise face detection based on color and depth data. Appl Comput Inform 10(1):1–13

    Article  Google Scholar 

  63. Pan Z, Lei J, Zhang Y, Sun X, Kwong S (2016) Fast motion estimation based on content property for low-complexity h. 265/hevc encoder. IEEE Trans Broadcast 62(3):675–684

    Article  Google Scholar 

  64. Pang Yanwei, Yuan Yuan, Li X (2009) Iterative subspace analysis based on feature line distance. IEEE Trans Image Process 18(4):903–907

    Article  MathSciNet  MATH  Google Scholar 

  65. Pardeshi SA, Talbar SN (2009) Face description with local invariant features: application to face recognition. In: 2009 2nd international conference on emerging trends in engineering and technology (ICETET). IEEE, pp 248–251

  66. Park SW, Savvides M (2010) A multifactor extension of linear discriminant analysis for face recognition under varying pose and illumination. EURASIP J Adv Signal Process 2010:6

    Article  Google Scholar 

  67. Purandare V, Talele KT (2014) Efficient heterogeneous face recognition using scale invariant feature transform. In: International conference on circuits, systems, communication and information technology applications (CSCITA), 2014. IEEE, pp 305–310

  68. Rabiner LR (1989) A tutorial on hidden markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286

    Article  Google Scholar 

  69. Rahulamathavan Y, Phan R, Chambers J, Parish D (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affective Comput 4(1):83–92

    Article  Google Scholar 

  70. Ramirez Rivera A, Rojas Castillo J, Chae O (2013) Local directional number pattern for face analysis: face and expression recognition. IEEE Trans Image Process 22(5):1740–1752

    Article  MathSciNet  MATH  Google Scholar 

  71. Roy S, Bandyopadhyay SK (2013) Face detection using a hybrid approach that combines hsv and rgb. Intern J Comput Sci Mobile Comput 2(3):127–136

    Google Scholar 

  72. Patil CS, Patil AJ (2013) A review paper on facial detection technique using pixel and color segmentation. Int J Comput Appl 62(1):21–24

    Google Scholar 

  73. Samaria FS (1994) Face recognition using hidden Markov models. PhD thesis, University of Cambridge

  74. Saudagare PV, Chaudhari DS (2012) Facial expression recognition using neural network–an overview. International Journal of Soft Computing and Engineering (IJSCE) 2(1):224–227

    Google Scholar 

  75. Shan C, Braspenning R (2010) Recognizing facial expressions automatically from video. In: Handbook of ambient intelligence and smart environments. Springer, pp 479–509

  76. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  77. Sharif A, Sharif MI, Riaz S, Shaheen A, Badini MK (2014) Face recognition: holistic approaches an analytical survey. Sci Intern 26(2):639–644

    Google Scholar 

  78. Sharma R, Patterh MS (2015) Face recognition using face alignment and pca techniques: a literature survey. Neural Comput 17(4):17–30

    Google Scholar 

  79. Siddiqi MH, Ali M, Eldib MEA, Khan A, Banos O, Khan AM, Lee S, Choo H (2017) Evaluating real-life performance of the state-of-the-art in facial expression recognition using a novel youtube-based datasets. Multimed Tools Appl 1–21

  80. Muhammad HS, Ali M, Idris M, Banos O, Lee S, Choo H (2016) A novel dataset for real-life evaluation of facial expression recognition methodologies. In: Canadian conference on artificial intelligence. Springer, pp 89–95

  81. Siddiqi MH, Ali R, Idris M, Khan AM, Kim ES, Whang MC, Lee S (2016) Human facial expression recognition using curvelet feature extraction and normalized mutual information feature selection. Multimed Tools Appl 75(2):935–959

    Article  Google Scholar 

  82. Siddiqi MH, Ali R, Khan AM, Kim ES, Kim GJ, Lee S (2015) Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection. Multimed Syst 21(6):541–555

    Article  Google Scholar 

  83. Siddiqi MH, Ali R, Khan AM, Park Y-T, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24 (4):1386–1398

    Article  MathSciNet  Google Scholar 

  84. Siddiqi MH, Ali R, Sattar A, Khan AM, Lee S (2014) Depth camera-based facial expression recognition system using multilayer scheme. IETE Tech Rev 31(4):277–286

    Article  Google Scholar 

  85. Siddiqi MH, Farooq F, Lee S (2012) A robust feature extraction method for human facial expressions recognition systems. In: Proceedings of the 27th conference on image and vision computing New Zealand. ACM, pp 464–468

  86. Siddiqi MH, Lee S, Lee Y-K, Khan AM, Truc PTH (2013) Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13(12):16682–16713

    Article  Google Scholar 

  87. Stoerring M, Andersen HJ, Granum E (1999) Skin colour detection under changing lighting conditions. In: 7th symposium on intelligent robotics systems. Citeseer

  88. Suresh AJ (2015) Face expression recognition using gabor features and probabilistic neural network. Int J Adv Res Comput Sci 6(5):86–90

    Google Scholar 

  89. Tian Q, Chen S (2017) Cross-heterogeneous-database age estimation through correlation representation learning. Neurocomputing 238:286–295

    Article  Google Scholar 

  90. Tian Y-L, Kanade T, Cohn JF (2005) Facial expression analysis. In: Handbook of face recognition. Springer, pp 247–275

  91. Turunen J et al (2011) A wavelet-based method for estimating damping in power systems

  92. Md ZU, Kim T-S, Song BC (2013) An optical flow feature-based robust facial expression recognition with hmm from video. Int J Innov Comput Inf Control 9(4):1409–1421

    Google Scholar 

  93. Uwechue OA, Pandya AS (2012) Human face recognition using third-order synthetic neural networks, vol 410. Springer Science & Business Media

  94. Verma A, Achyut Raj S, Midya A, Chakraborty J (2014) Face detection using skin color modeling and geometric feature. In: 2014 international conference on informatics, electronics & vision (ICIEV). IEEE, pp 1–6

  95. Wang X, Paliwal KK (2003) Feature extraction and dimensionality reduction algorithms and their applications in vowel recognition. Pattern Recogn 36(10):2429–2439

    Article  MATH  Google Scholar 

  96. Wang Z, Sun X (2012) Manifold adaptive kernel local fisher discriminant analysis for face recognition. J Multimed 7(6):387–393

    Google Scholar 

  97. Wei W, Song H, Li W, Shen P, Vasilakos A (2017) Gradient-driven parking navigation using a continuous information potential field based on wireless sensor network. Inf Sci 408:100–114

    Article  Google Scholar 

  98. Wu X, Zhao J (2010) Curvelet feature extraction for face recognition and facial expression recognition. In: 2010 sixth international conference on natural computation (ICNC), vol 3. IEEE, pp 1212–1216

  99. Xia Z, Wang X, Sun X, Liu Q, Xiong N (2016) Steganalysis of lsb matching using differences between nonadjacent pixels. Multimed Tools Appl 75(4):1947–1962

    Article  Google Scholar 

  100. Xia Z, Wang X, Zhang L, Qin Z, Sun X, Ren K (2016) A privacy-preserving and copy-deterrence content-based image retrieval scheme in cloud computing. IEEE Trans Inf Forensics Secur 11(11):2594–2608

    Article  Google Scholar 

  101. Zhang L, Tjondronegoro D (2010) Feature extraction and representation for face recognition. In: Oraves M (ed) Face recognition, pp 1–20

  102. Zhang L, Tjondronegoro D (2011) Facial expression recognition using facial movement features. IEEE Trans Affect Comput 2(4):219–229

    Article  Google Scholar 

  103. Zhao X, Shi X, Zhang S (2015) Facial expression recognition via deep learning. IETE Technical Review, (ahead-of-print):1–9

  104. Zheng W, Zhao L, Zou C (2004) A modified algorithm for generalized discriminant analysis. Neural Comput 16(6):1283–1297

    Article  MATH  Google Scholar 

  105. Zhou Z, Yang C-N, Chen B, Sun X, Liu Q, Jonathan QM (2016) Effective and efficient image copy detection with resistance to arbitrary rotation. IEICE Trans Inf Syst 99(6):1531–1540

    Article  Google Scholar 

  106. Zhu Z, Ji Q (2006) Robust real-time face pose and facial expression recovery. In: 2006 IEEE computer society conference on computer vision and pattern recognition, vol 1. IEEE, pp 681–688

Download references

Acknowledgements

The author would like to thank Faculty of Computer and Information Sciences, AlJouf University, Sakaka, Kingdom of Saudi Arabia.

Author information

Authors and Affiliations

  1. Department of Computer and Information Sciences, AlJouf University, Sakaka, Kingdom of Saudi Arabia

    Muhammad Hameed Siddiqi

Authors
  1. Muhammad Hameed Siddiqi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Hameed Siddiqi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Siddiqi, M.H. Accurate and robust facial expression recognition system using real-time YouTube-based datasets. Appl Intell 48, 2912–2929 (2018). https://doi.org/10.1007/s10489-017-1121-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-017-1121-y

Keywords

Search

Navigation