Skip to main content
SpringerLink
Log in

Efficient facial expression recognition using histogram of oriented gradients in wavelet domain

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expression recognition plays a significant role in human behavior detection. In this study, we present an efficient and fast facial expression recognition system. We introduce a new feature called W_HOG where W indicates discrete wavelet transform and HOG indicates histogram of oriented gradients feature. The proposed framework comprises of four stages: (i) Face processing, (ii) Domain transformation, (iii) Feature extraction and (iv) Expression recognition. Face processing is composed of face detection, cropping and normalization steps. In domain transformation, spatial domain features are transformed into the frequency domain by applying discrete wavelet transform (DWT). Feature extraction is performed by retrieving Histogram of Oriented Gradients (HOG) feature in DWT domain which is termed as W_HOG feature. For expression recognition, W_HOG feature is supplied to a well-designed tree based multiclass support vector machine (SVM) classifier with one-versus-all architecture. The proposed system is trained and tested with benchmark CK+, JAFFE and Yale facial expression datasets. Experimental results of the proposed method are effective towards facial expression recognition and outperforms existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Addison PS (2017) The illustrated wavelet transform handbook: introductory theory and applications in science, engineering, medicine and finance. CRC press, Boca Raton

    MATH  Google Scholar 

  2. Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Anal Mach Intell 28(12):2037–2041

    Article  Google Scholar 

  3. Al Chanti D, Caplier A (2017) Spontaneous facial expression recognition using sparse representation. In: 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017) (vol 5, no. 64–74, p 11), Porto, Portugal

  4. Ali G, Iqbal MA, Choi TS (2016) Boosted NNE collections for multicultural facial expression recognition. Pattern Recogn 55:14–27. https://doi.org/10.1016/j.patcog.2016.01.032

    Article  Google Scholar 

  5. Bousmalis K, Mehu M, Pantic M (2013) Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: a survey of related cues, databases, and tools. Image Vis Comput 31(2):203–221

    Article  Google Scholar 

  6. Burges CJ (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 2(2):121–167

    Article  Google Scholar 

  7. Burkert, P., Trier, F., Afzal, M. Z., Dengel, A., and Liwicki, M. (2015). Dexpression: deep convolutional neural network for expression recognition. arXiv preprint arXiv:1509.05371.URL http://arxiv.org/abs/1509.05371

  8. Chew, S. W., Lucey, P., Lucey, S., Saragih, J., Cohn, J. F., & Sridharan, S. (2011). Person-independent facial expression detection using constrained local models. In Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 I.E. International Conference on (pp. 915-920). IEEE

  9. Chiong R (ed) (2009) Intelligent systems for automated learning and adaptation: emerging trends and applications: emerging trends and applications. IGI Global, Hershey

    Google Scholar 

  10. Corneanu CA, Simón MO, Cohn JF, Guerrero SE (2016) Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans Pattern Anal Mach Intell 38(8):1548–1568

    Article  Google Scholar 

  11. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  12. Dailey MN, Joyce C, Lyons MJ, Kamachi M, Ishi H, Gyoba J, Cottrell GW (2010) Evidence and a computational explanation of cultural differences in facial expression recognition. Emotion 10(6):874–893

    Article  Google Scholar 

  13. Dalal, N., and Triggs, B. (2005). Histograms of oriented gradients for human detection. In IEEE computer society conference on computer vision and pattern recognition, CVPR 2005. (Vol. 1, pp. 886-893)

  14. Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists, San Francisco

    Google Scholar 

  15. Eskil MT, Benli KS (2014) Facial expression recognition based on anatomy. Comput Vis Image Underst 119:1–14

    Article  Google Scholar 

  16. Fan X, Tjahjadi T (2015) A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences. Pattern Recogn 48(11):3407–3416

    Article  Google Scholar 

  17. Fan X, Tjahjadi T (2017) A dynamic framework based on local Zernike moment and motion history image for facial expression recognition. Pattern Recogn 64:399–406

    Article  Google Scholar 

  18. Franco L, Treves A (2001) A neural network facial expression recognition system using unsupervised local processing. In 2nd International Symposium on Image and Signal Processing and Analysis (ISPA’01) (pp. 628-632)

  19. Guo G, Dyer CR (2003) Simultaneous feature selection and classifier training via linear programming: a case study for face expression recognition. In 2003 I.E. Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’03) (Vol. 1, pp. 346-352)

  20. Hegde GP, Seetha M (2017) Subspace based expression recognition using combinational Gabor based feature fusion. International Journal of Image, Graphics and Signal Processing 9(1):50

    Article  Google Scholar 

  21. Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In proceedings 2004 I.E. International Joint Conference on Neural Networks, (Vol. 2, pp. 985-990)

  22. Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42(2):513–529

    Article  Google Scholar 

  23. Iosifidis A, Tefas A, Pitas I (2017) Approximate kernel extreme learning machine for large scale data classification. Neurocomputing 219:210–220

    Article  Google Scholar 

  24. Jiang R, Ho AT, Cheheb I, Al-Maadeed N, Al-Maadeed S, Bouridane A (2017) Emotion recognition from scrambled facial images via many graph embedding. Pattern Recogn 67:245–251

    Article  Google Scholar 

  25. Jung H, Lee S, Park S, Kim B, Kim J, Lee I, Ahn C (2015) Development of deep learning-based facial expression recognition system. In Frontiers of computer vision (FCV), 2015 21st Korea-Japan joint workshop on (pp. 1-4). IEEE

  26. Jung H, Lee S, Yim J, Park S, Kim J (2015) Joint fine-tuning in deep neural networks for facial expression recognition. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2983-2991)

  27. Khan SA, Hussain A, Usman M (2017) Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features. Multimedia Tools and Applications, 1-33

  28. Kumbhar M, Jadhav A, Patil M (2012) Facial expression recognition based on image feature. International Journal of Computer and Communication Engineering 1(2):117–119

    Article  Google Scholar 

  29. Li W, Li M, Su Z, Zhu Z (2015) A deep-learning approach to facial expression recognition with candid images. In Machine Vision Applications (MVA), 2015 14th IAPR International Conference on (pp. 279-282). IEEE

  30. Liu M, Li S, Shan S, Wang R, Chen X (2014) Deeply learning deformable facial action parts model for dynamic expression analysis. In Asian conference on computer vision (pp. 143-157). Springer, Cham

    Google Scholar 

  31. Liu, M., Shan, S., Wang, R., & Chen, X. (2014). Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1749-1756)

  32. Liu P, Han S, Meng Z, Tong Y (2014) Facial expression recognition via a boosted deep belief network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1805-1812)

  33. Liu M, Li S, Shan S, Chen X (2015) Au-inspired deep networks for facial expression feature learning. Neurocomputing 159:126–136

    Article  Google Scholar 

  34. Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2Activity: recognizing complex activities from sensor data. In IJCAI (pp. 1617-1623)

  35. Liu L, Cheng L, Liu Y, Jia Y, Rosenblum DS (2016) Recognizing complex activities by a probabilistic interval-based model. In AAAI (Vol. 30, pp. 1266-1272)

  36. Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115

    Article  Google Scholar 

  37. Liu Y, Zhang L, Nie L, Yan Y, Rosenblum DS (2016) Fortune teller: predicting your career path. In AAAI (pp. 201-207)

  38. Liu Y, Zheng Y, Liang Y, Liu S, Rosenblum DS (2016) Urban water quality prediction based on multi-task multi-view learning

  39. Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628

    Article  Google Scholar 

  40. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 I.E. Computer Society Conference on (pp. 94-101). IEEE

  41. Lv, Y., Feng, Z., & Xu, C. (2014). Facial expression recognition via deep learning. In smart computing (SMARTCOMP), 2014 international conference on (pp. 303-308). IEEE

  42. Malisiewicz T, Gupta A, Efros AA (2011). Ensemble of exemplar-SVMs for object detection and beyond. In 2011 I.E. International Conference on Computer Vision (ICCV’11) (pp. 89-96)

  43. Meng Z, Liu P, Cai J, Han S, Tong Y (2017). Identity-aware convolutional neural network for facial expression recognition. In Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE international conference on (pp. 558-565). IEEE

  44. Nigam S, Khare A (2015) Multiresolution approach for multiple human detection using moments and local binary patterns. Multimedia Tools and Applications 74(17):7037–7062

    Article  Google Scholar 

  45. Nigam S, Khare A (2015) Multiscale local binary patterns for facial expression-based human emotion recognition, In Computational Vision and Robotics. Springer, New Delhi, pp 71–77

    Google Scholar 

  46. Nigam S, Khare A (2016) Integration of moment invariants and uniform local binary patterns for human activity recognition in video sequences. Multimedia Tools and Applications 75(24):17303–17332

    Article  Google Scholar 

  47. Nigam S, Singh R, Misra AK (2018) Towards intelligent human behavior detection for video surveillance. In Advancements in Computer Vision and Image Processing (pp. 34-73). IGI Global

  48. Nikitidis S, Tefas A, Nikolaidis N, Pitas I (2012) Subclass discriminant nonnegative matrix factorization for facial image analysis. Pattern Recogn 45(12):4080–4091

    Article  Google Scholar 

  49. Pantic, M. (2009). Facial expression recognition. In encyclopedia of biometrics (pp. 400-406). Springer US

  50. Poria S, Cambria E, Bajpai R, Hussain A (2017) A review of affective computing: from unimodal analysis to multimodal fusion. Information Fusion 37:98–125

    Article  Google Scholar 

  51. Preoţiuc-Pietro D, Liu Y, Hopkins D, Ungar L (2017). Beyond binary labels: political ideology prediction of twitter users. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Vol. 1, pp. 729-740)

  52. Rudovic O, Pantic M, Patras I (2013) Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans Pattern Anal Mach Intell 35(6):1357–1369

    Article  Google Scholar 

  53. Russell JA, Fernández-Dols JM (eds) (2017) The science of facial expression. Oxford University Press, Oxford

    Google Scholar 

  54. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  55. Shinohara Y, Otsu N (2004) Facial expression recognition using fisher weight maps. In sixth IEEE international conference on automatic face and gesture recognition (pp. 499-504)

  56. Singh AK, Shukla VP, Tiwari S, Biradar SR (2015) Wavelet based histogram of oriented gradients feature descriptors for classification of partially occluded objects. International Journal of Intelligent Systems and Applications 7(3):54–61

    Article  Google Scholar 

  57. Song I, Kim HJ, Jeon PB (2014). Deep learning for real-time robust facial expression recognition on a smartphone. In 2014 I.E. international conference on consumer electronics (ICCE), (pp. 564-567)

  58. Spiers DL (2016) Facial emotion detection using deep learning. Doctoral Dissertation, UPPSALA Universitet

  59. Tian Y, Kanade T, Cohn JF (2011) Facial expression recognition. In handbook of face recognition (pp. 487-519). Springer London

  60. UCSD Computer Vision. Yale facial expression database. http://vision.ucsd.edu/content/yale-facedatabase. Accessed 28 Dec 2017

  61. Uddin MZ, Lee JJ, Kim TS (2009) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55(4):2216–2224

    Article  Google Scholar 

  62. Valstar M, Pantic M (2006) Fully automatic facial action unit detection and temporal analysis. In IEEE computer society computer vision and pattern recognition workshop (CVPRW'06) (pp. 149-149)

  63. Valstar MF, Patras I, Pantic M (2005) Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In IEEE Computer Society Computer Vision and Pattern Recognition Workshop (CVPRW'05) (Vol. 3, pp. 76-76)

  64. Valstar M, Zafeiriou S, Pantic M (2017) Chapter 11: facial actions as social signals. In: Burgoon JK, Magnenat-Thalmann N, Pantic M, Vinciarelli A (eds) Social signal processing. Cambridge University Press. https://doi.org/10.1017/9781316676202. Available online at: www.cambridge.org/9781107161269

    Google Scholar 

  65. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  66. Wang Z, Wang S, Ji Q (2013) Capturing complex spatio-temporal relations among facial muscles for facial expression recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3422-3429)

  67. Yang MH (2002) Kernel Eigenfaces vs. Kernel Fisherfaces: face recognition using kernel methods. In proceedings of the 5th IEEE International Conference on automatic face and Gesture Recognition, pp. 215–220

  68. Yang J, Yang J-Y (2002) Generalized K–L transform based combined feature extraction. Pattern Recogn 35(1):295–297

    Article  Google Scholar 

  69. Zhang Z (1999) Feature based facial expression recognition: sensitivity analysis and experiment with a multi-layer perceptron. Int J Pattern Recognit Artif Intell 13(6):893–911

    Article  Google Scholar 

  70. Zhao X, Zhang S (2016) A review on facial expression recognition: feature extraction and classification. IETE Tech Rev 33(5):505–517

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work is supported by Science and Engineering Research Board, Department of Science and Technology, Government of India under grant number PDF/2016/003644.

Author information

Authors and Affiliations

  1. Computer Science and Engineering Department, SP Memorial Institute of Technology, Kaushambi, Uttar Pradesh, 212213, India

    Swati Nigam & A. K. Misra

  2. Department of Computer Science, Banasthali Vidyapith, Banasthali, Rajasthan, 304022, India

    Rajiv Singh

Authors
  1. Swati Nigam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rajiv Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. K. Misra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rajiv Singh.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nigam, S., Singh, R. & Misra, A.K. Efficient facial expression recognition using histogram of oriented gradients in wavelet domain. Multimed Tools Appl 77, 28725–28747 (2018). https://doi.org/10.1007/s11042-018-6040-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6040-3

Keywords

Search

Navigation