Skip to main content
SpringerLink
Log in

Facial appearance and texture feature-based robust facial expression recognition framework for sentiment knowledge discovery

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Facial sentiment analysis has been an enthusiastic research area for the last two decades. A fair amount of work has been done by researchers in this field due to its utility in numerous applications such as facial expression driven knowledge discovery. However, developing an accurate and efficient facial expression recognition system is still a challenging problem. Although many efficient recognition systems have been introduced in the past, the recognition rate is not satisfactory in general due to inherent limitations including light, pose variations, noise, and occlusion. In this paper, a hybrid approach of facial expression based sentiment analysis has been presented combining local and global features. Feature extraction is performed fusing the histogram of oriented gradients (HOG) descriptor with the uniform local ternary pattern (U-LTP) descriptor. These features are extracted from the entire face image rather than from individual components of faces like eyes, nose, and mouth. The most suitable set of HOG parameters are selected after analyzing them experimentally along with the ULTP descriptor, boosting performance of the proposed technique over face images containing noise and occlusions. Face sentiments are analyzed classifying them into seven universal emotional expressions: Happy, Angry, Fear, Disgust, Sad, Surprise, and Neutral. Extracted features via HOG and ULTP are fused into a single feature vector and this feature vector is fed into a Multi-class Support Vector Machine classifier for emotion classification. Three types of experiments are conducted over three public facial image databases including JAFFE, MMI, and CK+ to evaluate the recognition rate of the proposed technique during experimental evaluation; recognition accuracy in percent, i.e., 95.71, 98.20, and 99.68 are achieved for JAFFE, MMI, and CK+, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Mehrabian, A.: Nonverbal Communication. Transaction Publishers, Los Angeles (1972)

    Google Scholar 

  2. Ekman, P., Friesen, W.V., Hager, J.C.: A technique for the measurement of facial action. In: Facial action coding system (FACS), p. 22. Palo Alto, Consulting (1978)

  3. Zhang, Y., Ji, Q.: Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 27(5), 699–714 (2005)

    Article  Google Scholar 

  4. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  5. Song, K.T., Chien, S.C.: Facial expression recognition based on mixture of basic expressions and intensities. In: 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE (2012)

  6. Dhall, A., et al.: Emotion recognition using PHOG and LPQ features. In: Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on, IEEE (2011)

  7. Apte, S.: Facial Emotion Identification

  8. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)

    Article  Google Scholar 

  9. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, IEEE (2001)

  10. Rivera, A.R., Castillo, J.R., Chae, O.O.: Local directional number pattern for face analysis: face and expression recognition. IEEE Trans. Image Process. 22(5), 1740–1752 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  11. Jeyashree, T., et al.: An efficient algorithm for face and expression recognition. Int. J. Sci. Technol. 2(3), 172 (2014)

    Google Scholar 

  12. Turk, M.A., Pentland, A.P.: Face recognition using eigenfaces. In: Computer Vision and Pattern Recognition, 1991. Proceedings CVPR’91., IEEE Computer Society Conference on, IEEE (1991)

  13. Hsu, C.W., Chang, C.C., Lin, C.J.: A practical guide to support vector classification (2003)

  14. Kozma, L.: k Nearest Neighbors algorithm (kNN). Helsinki University of Technology (2008)

  15. Lee, Y.H., Han, W., Kim, Y.: Emotional recognition from facial expression analysis using bezier curve fitting. In: 2013 16th International Conference on Network-Based Information Systems, IEEE (2013)

  16. Al-Shabi, M., Cheah, W.P., Connie, T.: Facial expression recognition using a hybrid CNN-SIFT aggregator. arXiv preprint arXiv:1608.02833 (2016)

  17. Wang, X.H., Liu, A., Zhang, S.Q.: New facial expression recognition based on FSVM and KNN. Optik-Int. J. Light Electron Opt. 126(21), 3132–3134 (2015)

    Article  Google Scholar 

  18. Tong, Y., Chen, R., Cheng, Y.: Facial expression recognition algorithm using LGC based on horizontal and diagonal prior principle. Optik-Int. J. Light Electron Opt. 125(16), 4186–4189 (2014)

    Article  Google Scholar 

  19. Luo, Y., Wu, C.M., Zhang, Y.: Facial expression recognition based on fusion feature of PCA and LBP with SVM. Optik-Int. J. Light Electron Opt. 124(17), 2767–2770 (2013)

    Article  Google Scholar 

  20. Happy, S., Routray, A.: Robust facial expression classification using shape and appearance features. In: Advances in Pattern Recognition (ICAPR), 2015 Eighth International Conference on, IEEE (2015)

  21. Carcagnì, P., et al.: Facial expression recognition and histograms of oriented gradients: a comprehensive study. SpringerPlus 4(1), 1 (2015)

    Article  Google Scholar 

  22. Kumar, S., Bhuyan, M., Chakraborty, B.K.: Extraction of informative regions of a face for facial expression recognition. IET Comput. Vis. 10(6), 567–576 (2016)

    Article  Google Scholar 

  23. Guo, Y., et al.: EI3D: expression-invariant 3D face recognition based on feature and shape matching. Pattern Recognit. Lett. 83, 403–412 (2016)

    Article  Google Scholar 

  24. Sajjad, M., Ejaz, N., Baik, S.W.: Multi-kernel based adaptive interpolation for image super-resolution. Multimed. Tools Appl. 72(3), 2063–2085 (2014)

    Article  Google Scholar 

  25. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), IEEE (2005)

  26. Geng, C., Jiang, X.: SIFT features for face recognition. In: Computer Science and Information Technology, 2009. ICCSIT 2009. 2nd IEEE International Conference on, IEEE (2009)

  27. Dreuw, P., et al. SURF-Face: face recognition under viewpoint consistency constraints. In: BMVC (2009)

  28. Tan, X., Triggs, B.: Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans. Image Process. 19(6), 1635–1650 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  29. Ren, J., Jiang, X., Yuan, J.: Relaxed local ternary pattern for face recognition. In: ICIP (2013)

  30. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)

    Article  MATH  Google Scholar 

  31. Lucey, P., et al.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, IEEE (2010)

  32. Cohen, I., et al.: Learning Bayesian network classifiers for facial expression recognition both labeled and unlabeled data. In: Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, IEEE (2003)

  33. Cohen, I., et al.: Facial expression recognition from video sequences: temporal and static modeling. Comput. Vis. Image Underst. 91(1), 160–187 (2003)

    Article  Google Scholar 

  34. Bartlett, M.S., et al.: Recognizing facial expression: machine learning and application to spontaneous behavior. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), IEEE (2005)

  35. Valstar, M., Pantic, M.: Fully automatic facial action unit detection and temporal analysis. In: 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’06), IEEE (2006)

  36. Valstar, M.F., Patras, I., Pantic, M.: Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)-Workshops, IEEE (2005)

  37. Pantic, M., et al.: Web-based database for facial expression analysis. In: 2005 IEEE international conference on multimedia and Expo, IEEE (2005)

  38. Lyons, M., et al.: Coding facial expressions with gabor wavelets. In: Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on, IEEE (1998)

  39. Chao, W.L., Ding, J.J., Liu, J.Z.: Facial expression recognition based on improved local binary pattern and class-regularized locality preserving projection. Signal Process. 117, 1–10 (2015)

    Article  Google Scholar 

  40. Gu, W., et al.: Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recognit. 45(1), 80–91 (2012)

    Article  Google Scholar 

  41. Donia, M.M., Youssif, A.A., Hashad, A.: Spontaneous facial expression recognition based on histogram of oriented gradients descriptor. Comput. Inf. Sci. 7(3), 31 (2014)

    Google Scholar 

  42. Zhang, L., Tjondronegoro, D.: Facial expression recognition using facial movement features. IEEE Trans. Affect. Comput. 2(4), 219–229 (2011)

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by the MISP (Ministry of Science, ICT & Future Planning), Korea, under National program for Excellence in SW (2015-0-00938) supervised by the IITP (Institute for Information & communications Technology Promotion).

Author information

Authors and Affiliations

  1. Digital Image Processing Laboratory, Department of Computer Science, Islamia College Peshawar, Peshawar, Pakistan

    Muhammad Sajjad, Adnan Shah & Zahoor Jan

  2. Department of Mathematics, Islamia College Peshawar, Peshawar, Pakistan

    Syed Inayat Shah

  3. Digital Contents Research Institute, Department of Software, Sejong University, Seoul, South Korea

    Sung Wook Baik

  4. Department of Computer Science and Engineering, Sejong University, Seoul, South Korea

    Irfan Mehmood

Authors
  1. Muhammad Sajjad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adnan Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zahoor Jan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Syed Inayat Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sung Wook Baik
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Irfan Mehmood
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Irfan Mehmood.

Ethics declarations

Conflicts of interest

The authors declare no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sajjad, M., Shah, A., Jan, Z. et al. Facial appearance and texture feature-based robust facial expression recognition framework for sentiment knowledge discovery. Cluster Comput 21, 549–567 (2018). https://doi.org/10.1007/s10586-017-0935-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-017-0935-z

Keywords

Search

Navigation