Skip to main content
SpringerLink
Log in

Extraction of texture and geometrical features from informative facial regions for sign language recognition

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

In general, the most common form of gestures is made up of movements of the hand and/or arm associated with facial expressions. In this, the hand is used to make different message signs, while facial movements are used to reflect the mood and emotion of the person. In this paper, some sign language gestures are recognized only with the help of associated facial expressions. Existing facial expression based sign language recognition (SLR) methods only used facial geometric features to recognize sign language gestures. However, the performance of geometric feature-based SLR methods depends on the accuracy of tracking algorithms and the number of facial landmark points. Additionally, facial textures are more informative as compared to the geometric features of a face. Inspiring from these facts, we propose to recognize sign language gestures with the help of spatio-temporal characteristics of facial texture patterns. For this, a new face model is proposed by extracting texture features only from the informative regions of a face. The proposed face model can also be employed to extract the geometrical features of a face. The features extracted from the informative regions of a face are significantly discriminative, and so the proposed face model can track/encode the facial dynamics of the associated facial expressions of a sign. Finally, a 3-state hidden conditional random field is employed to model the texture variations of facial gestures. Experimental results on RWTH-BOSTON data-set show that proposed method can achieve upto 80.06% recognition rate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Engberg-Pedersen Elisabeth (1993) Space in danish sign language: the semantics and morphosyntax of the use of space in a visual language

  2. Fang G, Gao W, Zhao D (2007) Large-vocabulary continuous sign language recognition based on transition-movement models. IEEE Trans Syst Man Cybern Part A Syst Hum 37(1):1–9

    Article  Google Scholar 

  3. Agris UV, Knorr M, and Kraiss KF (2008) The significance of facial features for automatic sign language recognition. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp 1–6

  4. Assan M, Groebel K (1998) A vision-based sign language recognition system using tied-mixture density hmm. In: Gesture and Sign Language in Human-Computer Interaction, Lecture Notes in Computer Science, vol 1371, pp 97–109

  5. Nguyen TD, Ranganath S (2012) Facial expressions in american sign language: tracking and recognition. Pattern Recognit 45(5):1877–1891

    Article  MATH  Google Scholar 

  6. Rahulamathavan Y, Phan RC-W, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4(1):83–92

    Article  Google Scholar 

  7. Rudovic O, Pantic M, Patras I (2013) Coupled gaussian processes for pose-invariant facial expression recognition. IEEE Trans Pattern Anal Mach Intell 35(6):1357–1369

    Article  Google Scholar 

  8. Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Article  Google Scholar 

  9. Kim M, Pavlovic V (2010) Hidden conditional ordinal random fields for sequence classification. In: Machine Learning and Knowledge Discovery in Databases. Springer, Berlin, pp 51–65

  10. Cootes TF, Taylor CJ, Cooper DH, Graham J (1995) Active shape model-their training and application. Comput Vis. Image Underst 61(1):38–59

    Article  Google Scholar 

  11. Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685

    Article  Google Scholar 

  12. Ari I, Uyar A, Akarun L (2008) Facial feature tracking and expression recognition for sign language. In: 23rd International Symposium on Computer and Information Sciences, 2008. ISCIS ’08, pp 1–6

  13. Nguyen TD, Ranganath S (2008) Tracking facial features under occlusions and recognizing facial expressions in sign language. In: 8th IEEE International Conference on Automatic Face Gesture Recognition, 2008. FG ’08, pp 1–7

  14. Yang HD, Lee SW (July 2011) Combination of manual and non-manual features for sign language recognition based on conditional random field and active appearance model. In: 2011 International Conference on Machine Learning and Cybernetics (ICMLC), vol 4, pp 1726–1731

  15. Walecki R, Rudovic O, Pavlovic V, Pantic M (2015) Variable-state latent conditional random fields for facial expression recognition and action unit detection. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp 1–8

  16. Izard CE (1977) Human emotions. In: Springer Science and Business Media

  17. Zhang L, Tjondronegoro D, Chandran V (2011) Evaluation of texture and geometry for dimensional facial expression recognition. In: 2011 International Conference on Digital Image Computing Techniques and Applications (DICTA), pp 620–626

  18. Rabiner L (1989) A tutorial on hidden markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286

  19. Ong SCW, Ranganath S (2005) Automatic sign language analysis: a survey and the future beyond lexical meaning. IEEE Trans Pattern Anal Mach Intell 27(6):873–891

    Article  Google Scholar 

  20. Brand M, Oliver N, Pentland A (1997) Coupled hidden markov models for complex action recognition. In: 1997 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1997. Proceedings., pp 994–999

  21. Vogler C, Metaxas D (1999) Parallel hidden markov models for american sign language recognition. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999, vol 1, pp 116–122

  22. Wang SB, Quattoni A, Morency L, Demirdjian D, Darrell T (2006) Hidden conditional random fields for gesture recognition. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol 2, pp 1521–1527

  23. Ekman P, Friesen W (1978) Manual of the facial action coding system (FACS). Consulting Psychologists Press, Palo Alto

  24. Ojala T, Pietikainen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distribution. Pattern Recognit 29(1):51–59

    Article  Google Scholar 

  25. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  MATH  Google Scholar 

  26. Huang D, Shan C, Ardabilian M, Wang Y, Chen L (2011) Local binary patterns and its application to facial image analysis: a survey. IEEE Trans Syst Man Cybern Part C Appl Rev 41(6):765–781

    Article  Google Scholar 

  27. Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In: 11th International Workshop Image Analysis for Multimedia Interactive Services, pp 1–4

  28. Lyons M, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 21(12):1357–1362

    Article  Google Scholar 

  29. Kanade T, Cohn JF, Tian YingLi (2000) Comprehensive database for facial expression analysis. In: Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000. Proceedings, pp 46–53

  30. Belhumeur PN, Jacobs DW, Kriegman DJ, Kumar N (2013) Localizing parts of faces using a consensus of exemplars. IEEE Trans Pattern Anal Mach Intell 35(12):2930–2940

    Article  Google Scholar 

  31. Baker S, Matthews I (2004) Lucas-kanade 20 years on: a unifying framework. Int J Comput Vis 56(3):221–255

    Article  Google Scholar 

  32. Tzimiropoulos G, Pantic M (2013) Optimization problems for fast aam fitting in-the-wild. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp 593–600

  33. Von Agris U, Kraiss KF (2007) Towards a video corpus for signer-independent continuous sign language recognition. Gesture in Human-Computer Interaction and Simulation, Lisbon

    Google Scholar 

  34. Zahedi M, Keysers D, Deselaers T, Ney H (2005) Combination of tangent distance and an image distortion model for appearance-based sign language recognition. In: Pattern Recognition. Springer, Berlin, pp 401–408

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Electrical Engineering, Indian Institute of Technology (IIT) Guwahati, Guwahati, 781039, India

    Sunil Kumar, M. K. Bhuyan & Biplab Ketan Chakraborty

Authors
  1. Sunil Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. K. Bhuyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Biplab Ketan Chakraborty
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. K. Bhuyan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, S., Bhuyan, M.K. & Chakraborty, B.K. Extraction of texture and geometrical features from informative facial regions for sign language recognition. J Multimodal User Interfaces 11, 227–239 (2017). https://doi.org/10.1007/s12193-017-0241-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-017-0241-3

Keywords

Search

Navigation