Skip to main content
Springer Nature Link
Log in

Sign Language Recognition Based on Trajectory Modeling with HMMs

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2016)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9516))

Included in the following conference series:

Abstract

Sign language recognition targets on interpreting and understanding the sign language for convenience of communication between the deaf and the normal people, which has broad social impact. The problem is challenging due to the large variations for different signers and the subtle difference between sign words. In this paper, we propose a new method for isolated sign language recognition based on trajectory modeling with hidden Markov models (HMMs). In our approach, we first normalize and re-sample the raw trajectory data and partition the trajectory into multiple segments. To represent each trajectory segment, we proposed a new curve feature descriptor based on shape context. After that, hidden Markov model is used to model each isolated sign word for recognition. To evaluate the performance of our proposed algorithm, we have built a large isolated Chinese sign language vocabulary with Kinect 2.0. The dataset contains 100 unique isolated sign words, each of which is performed by 50 signers for 5 times. Experimental results demonstrate that the proposed method achieves a better performance compared with normal coordinate feature with HMM.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24(4), 509–522 (2002)

    Article  Google Scholar 

  2. Gales, M., Young, S.: The application of hidden markov models in speech recognition. Found. Trends Sig. Process. 1(3), 195–304 (2008)

    Article  Google Scholar 

  3. Grobel, K., Assan, M.: Isolated sign language recognition using hidden markov models. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pp. 162–167. IEEE (1997)

    Google Scholar 

  4. Hienz, H., Kraiss, K.-F., Bauer, B.: Continuous sign language recognition using hidden markov models. In: International Conference on Multimodal Interfaces, vol. 4, pp. 10–15 (1999)

    Google Scholar 

  5. Huang, J., Zhou, W., Li, H., Li, W.: Sign language recognition using 3D convolutional neural networks. In: IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE, 2015

    Google Scholar 

  6. Huang, J., Zhou, W., Li, H., Li, W.: Sign language recognition using real-sense. In: IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP), pp. 166–170. IEEE (2015)

    Google Scholar 

  7. Latecki, L.J., Lakämper, R.: Convexity rule for shape decomposition based on discrete contour evolution. Comput. Vis. Image Underst. 73(3), 441–454 (1999)

    Article  Google Scholar 

  8. Lin, Y., Chai, X., Zhou, Y., Chen, X.: Curve matching from the view of manifold for sign language recognition. In: Shan, S., Jawahar, C.V., Jawahar, C.V. (eds.) ACCV 2014 Workshops. LNCS, vol. 9010, pp. 233–246. Springer, Heidelberg (2014)

    Google Scholar 

  9. Murakami, K., Taguchi, H.: Gesture recognition using recurrent neural networks. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 237–242. ACM (1991)

    Google Scholar 

  10. Oz, C., Leu, M.C.: Recognition of finger spelling of American sign language with artificial neural network using position/orientation sensors and data glove. In: Wang, J., Liao, X.-F., Yi, Z. (eds.) ISNN 2005. LNCS, vol. 3497, pp. 157–164. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  11. Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Lost in quantization: improving particular object retrieval in large scale image databases. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2008)

    Google Scholar 

  12. Rabiner, L.R.: A tutorial on hidden markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989)

    Article  Google Scholar 

  13. Rabiner, L.R., Juang, B.-H.: An introduction to hidden markov models. IEEE ASSP Mag. 3(1), 4–16 (1986)

    Article  Google Scholar 

  14. Schlenzig, J., Hunter, E., Jain, R.: Vision based hand gesture interpretation using recursive estimation. In: Conference Record of the Twenty-Eighth Asilomar Conference on Signals, Systems and Computers, vol. 2, pp. 1267–1271. IEEE (1994)

    Google Scholar 

  15. Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., Moore, R.: Real-time human pose recognition in parts from single depth images. Commun. ACM 56(1), 116–124 (2013)

    Article  Google Scholar 

  16. Wang, H., Chai, X., Zhou, Y., Chen, X.: Fast sign language recognition benefited from low rank approximation. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1–6. IEEE (2015)

    Google Scholar 

  17. Wang, M., Hua, X.-S., Hong, R., Tang, J., Qi, G.-J., Song, Y.: Unified video annotation via multigraph learning. IEEE Trans. Circuits Syst. Video Technol. 19(5), 733–746 (2009)

    Article  Google Scholar 

  18. Wang, M., Ni, B., Hua, X.-S., Chua, T.-S.: Assistive tagging: a survey of multimedia tagging with human-computer joint exploration. ACM Comput. Surv. (CSUR) 44(4), 25 (2012)

    Article  Google Scholar 

  19. Wobbrock, J.O., Wilson, A.D., Li, Y.: Gestures without libraries, toolkits or training: a \(\$\)1 recognizer for user interface prototypes. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, pp. 159–168. ACM (2007)

    Google Scholar 

  20. Zafrulla, Z., Brashear, H., Starner, T., Hamilton, H., Presti, P.: American sign language recognition with the kinect. In: Proceedings of the 13th International Conference on Multimodal Interfaces, pp. 279–286. ACM (2011)

    Google Scholar 

  21. Zhang, J., Zhou, W., Li, H.: A threshold-based hmm-dtw approach for continuous sign language recognition. In: Proceedings of International Conference on Internet Multimedia Computing and Service, p. 237. ACM (2014)

    Google Scholar 

  22. Zhang, J., Zhou, W., Li, H.: A new system for chinese sign language recognition. In: IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP), pp. 534–538. IEEE (2015)

    Google Scholar 

Download references

Acknowledgement

This work was supported in part to Dr. Zhou by the Fundamental Research Funds for the Central Universities under contract No. WK2100060014 and WK2100060011 and the National Science Foundation of China under contract No. 61472378, and in part to Prof. Li by the National Science Foundation of China under contract No. 61272316.

Author information

Authors and Affiliations

  1. University of Science and Technology of China, Hefei, Anhui, People’s Republic of China

    Junfu Pu, Wengang Zhou, Jihai Zhang & Houqiang Li

Authors
  1. Junfu Pu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wengang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jihai Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Houqiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wengang Zhou .

Editor information

Editors and Affiliations

  1. University of Texas at San Antonio, San Antonio, USA

    Qi Tian

  2. Dept. of Information Engineering, University of Trento, Povo, Trento, Italy

    Nicu Sebe

  3. EECS, University of Central Florida, Orlando, Florida, USA

    Guo-Jun Qi

  4. EURECOM, Sophia-Antipolis, France

    Benoit Huet

  5. Hefei University of Technology, Hefei, Anhui, China

    Richang Hong

  6. School of Computing and Information, Hefei University of Technology, Hefei, Anhui, China

    Xueliang Liu

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Pu, J., Zhou, W., Zhang, J., Li, H. (2016). Sign Language Recognition Based on Trajectory Modeling with HMMs. In: Tian, Q., Sebe, N., Qi, GJ., Huet, B., Hong, R., Liu, X. (eds) MultiMedia Modeling. MMM 2016. Lecture Notes in Computer Science(), vol 9516. Springer, Cham. https://doi.org/10.1007/978-3-319-27671-7_58

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-27671-7_58

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-27670-0

  • Online ISBN: 978-3-319-27671-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics