Skip to main content

Advertisement

Springer Nature Link
Log in

Chinese Sign Language Recognition with Batch Sampling ResNet-Bi-LSTM

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Sign language has served as a communication medium between the Deaf community and society. Nonetheless, the practice of sign language is not common in Chinese society, along with a lack of professional sign language interpreters. Most existing studies on sign language recognition have only considered basic, simple, and static handshapes, which have not been practically implemented as real-world applications. To resolve the shortage of sign language interpreters, a sign language recognition application that interprets the sign language is required. Thus, the aim of this study was to develop and evaluate a sign language recognition framework using multi-modalities approach and spatio-temporal features that include dynamic handshapes. The proposed framework consists of three main parts, namely handshape recognition, movement tracking, and sign recognition. In this study, the use of hand skeletal data as features was also investigated, which were input to a bi-directional long short-term memory (Bi-LSTM) model for sign recognition. The proposed model was evaluated on a continuous Chinese sign language (CSL) dataset of 8 subjects with 1200 sample videos covering 100 signs. The experimental results demonstrated a true recognition rate of 98.75%, outperforming most of the state-of-the-art alternatives used for sign language recognition. The proposed sign recognition application can be deployed in public service sectors such as banks, hospitals, and police stations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Wang C, Gao W, Shan S. In: Proceedings of fifth IEEE international conference on automatic face gesture recognition. 2002. p. 411–16.

  2. Yao D, Jiang M, Huang Y, Abulizi A, Li H. Study of sign segmentation in the text of Chinese sign language. Univ Access Inf Soc. 2016;16:725. https://doi.org/10.1007/s10209-016-0506-8.

    Article  Google Scholar 

  3. Yang X, Chen X, Cao X, Wei S, Zhang X. Chinese sign language recognition based on an optimized tree-structure framework. IEEE J Biomed Health Inform. 2017;21(4):994.

    Article  Google Scholar 

  4. Crasborn O, Mesch J, Waters D, Nonhebel A, van der kooij E, Woll B, Bergman B. Sharing sign language data online: experiences from the echo project. Int J Corpus Linguist 2007;12:535. https://doi.org/10.1075/ijcl.12.4.06cra.

  5. Xuezhong L, Xiaomei O, Yan D. The genetic deafness in Chinese population. J Otol. 2006;1(1):1. https://doi.org/10.1016/S1672-2930(06)50001-7.

    Article  Google Scholar 

  6. Deafness, hearing loss. World Health Organization. 2020. https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss

  7. Chong TW, Lee BG. American sign language recognition using leap motion controller with machine learning approach. Sensors. 2018;18(1):3554. https://doi.org/10.3390/s18103554.

    Article  Google Scholar 

  8. Rastgoo R, Kiani K, Escalera S. Video-based isolated hand sign language recognition using a deep cascaded model. Multimed Tools Appl. 2020. https://doi.org/10.1007/s11042-020-09048-5.

    Article  Google Scholar 

  9. Klima E.S, Bellugi U. The signs of language. London: Harvard University Press; 2010. http://lcn.salk.edu/publications/SOL/SOL+-+6+Chinese+vs+American+Signs.pdf

  10. Koller O, Ney H, Bowden R. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). 2016. p. 3793–3802. https://doi.org/10.1109/CVPR.2016.412

  11. Siby J, Kader H, Jose J. Hand gesture recognition. Int J Innov Technol Res. 2015;32.

  12. Lamberti L. F. Camastra. In: Maino G, Foresti GL, editors. Image analysis and processing—ICIAP 2011. Berlin: Springer; 2011. p. 365–73.

    Chapter  Google Scholar 

  13. Bheda V, Radpour D. Using deep convolutional networks for gesture recognition in American sign language. 2017. arXiv:1710.06836 [CoRR].

  14. Pan T, Lo L, Yeh C, Li J, Liu H, Hu M. In: 2016 IEEE second international conference on multimedia big data (BigMM). 2016. p. 64–7.

  15. Ren Z, Yuan J, Meng J, Zhang Z. Robust part-based hand gesture recognition using kinect sensor. IEEE Trans Multimed. 2013;15(5):1110.

    Article  Google Scholar 

  16. Liang Z, Liao S, Hu B. 3d convolutional neural networks for dynamic sign language recognition. Comput J. 2018;61:1725. https://doi.org/10.1093/comjnl/bxy049.

    Article  Google Scholar 

  17. Chai X, Li G, Lin Y, Xu Z, Tang YB, Chen X. In: Proceeding IEEE international conference of automatic face and gesture recognition, Shanghai, China. 2013. p. 22–6.

  18. Soodtoetong N, Gedkhaw E. In: 2018 15th international conference on electrical engineering/electronics, computer, telecommunications and information technology (ECTI-CON). 2018. p. 70–3. https://doi.org/10.1109/ECTICon.2018.8619984.

  19. Garcia-Bautista G, Trujillo-Romero F, Morales SOC. Mexican sign language recognition using kinect and data time warping algorithm. In: 2017 international conference on electronics, communications and computers (CONIELECOMP). 2017. p. 1–5.

  20. Yang L, Chen J, Zhu W. Dynamic hand gesture recognition based on a leap motion controller and two-layer bidirectional recurrent neural network. Sensors. 2020;20:2106. https://doi.org/10.3390/s20072106.

    Article  Google Scholar 

  21. Khelil B, Amiri H. In: Proceeding 3rd international conference on automation, control, engineering and computer science, Hammamet, Tunisia. 2016. p. 20–2.

  22. Du Y, Liu S, Feng L, Chen M, Wu J. Hand gesture recognition with leap motion. 2017. arXiv:1711.04293 [CoRR].

  23. Mittal A, Kumar P, Roy PP, Balasubramanian R, Chaudhuri BB. A modified lstm model for continuous sign language recognition using leap motion. IEEE Sens J. 2019;19(16):7056.

    Article  Google Scholar 

  24. Biradar S, Tuppad AM. A static hand gesture classification system for American sign language (asl) fingerspelling and digits. Int J Latest Trends Eng Technol. 2016;7(1):695.

    Google Scholar 

  25. Bhavsar H, Trivedi J. Review on feature extraction methods of image based sign language recognition system. Indian J Comput Sci Eng. 2017;8(3):249.

    Google Scholar 

  26. Zamani M, Kanan HR. In: 2014 4th international conference on computer and knowledge engineering (ICCKE). 2014. p. 398–403.

  27. Belissen V. In: 20th international ACM SIGACCESS conference on computers and accessibility. 2018. p. 1–3.

  28. Camgoz NC, Hadfield S, Koller O, Bowden R. In: 2017 IEEE international conference on computer vision (ICCV). 2017. p. 3075–84. https://doi.org/10.1109/ICCV.2017.332.

  29. Khan AU, Borji A. In: 2018 IEEE/CVF conference on computer vision and pattern recognition. 2018. p. 4710–19. https://doi.org/10.1109/CVPR.2018.00495.

  30. Imran J, Raman B. Deep motion templates and extreme learning machine for sign language recognition. Visual Comput. 2020. https://doi.org/10.1007/s00371-019-01725-3.

  31. Cai M, Lu F, Sato Y. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR). 2020. p. 14380–14389. https://doi.org/10.1109/CVPR42600.2020.01440.

  32. Li Y, Yuan L, Vasconcelos N. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). 2019. p. 6929–38. https://doi.org/10.1109/CVPR.2019.00710.

  33. Toldo M, Maracani A, Michieli U, Zanuttigh P. Unsupervised domain adaptation in semantic segmentation: a review. 2020. arXiv:2005.10876 [CoRR].

  34. Preetham C, Ramakrishnan G, Kumar S, Tamse A, Krishnapura N. In: 2013 Texas instruments India educators’ conference. 2013. p. 328–31.

  35. Patil K, Pendharkar G, Gaikwad GN. American sign language detection. Int J Sci Res Publ. 2014;4(11):1.

    Google Scholar 

  36. Jingqiu W, Ting Z. In: The 26th Chinese control and decision conference (2014 CCDC). 2014. p. 1580–84.

  37. Lee BG, Lee SM. Smart wearable hand device for sign language interpretation system with sensors fusion. IEEE Sens J. 2018;18(3):1224.

    Article  Google Scholar 

  38. Fang B, Sun F, Liu H, Liu C. 3d human gesture capturing and recognition by the immu-based data glove. Neurocomputing. 2018;277:198. https://doi.org/10.1016/j.neucom.2017.02.101.

    Article  Google Scholar 

  39. Cheng J, Chen X, Liu A, Peng H. A novel phonology- and radical-coded Chinese sign language recognition framework using accelerometer and surface electromyography sensors. Sensors (Basel, Switzerland). 2015;15:23303. https://doi.org/10.3390/s150923303.

    Article  Google Scholar 

  40. Liu J, Shahroudy A, Xu D, Kot AC, Wang G. Skeleton-based action recognition using spatio-temporal lstm network with trust gates. IEEE Trans Pattern Anal Mach Intell. 2018;40(12):3007.

    Article  Google Scholar 

  41. Yong D, Wang W, Wang L. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR). 2015. p. 1110–18.

  42. Liu J, Wang G, Duan L, Abdiyeva K, Kot AC. Skeleton-based human action recognition with global context-aware attention lstm networks. IEEE Trans Image Process. 2018;27(4):1586.

    Article  MathSciNet  Google Scholar 

  43. Simon T, Joo H, Matthews IA, Sheikh Y. Hand keypoint detection in single images using multiview bootstrapping. 2017. arXiv:1704.07809 [CoRR].

  44. Cao Z, Hidalgo G, Simon T, Wei S, Sheikh Y. Openpose: realtime multi-person 2d pose estimation using part affinity fields. 2018. arXiv:1812.08008 [CoRR].

  45. Cheng J, Lu J, Zhang HC, Lei F, Sardar M, Bian XT, Zuo F, Shen ZH, Ni XW, Shi J. Combining cubic spline interpolation and fast Fourier transform to extend measuring range of reflectometry. Chin Phys Lett. 2018;35:050701. https://doi.org/10.1088/0256-307X/35/5/050701.

    Article  Google Scholar 

  46. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735–80. https://doi.org/10.1162/neco.1997.9.8.1735.

    Article  Google Scholar 

  47. Schuster M, Paliwal KK. Bidirectional recurrent neural networks. IEEE Trans Signal Process. 1997;45(11):2673.

    Article  Google Scholar 

  48. Meng L, Li R. An attention-enhanced multi-scale and dual sign language recognition network based on a graph convolution network. Sensors. 2021;21(4). https://doi.org/10.3390/s21041120. https://www.mdpi.com/1424-8220/21/4/1120.

  49. Yin F, Chai X, Chen X. Iterative reference driven metric learning for signer independent isolated sign language recognition. 2016;9911:434–50. https://doi.org/10.1007/978-3-319-46478-7_27.

  50. Özdemir O, Camgöz NC, Akarun L. In: 2016 24th signal processing and communication application conference (SIU). 2016. p. 1961–64. https://doi.org/10.1109/SIU.2016.7496151.

  51. Carreira J, Zisserman A. Quo vadis, action recognition? A new model and the kinetics dataset. 2017. arXiv:1705.07750 [CoRR].

Download references

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, Pukyong National University, Busan, 608-737, South Korea

    Wan-Young Chung

  2. School of Computer Science, Faculty of Science and Engineering, University of Nottingham Ningbo China, Ningbo, 315100, Zhejiang, China

    Haokai Xu & Boon Giin Lee

Authors
  1. Wan-Young Chung
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Haokai Xu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Boon Giin Lee
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Boon Giin Lee.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported in part by the Faculty of Science and Engineering, University of Nottingham Ningbo China under Grant BS123456. This work is also supported by the National Research Foundation of Korea (NRF) grant funded by the Korean Government (MIST) (2019R1A2C1089139).

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chung, WY., Xu, H. & Lee, B.G. Chinese Sign Language Recognition with Batch Sampling ResNet-Bi-LSTM. SN COMPUT. SCI. 3, 414 (2022). https://doi.org/10.1007/s42979-022-01341-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-022-01341-4

Keywords