Skip to main content
SpringerLink
Log in

Research on the improved gesture tracking algorithm in sign language synthesis

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Sign language synthesis technology is a bridge of communication between deaf and non-deaf people, and has important social value. Sign language recognition is the key to sign language synthesis technology, which is to acquire gesture data containing sign language information through hardware equipment and convert it into corresponding sound or text through relevant technology. The representation of sign language by computer virtual 3D mannequin is sign language synthesis. This paper reports on the construction of a video frame sequence based on interpolation transition video frame to improve the readability of the composition of sign language. Transfer of video frames is traditionally performed via the target tracking algorithm for gesture tracking arithmetic, but this can lead to several tracking errors because of the complexity of the hand background environment, the covering of the hand, multiple perspectives, the great similarity of different sign language words, and the interaction between the hand and other parts of the body. Therefore, in the present paper, in order to improve the accuracy of target tracking, a more accurate gesture tracking method is studied. Particle filter and optical flow algorithms are combined to build a more natural and smooth 3D transition frame image, which makes video-based sign language synthesis more realistic.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Cheok MJ, Omar Z, Jaward MH (2019) A review of hand gesture and sign language recognition techniques. Int J Mach Learn Cybern 10(1):131–153

    Article  Google Scholar 

  2. Wei C, Zhao J et al (2020) Semantic boundary detection with reinforcement learning for continuous sign language recognition. IEEE Trans Circuits Syst Video Technol 99:1

    Google Scholar 

  3. Chiu YH, Wu CH, Su HY, Cheng CJ (2007) Joint optimization of word alignment and epenthesis generation for Chinese to Taiwanese sign synthesis. IEEE Trans Pattern Anal Mach Intell 29(1):28–39

    Article  Google Scholar 

  4. Li Y, Chen X, Zhang X et al (2012) A sign-component-based framework for Chinese sign language recognition using accelerometer and sEMG data. IEEE Trans Biomed Eng 59(10):2695–2704

    Article  Google Scholar 

  5. Gineke A, Ten Holt AJ, Van Doom MJT, Reinders EA, Hendriks HDR (2011) Human-inspired search for redundancy in automatic sign language recognition. ACM Trans Appl Percept 8(2):1–15

    Article  Google Scholar 

  6. Gang H, Xing C, Ying Z, Xiaodong Z (2012) Towards Architecture-based management of platforms in the cloud. Front Comp Sci 6(4):388–397

    Article  MathSciNet  Google Scholar 

  7. Takayama N, Takahashi H (2019) Weakly-supervised learning for continuous sign language word recognition using DTW based forced alignment and isolated word HMM adjustment. IIEE J Trans Image Electr Vis Comput 7(2):88–96

    Google Scholar 

  8. Huang S, Mao C, Tao J et al (2018) A novel Chinese sign language recognition method based on key frame -centered clips. IEEE Signal Process Lett 25(3):442–446

    Article  Google Scholar 

  9. Li K, Zhou Z, Lee CH (2016) Sign transition modeling and a scalable solution to continuous sign language recognition for real-world application. ACM Trans Access Comput 8(2):1–23

    Article  Google Scholar 

  10. Runpeng C, Hu L, Changshui Z (2019) A deep neural framework for continuous sign language recognition by iterative training. IEEE Trans Multimed 21(7):1880–1891

    Article  Google Scholar 

  11. Cai X, Zhou W, Li H (2014) An effective representation for action recognition with human skeleton joints. Proc Spie Int Soc Opt Eng 9273:92731R

    Google Scholar 

  12. Koller O, Camgoz C, Ney H, Bowden R (2019) Weakly supervised learning with multi-stream CNN-LSTM-HMMs to discover sequential parallelism in sign language videos. IEEE Trans Pattern Anal Mach Intell 42(9):1–1

    Google Scholar 

  13. Suhail M K, Yidong C, Shaozi L, Xiaodong S, Jiangbin Z (2019) Technical Approaches to Chinese Sign Language Processing: A Review. IEEE Access, 7: 96926–96935.

  14. Tse-Yu P, Li-Yun L, Chung-Wei Y, Jhe-Wei L, Hou-Tim L, Min-Chun H (2018) Sign language recognition in complex background scene based on adaptive skin colour modelling and support vector machine. Int J Big Data Intell 5(1/2):21–30

    Article  Google Scholar 

  15. Anshul M, Pradeep K, Partha Pratim R, Raman B, Chaudhuri BB (2019) A modified LSTM model for continuous sign language recognition using leap motion. IEEE Sens J 19(16):7056–7063

    Article  Google Scholar 

  16. Wahid F, Tafreshi R, Langari R et al (2020) A multi-window majority voting strategy to improve hand gestured recognition accuracies using electromyography signal. IEEE Trans Neural Syst Rehabil Eng 28(2):427–436

    Article  Google Scholar 

  17. Sridhar S, Mueller F, Oulasvirta A, Theobalt C (2016) Fast and robust hand tracking using detection-guided optimization. IEICE Trans Fundam Electron Commun Comput Sci 1602:04124

    Google Scholar 

  18. Wang H, Chai X, Chen X (2019) A novel sign language recognition framework using hierarchical Grassmann covariance matrix. IEEE Trans Multimed 21(11):2806–2814

    Article  Google Scholar 

  19. Oberweger M, Riegler G, Wohlhart P, Lepetit V (2016) Efficiently creating 3D training data for fine hand pose estimation. IEICE Trans Fundam Electron Commun Comput Sci 1605:03389

    Google Scholar 

  20. Shin S, Kim WY (2020) Skeleton-based dynamic hand gesture recognition using a part-based GRU - RNN for gesture-based interface. IEEE Access 8:50236–50243

    Article  Google Scholar 

  21. Xing C, Ming L, Hao Z, Yun M, Ching-Hsien H (2022) DNNOff: offloading DNN-based intelligent IoT applications in mobile edge computing. IEEE Trans Industr Inf 18(4):2820–2829

    Article  Google Scholar 

  22. Zhang Z, Chen X, Lantz V, Yang J, Wang K (2009) Exploration on the feasibility of building muscle-computer interfaces using neck and shoulder motions. In: proceeding of the annual international conference of the IEEE engineering in medicine and biology society, Minneapolis, MN, pp. 7018–7021

  23. Yang X, Chen X, Cao X, Wei S, Zhang X (2017) Chinese sign language recognition based on an optimized tree-structure framework. IEEE J Biomed Health Inform 21(4):994–1004

    Article  Google Scholar 

  24. Chen X, Zhang J, Lin B, Chen Z, Wolter K, Min G (2022) Energy-efficient offloading for DNN-based smart IoT systems in cloud-edge environments. IEEE Trans Parallel Distrib Syst 33(3):683–697

    Article  Google Scholar 

  25. Chen X, Hu J, Chen Z, Lin B, Xiong N, Min G (2022) A reinforcement learning empowered feedback control system for industrial internet of things. IEEE Trans Industr Inf 18(4):2724–2733

    Article  Google Scholar 

  26. Huang G, Luo C, Wu K, Ma Y, Zhang Y, Liu X (2019) Software-defined infrastructure for decentralized data lifecycle governance: principled design and open challenges. In: IEEE international conference on distributed computing systems

  27. Ming Lim K, Tan AWC, Tan SC (2016) Block-based histogram of optical flow for isolated sign language recognition. J Vis Commun Image Represent 40:538–545

    Article  Google Scholar 

  28. Ko S-K, Kim CJ, Jung H, Cho C (2019) Neural sign language translation based on human key point estimation. Appl Sci 9(13):2683

    Article  Google Scholar 

  29. Ong SCW, Ranganath S (2005) Automatic sign language analysis: a survey and the future beyond lexical meaning. IEEE Trans Pattern Anal Mach Intell 27(6):873–891

    Article  Google Scholar 

  30. Fu L, Li Z, Ye Q et al (2022) Learning robust discriminant subspace based on joint L2, p- and L2, s-norm distance metrics. IEEE Trans Neural Netw Learn Syst 33(1):130–144. https://doi.org/10.1109/TNNLS.2020.3027588

    Article  MathSciNet  Google Scholar 

  31. Elakkiya R (2020) Machine learning based sign language recognition: a review and its research frontier. J Ambient Intell Hum Comput 12:1–20

    Google Scholar 

  32. Chen X, Yang L, Chen Z, Min G, Zheng X, Chunming R (2022) Resource allocation with workload-time windows for cloud-based software services: a deep reinforcement learning approach. IEEE Trans Cloud Comput. https://doi.org/10.1109/TCC.2022.3169157

    Article  Google Scholar 

  33. Rastgoo R, Kiani K, Escalera S (2020) Hand sign language recognition using multi-view hand skeleton. Expert Syst Appl 150:113336–113336

    Article  Google Scholar 

  34. Li D, Yu X, Xu C, Petersson L, Li H (2020) Transferring cross-domain knowledge for video sign language recognition. In: proceedings IEEE computer society conference on computer vision and pattern recognition, pp. 6204–6213

  35. Ye Q, Li Z, Fu L et al (2019) Nonpeaked discriminant analysis. IEEE Trans Neural Netw Learn Syst 30(12):3818–3832

    Article  MathSciNet  Google Scholar 

  36. Ye O, Huang P, Zhang Z, Zheng Y et al (2021) Multiview learning with robust double-sided twin SVM. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2021.3088519

    Article  Google Scholar 

  37. Huang S, Mao C, Tao J, Ye Z (2018) A novel chinese sign language recognition method based on key frame-centered clips. IEEE Signal Process Lett 25(3):442–446

    Article  Google Scholar 

Download references

Acknowledgements

The authors wish to thank the editors and reviewers. This work was supported in part by the 2021 Dongguan City College Young Teacher Development Fund Project (Project No. 2021QJY007Z).

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Macau University of Science and Technology, Macau, China

    Rong Lu

  2. College of Computer and Information, City College of Dongguan University of Technology, DongGuan, China

    Rong Lu

  3. College of General Education, Guangdong University of Science and Technology, DongGuan, China

    Quanyu Song

Authors
  1. Rong Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Quanyu Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rong Lu.

Ethics declarations

Conflict of interest

Not declared.

Data Availability Statement

The data used to support the findings of this study are included within the article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, R., Song, Q. Research on the improved gesture tracking algorithm in sign language synthesis. J Supercomput 79, 867–879 (2023). https://doi.org/10.1007/s11227-022-04705-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04705-y

Keywords

Search

Navigation