Skip to main content
SpringerLink
Log in

Real-time continuous handwritten trajectories recognition based on a regression-based temporal pyramid network

  • Research
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In the field of dynamic gesture trajectory recognition, it is difficult to real-time recognize its semantics on the continuous handwritten trajectories because of the difficulty of trajectory segment accurately. In this paper, focuses on the semantic recognition for the handwritten trajectories of continuous numeric characters, a regression-based time pyramid network real-time recognition method is proposed. Firstly, we use corner detection algorithms to obtain the corner points of the fingers, and then construct reasonable convex functions to obtain the unique fingertip point. Then, we perform hierarchical construction of the extracted fingertip trajectory features using a time pyramid, and then aggregate the features that have undergone spatial semantic modulation and temporal rate modulation. Finally, utilizing the idea of regression detection, we predict and classify the extracted trajectory features in a specialized fully connected layer with N neural nodes. According to the experimental results, our method achieved a recognition accuracy of up to 78.87%, while also achieving a recognition speed of 32.69 fps. Our method achieves a good balance between recognition accuracy and recognition speed, which indicates that our approach has significant advantages in real-time recognition of continuous handwritten trajectories.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

The data are available from the corresponding author on reasonable request.

References

  1. Jiang, H., Wachs, J.P., Duerstock, B.S.: An optimized real-time hands gesture recognition based interface for individuals with upper-level spinal cord injuries. J. Real-Time Image Proc. 11, 301–314 (2016)

    Article  Google Scholar 

  2. Liu, K., Kehtarnavaz, N.: Real-time robust vision-based hand gesture recognition using stereo images. J. Real-Time Image Proc. 11, 201–209 (2016)

    Article  Google Scholar 

  3. Zhang, T.: Application of AI-based real-time gesture recognition and embedded system in the design of English major teaching. Wirel. Netw. 27, 1–13 (2021)

    Google Scholar 

  4. Juan, W.: Gesture recognition and information recommendation based on machine learning and virtual reality in distance education. J. Intell. Fuzzy Syst. 40, 7509–7519 (2021)

    Article  Google Scholar 

  5. Magrofuoco, N., Roselli, P., Vanderdonckt, J.J.A.C.S.: Two-dimensional stroke gesture recognition. A survey. ACM Comput. Surv. 54(7), 1–36 (2021)

    Article  Google Scholar 

  6. Yao, J., et al.: Development of a wearable electrical impedance tomographic sensor for gesture recognition with machine learning. IEEE J. Biomed. Health Inform. 24(6), 1550–1556 (2020)

    Article  PubMed  Google Scholar 

  7. Lin, C.-S., Chen, P.-C., Pan, Y.-C., Chang, C.-M., Huang, K.-L.: The manipulation of real-time kinect-based robotic arm using double-hand gestures. J. Sens. 2020,1–9 (2020)

    Article  ADS  Google Scholar 

  8. Lu, W., Tong, Z., Chu, J.: Controller, dynamic hand gesture recognition with leap motion. IEEE Signal Process. Lett. 23, 1188–1192 (2016)

    Article  ADS  Google Scholar 

  9. Zhou, Y., Jiang, G., Lin, Y.: A novel finger and hand pose estimation technique for real-time hand gesture recognition. Pattern Recogn. 49, 102–114 (2016)

    Article  ADS  Google Scholar 

  10. De Smedt, Q., Wannous, H., Vandeborre, J.-P.: Heterogeneous hand gesture recognition using 3D dynamic skeletal data. Comput. Vis. Image Underst. 181, 60–72 (2019)

    Article  Google Scholar 

  11. Dahmani, D., Larabi, S.: User-independent system for sign language finger spelling recognition. J. Vis. Commun. Image Represent. 25(5), 1240–1250 (2014)

    Article  Google Scholar 

  12. Hsu, Y.-L., Chu, C.-L., Tsai, Y.-J., Wang, J.-S.: An inertial pen with dynamic time warping recognizer for handwriting and gesture recognition. IEEE Sens. J. 15(1), 154–163 (2015)

    Article  ADS  Google Scholar 

  13. Tang, J., Cheng, H., Zhao, Y., Guo, H.: Structured dynamic time warping for continuous hand trajectory gesture recognition. Pattern Recogn. 80, 21–31 (2018)

    Article  ADS  Google Scholar 

  14. Baranwal, N., Nandi, G.C., Singh, A.K.: Real-time gesture-based communication using possibility theory-based hidden Markov model: real-time gesture-based communication using PTBHMM. Comput. Intell. 33(4), 843–862 (2017)

    Article  MathSciNet  Google Scholar 

  15. Sagayam, K.M., Hemanth, D.J.: ABC algorithm based optimization of 1-D hidden Markov model for hand gesture recognition applications. Comput. Ind. 99, 313–323 (2018)

    Article  Google Scholar 

  16. Chen, H., Liu, X., Shi, J., Zhao, G.: Temporal hierarchical dictionary guided decoding for online gesture segmentation and recognition. IEEE Trans. Image Process. 29, 9689–9702 (2020)

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  17. Jian, C., Li, J., Zhang, M.: LSTM-based dynamic probability continuous hand gesture trajectory recognition. IET Image Proc. 13(12), 2314–2320 (2019)

    Article  Google Scholar 

  18. Ur Rehman, M., et al.: Dynamic hand gesture recognition using 3d-CNN and LSTM networks. Comput. Mater. Contin. 70(3), 4675–4690 (2021)

    Google Scholar 

  19. Yang, Z., Zheng, X.: Hand gesture recognition based on trajectories features and computation-efficient reused LSTM network. IEEE Sens. J. 21(15), 16945–16960 (2021)

    Article  ADS  Google Scholar 

  20. Jangpangi, M., Kumar, S., Bhardwaj, D., et al.: Handwriting recognition using Wasserstein metric in adversarial learning. SN Comput. Sci. 4(1), 1–10 (2023)

    Google Scholar 

  21. Alemayoh, T.T., Shintani, M., Lee, J.H., Okamoto, S.: Deep-learning-based character recognition from handwriting motion data captured using IMU and force sensors. Sensors (Basel, Switzerland) 22(20), 7840 (2022)

    Article  ADS  PubMed  Google Scholar 

  22. Ghosh, T., Sen, S., Obaidullah, S.M., Santosh, K.C., Roy, K., Pal, U.: Advances in online handwritten recognition in the last decades. Comput. Sci. Rev. 46, 100515 (2022)

    Article  MathSciNet  Google Scholar 

  23. Sun, F., Kong, T., Huang, W., Tan, C., Fang, B., Liu, H.: Feature pyramid reconfiguration with consistent loss for object detection. IEEE Trans. Image Process. 28(10), 5041–5051 (2019)

    Article  ADS  MathSciNet  Google Scholar 

  24. Huang, Y., Cao, X., Zhen, X., Han, J.: Attentive temporal pyramid network for dynamic scene classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 8497–8504 (2019)

  25. Yang, C., Xu, Y., Shi, J., Dai, B., Zhou, B.: Temporal pyramid network for action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 591–600 (2020)

  26. Li, Y., Liang, R., Wei, W., Wang, W., Zhou, J., Li, X.: Temporal pyramid network with spatial-temporal attention for pedestrian trajectory prediction. IEEE Trans. Netw. Sci. Eng. 9(3), 1006–1019 (2022)

    Article  Google Scholar 

  27. Jian, C., Liu, X., Zhang, M.: RD-Hand: a real-time regression-based detector for dynamic hand gesture. Appl. Intell. (Dordrecht, Netherlands). 52(1), 417–428 (2022)

    Google Scholar 

  28. Ali, H., Jirak, D., Wermter, S.: Snapture—a novel neural architecture for combined static and dynamic hand gesture recognition. Cogn. Comput. 15,2014-2033 (2023)

    Article  Google Scholar 

  29. Mazzia, V., Salvetti, F., Chiaberge, M.: Efficient-CapsNet: capsule network with self-attention routing. Sci. Rep. 11(1), 14634–14634 (2021)

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  30. Kowsari, K., Heidarysafa, M., Brown, D.E., Meimandi, K.J., Barnes, L.E.: Rmdl: random multimodel deep learning for classification. In: Proceedings of the 2nd International Conference on Information System and Data Mining, pp. 19–28 (2018)

  31. Jayasundara, V., Jayasekara, S., Jayasekara, H., Rajasegaran, J., Seneviratne, S., Rodrigo, R.: Textcaps: handwritten character recognition with very small datasets. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 254–262 (2019)

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 61672461 and 62073293.

Author information

Authors and Affiliations

  1. Computer Science and Technology College, Zhejiang University of Technology, Hangzhou, China

    Chengfeng Jian, Mengqi Wang & Meiyu Zhang

  2. Zhejiang Highway Technicians College, Hangzhou, China

    Min Ye

Authors
  1. Chengfeng Jian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mengqi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Min Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Meiyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Min Ye or Meiyu Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jian, C., Wang, M., Ye, M. et al. Real-time continuous handwritten trajectories recognition based on a regression-based temporal pyramid network. J Real-Time Image Proc 21, 10 (2024). https://doi.org/10.1007/s11554-023-01382-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11554-023-01382-9

Keywords

Search

Navigation