Abstract
Sign language is the communication medium between deaf and hearing people and has unique grammatical rules. Compared with isolated word recognition, continuous sign language recognition is more context-dependent, semantically complex, and challenging to segment temporally. The current research still needs to be improved regarding recognition accuracy, background interference resistance, and overfitting resistance. The unique coding and decoding structure of the Transformer model can be used for sign language recognition. However, its position encoding method and multi-headed self-attentive mechanism still need to be improved. This paper proposes a sign language recognition algorithm based on the improved Transformer target detection network model (SL-OTT). The continuous sign language recognition method based on the improved Transformer model computes each word vector in a continuous sign language sentence in multiple cycles by multiplexed position encoding with parameters to accurately grasp the position information between each word; adds learnable memory key-value pairs to the attention module to form a persistent memory module, and expands the number of attention heads and embedding dimension by linear high-dimensional mapping in equal proportion. The proposed method achieves competitive recognition results on the most authoritative continuous sign language dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Koller, O., Zargaran, O., Ney, H., et al.: Deep sign: hybrid CNN-HMM for continuous sign language recognition. In: British Machine Vision Conference 2016, pp. 1–2. British Machine Vision Association, York (2016)
Zhang, Z., Pu, J., Zhuang, L., Zhou, W., et al.: Continuous sign language recognition via reinforcement learning. In: International Conference on Image Processing (ICIP), pp. 285–289. IEEE Computer Society, Piscataway, NJ (2019)
Cui, R., Liu, H., Zhang, C.: Recurrent convolutional neural networks for continuous sign language recognition by staged optimization. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 7361–7369. IEEE Computer Society, Piscataway, NJ (2017)
Camgoz, N.C., Koller, O., Hadfield, S., et al.: Sign language Transformers: joint end-to-end sign language recognition and translation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10023–10033. IEEE Computer Society, Piscataway, NJ (2020)
Niu, Z., Mak, B.: Stochastic fine-grained labeling of multi-state sign glosses for continuous sign language recognition. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI, pp. 172–186. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-58517-4_11
Culati, A., Chiu, C.C., Qin, J., et al.: Conformer: convolution-augmented ‘Transformer for speech recognition. In: Proceedings of the INTERSPEECH 2020, pp. 5036–5040. International Speech Communication Association (ISCA), Baixas (2020)
Koller, O., Zargaran, S., Ney, H.: Re-sign: re-aligned end-to-end sequence modeling with deep recurrent CNN-HMMs. InL l/IEEE International Conference on Computer Vision and Pattern Recognition, pp. 4297–4305. IEEE Computer Society, Piscataway, NJ (2017)
Graves, A., Fernandezs, G.F., et al.: Connectionist temporal classification: labeling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 369–376 (2006)
Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. Adv. Neural Inform. Process. Syst. 5998–6008 (2017)
Molchanov, P., Yang, X., Gupta, S., et al.: Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4207–4215 (2016)
Ren, Z., Zhang, Y., Wang, S.: A hybrid framework for lung cancer classification. Electronics 11(10), 1614 (2022)
Wang, W., Pei, Y., Wang, S.H., Gorrz, J.M., Zhang, Y.D.: PSTCNN: Explainable COVID-19 diagnosis using PSO-guided self-tuning CNN. Biocell 47, 373–384 (2023)
Kollar, O., Camgoz, N.C., Ney, H., et al.: Weakly supervised learning with multi-stream CNN-LSTM-HMMs to discover sequential parallelism in sign language videos. IEEE Trans. Pattern Anal. Mach. Intell. 42(9), 2306–2320 (2019)
Hartigan, J.A., Wong, M.A.A.: K-means clustering algorithm. J. R. Stat. Soc. Ser. C (Appl. Stat.) 28(1), 100–108 (1979)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)
Huang, J., Zhou, W.G., Zhang, Q.L., et al.: Video-based sign language recognition without temporal segmentation. In: AAAI Conference on Artificial Intelligence, pp. 2–7. AAAI, New Orleans (2018)
Camgoz, N.C., Hadfield, S., Koller, O., et al.: SubUNets end-to-end hand shape and continuous sign language recognition. In: IEEE International Conference on Computer Vision (ICCV), pp. 3075–3084. IEEE, Venice (2017)
Pu, J., Zhou, W., Li, H.: Iterative alignment network for continuous sign language recognition. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4165–4174. IEEE Computer Society, Piscataway, NJ (2019)
Slimane, F., Bouguessa, M.: Context matters: self-attention for sign language recognition. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 7884–7891. Milan, Italy (2021)
Camgoz, N.C., Hadfield, S., Koller, O., et al.: Neural sign language translation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7784–7793. IEEE Computer Society, Piscataway (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Liu, L., Yang, Z., Liu, Y., Zhang, X., Yang, K. (2024). A Sign Language Recognition Based on Optimized Transformer Target Detection Model. In: Wang, B., Hu, Z., Jiang, X., Zhang, YD. (eds) Multimedia Technology and Enhanced Learning. ICMTEL 2023. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 535. Springer, Cham. https://doi.org/10.1007/978-3-031-50580-5_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-50580-5_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-50579-9
Online ISBN: 978-3-031-50580-5
eBook Packages: Computer ScienceComputer Science (R0)