Skip to main content

The Influence of Iconicity in Transfer Learning for Sign Language Recognition

  • Conference paper
  • First Online:
Natural Language Processing and Information Systems (NLDB 2024)

Abstract

Most sign language recognition research relies on Transfer Learning (TL) from vision-based datasets such as ImageNet. Some extend this to alternatively available language datasets, often focusing on signs with cross-linguistic similarities. This body of work examines the necessity of these likenesses on effective knowledge transfer by comparing TL performance between iconic signs of two different sign language pairs: Chinese to Arabic and Greek to Flemish. Google Mediapipe was utilised as an input feature extractor, enabling spatial information of these signs to be processed with a Multilayer Perceptron architecture and the temporal information with a Gated Recurrent Unit. Experimental results showed a 7.02% improvement for Arabic and 1.07% for Flemish when conducting iconic TL from Chinese and Greek respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://ustc-slr.github.io/datasets/2015_csl.

  2. 2.

    https://hamzah-luqman.github.io/KArSL/.

  3. 3.

    https://vcl.iti.gr/dataset/gsl/.

  4. 4.

    https://taalmaterialen.ivdnt.org/download/woordenboek-vgt/.

  5. 5.

    https://ieee-dataport.org/open-access/display-multimodal-medslset-medical-sign-language-set.

  6. 6.

    https://lsfb.info.unamur.be/.

References

  1. Holistic landmarks detection task guide | MediaPipe | Google for Developers — developers.google.com. https://developers.google.com/mediapipe/solutions/vision/holistic_landmarker. Accessed 17 Sep 2023

  2. Adaloglou, N., et al.: A comprehensive study on deep learning-based methods for sign language recognition. IEEE Trans. Multimedia 24, 1750–1762 (2022). https://doi.org/10.1109/TMM.2021.3070438

    Article  Google Scholar 

  3. Altaf, Y., Wahid, A., Kirmani, M.M.: Deep learning approach for sign language recognition using densenet201 with transfer learning. In: 2023 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS), pp. 1–6 (2023). https://doi.org/10.1109/SCEECS57921.2023.10063044

  4. Artiaga, K., Li, Y., Kuruoglu, E.E., Chan, W.K.V.: Cross-sign language transfer learning using domain adaptation with multi-scale temporal alignment. Multimedia Tools Appl. (2023). https://doi.org/10.1007/s11042-023-16703-0

    Article  Google Scholar 

  5. Bird, J.J., Ekárt, A., Faria, D.R.: British sign language recognition via late fusion of computer vision and leap motion with transfer learning to American sign language. Sensors 20(18) (2020). https://doi.org/10.3390/s20185151

  6. Brosens, C., Janssens, M., Verstraete, S., Vandamme, T., De Durpel, H.: Moving towards a functional approach in the Flemish Sign Language dictionary making process. In: Efthimiou, E., et al. (eds.) Proceedings of the LREC2022 10th Workshop on the Representation and Processing of Sign Languages: Multilingual Sign Language Resources, pp. 24–28. European Language Resources Association, Marseille, France (2022). https://aclanthology.org/2022.signlang-1.4

  7. Das, S., Imtiaz, M.S., Neom, N.H., Siddique, N., Wang, H.: A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier. Expert Systems with Applications 213, 118914 (2023). https://doi.org/10.1016/j.eswa.2022.118914, https://www.sciencedirect.com/science/article/pii/S0957417422019327

  8. Duy Khuat, B., Thai Phung, D., Thi Thu Pham, H., Ngoc Bui, A., Tung Ngo, S.: Vietnamese sign language detection using mediapipe. In: Proceedings of the 2021 10th International Conference on Software and Computer Applications, pp. 162–165. ICSCA ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3457784.3457810, https://doi.org/10.1145/3457784.3457810

  9. Fink, J., Frenay, B., Meurant, L., Cleve, A.: LSFB-CONT and LSFB-ISOL: two new datasets for vision-based sign language recognition. In: 2021 International Joint Conference on Neural Networks (IJCNN). IEEE (2021). https://doi.org/10.1109/ijcnn52387.2021.9534336

  10. Halvardsson, G., Peterson, J., Soto-Valero, C., Baudry, B.: Interpretation of Swedish sign language using convolutional neural networks and transfer learning, pp. 207 (2021). https://doi.org/10.1007/s42979-021-00612-w

  11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  12. Hu, H., Zhao, W., Zhou, W., Li, H.: Signbert+: hand-model-aware self-supervised pre-training for sign language understanding. IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI), pp. 1–20 (2023). https://doi.org/10.1109/TPAMI.2023.3269220

  13. Hu, H., Zhao, W., Zhou, W., Wang, Y., Li, H.: Signbert: pre-training of hand-model-aware representation for sign language recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11087–11096 (2021). https://doi.org/10.1109/ICCV48922.2021.01090

  14. Jaiswal, M., Sharmay, V., Sharmaz, A., Tomar, R.: Transfer learning with l2 norm regularization for classifying static two hand Hindi sign language gestures. In: 2020 IEEE 9th International Conference on Communication Systems and Network Technologies (CSNT), pp. 44–48 (2020). https://doi.org/10.1109/CSNT48778.2020.9115767

  15. Jiang, X., Hu, B., Satapathy, S.C., Wang, S., Zhang, Y.: Fingerspelling identification for Chinese sign language via alexnet-based transfer learning and Adam optimizer. Sci. Program. 2020, 3291426:1–3291426:13 (2020). https://doi.org/10.1155/2020/3291426

  16. Khodapanah Aghdam, E., et al.: Display multimodal medslset (medical sign language set) (2020). https://doi.org/10.21227/5gsb-fb69

  17. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015). https://doi.org/10.48550/arXiv.1412.6980

  18. Laines, D., Gonzalez-Mendoza, M., Ochoa-Ruiz, G., Bejarano, G.: Isolated sign language recognition based on tree structure skeleton images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 276–284 (2023). https://doi.org/10.1109/CVPRW59228.2023.00033

  19. Lakew, S.M., Karakanta, A., Federico, M., Negri, M., Turchi, M.: Adapting multilingual neural machine translation to unseen languages. In: Proceedings of the 16th International Conference on Spoken Language Translation. Association for Computational Linguistics, Hong Kong (2019). https://doi.org/10.48550/arXiv.1910.13998

  20. Liu, Y., et al.: Multilingual denoising pre-training for neural machine translation. Trans. Assoc. Comput. Linguis. 8, 726–742 (2020). https://doi.org/10.1162/tacl_a_00343

  21. Marivate, V., et al.: Investigating an approach for low resource language dataset creation, curation and classification: Setswana and Sepedi. arXiv preprint arXiv:2003.04986 (2020)

  22. Morocho-Cayamcela, M.E., Lim, W.: Fine-tuning a pre-trained convolutional neural network model to translate American sign language in real-time. 2019 International Conference on Computing, Networking and Communications (ICNC), pp. 100–104 (2019). https://doi.org/10.1109/ICCNC.2019.8685536

  23. Nishat, Z.K., Shopon, M.: Unsupervised pretraining and transfer learning-based Bangla sign language recognition. In: Proceedings of International Joint Conference on Computational Intelligence Algorithms for Intelligent Systems, pp. 529-540 (2020). https://doi.org/10.1007/978-981-15-3607-6_42

  24. Perlman, M., Little, H., Thompson, B., Thompson, R.L.: Iconicity in signed and spoken vocabulary: a comparison between American sign language, British sign language, English, and Spanish. Front. Psyc. 9 (2018). https://doi.org/10.3389/fpsyg.2018.01433

  25. Rathi, D.: Optimization of transfer learning for sign language recognition targeting mobile platform. In: International Journal on Recent and Innovation Trends in Computing and Communication. vol. 6, pp. 198–203 (2018). https://doi.org/10.48550/arXiv.1805.06618

  26. Shania, S., Naufal, M.F., Prasetyo, V.R., Azmi, M.S.B.: Translator of Indonesian sign language video using convolutional neural network with transfer learning. Indonesian J. Inf. Syst. (2022). https://doi.org/10.24002/ijis.v5i1.5865

  27. Sharma, S., Singh, S.: ISL recognition system using integrated mobile-net and transfer learning method. Expert Syst. Appl. 221, 119772 (2023). https://doi.org/10.1016/j.eswa.2023.119772

    Article  Google Scholar 

  28. Sidig, A.A.I., Luqman, H., Mahmoud, S., Mohandes, M.: KARSL: Arabic sign language database. ACM Trans. Asian Low-Resource Lang. Inf. Proc. (TALLIP) 20(1), 1–19 (2021). https://doi.org/10.1145/3423420

    Article  Google Scholar 

  29. Stokoe, W.C., Casterline, D.C., Croneberg, C.G.: A dictionary of American sign language on linguistic principles. (No Title) (1976)

    Google Scholar 

  30. Sultan, A., Makram, W., Kayed, M., Ali, A.A.: Sign language identification and recognition: a comparative study. Open Comput. Sci. 12(1), 191–210 (2022). https://doi.org/10.1515/comp-2022-0240

    Article  Google Scholar 

  31. Tarrés, L., Gállego, G.I., Duarte, A., Torres, J., Giró-i Nieto, X.: Sign language translation from instructional videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5624–5634 (2023). https://doi.org/10.1109/CVPRW59228.2023.00596

  32. Thakar, S., Shah, S., Shah, B., Nimkar, A.V.: Sign language to text conversion in real time using transfer learning. 2022 IEEE 3rd Global Conference for Advancement in Technology (GCAT), pp. 1–5 (2022). https://doi.org/10.1109/GCAT55367.2022.9971953

  33. Vázquez, R., Boggia, M., Raganato, A., Loppi, N.A., Grönroos, S.A., Tiedemann, J.: Latest development in the FoTran project – scaling up language coverage in neural machine translation using distributed training with language-specific components. In: Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pp. 311–312. European Association for Machine Translation, Ghent, Belgium (2022)

    Google Scholar 

  34. Vázquez-Enríquez, M., Alba-Castro, J.L., Docío-Fernández, L., Rodríguez-Banga, E.: Isolated sign language recognition with multi-scale spatial-temporal graph convolutional networks. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3457–3466 (2021). https://doi.org/10.1109/CVPRW53098.2021.00385

  35. Zakariah, M., Alotaibi, Y.A., Koundal, D., Guo, Y., Elahi, M.M.: Sign language recognition for Arabic alphabets using transfer learning technique. Comput. Intell. Neurosci. 2022 (2022). https://doi.org/10.1155/2022/4567989

  36. Zhou, H., Zhou, W., Qi, W., Pu, J., Li, H.: Improving sign language translation with monolingual data by sign back-translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1316–1325 (2021). https://doi.org/10.1109/GCAT55367.2022.9971953

  37. Östling, R., Börstell, C., Courtaux, S.: visual iconicity across sign languages: large-scale automated video analysis of iconic articulators and locations. Front. Psyc. 9 (2018).https://doi.org/10.3389/fpsyg.2018.00725

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Keren Artiaga .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Artiaga, K., Lynch, C., Afli, H., Hasanuzzaman, M. (2024). The Influence of Iconicity in Transfer Learning for Sign Language Recognition. In: Rapp, A., Di Caro, L., Meziane, F., Sugumaran, V. (eds) Natural Language Processing and Information Systems. NLDB 2024. Lecture Notes in Computer Science, vol 14762. Springer, Cham. https://doi.org/10.1007/978-3-031-70239-6_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-70239-6_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-70238-9

  • Online ISBN: 978-3-031-70239-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics