Abstract
Most sign language recognition research relies on Transfer Learning (TL) from vision-based datasets such as ImageNet. Some extend this to alternatively available language datasets, often focusing on signs with cross-linguistic similarities. This body of work examines the necessity of these likenesses on effective knowledge transfer by comparing TL performance between iconic signs of two different sign language pairs: Chinese to Arabic and Greek to Flemish. Google Mediapipe was utilised as an input feature extractor, enabling spatial information of these signs to be processed with a Multilayer Perceptron architecture and the temporal information with a Gated Recurrent Unit. Experimental results showed a 7.02% improvement for Arabic and 1.07% for Flemish when conducting iconic TL from Chinese and Greek respectively.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
References
Holistic landmarks detection task guide | MediaPipe | Google for Developers — developers.google.com. https://developers.google.com/mediapipe/solutions/vision/holistic_landmarker. Accessed 17 Sep 2023
Adaloglou, N., et al.: A comprehensive study on deep learning-based methods for sign language recognition. IEEE Trans. Multimedia 24, 1750–1762 (2022). https://doi.org/10.1109/TMM.2021.3070438
Altaf, Y., Wahid, A., Kirmani, M.M.: Deep learning approach for sign language recognition using densenet201 with transfer learning. In: 2023 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS), pp. 1–6 (2023). https://doi.org/10.1109/SCEECS57921.2023.10063044
Artiaga, K., Li, Y., Kuruoglu, E.E., Chan, W.K.V.: Cross-sign language transfer learning using domain adaptation with multi-scale temporal alignment. Multimedia Tools Appl. (2023). https://doi.org/10.1007/s11042-023-16703-0
Bird, J.J., Ekárt, A., Faria, D.R.: British sign language recognition via late fusion of computer vision and leap motion with transfer learning to American sign language. Sensors 20(18) (2020). https://doi.org/10.3390/s20185151
Brosens, C., Janssens, M., Verstraete, S., Vandamme, T., De Durpel, H.: Moving towards a functional approach in the Flemish Sign Language dictionary making process. In: Efthimiou, E., et al. (eds.) Proceedings of the LREC2022 10th Workshop on the Representation and Processing of Sign Languages: Multilingual Sign Language Resources, pp. 24–28. European Language Resources Association, Marseille, France (2022). https://aclanthology.org/2022.signlang-1.4
Das, S., Imtiaz, M.S., Neom, N.H., Siddique, N., Wang, H.: A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier. Expert Systems with Applications 213, 118914 (2023). https://doi.org/10.1016/j.eswa.2022.118914, https://www.sciencedirect.com/science/article/pii/S0957417422019327
Duy Khuat, B., Thai Phung, D., Thi Thu Pham, H., Ngoc Bui, A., Tung Ngo, S.: Vietnamese sign language detection using mediapipe. In: Proceedings of the 2021 10th International Conference on Software and Computer Applications, pp. 162–165. ICSCA ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3457784.3457810, https://doi.org/10.1145/3457784.3457810
Fink, J., Frenay, B., Meurant, L., Cleve, A.: LSFB-CONT and LSFB-ISOL: two new datasets for vision-based sign language recognition. In: 2021 International Joint Conference on Neural Networks (IJCNN). IEEE (2021). https://doi.org/10.1109/ijcnn52387.2021.9534336
Halvardsson, G., Peterson, J., Soto-Valero, C., Baudry, B.: Interpretation of Swedish sign language using convolutional neural networks and transfer learning, pp. 207 (2021). https://doi.org/10.1007/s42979-021-00612-w
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
Hu, H., Zhao, W., Zhou, W., Li, H.: Signbert+: hand-model-aware self-supervised pre-training for sign language understanding. IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI), pp. 1–20 (2023). https://doi.org/10.1109/TPAMI.2023.3269220
Hu, H., Zhao, W., Zhou, W., Wang, Y., Li, H.: Signbert: pre-training of hand-model-aware representation for sign language recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11087–11096 (2021). https://doi.org/10.1109/ICCV48922.2021.01090
Jaiswal, M., Sharmay, V., Sharmaz, A., Tomar, R.: Transfer learning with l2 norm regularization for classifying static two hand Hindi sign language gestures. In: 2020 IEEE 9th International Conference on Communication Systems and Network Technologies (CSNT), pp. 44–48 (2020). https://doi.org/10.1109/CSNT48778.2020.9115767
Jiang, X., Hu, B., Satapathy, S.C., Wang, S., Zhang, Y.: Fingerspelling identification for Chinese sign language via alexnet-based transfer learning and Adam optimizer. Sci. Program. 2020, 3291426:1–3291426:13 (2020). https://doi.org/10.1155/2020/3291426
Khodapanah Aghdam, E., et al.: Display multimodal medslset (medical sign language set) (2020). https://doi.org/10.21227/5gsb-fb69
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015). https://doi.org/10.48550/arXiv.1412.6980
Laines, D., Gonzalez-Mendoza, M., Ochoa-Ruiz, G., Bejarano, G.: Isolated sign language recognition based on tree structure skeleton images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 276–284 (2023). https://doi.org/10.1109/CVPRW59228.2023.00033
Lakew, S.M., Karakanta, A., Federico, M., Negri, M., Turchi, M.: Adapting multilingual neural machine translation to unseen languages. In: Proceedings of the 16th International Conference on Spoken Language Translation. Association for Computational Linguistics, Hong Kong (2019). https://doi.org/10.48550/arXiv.1910.13998
Liu, Y., et al.: Multilingual denoising pre-training for neural machine translation. Trans. Assoc. Comput. Linguis. 8, 726–742 (2020). https://doi.org/10.1162/tacl_a_00343
Marivate, V., et al.: Investigating an approach for low resource language dataset creation, curation and classification: Setswana and Sepedi. arXiv preprint arXiv:2003.04986 (2020)
Morocho-Cayamcela, M.E., Lim, W.: Fine-tuning a pre-trained convolutional neural network model to translate American sign language in real-time. 2019 International Conference on Computing, Networking and Communications (ICNC), pp. 100–104 (2019). https://doi.org/10.1109/ICCNC.2019.8685536
Nishat, Z.K., Shopon, M.: Unsupervised pretraining and transfer learning-based Bangla sign language recognition. In: Proceedings of International Joint Conference on Computational Intelligence Algorithms for Intelligent Systems, pp. 529-540 (2020). https://doi.org/10.1007/978-981-15-3607-6_42
Perlman, M., Little, H., Thompson, B., Thompson, R.L.: Iconicity in signed and spoken vocabulary: a comparison between American sign language, British sign language, English, and Spanish. Front. Psyc. 9 (2018). https://doi.org/10.3389/fpsyg.2018.01433
Rathi, D.: Optimization of transfer learning for sign language recognition targeting mobile platform. In: International Journal on Recent and Innovation Trends in Computing and Communication. vol. 6, pp. 198–203 (2018). https://doi.org/10.48550/arXiv.1805.06618
Shania, S., Naufal, M.F., Prasetyo, V.R., Azmi, M.S.B.: Translator of Indonesian sign language video using convolutional neural network with transfer learning. Indonesian J. Inf. Syst. (2022). https://doi.org/10.24002/ijis.v5i1.5865
Sharma, S., Singh, S.: ISL recognition system using integrated mobile-net and transfer learning method. Expert Syst. Appl. 221, 119772 (2023). https://doi.org/10.1016/j.eswa.2023.119772
Sidig, A.A.I., Luqman, H., Mahmoud, S., Mohandes, M.: KARSL: Arabic sign language database. ACM Trans. Asian Low-Resource Lang. Inf. Proc. (TALLIP) 20(1), 1–19 (2021). https://doi.org/10.1145/3423420
Stokoe, W.C., Casterline, D.C., Croneberg, C.G.: A dictionary of American sign language on linguistic principles. (No Title) (1976)
Sultan, A., Makram, W., Kayed, M., Ali, A.A.: Sign language identification and recognition: a comparative study. Open Comput. Sci. 12(1), 191–210 (2022). https://doi.org/10.1515/comp-2022-0240
Tarrés, L., Gállego, G.I., Duarte, A., Torres, J., Giró-i Nieto, X.: Sign language translation from instructional videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5624–5634 (2023). https://doi.org/10.1109/CVPRW59228.2023.00596
Thakar, S., Shah, S., Shah, B., Nimkar, A.V.: Sign language to text conversion in real time using transfer learning. 2022 IEEE 3rd Global Conference for Advancement in Technology (GCAT), pp. 1–5 (2022). https://doi.org/10.1109/GCAT55367.2022.9971953
Vázquez, R., Boggia, M., Raganato, A., Loppi, N.A., Grönroos, S.A., Tiedemann, J.: Latest development in the FoTran project – scaling up language coverage in neural machine translation using distributed training with language-specific components. In: Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pp. 311–312. European Association for Machine Translation, Ghent, Belgium (2022)
Vázquez-Enríquez, M., Alba-Castro, J.L., Docío-Fernández, L., Rodríguez-Banga, E.: Isolated sign language recognition with multi-scale spatial-temporal graph convolutional networks. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3457–3466 (2021). https://doi.org/10.1109/CVPRW53098.2021.00385
Zakariah, M., Alotaibi, Y.A., Koundal, D., Guo, Y., Elahi, M.M.: Sign language recognition for Arabic alphabets using transfer learning technique. Comput. Intell. Neurosci. 2022 (2022). https://doi.org/10.1155/2022/4567989
Zhou, H., Zhou, W., Qi, W., Pu, J., Li, H.: Improving sign language translation with monolingual data by sign back-translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1316–1325 (2021). https://doi.org/10.1109/GCAT55367.2022.9971953
Östling, R., Börstell, C., Courtaux, S.: visual iconicity across sign languages: large-scale automated video analysis of iconic articulators and locations. Front. Psyc. 9 (2018).https://doi.org/10.3389/fpsyg.2018.00725
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Artiaga, K., Lynch, C., Afli, H., Hasanuzzaman, M. (2024). The Influence of Iconicity in Transfer Learning for Sign Language Recognition. In: Rapp, A., Di Caro, L., Meziane, F., Sugumaran, V. (eds) Natural Language Processing and Information Systems. NLDB 2024. Lecture Notes in Computer Science, vol 14762. Springer, Cham. https://doi.org/10.1007/978-3-031-70239-6_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-70239-6_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70238-9
Online ISBN: 978-3-031-70239-6
eBook Packages: Computer ScienceComputer Science (R0)