Abstract
Sign languages (SLs) are an essential form of communication for hearing-impaired people. However, a communication barrier still exists between the deaf community and the hearing population due to the lack of accurate automated SL communication systems. In this work, a novel SL communication system running as a mobile application has been developed to facilitate the bi-directional communication between hearing-impaired and hearing people.The proposed system utilizes Natural Language Processing (NLP) techniques, along with linguistic rules to convert between spoken language and signs, taking into account the grammatical structure of a sign language. Additionally, the system employs sign language recognition (SLR) algorithms to transform video sequences to signs, as well as hand and pose estimation algorithms to model the 3D motion of signs. Moreover, a 3D human avatar representation is employed to animate the motion of each sign in a seamless manner. Finally, a new partition of the Greek SL (GSL) dataset is formed with 1825 videos from 12 signers captured in the wild to evaluate SLR performance under realistic conditions. The proposed SL communication system and its components are validated quantitatively in GSLW as well as qualitatively by means of questionnaires, demonstrating the user satisfaction with the system.
A. Stergioulas, C. Chatzikonstantinou, T. Chatzis, I. Papastratis—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Papastratis, I., Dimitropoulos, K., Konstantinidis, D., Daras, P.: Continuous sign language recognition through cross-modal alignment of video and text embeddings in a joint-latent space. IEEE Access 8, 91 170–91 180 (2020)
Papastratis, I., Chatzikonstantinou, C., Konstantinidis, D., Dimitropoulos, K., Daras, P.: Artificial intelligence technologies for sign language. Sensors 21(17), 5843 (2021)
Stefanidis, K., Konstantinidis, D., Kalvourtzis, A., Dimitropoulos, K., Daras, P.: 3d technologies and applications in sign language. Recent Advances in 3D Imaging, Modeling, and Reconstruction, pp. 50–78 (2020)
Adaloglou, N.M., et al.: A comprehensive study on deep learning-based methods for sign language recognition. IEEE Trans. Multimedia (2021)
Papastratis, I., Dimitropoulos, K., Daras, P.: Continuous sign language recognition through a context-aware generative adversarial network. Sensors 21(7), 2437 (2021)
Carreira, J., Zisserman, A.: Quo vadis, action recognition, a new model and the kinetics dataset. CoRR, abs/1705.07750, vol. 2, p. 3 (2017)
Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)
Kocabas, M., Athanasiou, N., Black, M.J.: Vibe: video inference for human body pose and shape estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5253–5263 (2020)
Stergioulas, A., Chatzis, T., Konstantinidis, D., Dimitropoulos, K., Daras, P.: 3d hand pose estimation via aligned latent space injection and kinematic losses. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1730–1739 (2021)
Zhang, J., Jiao, J., Chen, M., Qu, L., Xu, X., Yang, Q.: A hand pose tracking benchmark from stereo matching. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 982–986. IEEE (2017)
Zimmermann, C., Brox, T.: Learning to estimate 3d hand pose from single RGB images. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4903–4911 (2017)
Chatzis, T., Stergioulas, A., Konstantinidis, D., Dimitropoulos, K., Daras, P.: A comprehensive study on deep learning-based 3d hand pose estimation methods. Appl. Sci. 10(19), 6850 (2020)
Unreal engine: The most powerful real-time 3d creation tool. https://www.unrealengine.com. Accessed 24 Feb 2022
Easytv: Easing the access of europeans with disabilities to converging media and content, deliverable 6.5 (2020). https://easytvproject.eu/files/D6.5.pdf. Accessed 24 Feb 2022
Brooke, J.: Sus: a retrospective. J. Usabil. Stud. 8(2), 29–40 (2013)
Yüksel, A., Rimmington, M.: Customer-satisfaction measurement: performance counts. Cornell Hotel Restaurant Administ. Quar. 39(6), 60–70 (1998)
Author information
Authors and Affiliations
Contributions
Andreas Stergioulas, Christos Chatzikonstantinou, Theocharis Chatzis and Ilias Papastratis: equal contribbution
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Stergioulas, A. et al. (2023). Sign Language Communication Through an Interactive Mobile Application. In: Stephanidis, C., Antona, M., Ntoa, S., Salvendy, G. (eds) HCI International 2023 Posters. HCII 2023. Communications in Computer and Information Science, vol 1833. Springer, Cham. https://doi.org/10.1007/978-3-031-35992-7_52
Download citation
DOI: https://doi.org/10.1007/978-3-031-35992-7_52
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-35991-0
Online ISBN: 978-3-031-35992-7
eBook Packages: Computer ScienceComputer Science (R0)