Abstract
Despite existing solutions for accurate translation between written and spoken language, sign language is still not well-studied area. A reliable, robust and working in real-time translator of American Sign Language is a crucial bridge to facilitate communication between deaf and hearing people. In this paper we propose a method of sign language fingerspelling recognition using a modern architecture of convolutional neural network called Wide Residual Network trained with Snapshot Learning procedure. The model was trained on augmented datasets available at Surrey University and Massey University web pages using transfer learning. The final result is a robust classifier of all alphabet letters, which beats current state-of-the-art results. The outcomes encourage further research in this field for creating fully usable sign language translator.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Mitchell, R.E., Young, T.A., Bachleda, B., Karchmer, M.A.: How many people use ASL in the United States? Why estimates need updating. Sign Lang. Stud. 6(3), 306–335 (2006)
Rioux-Maldague, L., Giguère, P.: Sign language fingerspelling classification from depth and color images using a deep belief network. CoRR, abs/1503.05830 (2015)
Pigou, L., Dieleman, S., Kindermans, P.-J., Schrauwen, B.: Sign language recognition using convolutional neural networks. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014, Part I. LNCS, vol. 8925, pp. 572–578. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16178-5_40
Garcia, B., Viesca, S.: Real-time American sign language recognition with convolutional neural networks. In: Convolutional Neural Networks for Visual Recognition (2016)
Bheda, V., Radpour, D.: Using deep convolutional networks for gesture recognition in American sign language. CoRR, abs/1710.06836 (2017)
Ameen, S., Vadera, S.: A convolutional neural network to classify American sign language fingerspelling from depth and colour images. Expert Syst. 34(3), e12197 (2017)
Zagoruyko, S., Komodakis, N.: Wide residual networks. CoRR, abs/1605.07146 (2016)
Kang, B., Tripathi, S., Nguyen, T.Q.: Real-time sign language fingerspelling recognition using convolutional neural networks from depth map. CoRR, abs/1509.03001 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR, abs/1512.03385 (2015)
Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J.E., Weinberger, K.Q.: Snapshot ensembles: train 1, get M for free. CoRR, abs/1704.00109 (2017)
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S.E., Fu, C.-Y., Berg, A.C.: SSD: single shot multibox detector. CoRR, abs/1512.02325 (2015)
Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861 (2017)
Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 1, pp. 539–546. IEEE (2005)
University of Exeter: ASL Finger Spelling Dataset, 2 November 2017
Barczak, A.L.C., Reyes, N.H., Abastillas, M., Piccio, A., Susnjak, T.: A new 2D static hand gesture colour image dataset for ASL gestures. Res. Lett. Inf. Math. Sci. 15, 12–20 (2011)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167 (2015)
Tompson, J., Goroshin, R., Jain, A., LeCun, Y., Bregler, C.: Efficient object localization using convolutional networks. CoRR, abs/1411.4280 (2014)
Nesterov, Y.: Introductory Lectures on Convex Optimization. Springer US, New York (2004). https://doi.org/10.1007/978-1-4419-8853-9
Ruder, S.: An overview of multi-task learning in deep neural networks. CoRR, abs/1706.05098 (2017)
Acknowledgments
We thank Identt company for giving access to PC used to conduct experiments. Acknowledgments are directed also to dr Adam Gonczarek from the Wroclaw University of Science and Technology for leading the project of the recognition system. We thank Michał Kosturek and Piotr Grzybowski from scientific student assocation “medical.ml” at Wrocław University of Science and Technology, who implemented dictionary and hand localization modules respectively for the system.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Kania, K., Markowska-Kaczmar, U. (2018). American Sign Language Fingerspelling Recognition Using Wide Residual Networks. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J. (eds) Artificial Intelligence and Soft Computing. ICAISC 2018. Lecture Notes in Computer Science(), vol 10841. Springer, Cham. https://doi.org/10.1007/978-3-319-91253-0_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-91253-0_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-91252-3
Online ISBN: 978-3-319-91253-0
eBook Packages: Computer ScienceComputer Science (R0)