Skip to main content
SpringerLink
Log in

American and Russian Sign Language Dactyl Recognition and Text2Sign Translation

  • Conference paper
  • First Online:
Analysis of Images, Social Networks and Texts (AIST 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11832))

Abstract

Sign language is the main way to communicate for people from deaf community. However, common people mostly do not know sign language. In this paper, we overview several real-time sign language dactyl recognition systems using deep convolutional neural networks. These systems are able to recognize dactylized words gestured by signs for each letter. We evaluate our approach on American (ASL) and Russian (RSL) sign languages. This solution may help fasten the process of communication for deaf people. On the contrary, we also present the algorithm for generating sign animation from text information using text-to-sign video vocabulary, which helps to integrate sign language in dubbed TV and combining with speech recognition tool provide full translation from natural language to sign language.

I. Makarov—The work was supported by the Russian Science Foundation under grant 17-11-01294 and performed at National Research University Higher School of Economics, Russia.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://sl-data.ddns.net/.

  2. 2.

    https://www.spreadthesign.com.

  3. 3.

    https://www.xn--d1ascahfol.xn--p1ai/.

  4. 4.

    https://www.thesaurus.com/browse/.

  5. 5.

    https://github.com/animate1978/MB-Lab.

  6. 6.

    https://sl-data.ddns.net/.

References

  1. Barczak, A., Reyes, N., Abastillas, M., Piccio, A., Susnjak, T.: A new 2D static hand gesture colour image dataset for ASL gestures (2011)

    Google Scholar 

  2. Chakraborty, D., Garg, D., Ghosh, A., Chan, J.H.: Trigger detection system for American sign language using deep convolutional neural networks. In: Proceedings of the 10th International Conference on Advances in Information Technology, p. 4. ACM, New York (2018)

    Google Scholar 

  3. Choe, B.W., Min, J.-K., Cho, S.-B.: Online gesture recognition for user interface on accelerometer built-in mobile phones. In: Wong, K.W., Mendis, B.S.U., Bouzerdoum, A. (eds.) ICONIP 2010. LNCS, vol. 6444, pp. 650–657. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-17534-3_80

    Chapter  Google Scholar 

  4. Cui, R., Liu, H., Zhang, C.: Recurrent convolutional neural networks for continuous sign language recognition by staged optimization. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1610–1618. IEEE, New York (2017)

    Google Scholar 

  5. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes challenge 2012 (voc2012) results (2012). http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html

  6. Huang, J., Zhou, W., Li, H., Li, W.: Sign language recognition using 3D convolutional neural networks. In: 2015 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE, New York (2015)

    Google Scholar 

  7. Huenerfauth, M., Kacorri, H.: Best practices for conducting evaluations of sign language animation. J. Technol. Pers. Disabil. 3, 1–14 (2015)

    Google Scholar 

  8. Kamat, R., Danoji, A., Dhage, A., Puranik, P., Sengupta, S.: Monvoix-an android application for hearing impaired people. J. Commun. Technol. Electron. Comput. Sci. 8, 24–28 (2016)

    Article  Google Scholar 

  9. Kanevskiy, E., Tuzov, V.: Some questions of subject area terms appending to a semantic dictionary. Dialogue 2, 156–160 (2002). (in Russian)

    Google Scholar 

  10. Kau, L.J., Su, W.L., Yu, P.J., Wei, S.J.: A real-time portable sign language translation system. In: 2015 IEEE 58th International Midwest Symposium on Circuits and Systems (MWSCAS), pp. 1–4. IEEE, New York (2015)

    Google Scholar 

  11. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization, pp. 1–15. arXiv preprint arXiv:1412.6980arXiv:1412.6980 (2014)

  12. Koller, O., Zargaran, O., Ney, H., Bowden, R.: Deep sign: hybrid CNN-HMM for continuous sign language recognition. In: Proceedings of the British Machine Vision Conference 2016, pp. 1–12. BMVA, Durham, UK (2016)

    Google Scholar 

  13. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  14. Loïc Marie ASL Hand Gesture Dataset (2017). https://github.com/loicmarie/sign-language-alphabet-recognizer/tree/master/dataset

  15. Makarov, I., Veldyaykin, N., Chertkov, M., Pokoev, A.: American and Russian sign language dactyl recognition. In: 12th PErvasive Technologies Related to Assistive Environments Conference (PETRA 2019), pp. 1–7. ACM, New York (2019). https://doi.org/10.1145/3316782.3316786

  16. Makarov, I., Veldyaykin, N., Chertkov, M., Pokoev, A.: Russian sign language dactyl recognition. In: 42nd International Conference on Telecommunications and Signal Processing (TSP 2019), pp. 1–4. IEEE, New York (2019)

    Google Scholar 

  17. Mehta, D., et al.: VNect: real-time 3D human pose estimation with a single RGB camera. ACM Trans. Graph. (TOG) 36(4), 44 (2017)

    Article  Google Scholar 

  18. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119. Curran Associates Inc., New York (2013)

    Google Scholar 

  19. Mirman, D., Strauss, T.J., Dixon, J.A., Magnuson, J.S.: Effect of representational distance between meanings on recognition of ambiguous spoken words. Cogn. Sci. 34(1), 161–173 (2010)

    Article  Google Scholar 

  20. Molchanov, P., Gupta, S., Kim, K., Kautz, J.: Hand gesture recognition with 3D convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–7. IEEE, New York (2015)

    Google Scholar 

  21. Mozgovoy, M.: Machine semantic analysis of Russian language and its applications, vol. 1. SPBGU, St.-Petersburg, Russia (2006). (in Russian)

    Google Scholar 

  22. Pigou, L., Dieleman, S., Kindermans, P.-J., Schrauwen, B.: Sign language recognition using convolutional neural networks. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8925, pp. 572–578. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16178-5_40

    Chapter  Google Scholar 

  23. Prasuhn, L., Oyamada, Y., Mochizuki, Y., Ishikawa, H.: A hog-based hand gesture recognition system on a mobile device. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 3973–3977. IEEE, New York (2014)

    Google Scholar 

  24. Preetham, C., Ramakrishnan, G., Kumar, S., Tamse, A., Krishnapura, N.: Hand talk-implementation of a gesture recognizing glove. In: 2013 Texas Instruments India Educators, pp. 328–331. IEEE, New York (2013)

    Google Scholar 

  25. Redmon, J., Farhadi, A.: Yolo9000: better, faster, stronger. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7263–7271. IEEE, New York, July 2017

    Google Scholar 

  26. rrupeshh ASL Hand Gesture Dataset (2018). https://github.com/rrupeshh/Simple-Sign-Language-Detector/tree/master/mydata

  27. Sign Language MNIST (2018). https://www.kaggle.com/datamunge/sign-language-mnist

  28. Taskiran, M., Killioglu, M., Kahraman, N.: A real-time system for recognition of American sign language by using deep learning. In: 2018 41st International Conference on Telecommunications and Signal Processing (TSP), pp. 1–5. IEEE, New York (2018)

    Google Scholar 

  29. The RSL alphabet dataset (2019). https://github.com/hse-sl/rsl-alphabet-dataset

  30. thtrieu (2017). https://github.com/thtrieu/darkflow

  31. Tuzov, V.: Computer semantics of Russian. Sankt-Petersburg State University, vol. 1, pp. 1–6 (2004)

    Google Scholar 

  32. Uchida, T., et al.: Sign language support system for viewing sports programs. In: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 339–340. ACM (2017)

    Google Scholar 

  33. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)

    Article  Google Scholar 

  34. Wang, X., Tarrío, P., Metola, E., Bernardos, A.M., Casar, J.R.: Gesture recognition using mobile phone’s inertial sensors. In: Omatu, S., De Paz Santana, J.F., González, S.R., Molina, J.M., Bernardos, A.M., Rodríguez, J.M.C. (eds.) Distributed Computing and Artificial Intelligence. AISC, vol. 151, pp. 173–184. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28765-7_21

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Research University Higher School of Economics, Moscow, Russia

    Ilya Makarov, Nikolay Veldyaykin, Maxim Chertkov & Aleksei Pokoev

Authors
  1. Ilya Makarov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nikolay Veldyaykin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maxim Chertkov
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aleksei Pokoev
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ilya Makarov .

Editor information

Editors and Affiliations

  1. RWTH Aachen University, Aachen, Germany

    Wil M. P. van der Aalst

  2. University of Ljubljana, Ljubljana, Slovenia

    Vladimir Batagelj

  3. National Research University Higher School of Economics, Moscow, Russia

    Dmitry I. Ignatov

  4. Krasovskii Institute of Mathematics and Mechanics, Yekaterinburg, Russia

    Michael Khachay

  5. National Research University Higher School of Economics, Moscow, Russia

    Valentina Kuskova

  6. University of Oslo, Oslo, Norway

    Andrey Kutuzov

  7. National Research University Higher School of Economics, Moscow, Russia

    Sergei O. Kuznetsov

  8. National Research University Higher School of Economics, Moscow, Russia

    Irina A. Lomazova

  9. Lomonosov Moscow State University, Moscow, Russia

    Natalia Loukachevitch

  10. LORIA, Vandœuvre-lès-Nancy, France

    Amedeo Napoli

  11. University of Florida, Gainesville, FL, USA

    Panos M. Pardalos

  12. Ca Foscari University of Venice, Venice, Italy

    Marcello Pelillo

  13. National Research University Higher School of Economics, Nizhny Novgorod, Russia

    Andrey V. Savchenko

  14. Kazan Federal University, Kazan, Russia

    Elena Tutubalina

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Makarov, I., Veldyaykin, N., Chertkov, M., Pokoev, A. (2019). American and Russian Sign Language Dactyl Recognition and Text2Sign Translation. In: van der Aalst, W., et al. Analysis of Images, Social Networks and Texts. AIST 2019. Lecture Notes in Computer Science(), vol 11832. Springer, Cham. https://doi.org/10.1007/978-3-030-37334-4_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-37334-4_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-37333-7

  • Online ISBN: 978-3-030-37334-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation