Skip to main content
SpringerLink
Log in

A real-time approach to recognition of Turkish sign language by using convolutional neural networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Sign language is a form of visual communication used by people with hearing problems to express themselves. The main purpose of this study is to make life easier for these people. In this study, a data set was created using 3200 RGB images for 32 classes (32 static words) taken from three different people. Data augmentation methods were applied to the data sets, and the number of images increased from 3200 to 19,200, 600 per class. A 10-layer convolutional neural network model was created for the classification of the signs, and VGG166, Inception, and ResNet deep network architectures, which are deep learning methods, were applied by using the transfer learning method. Also, the signs are classified using the support vector machines and k-nearest neighbor methods, which are the traditional machine learning methods, by using features obtained from the last layer of the convolutional neural network. The most successful method was determined by comparing the obtained results according to time and performance ratios. In addition to these analyses, an interface was developed. By using the interface, the static words belonging to Turkish sign language (TSL) are translated into real-time written language. With the real-time system designed, its success in recognizing the static words of TSL signs and printing its prediction on the computer screen was evaluated.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Wadhawan A, Kumar P (2020) Deep Learning Based Sign Language Recognition System for Static Signs. Neural Comput Appl 32:7957–7968. https://doi.org/10.1007/s00521-019-04691-y

    Article  Google Scholar 

  2. Sadeddine K, Chelali ZF, Djeradi R, Djeradi A, Benabderrahmane S (2021) Recognition of user-dependent and independent static hand gestures: Application to sign language. J Vis Commun Image Represent 79(103193):1047–3203. https://doi.org/10.1016/j.jvcir.2021.103193

    Article  Google Scholar 

  3. Vogler C, Metaxas D (1998) ASL recognition based on a coupling between HMMs and 3D motion analysis. In Proc IEEE Int Conf Comput Vision. https://doi.org/10.1109/iccv.1998.710744

    Article  Google Scholar 

  4. Starner T, Mann S, Rhodes B, Healey J, Russell KB, Levine J, & Pentland A (1995) Wearable Computing and Augmented Reality. M.I.T. Media Lab Vision and Modeling Group Technical Report.

  5. Dong C, Leu MC, Yin M (2015) American Sign Language alphabet recognition using Microsoft Kinect, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). doi: https://doi.org/10.1109/CVPRW.2015.7301347

  6. Chuan CH, Regina E, & Guardino C (2014) American sign language recognition using leap motion sensor. In Proceedings - 2014 13th International Conference on Machine Learning and Applications, ICMLA 2014. https://doi.org/10.1109/ICMLA.2014.110

  7. LeapMotion Sensor. (2020). https://www.hackster.io/leapmotion/projects . Accessed November 11, 2020.

  8. Cyberglove Sensor (2017). http://www.cyberglovesystems.com/cyberglove-ii. Accessed November 11, 2020.

  9. Microsoft Kinect Sensor (2019). https://www.instructables.com/id/Kinect-Projects/. Accessed November 11, 2020.

  10. Cerna LR, Cardenas EE, Miranda DG, Menotti D, Chavez GC (2020) A Multimodal Libras-Ufop Brazilian Sign Language Dataset Of Minimal Pairs Using a Microsoft Kinect Sensor. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2020.114179

    Article  Google Scholar 

  11. Usta YH, Ghosheh M, & Isler Y (2017) El protezlerinde kullanilan yüzey EMG elektrotlari ile kuvvet sensörlerinin karşilaştirilmasi. In 2016 Medical Technologies National Conference, TIPTEKNO 2016. https://doi.org/10.1109/TIPTEKNO.2016.7863104.

  12. Arpacı SA, Kalıpsız O (2018) Yazılım Hata Sınıflandırmasında Farklı Naive Bayes Tekniklerin Kıyaslanması. Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi. https://doi.org/10.28948/ngumuh.383709

    Article  Google Scholar 

  13. Cunningham P & Delany SJ (2007) K -Nearest Neighbour Classifiers. Multiple Classifier Systems.

  14. Güven F (2019) Using Text Representation And Deep Learning Methods For Turkish Text Classification, Çukurova Üniversitesi, Bilgisayar Mühendisliği Anabilim Dalı , Master Thesis.

  15. Ezel E (2018) Image-based turkish sign language recognition using deep learning method. Master Thesis, Department of Computer Engineering, Selçuk University.

  16. Rao GA, Kishore PVV (2018) Selfie video based continuous Indian sign language recognition system. Ain Shams Eng J 9(4):1929–1939

    Article  Google Scholar 

  17. Madylova A (2009) Kosinüs Benzerliğini Kullanarak Belgeler Arası Anlamsal Benzerliği Kavramsal Sözlüğe Dayalı Hesaplama Yöntemi. Master Thesis. İstanbul Technical University.

  18. Öz C, Eken B, Şahin B, Akbulut E, & Akbulut F (2011) Saklı Markov Model Kullanarak Türkçe Konuşmayı ve Yazıyı İşaret Diline Çevirme. TMMOB Elektrik Mühendisleri Odası, Elektrik, Elektronik ve Bilgisayar Sempozyumu 2011, Fırat Üniversitesi, Elazığ.

  19. Haberdar H (2005) Real time Turkish sign language recognition system from video using hidden Markov models. Master Thesis, Computer Engineering, Yıldız Technical University.

  20. Altun O (2010) İlgi bölgeleri ve yerel tanımlayıcılar ile genelleştirilmiş Hough dönüşümü ve en az eylemsizlik ekseni tabanlı hizalama yaklaşımı ile Türk işaret dili tanıma sistemi. PhD Thesis, Department of Computer Engineering, Yıldız Technical University.

  21. Memiş A (2013) Sign language recognition using spatio-temporal features on Kinect rgb video sequences and depth maps. Master Thesis, Department of Computer Engineering, Yıldız Technical University.

  22. Memiş A, Albayrak S (2013) A Kinect based sign language recognition system using spatio-temporal features. In Sixth International Conference on Machine Vision (ICMV 2013) (Vol. 9067). https://doi.org/10.1117/12.2051018

  23. Akış AH (2018) A low cost learning based sign language recognition system. Master Thesis, Electrical and Electronics Engineering, Middle East Technical University.

  24. Özdemir O (2018) Isolated sign language classification using hand descriptors and trajectory based methods. Master Thesis, Computer Engineering, Boğaziçi University.

  25. Kın ZB (2019) Classification of turkish sign language alphabet with deep learning method. Master Thesis, Electrical and Electronics Engineering, Başkent University.

  26. Turkish sign Language Dictionary, MEB Yayınları, Ankara (2015) http://orgm.meb.gov.tr/alt_sayfalar/duyurular/1.pdf . Accessed: November 11, 2020.

  27. Eryiğit G, Eryiğit C, Karabüklü S, Kelepir M, Özkul A, Pamay T, Köse H (2020) Building the first comprehensive machine-readable Turkish sign language resource: methods, challenges and solutions. Lang Res Eval 54(1):97–121

    Article  Google Scholar 

  28. Hinton GE, Mcclelland JL, & Rumelhart DE (1986) Distributed representations, Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations.". Volume 2: Psychological and Biological Models (Vol. 2).

  29. Wang P, Fan E, Wang P (2021) Comparative analysis of image classification algorithms based on traditional machine learning and deep learning. Pattern Recogn Lett 141:61–67. https://doi.org/10.1016/j.patrec.2020.07.042

    Article  Google Scholar 

  30. Lee H, Kwon H (2017) Going Deeper With Contextual CNN for Hyperspectral Image Classification. IEEE Trans Image Process 26(10):4843–4855. https://doi.org/10.1109/TIP.2017.2725580

    Article  MathSciNet  Google Scholar 

  31. Han D, Liu Q, Fan W (2018) A new image classification method using CNN transfer learning and web data augmentation. Expert Syst Appl 95:43–56. https://doi.org/10.1016/j.eswa.2017.11.028

    Article  Google Scholar 

  32. Goodfellow I, Bengio Y, & Courville A (2016) Structured Probabilistic Models for Deep Learning. In Deep Learning.

  33. Aggarwal A (2018) Resurgence of Artificial Intelligence. https://scryanalytics.ai/resurgence-of-artificial-intelligence-during-1983-2010/ Accessed November 11, 2020.

  34. Vapnik V (1963) Pattern recognition using generalized portrait method. Automation and Remote Control.

  35. Osisanwo FY, Akinsola JET, Awodele O, Hinmikaiye JO, Olakanmi O, Akinjobi J (2017) Supervised Machine Learning Algorithms: Classification and Comparison. Int J Comput Trends Technol. https://doi.org/10.14445/22312803/ijctt-v48p126

    Article  Google Scholar 

  36. Srivastava DK, Bhambhu L (2010) Data classification using support vector machine. J Theor Appl Inf Technol 12(1):1–7

    Google Scholar 

  37. Huang S, Nianguang CAI, Penzuti Pacheco P, Narandes S, Wang Y, Wayne XU (2018) Applications of support vector machine (SVM) learning in cancer genomics. Cancer Genomics Proteomics. https://doi.org/10.21873/cgp.20063

    Article  Google Scholar 

  38. Hunt KJ, Sbarbaro D (1991) Neural networks for nonlinear internal model control. IEE Proc D (Control Theory Appl) 138(5):431–438. https://doi.org/10.1049/ip-d.1991.0059

    Article  MATH  Google Scholar 

  39. Özcan T, Baştürk A (2021) ERUSLR: A new Turkish sign language dataset and its recognition using hyperparameter optimization aided convolutional neural network. J Faculty Eng Architect Gazi Univ 36:527–542. https://doi.org/10.17341/gazimmfd.746793

    Article  Google Scholar 

  40. Çelik Ö, Odabaş A (2020) Sign2Text: Turkish Sign Language recognition using Convolutional Neural Networks. Eur J Sci Technol 19:923–934. https://doi.org/10.31590/ejosat.747231

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Electronics Engineering, Başkent University, 06790, Ankara, Turkey

    Selda Güney

  2. Turkish Aerospace Industries, Inc., 06980, Kahramankazan, Ankara, Turkey

    Mehmet Erkuş

Authors
  1. Selda Güney
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mehmet Erkuş
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Selda Güney.

Ethics declarations

Conflict of interests

The authors did not receive support from any organization for the submitted work. The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Güney, S., Erkuş, M. A real-time approach to recognition of Turkish sign language by using convolutional neural networks. Neural Comput & Applic 34, 4069–4079 (2022). https://doi.org/10.1007/s00521-021-06664-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06664-6

Keywords

Search

Navigation