Skip to main content
SpringerLink
Log in

Symmetric mean binary pattern-based Pakistan sign language recognition using multiclass support vector machines

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Sign languages are a fundamental source of communication for the deaf community, generated through movements of the human body. Similar to the natural languages, sign languages vary from region to region, and from nation to nation. Pakistan sign language (PSL) is inspired by Urdu, national language of Pakistan. It has 38 alphabet signs, and out of them, 36 are represented by the static hand gestures. Automatic recognition of a sign language helps in interaction between the hearing and deaf individuals. The number of quality efforts in context of Pakistan sign language recognition is quite limited, leaving a fair room for addressing open issues of research, i.e., (i) efficient hand detection under complex backgrounds and (ii) extracting signer independent feature vector that should not only be discriminant for all the PSL alphabets but reduced dimension as well. In this research, recognition of PSL static alphabets is addressed where the task of hand localization is accomplished through faster regional-convolutional neural networks (faster R-CNN). Feature extraction is achieved through presenting symmetric mean-based binary patterns (sMBP) that extend uniform local binary patterns. The proposed feature vector not only suppresses the noise but preserves the rotation invariance as well. Classification task is accomplished through error correction output codes (ECOC)-based support vector machines using linear, polynomial and radial basis function kernels with one-vs-one and one-vs-all modalities. The proposed technique is validated through PSL dataset, created by seven native signers, having 7174 images. The comparative results clearly demonstrate the authority of the proposed technique over all of its baseline and competitor techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Ahmed H, Gilani SO, Jamil M, Ayaz Y, Shah SIA (2016) Monocular vision-based signer-independent Pakistani sign-language recognition system using supervised learning. Indian J Sci Technol 9(25):1–16

    Article  Google Scholar 

  2. Ali SA (2013) Detection of Urdu sign language using harr algorithms. Int J Invent Eng Sci (IJIES) 1(6):50–54

    Google Scholar 

  3. Alvi AK, Azhar MYB, Usman M, Mumtaz S, Rafiq S, Rehman RU, Ahmed I (2004) Pakistan sign language recognition using statistical template matching. Int J Inf Technol 1(1):1–12

    Google Scholar 

  4. Alzohairi R, Alghonaim R, Alshehri W, Aloqeely S, Alzaidan M, Bchir O (2018) Image based Arabic sign language recognition system. Int J Adv Comput Sci Appl (IJACSA) 9(3):185–194

    Google Scholar 

  5. Bagheri MA, Gao Q, Escalera S (2012) Efficient pairwise classification using local cross off strategy. In: Canadian conference on artificial intelligence, Springer, pp 25–36

  6. Bambach S, Lee S, Crandall DJ, Yu C (2015) Lending a hand: detecting hands and recognizing activities in complex egocentric interactions. In: Proceedings of the IEEE international conference on computer vision, pp 1949–1957

  7. Bantupalli K, Xie Y (2018) American sign language recognition using deep learning and computer vision. In: 2018 IEEE International conference on big data (Big Data), IEEE, pp 4896–4899

  8. Bilal S, Akmeliawati R, El Salami MJ, Shafie AA (2011) Vision-based hand posture detection and recognition for sign language—a study. In: 2011 4th International conference on mechatronics (ICOM), IEEE, pp 1–6

  9. Chong TW, Lee BG (2018) American sign language recognition using leap motion controller with machine learning approach. Sensors 18(10):3554

    Article  Google Scholar 

  10. Cooper H, Ong EJ, Pugeault N, Bowden R (2012) Sign language recognition using sub-units. J Mach Learn Res 13(Jul):2205–2231

    Google Scholar 

  11. Cui R, Liu H, Zhang C (2019) A deep neural framework for continuous sign language recognition by iterative training. IEEE Trans Multimed 21(7):1880–1891

    Article  Google Scholar 

  12. Dojer N, Bednarz P, Podsiadło A, Wilczyński B (2013) Bnfinder2: faster Bayesian network learning and Bayesian classification. Bioinformatics 29(16):2068–2070

    Article  Google Scholar 

  13. Escalera S, Pujol O, Radeva P (2010) Error-correcting output codes library. J Mach Learn Res 11(Feb):661–664

    MATH  Google Scholar 

  14. Halim Z, Abbas G (2015) A Kinect-based sign language hand gesture recognition system for hearing-and speech-impaired: a pilot study of pakistani sign language. Assist Technol 27(1):34–43

    Article  Google Scholar 

  15. Hassan S, Abolarinwa J, Alenoghena C, Bala S, David M, Enenche P (2018) Intelligent sign language recognition using image processing techniques: a case of Hausa sign language. ATBU J Sci Technol Educ 6(2):127–134

    Google Scholar 

  16. Jalal MA, Chen R, Moore RK, Mihaylova L (2018) American sign language posture understanding with deep neural networks. In: 2018 21st international conference on information fusion (FUSION), IEEE, pp 573–579

  17. Jasim M, Hasanuzzaman M (2014) Sign language interpretation using linear discriminant analysis and local binary patterns. In: 2014 International conference on informatics, electronics and vision (ICIEV), IEEE, pp 1–5

  18. Josepha KJJ, Thangaswamyb SS (2021) Recognition of hand signs based on geometrical features using machine learning and deep learning approaches. Revista Argentina de Clínica Psicológica 30(3):175–183

    Google Scholar 

  19. Kausar S, Javed MY (2011) A survey on sign language recognition. In: 2011 Frontiers of information technology, IEEE, pp 95–98

  20. Kausar S, Javed MY, Sohail S (2008) Recognition of gestures in Pakistani sign language using fuzzy classifier. In: Proceedings of the 8th conference on Signal processing, computational geometry and artificial vision, World Scientific and Engineering Academy and Society (WSEAS), pp 101–105

  21. Kausar S, Javed MY, Tehsin S, Riaz M (2016) Vision-based classification of Pakistani sign language. Int J Image Graph Signal Process 8(2):9

    Article  Google Scholar 

  22. Khan N, Shahzada A, Ata S, Abid A, Khan Y, ShoaibFarooq M (2014) A vision based approach for Pakistan sign language alphabets recognition. Pensee 76(3):274–285

    Google Scholar 

  23. Kindiroglu A, Yalcin H, Aran O, Hruz M, Campr P, Akarun L, Karpov A (2011) Multi-lingual fingerspelling recognition for handicapped kiosk. Pattern Recognit Image Anal 21(3):402

    Article  Google Scholar 

  24. Koller O, Zargaran S, Ney H, Bowden R (2018) Deep sign: enabling robust statistical continuous sign language recognition via hybrid CNN-HMMs. Int J Comput Vis 126(12):1311–1325

    Article  Google Scholar 

  25. Kumar N (2017) Sign language recognition for hearing impaired people based on hands symbols classification. In: 2017 International conference on computing, communication and automation (ICCCA), IEEE, pp 244–249

  26. Kumar P, Saini R, Roy PP, Dogra DP (2018) A position and rotation invariant framework for sign language recognition (SLR) using Kinect. Multimed Tools Appl 77(7):8823–8846

    Article  Google Scholar 

  27. Lee YH, Tsai CY (2009) Taiwan sign language (TSL) recognition based on 3D data and neural networks. Expert Syst Appl 36(2):1123–1128

    Article  Google Scholar 

  28. Mahmud I, Tabassum T, Uddin MP, Ali E, Nitu AM, Afjal MI (2018) Efficient noise reduction and hog feature extraction for sign language recognition. In: 2018 International conference on advancement in electrical and electronic engineering (ICAEEE), IEEE, pp 1–4

  29. Malik MSA, Kousar N, Abdullah T, Ahmed M, Rasheed F, Awais M (2018) Pakistan sign language detection using PCA and KNN. Int J Adv Comput Sci Appl 9(54):78–81

    Google Scholar 

  30. Mujahid A, Awan MJ, Yasin A, Mohammed MA, Damaševičius R, Maskeliūnas R, Abdulkareem KH (2021) Real-time hand gesture recognition based on deep learning YOLOv3 model. Appl Sci 11(9):4164

    Article  Google Scholar 

  31. Munib Q, Habeeb M, Takruri B, Al-Malik HA (2007) American sign language (ASL) recognition based on Hough transform and neural networks. Expert Syst Appl 32(1):24–37

    Article  Google Scholar 

  32. Neethu P, Suguna R, Sathish D (2020) An efficient method for human hand gesture detection and recognition using deep learning convolutional neural networks. Soft Comput 24(20):15239–15248

    Article  Google Scholar 

  33. Oz C, Leu MC (2011) American sign language word recognition with a sensory glove using artificial neural networks. Eng Appl Artif Intell 24(7):1204–1213

    Article  Google Scholar 

  34. Panchal TH, Patel PR (2018) A novel approach of sign recognition for Indian sign. Int J Sci Res Sci Eng Technol (IJSRSET) 4(4):974–978

    Google Scholar 

  35. Paulraj M, Yaacob S, Desa H, Hema C, Ridzuan WM, Ab Majid W (2008) Extraction of head and hand gesture features for recognition of sign language. In: 2008 international conference on electronic design, IEEE, pp 1–6

  36. Pu J, Zhou W, Li H (2019) Iterative alignment network for continuous sign language recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4165–4174

  37. Raees M, Ullah S, Rahman SU, Rabbi I (2016) Image based recognition of Pakistan sign language. J Eng Res 1(4):1–21

    Google Scholar 

  38. Rahaman MA, Jasim M, Ali MH, Hasanuzzaman M (2020) Bangla language modeling algorithm for automatic recognition of hand-sign-spelled Bangla sign language. Front Comput Sci 14(3):143302

    Article  Google Scholar 

  39. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

  40. Santa U, Tazreen F, Chowdhury SA (2017) Bangladeshi hand sign language recognition from video. In: 2017 20th International conference of computer and information technology (ICCIT), IEEE, pp 1–4

  41. Saqib S, Kazmi SAR (2018) Recognition of static gestures using correlation and cross-correlation. Int J Adv Appl Sci 5:11–18

    Article  Google Scholar 

  42. Shah SMS, Naqvi HA, Khan JI, Ramzan M, Khan HU et al (2018) Shape based Pakistan sign language categorization using statistical features and support vector machines. IEEE Access 6:59242–59252

    Article  Google Scholar 

  43. Tariq M, Iqbal A, Zahid A, Iqbal Z, Akhtar J (2012) Sign language localization: learning to eliminate language dialects. In: 2012 15th International multitopic conference (INMIC), IEEE, pp 17–22

  44. Tauseef H, Fahiem MA, Farhan S (2009) Recognition and translation of hand gestures to Urdu alphabets using a geometrical classification. In: 2009 Second International Conference in Visualisation, IEEE, pp 213–217

  45. Tolentino LKS, Juan ROS, Thio-ac AC, Pamahoy MAB, Forteza JRR, Garcia XJO (2019) Static sign language recognition using deep learning. Int J Mach Learn Comput 9(6):821–827

    Article  Google Scholar 

  46. Vargas LP, Barba L, Torres C, Mattos L (2011) Sign language recognition system using neural network for digital hardware implementation. J Phys Conf Ser 274:012051

    Article  Google Scholar 

  47. Wang J, Xu X, Li M (2015) The study of gesture recognition based on SVM with LBP and PCA. J Image Graph 3(1):16–19

    Google Scholar 

  48. WHO (xxxx) Deafness and hearing loss. https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss. Accessed 05 March 2020

  49. Yuan T, Sah S, Ananthanarayana T, Zhang C, Bhat A, Gandhi S, Ptucha R (2019) Large scale sign language interpretation. In: 2019 14th IEEE International conference on automatic face and gesture recognition (FG 2019), IEEE, pp 1–5

Download references

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Department of Computer Science and Software Engineering, International Islamic University, Islamabad, Pakistan

    Syed Muhammad Saqlain Shah, Syed Husnain Abbas & Anwar Ghani

  2. Department of Computer Science, Kent State University, Ohio, USA

    Javed I. Khan

Authors
  1. Syed Muhammad Saqlain Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Javed I. Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Syed Husnain Abbas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anwar Ghani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Syed Muhammad Saqlain Shah.

Ethics declarations

Conflict of interest

There have been no involvements that might raise the question of bias in the work reported or in the Conclusions, implications, or opinions stated. The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shah, S.M.S., Khan, J.I., Abbas, S.H. et al. Symmetric mean binary pattern-based Pakistan sign language recognition using multiclass support vector machines. Neural Comput & Applic 35, 949–972 (2023). https://doi.org/10.1007/s00521-022-07804-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07804-2

Keywords

Search

Navigation