Abstract
Sign languages has extensive applications among differently-abled to communicate with their surroundings. With the development of different sensing technologies, several new human-computer interaction techniques (HCI) have been established to recognize hand gestures. Computer vision-based methods have shown significant utility for such applications. However, these methods are strongly dependent on the lighting conditions. The surface electromyography (sEMG) technique is invariant to lighting conditions and can easily reflect human motion intention. In this work, sEMG based sign language recognition model was developed using an efficient machine learning pipeline.
Two sEMG datasets were recorded for predefined hand gestures using wireless sensors. These signals were mainly acquired against 24 manual alphabets (ASL-24) and ten digits(ASL-10) of American Sign Language (ASL). The collected data sets were preprocessed, and around 450 well-established feature was extracted from each sEMG channel. We applied an ensemble feature selection approach combining four diverse filter-based feature selection methods (ANOVA, Chi-square, Mutual Info, ReliefF). A newly proposed feature combiner that exploits feature–feature and feature–class correlation thresholds is used to combine feature subsets formed across the ensemble. The resulting features comprise reduced & most representative feature subsets and are further used in the pipeline for classifying ASL gestures.
Using the CatBoost algorithm, the pipeline presented excellent average classification accuracy(99.91% on ASL-24) and other performance parameters for recognizing ASL gestures. The pipeline was also applied and validated on a benchmark dataset (Ninapro database 5, exercise A) and achieved similar outcomes. The result highlights the feasibility of using sEMG based approach as better options to computer-vision-based techniques to build an accurate and robust Sign Language Recognition system (SLRS). Moreover, efforts were made to find the optimal number of sensors and features for recognition task on (ASL-10 dataset) without impacting the overall reliability and accuracy of the system. The experiments results can be used to enhance the performance of various wearable sEMG sensor based HCI applications.
Similar content being viewed by others
Data Availability
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
References
Aboy M, Hornero R, Abásolo D, Álvarez D (2006) Interpretation of the lempel-ziv complexity measure in the context of biomedical signal analysis. IEEE Trans Biomed Eng 53(11):2282–2288
Ahmed MA, Zaidan BB, Zaidan AA, Salih MM, Lakulu MMB (2018) A review on systems-based sensory gloves for sign language recognition state of the art between 2007 and 2017. Sensors 18(7):2208
Ahsan MR, Ibrahimy MI, Khalifa OO et al (2009) Emg signal classification for human computer interaction: a review. Eur J Sci Res 33(3):480–501
Ali Khan S, Hussain A, Basit A, Akram S (2014) Kruskal-wallis-based computationally efficient feature selection for face recognition. Sci World J, vol 2014
Anderson R, Wiryana F, Ariesta MC, Kusuma GP et al (2017) Sign language recognition application systems for deaf-mute people: a review based on input-process-output. Procedia Comput Sci 116:441–448
Atzori M, Gijsberts A, Castellini C, Caputo B, Hager A-GM, Elsig S, Giatsidis G, Bassetto F, Müller H (2014) Electromyography data for non-invasive naturally-controlled robotic hand prostheses. Sci Data 1(1):1–13
Barbhuiya AA, Karsh RK, Jain R (2021) Cnn based feature extraction and classification for sign language. Multimed Tools Appl 80(2):3051–3069
Batista GE, Keogh EJ, Tataw OM, De Souza VM (2014) Cid: an efficient complexity-invariant distance for time series. Data Min Knowl Disc 28 (3):634–669
Battison R (1978) Lexical borrowing in american sign language
Bheda V, Radpour D (2017) Using deep convolutional networks for gesture recognition in american sign language. arXiv:1710.06836
Bin Munir M, Alam FR, Ishrak S, Hussain S, Shalahuddin M, Islam MN (2021) A machine learning based sign language interpretation system for communication with deaf-mute people. In: Proceedings of the XXI international conference on human computer interaction, pp 1–9
Blum AL, Langley P (1997) Selection of relevant features and examples in machine learning. Artif Intell 97(1-2):245–271
Cardenas EJE, Chavez GC (2020) Multimodal hand gesture recognition combining temporal and pose information based on cnn descriptors and histogram of cumulative magnitudes. J Vis Commun Image Represent 71:102772
Chang Y-W, Lin C-J (2008) Feature ranking using linear svm. In: Causation and prediction challenge. PMLR, pp 53–64
Chen C-W, Tsai Y-H, Chang F-R, Lin W-C (2020) Ensemble feature selection in medical datasets: combining filter, wrapper, and embedded feature selection results. Expert Syst 37(5):12553
Chicco D, Jurman G (2020) The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC Genomics 21(1):1–13
Chowdhury RH, Reaz MB, Ali MABM, Bakar AA, Chellappan K, Chang T (2013) Surface electromyography signal processing and classification techniques. Sensors (Basel Switzerland) 13(9):12431–12466
Chuan C-H, Regina E, Guardino C (2014) American sign language recognition using leap motion sensor. In: 2014 13th International conference on machine learning and applications. IEEE, pp 541–544
Cooper H, Holt B, Bowden R (2011) Sign language recognition. In: Visual analysis of humans. Springer, pp 539–562
Day S (2002) Important factors in surface emg measurement. Bortec Biomed Ltd Pub:1–17
De la Rosa R, Alonso A, Carrera A, Durán R, Fernández P (2010) Man-machine interface system for neuromuscular training and evaluation based on emg and mmg signals. Sensors (Basel Switzerland) 10(12):11100–11125
Dietterich TG (1998) Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computat 10(7):1895–1923
Dinno A (2015) Nonparametric pairwise multiple comparisons in independent groups using dunn’s test. Stata J 15(1):292–300
Dorogush AV, Ershov V, Gulin A (2018) Catboost: gradient boosting with categorical features support. arXiv:1810.11363
Dunn OJ (1964) Multiple comparisons using rank sums. Technometrics 6(3):241–252
Erkilinc MS, Sahin F (2011) Camera control with emg signals using principal component analysis and support vector machines. In: 2011 IEEE international systems conference. IEEE, pp 417–421
Fatmi R, Rashad S, Integlia R (2019) Comparing ann, svm, and hmm based machine learning methods for american sign language recognition using wearable motion sensors. In: 2019 IEEE 9th annual computing and communication workshop and conference (CCWC)
Fels SS, Hinton GE (1993) Glove-talk: a neural network interface between a data-glove and a speech synthesizer. IEEE Trans Neural Netw 4(1):2–8
Feng Y, Uchidiuno UA, Zahiri HR, George I, Park AE, Mentis H (2021) Comparison of kinect and leap motion for intraoperative image interaction. Surg Innov 28(1):33–40
Ferri C, Hernández-Orallo J, Modroiu R (2009) An experimental comparison of performance measures for classification. Pattern Recognit Lett 30 (1):27–38
Friedrich R, Siegert S, Peinke J, Siefert M, Lindemann M, Raethjen J, Deuschl G, Pfister G et al (2000) Extracting model equations from experimental data. Phys Lett A 271(3):217–222
Garcia B, Viesca SA (2016) Real-time american sign language recognition with convolutional neural networks. Convolutional Neural Netw Vis Recognit 2:225–232
Genuer R, Poggi J-M, Tuleau-Malot C (2015) Vsurf: an r package for variable selection using random forests. R Journal 7(2):19–33
Gomez-Donoso F, Orts-Escolano S, Cazorla M (2019) Accurate and efficient 3d hand pose regression for robot hand teleoperation using a monocular rgb camera. Expert Syst Appl 136:327–337
Goswami T, Javaji SR (2021) Cnn model for american sign language recognition. In: ICCCE 2020. Springer, pp 55–61
Grandini M, Bagli E, Visani G (2020) Metrics for multi-class classification: an overview. arXiv:2008.05756
Güler NF, Koçer S (2005) Classification of emg signals using pca and fft. J Med Syst 29(3):241–250
Guo D, Zhou W, Li H, Wang M (2018) Hierarchical lstm for sign language translation. In: Proceedings of the AAAI conference on artificial intelligence, vol 32
Guyon I, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3(Mar):1157–1182
Haq AU, Zhang D, Peng H, Rahman SU (2019) Combining multiple feature-ranking techniques and clustering of variables for feature selection. IEEE Access 7:151482–151492. https://doi.org/10.1109/ACCESS.2019.2947701
Hoque N, Singh M, Bhattacharyya DK (2018) Efs-mi: an ensemble feature selection method for classification. Complex Intell Syst 4(2):105–118
Hudgins B, Parker P, Scott RN (1993) A new strategy for multifunction myoelectric control. IEEE Trans Biomed Eng 40(1):82–94
Isaacs J, Foo S (2004) Hand pose estimation for american sign language recognition. In: Thirty-sixth southeastern symposium on system theory, 2004. Proceedings of the. IEEE, pp 132–136
Jain A, Zongker D (1997) Feature selection: evaluation, application, and small sample performance. IEEE Trans Pattern Anal Mach Intell 19(2):153–158. https://doi.org/10.1109/34.574797
Jones E, Oliphant T, Peterson P et al (2001) SciPy: open source scientific tools for python. http://www.scipy.org/. Accessed 10 June 2022
Jurman G, Riccadonna S, Furlanello C (2012) A comparison of mcc and cen error measures in multi-class prediction
Kadhim RA, Khamees M (2020) A real-time american sign language recognition system using convolutional neural network for real datasets. TEM J 9 (3):937
Kanoga S, Kanemura A, Asoh H (2020) Are armband semg devices dense enough for long-term use?—sensor placement shifts cause significant reduction in recognition accuracy. Biomed Signal Process Contr 60:101981
Kerber F, Schardt P, Löchtefeld M (2015) Wristrotate: a personalized motion gesture delimiter for wrist-worn devices. In: Proceedings of the 14th international conference on mobile and ubiquitous multimedia, pp 218–222
Khan SM, Khan AA, Farooq O (2019) Selection of features and classifiers for emg-eeg-based upper limb assistive devices—a review. IEEE Rev Biomed Eng 13:248–260
Kleiman R, Page D (2019) Aucμ: a performance metric for multi-class machine learning models. In: International conference on machine learning. PMLR, pp 3439–3447
Koller O (2020) Quantitative survey of the state of the art in sign language recognition. arXiv:2008.09918
Kosmidou VE, Hadjileontiadis LJ, Panas SM (2006) Evaluation of surface emg features for the recognition of american sign language gestures. In: 2006 International conference of the IEEE engineering in medicine and biology society, pp 6197–6200. https://doi.org/10.1109/IEMBS.2006.259428
Kuroda T, Tabata Y, Goto A, Ikuta H, Murakami M et al (2004) Consumer price data-glove for sign language recognition. In: Proceeding ICDVRAT, pp 253–258
Lee CK, Ng KK, Chen C-H, Lau HC, Chung S, Tsoi T (2021) American sign language recognition and training method with recurrent neural network. Expert Syst Appl 167:114403
Li L, Jiang S, Shull PB, Gu G (2018) Skingest: artificial skin for gesture recognition via filmy stretchable strain sensors. Adv Robot 32 (21):1112–1121
Liddell SK, Johnson RE (1989) American sign language: the phonological base. Sign Language Studies 64(1):195–277
Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems 30. Curran Associates, Inc., pp 4765–4774. http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf. Accessed 10 June 2022
Masood S, Srivastava A, Thuwal HC, Ahmad M (2018) Real-time sign language gesture (word) recognition from video sequences using cnn and rnn. In: Intelligent engineering informatics. Springer, pp 623–632
McKight PE, Najab J (2010) Kruskal-wallis test. corsini Encyclo Psychol:1–1
Mehdi SA, Khan YN (2002) Sign language recognition using sensor gloves. In: Proceedings of the 9th international conference on neural information processing, 2002. ICONIP’02., vol 5, pp 2204–22065. https://doi.org/10.1109/ICONIP.2002.1201884
Miften FS, Diykh M, Abdulla S, Siuly S, Green JH, Deo RC (2021) A new framework for classification of multi-category hand grasps using emg signals. Artif Intell Med 112:102005
Müller M (2007) Information retrieval for music and motion. Springer, vol 2
Munib Q, Habeeb M, Takruri B, Al-Malik HA (2007) American sign language (asl) recognition based on hough transform and neural networks. Expert Syst Appl 32(1):24–37
NIDCD (2021) American sign language. https://www.nidcd.nih.gov/health/american-sign-language. Accessed 26 May 2021
Nishad A, Upadhyay A, Pachori RB, Acharya UR (2019) Automated classification of hand movements using tunable-q wavelet transform based filter-bank with surface electromyogram signals. Futur Gener Comput Syst 93:96–110
Olsson JOS, Oard DW (2006) Combining feature selectors for text classification. In: Proceedings of the 15th ACM international conference on information and knowledge management, pp 798–799
Oz C, Leu MC (2005) Recognition of finger spelling of american sign language with artificial neural network using position/orientation sensors and data glove. In: International symposium on neural networks. Springer, pp 157–164
Oz C, Leu MC (2007) Linguistic properties based on american sign language isolated word recognition with artificial neural networks using a sensory glove and motion tracker. Neurocomputing 70(16-18):2891–2901
Oz C, Leu MC (2011) American sign language word recognition with a sensory glove using artificial neural networks. Eng Appl Artif Intell 24 (7):1204–1213
Paudyal P, Banerjee A, Gupta SK (2016) Sceptre: a pervasive, non-invasive, and programmable gesture recognition technology. In: Proceedings of the 21st international conference on intelligent user interfaces, pp 282–293
Pires R, Falcari T, Campo AB, Pulcineli BC, Hamill J, Ervilha UF (2019) Using a support vector machine algorithm to classify lower-extremity emg signals during running shod/unshod with different foot strike patterns. J Appl Biomechan 35(1):87–90
Pizzolato S, Tagliapietra L, Cognolato M, Reggiani M, Müller H, Atzori M (2017) Comparison of six electromyography acquisition setups on hand movement classification tasks. PloS One 12(10):0186132
Poizner H, Tallal P (1987) Temporal processing in deaf signers. Brain Lang 30(1):52–62
Prokhorenkova L, Gusev G, Vorobev A, Dorogush AV, Gulin A (2018) Catboost: unbiased boosting with categorical features. In: Advances in neural information processing systems, pp 6638–6648
Pugeault N, Bowden R (2011) Spelling it out: real-time asl fingerspelling recognition. In: 2011 IEEE international conference on computer vision workshops (ICCV workshops). IEEE, pp 1114–1119
Rao GA, Syamala K, Kishore P, Sastry A (2018) Deep convolutional neural networks for sign language recognition. In: 2018 Conference on signal processing and communication engineering systems (SPACES). IEEE, pp 194–197
Rashid O, Al-Hamadi A, Michaelis B (2010) Utilizing invariant descriptors for finger spelling american sign language using svm. In: International symposium on visual computing. Springer, pp 253–263
Rastgoo R, Kiani K, Escalera S (2018) Multi-modal deep hand sign language recognition in still images using restricted boltzmann machine. Entropy 20(11):809
Rastgoo R, Kiani K, Escalera S (2020) Hand sign language recognition using multi-view hand skeleton. Expert Syst Appl 150:113336
Remeseiro B, Bolon-Canedo V (2019) A review of feature selection methods in medical applications. Comput Bio Med 112:103375
Rivera-Acosta M, Ruiz-Varela JM, Ortega-Cisneros S, Rivera J, Parra-Michel R, Mejia-Alvarez P (2021) Spelling correction real-time american sign language alphabet translation system based on yolo network and lstm. Electronics 10(9):1035
Rodríguez-Tapia B, Soto I, Martínez DM, Arballo NC (2020) Myoelectric interfaces and related applications: current state of emg signal processing–a systematic review. IEEE Access 8:7792–7805
Salo F, Injadat M, Moubayed A, Nassif AB, Essex A (2019) Clustering enabled classification using ensemble feature selection for intrusion detection. In: 2019 International conference on computing, networking and communications (ICNC). IEEE, pp 276–281
Savur C, Sahin F (2015) Real-time american sign language recognition system using surface emg signal. In: 2015 IEEE 14th international conference on machine learning and applications (ICMLA). IEEE, pp 497–502
Savur C, Sahin F (2016) American sign language recognition system by using surface emg signal. In: 2016 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, pp 002872–002877
Schreiber T, Schmitz A (1997) Discrimination power of measures for nonlinearity in a time series. Phys Rev E 55(5):5443
Sharma R, Pachori RB (2015) Classification of epileptic seizures in eeg signals based on phase space representation of intrinsic mode functions. Expert Syst Appl 42(3):1106–1117
Simons EDMGF, Fennig CD (2021) Ethnologue: languages of the world. http://www.ethnologue.com. Accessed 26 May 2021
Starner T, Pentland A (1997) Real-time american sign language recognition from video using hidden markov models. In: Motion-based recognition. Springer, pp 227–243
Sun C, Zhang T, Bao B-K, Xu C (2013) Latent support vector machine for sign language recognition with kinect. In: 2013 IEEE international conference on image processing. IEEE, pp 4190–4194
Taylor J (2016) Real-time translation of american sign language using wearable technology
Too J, Abdullah A, Saad NM, Ali NM, Musa H (2018) A detail study of wavelet families for emg pattern recognition. Int J Electr Comput Eng (IJECE) 8(6):4221–4229
Van der Maaten L, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res, vol 9(11)
Wadhawan A, Kumar P (2019) Sign language recognition systems: a decade systematic literature review. Arch Computat Methods Eng:1–29
Wadhawan A, Kumar P (2020) Deep learning-based sign language recognition system for static signs. Neural Comput Appl 32(12):7957–7968
Wang H, Khoshgoftaar TM, Napolitano A (2012) Software measurement data reduction using ensemble techniques. Neurocomputing 92:124–132
Wattenberg M, Viégas F, Johnson I (2016) How to use t-sne effectively. Distill. https://doi.org/10.23915/distill.00002https://doi.org/10.23915/distill.00002
Wu J, Sun L, Jafari R (2016) A wearable system for recognizing american sign language in real-time using imu and surface emg sensors. IEEE J Biomed Health Inform 20(5):1281–1290. https://doi.org/10.1109/JBHI.2016.2598302
Wu J, Tian Z, Sun L, Estevez L, Jafari R (2015) Real-time american sign language recognition using wrist-worn motion and surface emg sensors. In: 2015 IEEE 12th international conference on wearable and implantable body sensor networks (BSN). IEEE, pp 1–6
Wu J, Tian Z, Sun L, Estevez L, Jafari R (2015) Real-time american sign language recognition using wrist-worn motion and surface emg sensors. In: 2015 IEEE 12th international conference on wearable and implantable body sensor networks (BSN). https://doi.org/10.1109/BSN.2015.7299393, pp 1–6
Yeh C-CM, Zhu Y, Ulanova L, Begum N, Ding Y, Dau HA, Silva DF, Mueen A, Keogh E (2016) Matrix profile i: all pairs similarity joins for time series: a unifying view that includes motifs, discords and shapelets. In: 2016 IEEE 16th international conference on data mining (ICDM). Ieee, pp 1317–1322
Yu E, Cho S (2006) Ensemble based on ga wrapper feature selection. Comput Industr Eng 51(1):111–116
Zafrulla Z, Brashear H, Starner T, Hamilton H, Presti P (2011) American sign language recognition with the kinect. In: Proceedings of the 13th international conference on multimodal interfaces, pp 279–286
Zamani M, Kanan HR (2014) Saliency based alphabet and numbers of american sign language recognition using linear feature extraction. In: 2014 4th International conference on computer and knowledge engineering (ICCKE). IEEE, pp 398–403
Zhang J, Bi H, Chen Y, Wang M, Han L, Cai L (2019) Smarthandwriting: handwritten chinese character recognition with smartwatch. IEEE Internet Things J 7(2):960–970
Zhang Y, Gong D-W, Cheng J (2017) Multi-objective particle swarm optimization approach for cost-based feature selection in classification. IEEE/ACM Trans Computat Bio Bioinform 14(1):64–75. https://doi.org/10.1109/TCBB.2015.2476796
Zhao W (2016) A concise tutorial on human motion tracking and recognition with microsoft kinect. Sci China Inf Sci 59(9):1–5
Zheng M, Crouch M, Eggleston MS (2021) Surface electromyography as a natural human-machine interface: a review. arXiv:2101.04658
Zia ur Rehman M, Gilani SO, Waris A, Niazi IK, Slabaugh G, Farina D, Kamavuako EN (2018) Stacked sparse autoencoders for emg-based classification of hand motions: a comparative multi day analyses between surface and intramuscular emg. Appl Sci 8(7):1126
Acknowledgements
We thank Dr. Shiru Sharma, Associate Professor, School of Biomedical Engineering IIT BHU, for providing the MyoArmband for data collection. We also thank Dr. Alok Prakash, School of Biomedical Engineering IIT BHU, for help in deciding the experimental protocol for data collection.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
None of the authors have any conflict of interest to declare.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Singh, S.K., Chaturvedi, A. A reliable and efficient machine learning pipeline for american sign language gesture recognition using EMG sensors. Multimed Tools Appl 82, 23833–23871 (2023). https://doi.org/10.1007/s11042-022-14117-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-022-14117-y