Skip to main content
SpringerLink
Log in

A Survey: The Sensor-Based Method for Sign Language Recognition

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14430))

Included in the following conference series:

  • 355 Accesses

Abstract

Sign language is a crucial communication carrier among deaf people to express and exchange their thoughts and emotions. However, ordinary individuals cannot acquire proficiency in sign language in the short term, which leads to deaf people facing huge barriers with the sound community. Regarding this conundrum, it is valuable to investigate Sign Language Recognition (SLR) equipped with sensors which collect data for the following computer vision processing. This study has reviewed the sensor-based SLR methods, which can transform heterogeneous signals from various underlying sensors into high-level motion representations. Specifically, we have summarized current developments in sensor-based SLR techniques from the perspective of modalities. Addtionally, we have also distilled the sensor-based SLR paradigm and compared the state-of-the-art works, including computer vision. Following that, we have concluded the research opportunities and future work expectations.

This work was supported in part by the Postgraduate Scientific Research Innovation Practice Program of Tianjin University of Technology (YJ2247).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abdullah, A., Abdul-Kadir, N.A., Che Harun, F.K.: An optimization of IMU sensors-based approach for Malaysian sign language recognition. In: ICCED, pp. 1–4 (2020)

    Google Scholar 

  2. Alaoui, F., Fourati, H., Kibangou, A., Robu, B., Vuillerme, N.: Kick-scooters identification in the context of transportation mode detection using inertial sensors: methods and accuracy. J. Intell. Transport. Syst. (2023). https://doi.org/10.1080/15472450.2022.2141118

    Article  Google Scholar 

  3. Alosail, D., Aldolah, H., Alabdulwahab, L., Bashar, A., Khan, M.: Smart glove for bi-lingual sign language recognition using machine learning. In: IDCIoT, pp. 409–415 (2023)

    Google Scholar 

  4. Barioul, R., Ghribi, S.F., Ben Jmaa Derbel, H., Kanoun, O.: Four sensors bracelet for American sign language recognition based on wrist force myography. In: CIVEMSA, pp. 1–5 (2020)

    Google Scholar 

  5. Barraza Madrigal, J.A., Contreras Rodríguez, L.A., Cardiel Pérez, E., Hernández Rodríguez, P.R., Sossa, H.: Hip and lower limbs 3D motion tracking using a double-stage data fusion algorithm for IMU/MARG-based wearables sensors. Biomed. Signal Process. Control 86, 104938 (2023)

    Article  Google Scholar 

  6. Ben Haj Amor, A., El Ghoul, O., Jemni, M.: Deep learning approach for sign language’s handshapes recognition from EMG signals. In: ITSIS, pp. 1–5 (2022)

    Google Scholar 

  7. Boukhechba, M., Cai, L., Wu, C., Barnes, L.E.: ActiPPG: using deep neural networks for activity recognition from wrist-worn photoplethysmography (PPG) sensors. Smart Health 14, 100082 (2019)

    Article  Google Scholar 

  8. Chen, H., Feng, D., Hao, Z., Dang, X., Niu, J., Qiao, Z.: Air-CSL: Chinese sign language recognition based on the commercial WiFi devices. Wirel. Commun. Mob. Comput. 2022 (2022). https://doi.org/10.1155/2022/5885475

  9. Choi, J., Hwang, G., Lee, J.S., Ryu, M., Lee, S.J.: Weighted knowledge distillation of attention-LRCN for recognizing affective states from PPG signals. Expert Syst. Appl. 120883 (2023)

    Google Scholar 

  10. Chu, X., Liu, J., Shimamoto, S.: A sensor-based hand gesture recognition system for Japanese sign language. In: LifeTech, pp. 311–312 (2021)

    Google Scholar 

  11. DiFilippo, N.M., Jouaneh, M.K.: Characterization of different Microsoft Kinect sensor models. IEEE Sens. J. 15(8), 4554–4564 (2015)

    Article  Google Scholar 

  12. Dweik, A., Qasrawi, H., Shawar, D.: Smart glove for translating Arabic sign language “SGTArSL”. In: ICCTA, pp. 49–53 (2021)

    Google Scholar 

  13. Fouts, T., Hindy, A., Tanner, C.: Sensors to sign language: a natural approach to equitable communication. In: ICASSP, pp. 8462–8466 (2022)

    Google Scholar 

  14. Galka, J., Masior, M., Zaborski, M., Barczewska, K.: Inertial motion sensing glove for sign language gesture acquisition and recognition. IEEE Sens. J. 16(16), 6310–6316 (2016)

    Article  Google Scholar 

  15. Godiyal, A.K., Singh, U., Anand, S., Joshi, D.: Analysis of force myography based locomotion patterns. Measurement 140, 497–503 (2019)

    Article  Google Scholar 

  16. Gupta, R., Bhatnagar, A.S.: Multi-stage Indian sign language classification with sensor modality assessment. In: ICACCS, vol. 1, pp. 18–22 (2021)

    Google Scholar 

  17. Gurbuz, S.Z., et al.: ASL recognition based on Kinematics derived from a multi-frequency RF sensor network. IEEE Sens. J. 1–4 (2020)

    Google Scholar 

  18. Han, J., Shao, L., Xu, D., Shotton, J.: Enhanced computer vision with Microsoft Kinect sensor: a review. IEEE T. Cybern. 43(5), 1318–1334 (2013)

    Article  Google Scholar 

  19. Hu, H., Wang, W., Zhou, W., Zhao, W., Li, H.: Model-aware gesture-to-gesture translation. In: CVPR, pp. 16423–16432 (2021)

    Google Scholar 

  20. Ji, L., Liu, J., Shimamoto, S.: Recognition of Japanese sign language by sensor-based data glove employing machine learning. In: LifeTech, pp. 256–258 (2022)

    Google Scholar 

  21. Kania, M., Korzeniewska, E., Zawiślak, R., Nikitina, A., Krawczyk, A.: Wearable solutions for the sign language. In: MEES, pp. 1–4 (2022)

    Google Scholar 

  22. Kudrinko, K., Flavin, E., Zhu, X., Li, Q.: Wearable sensor-based sign language recognition: a comprehensive review. IEEE Rev. Biomed. Eng. 14, 82–97 (2021)

    Article  Google Scholar 

  23. Kumar, P., Gauba, H., Roy, P.P., Dogra, D.P.: A multimodal framework for sensor based sign language recognition. Neurocomputing 259(SI), 21–38 (2017)

    Article  Google Scholar 

  24. Kwon, J., Nam, H., Chae, Y., Lee, S., Kim, I.Y., Im, C.H.: Novel three-axis accelerometer-based silent speech interface using deep neural network. Eng. Appl. Artif. Intell. 120, 105909 (2023)

    Article  Google Scholar 

  25. Liu, C., Liu, J., Shimamoto, S.: Sign language estimation scheme employing Wi-Fi signal. In: SAS, pp. 1–5 (2021)

    Google Scholar 

  26. Ma, Y., Zhao, S., Wang, W., Li, Y., King, I.: Multimodality in meta-learning: a comprehensive survey. Knowl.-Based Syst. 250, 108976 (2022)

    Article  Google Scholar 

  27. Maharjan, P., et al.: A human skin-inspired self-powered flex sensor with thermally embossed microstructured triboelectric layers for sign language interpretation. Nano Energy 76, 105071 (2020)

    Article  Google Scholar 

  28. Mitra, S., Acharya, T.: Gesture recognition: a survey. IEEE Trans. Syst. Man Cybern.-Syst. 37(3), 311–324 (2007)

    Article  Google Scholar 

  29. Muralidharan, N.T., Rohidh, M.R., Harikumar, M.E.: Modelling of sign language smart glove based on bit equivalent implementation using flex sensor. In: WiSPNET, pp. 99–104 (2022)

    Google Scholar 

  30. Nhu, C.T., Dang, P.N., Thanh, V.N.T., Thuy, H.T.T., Thanh, V.D., Thanh, T.B.: A sign language recognition system using ionic liquid strain sensor. In: ISMEE, pp. 263–267 (2021)

    Google Scholar 

  31. Qahtan, S., Alsattar, H.A., Zaidan, A.A., Deveci, M., Pamucar, D., Martinez, L.: A comparative study of evaluating and benchmarking sign language recognition system-based wearable sensory devices using a single fuzzy set. Knowl.-Based Syst. 269, 110519 (2023)

    Article  Google Scholar 

  32. Qin, Y., Pan, S., Zhou, W., Pan, D., Li, Z.: WiASL: American sign language writing recognition system using commercial WiFi devices. Measurement 218, 113125 (2023)

    Article  Google Scholar 

  33. Rakun, E., Andriani, M., Wiprayoga, I.W., Danniswara, K., Tjandra, A.: Combining depth image and skeleton data from Kinect for recognizing words in the sign system for Indonesian language (SIBI [Sistem Isyarat Bahasa Indonesia]). In: ICACSIS, pp. 387–392 (2013)

    Google Scholar 

  34. Rashid, A., Hasan, O.: Wearable technologies for hand joints monitoring for rehabilitation: a survey. Microelectron. J. 88, 173–183 (2019)

    Article  Google Scholar 

  35. Saggio, G., Riillo, F., Sbernini, L., Quitadamo, L.R.: Resistive flex sensors: a survey. Smart Mater. Struct. 25(1), 013001 (2016)

    Google Scholar 

  36. Saif, R., Ahmad, M., Naqvi, S.Z.H., Aziz, S., Khan, M.U., Faraz, M.: Multi-channel EMG signal analysis for Italian sign language interpretation. In: ICETST, pp. 1–5 (2022)

    Google Scholar 

  37. Sarkar, B., Takeyeva, D., Guchhait, R., Sarkar, M.: Optimized radio-frequency identification system for different warehouse shapes. Knowl.-Based Syst. 258, 109811 (2022)

    Article  Google Scholar 

  38. Sharma, A., Ansari, M.Z., Cho, C.: Ultrasensitive flexible wearable pressure/strain sensors: parameters, materials, mechanisms and applications. Sens. Actuat. A 347, 113934 (2022)

    Article  Google Scholar 

  39. Subedi, B., Dorji, K.U., Wangdi, P., Dorji, T., Muramatsu, K.: Sign language translator of Dzongkha alphabets using Arduino. In: i-PACT, pp. 1–6 (2021)

    Google Scholar 

  40. Suri, A., Singh, S.K., Sharma, R., Sharma, P., Garg, N., Upadhyaya, R.: Development of sign language using flex sensors. In: ICOSEC, pp. 102–106 (2020)

    Google Scholar 

  41. Sze, F.: From gestures to grammatical non-manuals in sign language: a case study of polar questions and negation in Hong Kong sign language. Lingua 267, 103188 (2022)

    Article  Google Scholar 

  42. Ul Islam, M.R., Bai, S.: A novel approach of FMG sensors distribution leading to subject independent approach for effective and efficient detection of forearm dynamic movements. Biomed. Eng. Adv. 4, 100062 (2022)

    Article  Google Scholar 

  43. Venugopalan, A., Reghunadhan, R.: Applying deep neural networks for the automatic recognition of sign language words: a communication aid to deaf agriculturists. Expert Syst. Appl. 185, 115601 (2021)

    Article  Google Scholar 

  44. Wang, Z., et al.: Hear sign language: a real-time end-to-end sign language recognition system. IEEE Trans. Mob. Comput. 21(7), 2398–2410 (2022)

    Google Scholar 

  45. Wu, J., Sun, L., Jafari, R.: A wearable system for recognizing American sign language in real-time using IMU and surface EMG sensors. IEEE J. Biomed. Health Inform. 20(5, SI), 1281–1290 (2016)

    Article  Google Scholar 

  46. Yang, H.D.: Sign language recognition with the Kinect sensor based on conditional random fields. IEEE Sens. J. 15(1), 135–147 (2015)

    Google Scholar 

  47. Yang, X., Chen, X., Cao, X., Wei, S., Zhang, X.: Chinese sign language recognition based on an optimized tree-structure framework. IEEE J. Biomed. Health Inform. 21(4), 994–1004 (2017)

    Article  Google Scholar 

  48. Zhang, N., Zhang, J., Ying, Y., Luo, C., Li, J.: Wi-phrase: deep residual-multihead model for WiFi sign language phrase recognition. IEEE Internet Things J. 9(18), 18015–18027 (2022)

    Article  Google Scholar 

  49. Zhang, Y., Xu, W., Zhang, X., Li, L.: Sign annotation generation to alphabets via integrating visual data with somatosensory data from flexible strain sensor-based data glove. Measurement 202, 111700 (2022)

    Article  Google Scholar 

  50. Zhao, T., Liu, J., Wang, Y., Liu, H., Chen, Y.: Towards low-cost sign language gesture recognition leveraging wearables. IEEE Trans. Mob. Comput. 20(4), 1685–1701 (2021)

    Article  Google Scholar 

  51. Zhou, H., Zhou, W., Zhou, Y., Li, H.: Spatial-temporal multi-cue network for sign language recognition and translation. IEEE Trans. Multimed. 24, 768–779 (2022)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, People’s Republic of China

    Tian Yang, Cong Shen, Xinyue Wang & Xiaoyu Ma

  2. Engineering Research Center of Learning-Based Intelligent System, Ministry of Education, Tianjin, People’s Republic of China

    Tian Yang, Cong Shen, Xinyue Wang & Xiaoyu Ma

  3. R &D Center (China), Intel-Mobileye, Shanghai, People’s Republic of China

    Chen Ling

Authors
  1. Tian Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cong Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinyue Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoyu Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chen Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cong Shen .

Editor information

Editors and Affiliations

  1. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

  2. Xiamen University, Xiamen, China

    Hanzi Wang

  3. Beijing University of Posts and Telecommunications, Beijing, China

    Zhanyu Ma

  4. Sun Yat-sen University, Guangzhou, China

    Weishi Zheng

  5. Peking University, Beijing, China

    Hongbin Zha

  6. Chinese Academy of Sciences, Beijing, China

    Xilin Chen

  7. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  8. Xiamen University, Xiamen, China

    Rongrong Ji

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yang, T., Shen, C., Wang, X., Ma, X., Ling, C. (2024). A Survey: The Sensor-Based Method for Sign Language Recognition. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14430. Springer, Singapore. https://doi.org/10.1007/978-981-99-8537-1_21

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8537-1_21

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8536-4

  • Online ISBN: 978-981-99-8537-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation