skip to main content
10.1145/1891903.1891926acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

Automatic recognition of sign language subwords based on portable accelerometer and EMG sensors

Published: 08 November 2010 Publication History

Abstract

Sign language recognition (SLR) not only facilitates the communication between the deaf and hearing society, but also serves as a good basis for the development of gesture-based human-computer interaction (HCI). In this paper, the portable input devices based on accelerometers and surface electromyography (EMG) sensors worn on the forearm are presented, and an effective fusion strategy for combination of multi-sensor and multi-channel information is proposed to automatically recognize sign language at the subword classification level. Experimental results on the recognition of 121 frequently used Chinese sign language subwords demonstrate the feasibility of developing SLR system based on the presented portable input devices and that our proposed information fusion method is effective for automatic SLR. Our study will promote the realization of practical sign language recognizer and multimodal human-computer interfaces.

References

[1]
Ong, S. C. W. and Ranganath, S. 2005. Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning. IEEE Trans. Pattern Anal. Mach. Intell., 27(6): 873--891.
[2]
Fang, G., Gao, W., Zhao, D. 2003. Large vocabulary sign language recognition based on hierarchical decision trees. In Proc. the 5th int. conf. on Multimodal interfaces (ICMI'03), Vancouver, British Columbia, Canada, 125--131.
[3]
Mitra S., and Acharya, T. 2007. Gesture recognition: A survey. IEEE Transactions on Systems Man and Cybernetics Part C-Applications and Reviews, 37(3): 311--324.
[4]
Kelly, D., Delannoy, R., Mc Donald, J. 2009. A framework for continuous multimodal sign language recognition. In Proc. of the Int. Conf. on Multimodal interfaces (ICMI-MLMI'09), Cambridge, Massachusetts, USA, 351--358.
[5]
Oskoei M. A., Hu, H. S. 2007. Myoelectric control systems-A survey. Biomedical Signal Processing and Control, 2(4): 275--294.
[6]
Saponas, T. S., Tan, D. S., Morris, D., Balakrishnan, R. 2008. Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In Proceeding of the 26th annual SIGCHI conference on Human factors in computing systems (CHI'08), Florence, Italy, 515--524.
[7]
Chen, X., Zhang, X., Zhao, Z., Yang, J., Lantz, V. and Wang, K. 2007. Hand Gesture Recognition Research Based on Surface EMG Sensors and 2D-accelerometers. In Proceedings of the 2007 11th IEEE International Symposium on Wearable Computers, Boston, MA, USA, 1--4.
[8]
Kim J., Wagner J., Rehm M., and André E., 2008. Bichannel Sensor Fusion for Automatic Sign Language Recognition. In Proc. IEEE Int. Conf. Automatic Face and Gesture Recognition, Amsterdam, Netherlands.
[9]
Zhang, X., Chen, X., Wang, W., Yang, J., Lantz, V., and Wang, K. 2009. Hand gesture recognition and virtual game control based on 3D accelerometer and EMG sensors. In Proceeding of the 13th international conference on Intelligent user interfaces (IUI'09), Sanibel Island, USA, 401--406.
[10]
Wang, C. L., Gao, W., and Shan, S. G. 2002. An approach based on phonemes to large vocabulary Chinese sign language recognition. In Proc. of IEEE Int. Conf. on Automatic Face and Gesture Recognition, Washington, DC, USA, 411--416.
[11]
Kosmidou, V. E., Hadjileontiadis, L. J. 2009. Sign language recognition using intrinsic-mode sample entropy on sEMG and accelerometer data. IEEE Trans. Biomed. Eng., 56(12): 2879--2890.
[12]
Department for Education and Employment in China Disabled Persons' Federation, China Association of the Deaf. 2003. Chinese Sign Language (Revised edition). Beijing: HuaXia Press. (in Chinese)
[13]
Gao, S. L., Yu, P. 1998. Atlas of Human Anatomy (Revision). Shanghai: Shanghai Scientific & Technical Publishers. (in Chinese)
[14]
Zhang, L., Chen, X., Wang, C., Chen, Y., Gao, W. 2005. Recognition of sign language subwords based on boosted hidden Markov models. In Proceedings of the 7th Int. Conf. on Multimodal interfaces, Torento, Italy, 282--287.
[15]
Starner, T., Weaver, J. and Penland, A. 1998. Real-time American sign language recognition using desk and wearable computer based video. IEEE Trans. Pattern Analysis and Machine Intelligence, 20(12): 1371--1375.
[16]
Vogler, C. and Metaxas, D. 1999. ASL Recognition Based on a Coupling Between HMMs and 3D Motion Analysis. In Proceedings of the Sixth International Conference on Computer Vision, Bombay, 363--369.
[17]
Mäntyjärvi, J., Kela, J., Korpipää, P., Kallio, S. 2004. Enabling fast and effortless customisation in accelerometer based gesture interaction. In Proceedings of the 3rd international conference on Mobile and ubiquitous multimedia (MUM'04), College Park, Maryland, 25--31.
[18]
Brashear, H., Starner, T., Lukowicz, P., Junker H. 2003. Using Multiple Sensors for Mobile Sign Language Recognition. In Proceedings of the 7th IEEE International Symposium on Wearable Computers, White Plains, NY, USA, 45--52.
[19]
Costanza, E., Inverso, S. A., Allen, R. 2005. Toward subtle intimate interfaces for mobile devices using an EMG controller. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI'05), Portland, Oregon, USA, 481--489.
[20]
Huang, Y., Englehart, K. B., Hudgins, B., Chan, A. D. C. 2005. A Gaussian mixture model based classification scheme for myoelectric control of powered upper limb prostheses. IEEE Transactions on Biomedical Engineering, 52(11): 1801--1811.
[21]
Duda, R. O., Hart, P. E. and Stork, D. G. 2001. Pattern Classification, Second Edition, New York: John Wiley, Section 10.4.4.
[22]
Liu C. and Wechsler H. 2000. Robust coding schemes for indexing and retrieval from large face databases. IEEE Trans. Image Processing, 9(1): 132--137.

Cited By

View all
  • (2024)Pattern Recognition in the Processing of Electromyographic Signals for Selected Expressions of Polish Sign LanguageSensors10.3390/s2420671024:20(6710)Online publication date: 18-Oct-2024
  • (2024)American Sign Language Recognition and Translation Using Perception Neuron Wearable Inertial Motion Capture SystemSensors10.3390/s2402045324:2(453)Online publication date: 11-Jan-2024
  • (2024)EITPose: Wearable and Practical Electrical Impedance Tomography for Continuous Hand Pose EstimationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642663(1-10)Online publication date: 11-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI-MLMI '10: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction
November 2010
311 pages
ISBN:9781450304146
DOI:10.1145/1891903
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2010

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. accelerometer
  2. gesture-based interfaces
  3. sign language recognition
  4. surface electromyography

Qualifiers

  • Research-article

Funding Sources

Conference

ICMI-MLMI '10
Sponsor:

Acceptance Rates

ICMI-MLMI '10 Paper Acceptance Rate 41 of 100 submissions, 41%;
Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)20
  • Downloads (Last 6 weeks)2
Reflects downloads up to 12 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Pattern Recognition in the Processing of Electromyographic Signals for Selected Expressions of Polish Sign LanguageSensors10.3390/s2420671024:20(6710)Online publication date: 18-Oct-2024
  • (2024)American Sign Language Recognition and Translation Using Perception Neuron Wearable Inertial Motion Capture SystemSensors10.3390/s2402045324:2(453)Online publication date: 11-Jan-2024
  • (2024)EITPose: Wearable and Practical Electrical Impedance Tomography for Continuous Hand Pose EstimationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642663(1-10)Online publication date: 11-May-2024
  • (2023)Sign Language Recognition Using the Electromyographic Signal: A Systematic Literature ReviewSensors10.3390/s2319834323:19(8343)Online publication date: 9-Oct-2023
  • (2023)A Framework and Call to Action for the Future Development of EMG-Based Input in HCIProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580962(1-23)Online publication date: 19-Apr-2023
  • (2022)A Machine Translation System from Indian Sign Language to English TextInternational Journal of Information Technologies and Systems Approach10.4018/IJITSA.31341915:1(1-23)Online publication date: 28-Oct-2022
  • (2022)Show of Hands: Leveraging Hand Gestural Cues in Virtual Meetings for Intelligent Impromptu Polling InteractionsProceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511153(292-309)Online publication date: 22-Mar-2022
  • (2021)Design of Effective Smart Communication System for Impaired PeopleJournal of Electrical Engineering and Automation10.36548/jeea.2020.4.0062:4(181-194)Online publication date: 8-Mar-2021
  • (2021)Wearable Sensor-Based Sign Language Recognition: A Comprehensive ReviewIEEE Reviews in Biomedical Engineering10.1109/RBME.2020.301976914(82-97)Online publication date: 2021
  • (2021)A deep learning based approach for Arabic Sign language alphabet recognition using electromyographic signals2021 8th International Conference on ICT & Accessibility (ICTA)10.1109/ICTA54582.2021.9809780(01-04)Online publication date: 8-Dec-2021
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media