skip to main content
10.1145/3025453.3025807acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

EchoFlex: Hand Gesture Recognition using Ultrasound Imaging

Published: 02 May 2017 Publication History

Abstract

Recent improvements in ultrasound imaging enable new opportunities for hand pose detection using wearable devices. Ultrasound imaging has remained under-explored in the HCI community despite being non-invasive, harmless and capable of imaging internal body parts, with applications including smart-watch interaction, prosthesis control and instrument tuition. In this paper, we compare the performance of different forearm mounting positions for a wearable ultrasonographic device. Location plays a fundamental role in ergonomics and performance since the anatomical features differ among positions. We also investigate the performance decrease due to cross-session position shifts and develop a technique to compensate for this misalignment. Our gesture recognition algorithm combines image processing and neural networks to classify the flexion and extension of 10 discrete hand gestures with an accuracy above 98%. Furthermore, this approach can continuously track individual digit flexion with less than 5% NRMSE, and also differentiate between digit flexion at different joints.

Supplementary Material

ZIP File (pn2761-file4.zip)
suppl.mov (pn2761.mp4)
Supplemental video
suppl.mov (pn2761p.mp4)
Supplemental video
MP4 File (p1923-mcintosh.mp4)

References

[1]
Akhlaghi, N., Baker, C. A., Lahlou, M., Zafar, H., Murthy, K. G., Rangwala, H. S., Kosecka, J., Joiner, W. M., Pancrazio, J. J., and Sikdar, S. Real-time classification of hand motions using ultrasound imaging of forearm muscles. IEEE Transactions on Biomedical Engineering 63, 8 (Aug 2016), 1687--1698.
[2]
Amma, C., Krings, T., Böer, J., and Schultz, T. Advancing muscle-computer interfaces with high-density electromyography. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, ACM (2015), 929--938.
[3]
Bhuyan, A., Choe, J. W., Lee, B. C., Wygant, I. O., Nikoozadeh, A., Oralkan, Ö., and Khuri-Yakub, B. T. Integrated circuits for volumetric ultrasound imaging with 2-d cmut arrays. IEEE transactions on biomedical circuits and systems 7, 6 (2013), 796--804.
[4]
Bradski, G., et al. The opencv library. Doctor Dobbs Journal 25, 11 (2000), 120--126.
[5]
Castellini, C., and Passig, G. Ultrasound image features of the wrist are linearly related to finger positions. In Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Conference on, IEEE (2011), 2108--2114.
[6]
Castellini, C., Passig, G., and Zarka, E. Using ultrasound images of the forearm to predict finger positions. Neural Systems and Rehabilitation Engineering, IEEE Transactions on 20, 6 (2012), 788--797.
[7]
Castellini, C., and van der Smagt, P. Surface emg in advanced hand prosthetics. Biological cybernetics 100, 1 (2009), 35--47.
[8]
Chen, X., Zheng, Y.-P., Guo, J.-Y., and Shi, J. Sonomyography (smg) control for powered prosthetic hand: a study with normal subjects. Ultrasound in medicine & biology 36, 7 (2010), 1076--1088.
[9]
Cheng, J., Amft, O., Bahle, G., and Lukowicz, P. Designing sensitive wearable capacitive sensors for activity recognition. Sensors Journal, IEEE 13, 10 (2013), 3935--3947.
[10]
Dementyev, A., and Paradiso, J. A. Wristflex: Low-power gesture input with wrist-worn pressure sensors. In Proceedings of the 27th annual ACM symposium on User interface software and technology, ACM (2014), 161--166.
[11]
Farnebäck, G. Two-frame motion estimation based on polynomial expansion. In Image analysis. Springer, 2003, 363--370.
[12]
Spectra Symbol Flex Sensor 4.5. http://www.spectrasymbol.com/wp-content/themes/ spectra/images/datasheets/FlexSensor.pdf.
[13]
GE Vscan. http://www3.gehealthcare.com/en/products/ categories/ultrasound/vscan_portfolio/vscan.
[14]
González, D. S., and Castellini, C. A realistic implementation of ultrasound imaging as a human-machine interface for upper-limb amputees. Frontiers in neurorobotics 7, 17 (2013).
[15]
Guo, J.-Y., Zheng, Y.-P., Huang, Q.-H., Chen, X., and He, J.-F. Comparison of sonomyography and electromyography of forearm muscles in the guided wrist extension. In Medical Devices and Biosensors, 2008. ISSS-MDBS 2008. 5th International Summer School and Symposium on, IEEE (2008), 235--238.
[16]
Guo, J.-Y., Zheng, Y.-P., Kenney, L. P., Bowen, A., Howard, D., and Canderle, J. J. A comparative evaluation of sonomyography, electromyography, force, and wrist angle in a discrete tracking task. Ultrasound in medicine & biology 37, 6 (2011), 884--891.
[17]
Hamid Muhammed, H., and Raghavendra, J. Optomyography (omg): A novel technique for the detection of muscle surface displacement using photoelectric sensors. In 10th International Conference on Bioelectromagnetism, ISBEM 2015, vol. 10 (2015).
[18]
Hettiarachchi, N., Ju, Z., and Liu, H. A new wearable ultrasound muscle activity sensing system for dexterous prosthetic control. In Systems, Man, and Cybernetics (SMC), 2015 IEEE International Conference on, IEEE (2015), 1415--1420.
[19]
Hodges, P., Pengel, L., Herbert, R., and Gandevia, S. Measurement of muscle contraction with ultrasound imaging. Muscle & nerve 27, 6 (2003), 682--692.
[20]
HydroAid Ultrasound Hydrogel Pad. http://www.civco.com/mmi/ultrasound/ accessories/acoustic-standoffs/ Disposable-AquaFlex-Acoustic-Standoff-Pads-610--1322. htm.
[21]
Islam, M. A., Sundaraj, K., Ahmad, R. B., and Ahamed, N. U. Mechanomyogram for muscle function assessment: a review. PloS one 8, 3 (2013), e58902.
[22]
Kerber, F., Löchtefeld, M., Krüger, A., McIntosh, J., McNeill, C., and Fraser, M. Understanding same-side interactions with wrist-worn devices. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction, NordiCHI '16, ACM (NY, NY, USA, 2016), 28:1--28:10.
[23]
Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., and Olivier, P. Digits: freehand 3d interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology, ACM (2012), 167--176.
[24]
Kono, M., Ueki, H., and Umemura, S.-i. Near-infrared finger vein patterns for personal identification. Applied Optics 41, 35 (2002), 7429--7436.
[25]
Korstanje, J.-W. H., Selles, R. W., Stam, H. J., Hovius, S. E., and Bosch, J. G. Development and validation of ultrasound speckle tracking to quantify tendon displacement. Journal of biomechanics 43, 7 (2010), 1373--1379.
[26]
Loram, I. D., Maganaris, C. N., and Lakie, M. Use of ultrasound to make noninvasive in vivo measurement of continuous changes in human muscle contractile length. Journal of applied physiology 100, 4 (2006), 1311--1323.
[27]
Mujibiya, A., Cao, X., Tan, D. S., Morris, D., Patel, S. N., and Rekimoto, J. The sound of touch: On-body touch and gesture sensing based on transdermal ultrasound propagation. In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces, ITS '13, ACM (NY, NY, USA, 2013), 189--198.
[28]
Orizio, C., and Gobbo, M. Mechanomyography. Wiley Encyclopedia of Biomedical Engineering (2006).
[29]
Ortenzi, V., Tarantino, S., Castellini, C., and Cipriani, C. Ultrasound imaging for hand prosthesis control: a comparative study of features and classification methods. In Rehabilitation Robotics (ICORR), 2015 IEEE International Conference on, IEEE (2015), 1--6.
[30]
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. Scikit-learn: Machine learning in python. The Journal of Machine Learning Research 12 (2011), 2825--2830.
[31]
Rautaray, S. S., and Agrawal, A. Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review 43, 1 (2015), 1--54.
[32]
Rekimoto, J. Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In Wearable Computers, 2001. Proceedings. Fifth International Symposium on, IEEE (2001), 21--27.
[33]
Rossi, M., Benatti, S., Farella, E., and Benini, L. Hybrid emg classifier based on hmm and svm for hand gesture recognition in prosthetics. In Industrial Technology (ICIT), 2015 IEEE International Conference on, IEEE (2015), 1700--1705.
[34]
Shi, J., Hu, S.-X., Liu, Z., Guo, J.-Y., Zhou, Y.-J., and Zheng, Y.-P. Recognition of finger flexion from ultrasound image with optical flow: A preliminary study. In Biomedical Engineering and Computer Science (ICBECS), 2010 International Conference on, IEEE (2010), 1--4.
[35]
Sikdar, S., Rangwala, H., Eastlake, E. B., Hunt, I. A., Nelson, A. J., Devanathan, J., Shin, A., and Pancrazio, J. J. Novel method for predicting dexterous individual finger movements by imaging muscle activity using a wearable ultrasonic system. Neural Systems and Rehabilitation Engineering, IEEE Transactions on 22, 1 (2014), 69--76.
[36]
Smith, R. J., Tenore, F., Huberdeau, D., Cummings, R. E., and Thakor, N. V. Continuous decoding of finger position from surface emg signals for the control of powered prostheses. In Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30th Annual International Conference of the IEEE, IEEE (2008), 197--200.
[37]
Tenore, F. V., Ramos, A., Fahmy, A., Acharya, S., Etienne-Cummings, R., and Thakor, N. V. Decoding of individuated finger movements using surface electromyography. Biomedical Engineering, IEEE Transactions on 56, 5 (2009), 1427--1434.
[38]
Wojtczak, J., and Bonadonna, P. Pocket mobile smartphone system for the point-of-care submandibular ultrasonography. The American journal of emergency medicine 31, 3 (2013), 573--577.
[39]
Wolf, M. T., Assad, C., Stoica, A., You, K., Jethani, H., Vernacchia, M. T., Fromm, J., and Iwashita, Y. Decoding static and dynamic arm and hand gestures from the jpl biosleeve. In Aerospace Conference, 2013 IEEE, IEEE (2013), 1--9.
[40]
Yamakawa, S., and Nojima, T. A proposal for a mmg-based hand gesture recognition method. In Adjunct proceedings of the 25th annual ACM symposium on User interface software and technology, ACM (2012), 89--90.
[41]
Yang, Y., Tian, H., Yan, B., Sun, H., Wu, C., Shu, Y., Wang, L.-G., and Ren, T.-L. A flexible piezoelectric micromachined ultrasound transducer. RSC Advances 3, 47 (2013), 24900--24905.
[42]
You, K.-J., Rhee, K.-W., and Shin, H.-C. Finger motion decoding using emg signals corresponding various arm postures. Experimental neurobiology 19, 1 (2010), 54--61.
[43]
Zhang, Y., Xiao, R., and Harrison, C. Advancing hand gesture recognition with high resolution electrical impedance tomography. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST '16, ACM (NY, NY, USA, 2016), 843--850.
[44]
Zheng, Y.-P., Chan, M., Shi, J., Chen, X., and Huang, Q.-H. Sonomyography: Monitoring morphological changes of forearm muscles in actions with the feasibility for the control of powered prosthesis. Medical engineering & physics 28, 5 (2006), 405--415.

Cited By

View all
  • (2025)Grasping control using 2D and 3D visual integration for robotic prosthetic handIntelligent Service Robotics10.1007/s11370-024-00572-z18:1(185-194)Online publication date: 8-Jan-2025
  • (2024)EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a WristbandProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642910(1-21)Online publication date: 11-May-2024
  • (2024)EITPose: Wearable and Practical Electrical Impedance Tomography for Continuous Hand Pose EstimationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642663(1-10)Online publication date: 11-May-2024
  • Show More Cited By

Index Terms

  1. EchoFlex: Hand Gesture Recognition using Ultrasound Imaging

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. computer vision
    2. gesture recognition
    3. interactive ultrasound imaging
    4. machine learning

    Qualifiers

    • Research-article

    Funding Sources

    • EPSRC

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)261
    • Downloads (Last 6 weeks)28
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Grasping control using 2D and 3D visual integration for robotic prosthetic handIntelligent Service Robotics10.1007/s11370-024-00572-z18:1(185-194)Online publication date: 8-Jan-2025
    • (2024)EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a WristbandProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642910(1-21)Online publication date: 11-May-2024
    • (2024)EITPose: Wearable and Practical Electrical Impedance Tomography for Continuous Hand Pose EstimationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642663(1-10)Online publication date: 11-May-2024
    • (2024)MAF: Exploring Mobile Acoustic Field for Hand-to-Face Gesture InteractionsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642437(1-20)Online publication date: 11-May-2024
    • (2024)Hand Gesture Classification Based on Forearm Ultrasound Video Snippets Using 3D Convolutional Neural Networks2024 IEEE Ultrasonics, Ferroelectrics, and Frequency Control Joint Symposium (UFFC-JS)10.1109/UFFC-JS60046.2024.10794148(1-4)Online publication date: 22-Sep-2024
    • (2024)Towards Natural Multi-DoF Prosthetic Control with Distributed Ultrasound2024 IEEE Ultrasonics, Ferroelectrics, and Frequency Control Joint Symposium (UFFC-JS)10.1109/UFFC-JS60046.2024.10793502(1-6)Online publication date: 22-Sep-2024
    • (2024)Ultrasound as a Neurorobotic Interface: A ReviewIEEE Transactions on Systems, Man, and Cybernetics: Systems10.1109/TSMC.2024.335896054:6(3534-3546)Online publication date: Jun-2024
    • (2024)EchoGest: Soft Ultrasonic Waveguides Based Sensing Skin for Subject-Independent Hand Gesture RecognitionIEEE Transactions on Neural Systems and Rehabilitation Engineering10.1109/TNSRE.2024.341413632(2366-2375)Online publication date: 2024
    • (2024)DANN-Repositing Strategy for Zero Retraining Long-Term Hand Gesture Recognition Using Wearable A-Mode UltrasoundIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.346093973(1-11)Online publication date: 2024
    • (2024)High Performance Wearable Ultrasound as a Human-Machine Interface for Wrist and Hand Kinematic TrackingIEEE Transactions on Biomedical Engineering10.1109/TBME.2023.330795271:2(484-493)Online publication date: Feb-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media