skip to main content
research-article
Public Access

Force-Aware Interface via Electromyography for Natural VR/AR Interaction

Published: 30 November 2022 Publication History

Abstract

While tremendous advances in visual and auditory realism have been made for virtual and augmented reality (VR/AR), introducing a plausible sense of physicality into the virtual world remains challenging. Closing the gap between real-world physicality and immersive virtual experience requires a closed interaction loop: applying user-exerted physical forces to the virtual environment and generating haptic sensations back to the users. However, existing VR/AR solutions either completely ignore the force inputs from the users or rely on obtrusive sensing devices that compromise user experience.
By identifying users' muscle activation patterns while engaging in VR/AR, we design a learning-based neural interface for natural and intuitive force inputs. Specifically, we show that lightweight electromyography sensors, resting non-invasively on users' forearm skin, inform and establish a robust understanding of their complex hand activities. Fuelled by a neural-network-based model, our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. Through an interactive psychophysical study, we show that human perception of virtual objects' physical properties, such as stiffness, can be significantly enhanced by our interface. We further demonstrate that our interface enables ubiquitous control via finger tapping. Ultimately, we envision our findings to push forward research towards more realistic physicality in future VR/AR.

Supplemental Material

MP4 File
presentation

References

[1]
Heba Abdelnasser, Khaled Harras, and Moustafa Youssef. 2018. A ubiquitous WiFi-based fine-grained gesture recognition system. IEEE Transactions on Mobile Computing 18, 11 (2018), 2474--2487.
[2]
Heba Abdelnasser, Moustafa Youssef, and Khaled A Harras. 2015. Wigest: A ubiquitous wifi-based gesture recognition system. In 2015 IEEE conference on computer communications (INFOCOM). IEEE, 1472--1480.
[3]
Md Rezwanul Ahsan, Muhammad I Ibrahimy, Othman O Khalifa, et al. 2009. EMG signal classification for human computer interaction: a review. European Journal of Scientific Research 33, 3 (2009), 480--501.
[4]
Deni Andrean, Daniel S Pamungkas, and Sumantri Kurniawan Risandriya. 2019. Controlling Robot Hand Using FFT as Input to the NN Algorithm. In Journal of Physics: Conference Series, Vol. 1230. IOP Publishing, 012030.
[5]
Gopala K Anumanchipalli, Josh Chartier, and Edward F Chang. 2019. Speech synthesis from neural decoding of spoken sentences. Nature 568, 7753 (2019), 493--498.
[6]
Manfredo Atzori and Henning Müller. 2015. The Ninapro database: a resource for sEMG naturally controlled robotic hand prosthetics. In 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 7151--7154.
[7]
Tara Baldacchino, William R Jacobs, Sean R Anderson, Keith Worden, and Jennifer Rowson. 2018. Simultaneous force regression and movement classification of fingers via surface EMG within a unified Bayesian framework. Frontiers in bioengineering and biotechnology 6 (2018), 13.
[8]
Berj Bardizbanian, Jennifer Keating, Xinming Huang, and Edward A Clancy. 2020a. Estimating Individual and Combined Fingertip Forces From Forearm EMG During Constant-Pose, Force-Varying Tasks. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 3134--3137.
[9]
Berj Bardizbanian, Ziling Zhu, Jianan Li, Xinming Huang, Chenyun Dai, Carlos Martinez-Luna, Benjamin E McDonald, Todd R Farrell, and Edward A Clancy. 2020b. Efficiently training two-DoF hand-wrist EMG-force models. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 369--373.
[10]
Elisabeth Baumgartner, Christiane B. Wiebel, and Karl R. Gegenfurtner. 2013. Visual and haptic representations of material properties. Multisensory Research 26, 5 (2013), 429--455.
[11]
Vincent Becker, Pietro Oldrati, Liliana Barrios, and Gábor Sörös. 2018. Touchsense: classifying finger touches and measuring their force with an electromyography armband. In Proceedings of the 2018 ACM International Symposium on Wearable Computers. 1--8.
[12]
Hrvoje Benko, Christian Holz, Mike Sinclair, and Eyal Ofek. 2016. Normaltouch and texturetouch: High-fidelity 3d haptic shape rendering on handheld virtual reality controllers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 717--728.
[13]
Joanna Bergström and Kasper Hornbæk. 2019. Human-Computer Interaction on the Skin. ACM Computing Surveys (CSUR) 52, 4 (2019), 1--14.
[14]
Reza Boostani and Mohammad Hassan Moradi. 2003. Evaluation of the forearm EMG signal features for the control of a prosthetic hand. Physiological measurement 24, 2 (2003), 309.
[15]
Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 2019. 3d hand shape and pose from images in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10843--10852.
[16]
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7291--7299.
[17]
Claudio Castellini and Risto Koiva. 2012. Using surface electromyography to predict single finger forces. In 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob). IEEE, 1266--1272.
[18]
Claudio Castellini and Patrick Van Der Smagt. 2009. Surface EMG in advanced hand prosthetics. Biological cybernetics 100, 1 (2009), 35--47.
[19]
Lele Chen, Chen Cao, Fernando De la Torre, Jason Saragih, Chenliang Xu, and Yaser Sheikh. 2021. High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13059--13069.
[20]
Minsu Cho, Younggeol Cho, and Kyung-Soo Kim. 2022. Training Strategy and sEMG Sensor Positioning for Finger Force Estimation at Various Elbow Angles. International Journal of Control, Automation and Systems 20, 5 (2022), 1621--1631.
[21]
Inrak Choi, Heather Culbertson, Mark R Miller, Alex Olwal, and Sean Follmer. 2017. Grabity: A wearable haptic interface for simulating weight and grasping in virtual reality. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 119--130.
[22]
Inrak Choi, Elliot W Hawkes, David L Christensen, Christopher J Ploch, and Sean Follmer. 2016. Wolverine: A wearable haptic interface for grasping in virtual reality. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 986--993.
[23]
Inrak Choi, Eyal Ofek, Hrvoje Benko, Mike Sinclair, and Christian Holz. 2018. Claw: A multifunctional handheld haptic controller for grasping, touching, and triggering in virtual reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--13.
[24]
Hang Chu, Shugao Ma, Fernando De la Torre, Sanja Fidler, and Yaser Sheikh. 2020. Expressive telepresence via modular codec avatars. In European Conference on Computer Vision. Springer, 330--345.
[25]
Wang Dangxiao, Guo Yuan, Liu Shiyi, Yuru Zhang, Xu Weiliang, and Xiao Jing. 2019. Haptic display for virtual reality: progress and challenges. Virtual Reality & Intelligent Hardware 1, 2 (2019), 136--162.
[26]
Yu Du, Yongkang Wong, Wenguang Jin, Wentao Wei, Yu Hu, Mohan S Kankanhalli, and Weidong Geng. 2017. Semi-Supervised Learning for Surface EMG-based Gesture Recognition. In IJCAI. 1624--1630.
[27]
Kiana Ehsani, Shubham Tulsiani, Saurabh Gupta, Ali Farhadi, and Abhinav Gupta. 2020a. Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[28]
Kiana Ehsani, Shubham Tulsiani, Saurabh Gupta, Ali Farhadi, and Abhinav Gupta. 2020b. Use the force, luke! learning to predict physical forces by simulating effects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 224--233.
[29]
Marc O Ernst and Martin S Banks. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 6870 (2002), 429--433.
[30]
Navid Fallahinia and Stephen Mascaro. 2021a. Real-Time Tactile Grasp Force Sensing Using Fingernail Imaging via Deep Neural Networks. arXiv preprint arXiv:2109.15231 (2021).
[31]
Navid Fallahinia and Stephen A Mascaro. 2020. Comparison of Constrained and Unconstrained Human Grasp Forces Using Fingernail Imaging and Visual Servoing. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2668--2674.
[32]
Navid Fallahinia and Stephen A Mascaro. 2021b. Feasibility study of force measurement for multi-digit unconstrained grasping via fingernail imaging and visual servoing. ASME Letters in Dynamic Systems and Control 1, 2 (2021).
[33]
Yinfeng Fang, Dalin Zhou, Kairu Li, Zhaojie Ju, and Honghai Liu. 2019. Attribute-driven granular model for EMG-based pinch and fingertip force grand recognition. IEEE transactions on cybernetics 51, 2 (2019), 789--800.
[34]
Dario Farina and Aleš Holobar. 2016. Characterization of human motor units from surface EMG decomposition. Proc. IEEE 104, 2 (2016), 353--373.
[35]
Sharlene N Flesher, John E Downey, Jeffrey M Weiss, Christopher L Hughes, Angelica J Herrera, Elizabeth C Tyler-Kabara, Michael L Boninger, Jennifer L Collinger, and Robert A Gaunt. 2021. A brain-computer interface that evokes tactile sensations improves robotic arm control. Science 372, 6544 (2021), 831--836.
[36]
Alycia Gailey, Panagiotis Artemiadis, and Marco Santello. 2017. Proof of concept of an online EMG-based decoding of hand postures and individual digit forces for prosthetic hand control. Frontiers in neurology 8 (2017), 7.
[37]
Eric J Gonzalez, Eyal Ofek, Mar Gonzalez-Franco, and Mike Sinclair. 2021. X-Rings: A Hand-mounted 360 Shape Display for Grasping in Virtual Reality. In The 34th Annual ACM Symposium on User Interface Software and Technology. 732--742.
[38]
Thomas Grieve, Lucas Lincoln, Yu Sun, John M Hollerbach, and Stephen A Mascaro. 2010. 3d force prediction using fingernail imaging with automated calibration. In 2010 IEEE Haptics Symposium. IEEE, 113--120.
[39]
Thomas R Grieve, John M Hollerbach, and Stephen A Mascaro. 2015a. 3-d fingertip touch force prediction using fingernail imaging with automated calibration. IEEE Transactions on Robotics 31, 5 (2015), 1116--1129.
[40]
Thomas R Grieve, John M Hollerbach, and Stephen A Mascaro. 2015b. Optimizing fingernail imaging calibration for 3d force magnitude prediction. IEEE transactions on haptics 9, 1 (2015), 69--79.
[41]
Paras Gulati, Qin Hu, and S Farokh Atashzar. 2021. Toward Deep Generalization of Peripheral EMG-Based Human-Robot Interfacing: A Hybrid Explainable Solution for NeuroRobotic Systems. IEEE Robotics and Automation Letters 6, 2 (2021), 2650--2657.
[42]
Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, et al. 2020. MEgATrack: monochrome egocentric articulated hand-tracking for virtual reality. ACM Transactions on Graphics (TOG) 39, 4 (2020), 87--1.
[43]
Chris Harrison and Scott E Hudson. 2008. Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces. In Proceedings of the 21st annual ACM symposium on User interface software and technology. 205--208.
[44]
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black. 2021. Stochastic Scene-Aware Motion Prediction. In Proceedings of the International Conference on Computer Vision 2021.
[45]
Mamoru Hirota, Ayumu Tsuboi, Masayuki Yokoyama, and Masao Yanagisawa. 2018. Gesture recognition of air-tapping and its application to character input in VR space. In SIGGRAPH Asia 2018 Posters. 1--2.
[46]
Leigh R Hochberg, Daniel Bacher, Beata Jarosiewicz, Nicolas Y Masse, John D Simeral, Joern Vogel, Sami Haddadin, Jie Liu, Sydney S Cash, Patrick Van Der Smagt, et al. 2012. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485, 7398 (2012), 372--375.
[47]
At L Hof. 1991. Errors in frequency parameters of EMG power spectra. IEEE transactions on biomedical engineering 38, 11 (1991), 1077--1088.
[48]
Ruochen Hu, Xiang Chen, Haotian Zhang, Xu Zhang, and Xun Chen. 2022. A Novel Myoelectric Control Scheme Supporting Synchronous Gesture Recognition and Muscle Force Estimation. IEEE Transactions on Neural Systems and Rehabilitation Engineering (2022).
[49]
Wonjun Hwang and Soo-Chul Lim. 2017. Inferring interaction force from visual information without using physical force sensors. Sensors 17, 11 (2017), 2455.
[50]
James N Ingram, Konrad P Körding, Ian S Howard, and Daniel M Wolpert. 2008. The statistics of natural hand movements. Experimental brain research 188, 2 (2008), 223--236.
[51]
Sumit Jain, Yuting Ye, and C Karen Liu. 2009. Optimization-based interactive motion synthesis. ACM Transactions on Graphics (TOG) 28, 1 (2009), 1--12.
[52]
Haider Ali Javaid, Mohsin Islam Tiwana, Ahmed Alsanad, Javaid Iqbal, Muhammad Tanveer Riaz, Saeed Ahmad, and Faisal Abdulaziz Almisned. 2021. Classification of Hand Movements Using MYO Armband on an Embedded Platform. Electronics 10, 11 (2021), 1322.
[53]
Yong-Un Jo and Do-Chang Oh. 2020. REAL-TIME HAND GESTURE CLASSIFICATION USING CRNN WITH SCALE AVERAGE WAVELET TRANSFORM. Journal of Mechanics in Medicine and Biology 20, 10 (2020), 2040028.
[54]
Hanbyul Joo, Tomas Simon, and Yaser Sheikh. 2018. Total capture: A 3d deformation model for tracking faces, hands, and bodies. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8320--8329.
[55]
Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. 2018. End-to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7122--7131.
[56]
David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 167--176.
[57]
Jonghyun Kim, Youngmo Jeong, Michael Stengel, Kaan Akşit, Rachel Albert, Ben Boudaoud, Trey Greer, Joohwan Kim, Ward Lopes, Zander Majercik, et al. 2019. Foveated AR: dynamically-foveated augmented reality display. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--15.
[58]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings.
[59]
Babis Koniaris, Ivan Huerta, Maggie Kosek, Karen Darragh, Charles Malleson, Joanna Jamrozy, Nick Swafford, Jose Guitian, Bochang Moon, Ali Israr, et al. 2016. Iridium: immersive rendered interactive deep media. In ACM SIGGRAPH 2016 VR Village. 1--2.
[60]
Eike Langbehn, Frank Steinicke, Markus Lappe, Gregory F Welch, and Gerd Bruder. 2018. In the blink of an eye: leveraging blink-induced suppression for imperceptible position and orientation redirection in virtual reality. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--11.
[61]
Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--19.
[62]
Ming C Lin and Miguel Otaduy. 2008. Haptic rendering: foundations, algorithms, and applications. CRC Press.
[63]
Pu Liu, Donald R Brown, Edward A Clancy, Francois Martel, and Denis Rancourt. 2013. EMG-force estimation for multiple fingers. In 2013 IEEE Signal Processing in Medicine and Biology Symposium (SPMB). IEEE, 1--6.
[64]
Yilin Liu, Shijia Zhang, and Mahanth Gowda. 2021. NeuroPose: 3D Hand Pose Tracking using EMG Wearables. In Proceedings of the Web Conference 2021. 1471--1482.
[65]
Yiyue Luo, Yunzhu Li, Pratyusha Sharma, Wan Shou, Kui Wu, Michael Foshey, Beichen Li, Tomás Palacios, Antonio Torralba, and Wojciech Matusik. 2021a. Learning human-environment interactions using conformal tactile textiles. Nature Electronics 4, 3 (2021), 193--201.
[66]
Yiyue Luo, Kui Wu, Tomás Palacios, and Wojciech Matusik. 2021b. KnitUI: Fabricating interactive and sensing textiles with machine knitting. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--12.
[67]
Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang, Yuecheng Li, Fernando De La Torre, and Yaser Sheikh. 2021. Pixel Codec Avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 64--73.
[68]
Miles Macklin, Matthias Müller, and Nuttapong Chentanez. 2016. XPBD: position-based simulation of compliant constrained dynamics. In Proceedings of the 9th International Conference on Motion in Games. 49--54.
[69]
He Mao, Peng Fang, and Guanglin Li. 2021. Simultaneous estimation of multi-finger forces by surface electromyography and accelerometry signals. Biomedical Signal Processing and Control 70 (2021), 103005.
[70]
Itzel Jared Rodríguez Martínez, Andrea Mannini, Francesco Clemente, and Christian Cipriani. 2020. Online grasp force estimation from the transient EMG. IEEE Transactions on Neural Systems and Rehabilitation Engineering 28, 10 (2020), 2333--2341.
[71]
Itzel Jared Rodriguez Martinez, Andrea Mannini, Francesco Clemente, Angelo Maria Sabatini, and Christian Cipriani. 2020. Grasp force estimation from the transient EMG using high-density surface recordings. Journal of Neural Engineering 17, 1 (2020), 016052.
[72]
W Melzer, E Rios, and MF Schneider. 1984. Time course of calcium release and removal in skeletal muscle fibers. Biophysical journal 45, 3 (1984), 637--641.
[73]
Matthias Müller, Bruno Heidelberger, Marcus Hennix, and John Ratcliff. 2007. Position based dynamics. Journal of Visual Communication and Image Representation 18, 2 (2007), 109--118.
[74]
Jose María Méndez and Lidia Martínez. 2021. Virtual Method Studio. Retrieved 2021-12-01 from http://obi.virtualmethodstudio.com/index.html
[75]
Alejandro Newell, Kaiyu Yang, and Jia Deng. 2016. Stacked hourglass networks for human pose estimation. In European conference on computer vision. Springer, 483--499.
[76]
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016).
[77]
Yun Suen Pai, Tilman Dingler, and Kai Kunze. 2019. Assessing hands-free interactions for VR using eye gaze and electromyography. Virtual Reality 23, 2 (2019), 119--131.
[78]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019), 8026--8037.
[79]
Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1--12.
[80]
Tu-Hoa Pham, Abderrahmane Kheddar, Ammar Qammaz, and Antonis A Argyros. 2015. Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2810--2819.
[81]
Tu-Hoa Pham, Nikolaos Kyriazis, Antonis A Argyros, and Abderrahmane Kheddar. 2017. Hand-object contact force estimation from markerless visual tracking. IEEE transactions on pattern analysis and machine intelligence 40, 12 (2017), 2883--2896.
[82]
Angkoon Phinyomark and Erik Scheme. 2018. EMG pattern recognition in the era of big data and deep learning. Big Data and Cognitive Computing 2, 3 (2018), 21.
[83]
S. Pizzolato, L. Tagliapietra, M. Cognolato, M. Reggiani, H. Müller, and M. Atzori. 2017. Comparison of six electromyography acquisition setups on hand movement classification tasks. PLoS One 12, 10 (2017), e0186132.
[84]
Wen Qi, Hang Su, Junhao Zhang, Rong Song, Giancarlo Ferrigno, Elena De Momi, and Andrea Aliverti. 2021. Active Learning Strategy of Finger Flexion Tracking using sEMG for Robot Hand Control. In 2021 6th IEEE International Conference on Advanced Robotics and Mechatronics (ICARM). IEEE, 753--758.
[85]
Elahe Rahimian, Soheil Zabihi, Amir Asif, S Farokh Atashzar, and Arash Mohammadi. 2021. Few-Shot Learning for Decoding Surface Electromyography for Hand Gesture Recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1300--1304.
[86]
Javier Romero, Dimitrios Tzionas, and Michael J Black. 2017. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics (ToG) 36, 6 (2017), 1--17.
[87]
Farshid Salemi Parizi, Wolf Kienzle, Eric Whitmire, Aakar Gupta, and Hrvoje Benko. 2021. RotoWrist: Continuous Infrared Wrist Angle Tracking using a Wristband. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology. 1--11.
[88]
Christopher Spiewak, M Islam, A Zaman, and Mohammad Habibur Rahman. 2018. A comprehensive study on EMG feature extraction and classifiers. Open Access Journal of Biomedical Engineering and Biosciences 1, 1 (2018), 1--10.
[89]
Shriya S Srinivasan, Samantha Gutierrez-Arango, Ashley Chia-En Teng, Erica Israel, Hyungeun Song, Zachary Keith Bailey, Matthew J Carty, Lisa E Freed, and Hugh M Herr. 2021. Neural interfacing architecture enables enhanced motor control and residual limb functionality postamputation. Proceedings of the National Academy of Sciences 118, 9 (2021).
[90]
Lee Stearns, Uran Oh, Leah Findlater, and Jon E Froehlich. 2018. TouchCam: Realtime Recognition of Location-Specific On-Body Gestures to Support Users with Visual Impairments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1--23.
[91]
Qi Sun, Anjul Patney, Li-Yi Wei, Omer Shapira, Jingwan Lu, Paul Asente, Suwen Zhu, Morgan McGuire, David Luebke, and Arie Kaufman. 2018. Towards virtual reality infinite walking: dynamic saccadic redirection. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--13.
[92]
Tianyun Sun, Qin Hu, Jacqueline Libby, and S Farokh Atashzar. 2022. Deep Heterogeneous Dilation of LSTM for Transient-phase Gesture Prediction through High-density Electromyography: Towards Application in Neurorobotics. IEEE Robotics and Automation Letters (2022).
[93]
Yu Sun, John M Hollerbach, and Stephen A Mascaro. 2008. Predicting fingertip forces by imaging coloration changes in the fingernail and surrounding skin. IEEE Transactions on Biomedical Engineering 55, 10 (2008), 2363--2371.
[94]
Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojciech Matusik. 2019. Learning the signatures of the human grasp using a scalable tactile glove. Nature 569, 7758 (2019), 698--702.
[95]
Paul M. Torrens and Simin Gu. 2021. Real-Time Experiential Geosimulation in Virtual Reality with Immersion-Emission. In Proceedings of the 4th ACM SIGSPATIAL International Workshop on GeoSpatial Simulation (Beijing, China) (GeoSim '21). Association for Computing Machinery, New York, NY, USA, 19--28.
[96]
Ayumu Tsuboi, Mamoru Hirota, Junki Sato, Masayuki Yokoyama, and Masao Yanagisawa. 2017. A proposal for wearable controller device and finger gesture recognition using surface electromyography. In SIGGRAPH Asia 2017 Posters. 1--2.
[97]
Mickeal Verschoor, Dan Casas, and Miguel A Otaduy. 2020. Tactile rendering based on skin stress optimization. ACM Transactions on Graphics (TOG) 39, 4 (2020), 90--1.
[98]
Christian Von Hardenberg and François Bérard. 2001. Bare-hand human-computer interaction. In Proceedings of the 2001 workshop on Perceptive user interfaces. 1--8.
[99]
Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao. 2018. Dense 3d regression for hand pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5147--5156.
[100]
Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 851--860.
[101]
Eric Whitmire, Hrvoje Benko, Christian Holz, Eyal Ofek, and Mike Sinclair. 2018. Haptic revolver: Touch, shear, texture, and shape rendering on a reconfigurable virtual reality controller. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--12.
[102]
Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V Shenoy. 2021. High-performance brain-to-text communication via handwriting. Nature 593, 7858 (2021), 249--254.
[103]
Changcheng Wu, Qingqing Cao, Fei Fei, Dehua Yang, Baoguo Xu, Hong Zeng, and Aiguo Song. 2020. sEMG Feature Optimization Strategy for Finger Grip Force Estimation. In International Conference on Intelligent Robotics and Applications. Springer, 184--194.
[104]
Changcheng Wu, Qingqing Cao, Fei Fei, Dehua Yang, Baoguo Xu, Guanglie Zhang, Hong Zeng, and Aiguo Song. 2021. Optimal strategy of sEMG feature and measurement position for grasp force estimation. PloS one 16, 3 (2021), e0247883.
[105]
Feng Xu, Yang Zheng, and Xiaogang Hu. 2020b. Real-time finger force prediction via parallel convolutional neural networks: a preliminary study. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 3126--3129.
[106]
Xuhai Xu, Haitian Shi, Xin Yi, WenJia Liu, Yukang Yan, Yuanchun Shi, Alex Mariakakis, Jennifer Mankoff, and Anind K Dey. 2020a. Earbuddy: Enabling on-face interaction via wireless earbuds. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--14.
[107]
Zhiqi Yin, Zeshi Yang, Michiel Van De Panne, and KangKang Yin. 2021. Discovering diverse athletic jumping strategies. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1--17.
[108]
Shigeo Yoshida, Yuqian Sun, and Hideaki Kuzuoka. 2020. Pocopo: Handheld pin-based shape display for haptic rendering in virtual reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--13.
[109]
Cheng Zhang, Qiuyue Xue, Anandghan Waghmare, Ruichen Meng, Sumeet Jain, Yizeng Han, Xinyu Li, Kenneth Cunefare, Thomas Ploetz, Thad Starner, et al. 2018. Finger-Ping: Recognizing fine-grained hand poses using active acoustic on-body sensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--10.
[110]
Qin Zhang, Li Fang, Qining Zhang, and Caihua Xiong. 2022. Simultaneous estimation of joint angle and interaction force towards sEMG-driven human-robot interaction during constrained tasks. Neurocomputing 484 (2022), 38--45.
[111]
Yihui Zhao, Zhiqiang Zhang, Zhenhong Li, Zhixin Yang, Abbas A Dehghani-Sanij, and Shengquan Xie. 2020. An EMG-driven musculoskeletal model for estimating continuous wrist motion. IEEE Transactions on Neural Systems and Rehabilitation Engineering 28, 12 (2020), 3113--3120.
[112]
Yixin Zhu, Chenfanfu Jiang, Yibiao Zhao, Demetri Terzopoulos, and Song-Chun Zhu. 2016. Inferring forces and learning human utilities from videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3823--3833.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 41, Issue 6
December 2022
1428 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3550454
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 November 2022
Published in TOG Volume 41, Issue 6

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. electromyography
  2. force-aware neural interface
  3. haptic perception
  4. machine learning

Qualifiers

  • Research-article

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)486
  • Downloads (Last 6 weeks)62
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Study on Finger Gesture Interface Using One-Channel EMGIEEE Access10.1109/ACCESS.2025.352768413(9606-9614)Online publication date: 2025
  • (2024)Realidad Virtual y Aumentada en la Educación: Potencial y Aplicaciones PrácticasInnDev10.69583/inndev.v3n2.2024.1333:2(1-16)Online publication date: 31-Aug-2024
  • (2024)Using Immersive Virtual Reality to Study Road-Crossing Sustainability in Fleeting Moments of Space and TimeSustainability10.3390/su1603132716:3(1327)Online publication date: 4-Feb-2024
  • (2024)Prediction and Fitting of Nonlinear Dynamic Grip Force of the Human Upper Limb Based on Surface Electromyographic SignalsSensors10.3390/s2501001325:1(13)Online publication date: 24-Dec-2024
  • (2024)Ten Traps for Non-Representational Theory in Human GeographyGeographies10.3390/geographies40200164:2(253-286)Online publication date: 18-Apr-2024
  • (2024)Proxy Asset Generation for Cloth Simulation in GamesACM Transactions on Graphics10.1145/365817743:4(1-12)Online publication date: 19-Jul-2024
  • (2024)Media2Face: Co-speech Facial Animation Generation With Multi-Modality GuidanceACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657413(1-13)Online publication date: 13-Jul-2024
  • (2024)VRMM: A Volumetric Relightable Morphable Head ModelACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657406(1-11)Online publication date: 13-Jul-2024
  • (2024)Unveiling Privacy Vulnerabilities: Investigating the Role of Structure in Graph DataProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3672013(4059-4070)Online publication date: 25-Aug-2024
  • (2024)Finger Tracking Using Wrist-Worn EMG SensorsIEEE Transactions on Mobile Computing10.1109/TMC.2024.343901823:12(14099-14110)Online publication date: Dec-2024
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media