Abstract
Vivid haptic feedback remains a challenge in truly immersive virtual reality and augmented reality. As the tactile sensitivity among different individuals and different parts of the hand within a person varies widely, a universal method to encode tactile information into faithful feedback in hands according to sensitivity features is urgently needed. In addition, existing haptic interfaces worn on the hand are usually bulky, rigid and tethered by cables, which is a hurdle for accurately and naturally providing haptic feedbacks. Here we report a soft, ultrathin, miniaturized and wireless electrotactile system (WeTac) that delivers current through the hand to induce tactile sensations as the skin-integrated haptic interface. With a relatively high pixel density over the whole hand area, the WeTac can provide tactile stimulation and measure the sensation thresholds of users in a flexible way. By mapping the thresholds for different electrical parameters, personalized threshold data can be acquired to reproduce virtual touching sensations on the hand with optimized stimulation intensity and avoid causing pain. With an accurate control of sensation level, temporal and spatial perception, it allows providing personalized feedback when users interact with virtual objects. This technique is promising for a more vivid touching experience in the virtual world and in human–machine interactions.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
All data needed to evaluate the conclusions in the papers are present in the paper and/or the Supplementary Information. Volunteer threshold data and current output data are available from https://github.com/zjkhurry/WeTac. The data are available via Zenodo at https://doi.org/10.5281/zenodo.6919395 (ref. 41).
Code availability
The code that supports the Bluetooth-compatible Android application for communicating by mobile device, and VR interaction with WeTac system within this paper and other findings of this study are available from https://github.com/zjkhurry/WeTac. The code is available via Zenodo at https://doi.org/10.5281/zenodo.6919395 (ref. 41).
References
Wang, B. et al. Flexible and stretchable metal oxide nanofiber networks for multimodal and monolithically integrated wearable electronics. Nat. Commun. 11, 2405 (2020).
Liu, Y. et al. Epidermal electronics for respiration monitoring via thermo-sensitive measuring. Mater. Today Phys. 13, 100199 (2020).
Song, E. et al. Miniaturized electromechanical devices for the characterization of the biomechanics of deep tissue. Nat. Biomed. Eng. 5, 759–771 (2021).
Liu, Y. et al. Electronic skin from high-throughput fabrication of intrinsically stretchable lead zirconate titanate elastomer. Research 2020, 1085417 (2020).
You, I. et al. Artificial multimodal receptors based on ion relaxation dynamics. Science 370, 961–965 (2020).
Zhu, M. et al. Haptic-feedback smart glove as a creative human–machine interface (HMI) for virtual/augmented reality applications. Sci. Adv. 6, eaaz8693 (2020).
Huang, Z. et al. Three-dimensional integrated stretchable electronics. Nat. Electron. 1, 473–480 (2018).
Liu, Y. et al. Electronic skin as wireless human-machine interfaces for robotic VR. Sci. Adv. 8, eabl6700 (2022).
Li, D. et al. Miniaturization of mechanical actuators in skin-integrated electronics for haptic interfaces. Microsyst. Nanoeng. 7, 1–9 (2021).
Jayant, C., Acuario, C., Johnson, W. A., Hollier, J. & Ladner, R. E. VBraille: Haptic Braille Perception using a Touch-screen and Vibration on Mobile Phones. in ASSETS’10 - Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility (ed. Barreto, A.) 2–3 (Association for Computing Machinery, 2010).
Gu, G. et al. A soft neuroprosthetic hand providing simultaneous myoelectric control and tactile feedback. Nat. Biomed. Eng. https://doi.org/10.1038/s41551-021-00767-0 (2021).
Akhtar, A., Sombeck, J., Boyce, B. & Bretl, T. Controlling sensation intensity for electrotactile stimulation in human-machine interfaces. Sci. Robot. 3, eaap9770 (2018).
Yu, X. et al. Skin-integrated wireless haptic interfaces for virtual and augmented reality. Nature 575, 473–479 (2019).
Perret, J. & Poorten, E. Vander. Review Paper : Commercial Haptic Gloves. in The Industrial Track of EuroVR 2018 : Proceedings of the 15th Annual EuroVR Conference (eds. Helin, K., Poyade, M., D’Cruz, M. & Eastgate, R.) 39–46 (VTT Technical Research Centre of Finland, 2018).
Kaczmarek, K. A., Webster, J. G., Bach-y-Rita, P. & Tompkins, W. J. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Trans. Biomed. Eng. 38, 1–16 (1991).
Kajimoto, H., Kawakami, N., Maeda, T. & Tachi, S. Tactile Feeling Display using Functional Electrical Stimulation. in Icat’99 (ed. Tachi, S.) 107 (The Virtual Reality Society of Japan, 1999).
Vallbo, A. B. & Johansson, R. S. Properties of cutaneous mechanoreceptors in the human hand related to touch sensation. Hum. Neurobiol. 3, 3–14 (1984).
Mazzotta, A., Carlotti, M. & Mattoli, V. Conformable on-skin devices for thermo-electro-tactile stimulation: materials, design, and fabrication. Mater. Adv. 2, 1787–1820 (2021).
Jung, Y. H., Kim, J. H. & Rogers, J. A. Skin-integrated vibrohaptic interfaces for virtual and augmented reality. Adv. Funct. Mater. https://doi.org/10.1002/adfm.202008805 (2020).
Huang, Y. et al. Recent advances in multi-mode haptic feedback technologies towards wearable interfaces. Mater. Today Phys. 22, 100602 (2021).
Zárate, J. J. & Shea, H. Using pot-magnets to enable stable and scalable electromagnetic tactile displays. IEEE Trans. Haptics 10, 106–112 (2017).
Sonar, H. A. & Paik, J. Soft pneumatic actuator skin with piezoelectric sensors for vibrotactile feedback. Front. Robot. AI 2, 1–11 (2016).
Sonar, H. A., Gerratt, A. P., Lacour, S. P. & Paik, J. Closed-loop haptic feedback control using a self-sensing soft pneumatic actuator skin. Soft Robot. 7, 22–29 (2020).
Acome, E. et al. Hydraulically amplified self-healing electrostatic actuators with muscle-like performance. Science 359, 61–65 (2018).
Cacucciolo, V. et al. Stretchable pumps for soft machines. Nature 572, 516–519 (2019).
Nakajima, M., Makino, Y. & Shinoda, H. Remote cooling sensation presentation controlling mist in midair. In Proc. 2020 IEEE/SICE International Symposium on System Integration (SII) 2020 1238–1241 (IEEE, 2020); https://doi.org/10.1109/SII46433.2020.9025959
Long, B., Seah, S. A., Carter, T. & Subramanian, S. Rendering volumetric haptic shapes in mid-air using ultrasound. ACM Trans. Graph. 33, 181 (2014).
Freeman, E. & Wilson, G. Perception of Ultrasound Haptic Focal Point Motion. in ICMI 2021 - Proceedings of the 2021 International Conference on Multimodal Interaction (eds. Hammal, Z. et al.) 697–701 (Association for Computing Machinery, 2021). https://doi.org/10.1145/3462244.3479950
Nakajima, M., Hasegawa, K., Makino, Y. & Shinoda, H. Spatiotemporal pinpoint cooling sensation produced by ultrasound-driven mist vaporization on skin. IEEE Trans. Haptics 14, 874–884 (2021).
Ying, M. et al. Silicon nanomembranes for fingertip electronics. Nanotechnology 23, 344004 (2012).
Withana, A., Groeger, D. & Steimle, J. Tacttoo: A thin and feel-through tattoo for on-skin tactile output. in UIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (eds. Baudisch, P. & Schmidt, A.) 365–378 (Association for Computing Machinery, 2018); https://doi.org/10.1145/3242587.3242645
Xu, B. et al. An epidermal stimulation and sensing platform for sensorimotor prosthetic control, management of lower back exertion, and electrical muscle activation. Adv. Mater. 28, 4462–4471 (2016).
Kajimoto, H., Suzuki, M. & Kanno, Y. HamsaTouch: Tactile Vision Substitution with Smartphone and Electro-Tactile Display. in CHI EA ’14: CHI ’14 Extended Abstracts on Human Factors in Computing Systems (eds. Jones, M. & Palanque, P.) 1273–1278 (Association for Computing Machinery, 2014). https://doi.org/10.1145/2559206.2581164
Larson, C. et al. Highly stretchable electroluminescent skin for optical signaling and tactile sensing. Science 351, 1071–1074 (2016).
Kim, C. C., Lee, H. H., Oh, K. H. & Sun, J. Y. Highly stretchable, transparent ionic touch panel. Science 353, 682–687 (2016).
Effects of Current on Human Beings and Livestock IEC TS 60479-1 (2016). International Electrotechnical Commission
Schaning, M. A. & Kaczmarek, K. A. A high-voltage bipolar transconductance amplifier for electrotactile stimulation. IEEE Trans. Biomed. Eng. 55, 2433–2443 (2008).
Tang, H. & Beebe, D. J. A microfabricated electrostatic haptic display for persons with visual impairments. IEEE Trans. Rehabil. Eng. 6, 241–248 (1998).
Hemmerling, M. et al. Sparkle: Hover Feedback with Touchable Electric Arcs. in CHI ’17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (eds. Mark, G. & Fussell, S.) 3705–3717 (Association for Computing Machinery, 2017); https://doi.org/10.1007/978-3-319-53135-9_17
Johansson, R. S. & Vallbo, A. B. Tactile sensibility in the human hand: relative and absolute densities of four types of mechanoreceptive units in glabrous skin. J. Physiol. 286, 283–300 (1979).
Yao, K. et al. Encoding of tactile information in hand via skin-integrated wireless haptic interface. Zenodo https://doi.org/10.5281/zenodo.6919395 (2022).
Acknowledgements
We thank S. K. Nejad for offering circuit design advice, S. Y. Au, J. Liu, M. Yan, J. Chen, Q. Zhuang, Grace Hamor and Melisa M. Ronato for helping with human experiments, and M. Park for helping with equipment usage. This work was supported by City University of Hong Kong (grants numbers 9667221 and 9680322), Research Grants Council of the Hong Kong Special Administrative Region (grants numbers 21210820 and 11213721), National Natural Science Foundation of China (grants number 62122002), Innovation and Technology Fund of Innovation and Technology Commission (GHP/095/20GD), and Shenzhen Science and Technology Innovation Commission (grants number JCYJ20200109110201713) (X.Y.). This work is also sponsored by InnoHK Project on Project 2.2 - AI-based 3D ultrasound imaging algorithm at Hong Kong Centre for Cerebro-Cardiovascular Health Engineering (COCHE), Center of Flexible Electronics Technology, and Qiantang Science & Technology Innovation Center (X.Y.). This work is also supported from Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01), ZJ Lab, and Shanghai Center for Brain Science and Brain-Inspired Technology (E.S.). This work is also financially supported by the Regional Joint Fund of the National Science Foundation of China (U21A20492), and the Sichuan Science and Technology Program (Grant Nos. 2022YFH0081, 2022YFG0012, 2022YFG0013, the Sichuan Province Key Laboratory of Display Science and Technology (J.Y.).
Author information
Authors and Affiliations
Contributions
K.Y., J. Zhou and X.Y. conceived the project and designed the studies. K.Y. and J. Zhou designed and developed the device, circuits and programs. Q.H. and M.W. worked on testing experiments, data analysis and visualization. C.K.Y., Jian Li, J.S. Y.L., S.H., Y.H., D.L., T.H.W., Jiyu Li, R.S., H.L., B.Z., W.P., Y.Z. and J. Zhu assisted in the fabrication and human experiments. X.H., Z.T., H.J., Z.G. and Y.G. worked on materials characterizations. E.S., M.H., H.Z., J.Y., L.W. and W.J.L. evaluated the experiments and contributed ideas. K.Y., J. Zhou, Q.H., M.W. and X.Y. composed and edited the manuscript. All authors discussed the results and provided valuable feedbacks on the manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Machine Intelligence thanks Huanbo Sun, Sheng Xu, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Illustration of the QI-compatible wireless charging ability.
a, Photograph showing the flexible PCB of driver unit circuit with foldable part of QI wireless charging module. b, Photograph showing a soft driver unit getting charged on a commercial QI-compatible wireless charger. c, Current curve of the battery during wireless charging for about 15 min. 200 points averaging smoothed curve indicated in orange. d, The growing battery voltage versus charging time (n = 3 times of the independent charging process; error bar, SD).
Extended Data Fig. 2 Mechanical performance characterizations of the PAAm hydrogel.
a, Photograph of PAAm hydrogel sample during tensile test. b, Stress-strain curve of PAAm hydrogel. c, Peel-off strength test of a hydrogel film to the silicone skin replica with a peeling angle of 90°. Inset: the detailed morphology photo of the skin replica. d, The measured peel strength during the peeling. The maximum strength is ~10mN/cm. e, Photograph showing a PAAm hydrogel patch adhered to the skin of fingertip. f, Photograph showing peeling off the hydrogel patch after wearing it for 1 hour, and still adhered to the skin.
Extended Data Fig. 3 Current control module performance and monitoring results.
a, Diagram showing how the hand is in series in the closed circuit and the position of CCM in this circuit. A virtual ground (VGND) whose potential changes according to permitted current is connected to the common cathode electrode (CE). SE, stimulating electrode. GND, ground. b, Detail schematic of the CCM and current monitoring circuit. Current control part is circled with red dashed lines, mainly composed of a current mirror where the controlled current of the 2nd transistor (ICTL)is identical to the reference current through the 1st transistor (IREF). Monitoring part is circled with blue dashed lines, mainly composed of a fixed resistor (typically 50 Ohm) and an operational amplifier functions as a voltage follower, where the current passing through the resistor is transformed into voltage signal and the duplicated signal being sensed by a 14 bit analogue-digital converter (ADC) on the MCU. c, Measured current under different wireless commands using commercial data acquisition system (Keithley DAQ 6510) d, Zoomed-in details of the current waveform when switching intensities. e, Voltage value between two sides of the resistor, sensed by the ADC of MCU. f, Zoomed in details of the voltage waveform. Time scales of e is identical with c while f is identical with d.
Extended Data Fig. 4 Thermal images of the driver unit during working.
a, Thermal image of the bare FPCB. b, Thermal image of the driver unit encapsulated in Ecoflex. c, Optical photograph of the intact driver unit worn on a user’s forearm. d, Thermal image of the intact driver unit worn on a user’s forearm, after WeTac continuously worked for 10 min.
Extended Data Fig. 5
Threshold maps of female volunteers at fixed fs of 25 Hz. Threshold maps of different sensation level of each volunteer were aligned in a column. Ages of volunteers were labeled at the top of each column.
Extended Data Fig. 6
Threshold maps of male volunteers at fixed fS of 25 Hz. Threshold maps of different sensation level of each volunteer were aligned in a column. Ages of volunteers were labeled at the top of each column.
Extended Data Fig. 7 Independent current intensity modulation in simultaneously activated channels.
a, The current plot of all channels in tennis-grabbing demo. b, Detailed plot showing one period of currents of 11 simultaneously activated channels, whose phases were shifted sequentially. c, Flow chart showing the control strategy of the modulation method. d, Actual output of DAC for controlling current intensity of every channel as an assistive evidence of the method.
Extended Data Fig. 8 Pressure-encoded electrotactile stimulation in the human–machine interaction applications.
a, Sensor configurations on both user’s hand and the robotic hand. b, Photographs showing 3 actions (knocking, slapping, and hitting) with different strengths. c, Recorded pressure of 3 actions in b, correspondingly. d, Corresponding stimulation waveforms for three actions in c. e, Photographs showing the user controls the robotic hand to grasp a plastic cup, lift it and then drop it. f, Recorded pressure on 5 fingers. g, Corresponding stimulation waveforms on five fingers.
Extended Data Fig. 9 Schematic diagrams showing the method to integrate WeTac system with VR environment.
a, the block diagram showing the communication, control and stimulation signal transmitting among different parts. b, Flow chart showing the control logic of the collision-detection-stimulation program in Unity3D. The Bluetooth word mark and logos are registered trademarks owned by Bluetooth SIG, Inc.
Extended Data Fig. 10 WeTac enabled virtual touching experience in VR environment.
a,b, Photographs and synchronized screen captures showing the interaction between the user’s hand avatar and the virtual tennis ball. c, The stimulation current during the whole grasping-lifting-dropping process (for~2.6 s). d, The felt sensation levels on 5 fingertips in situations of grasping a real tennis ball (Real), perceiving the tuned stimulation without VR (Blind), and perceiving real-time stimulation when grasping a virtual tennis in VR (VR) (n = 9 random users; error bars, SD). e,f, Photographs and synchronized screen captures showing the interaction between the user’s hand avatar and the virtual cactus. The user was triggered to show an instinctive hand-withdrawal reflex when stimulated. g, The stimulation current when user’s finger contacts the cactus. h, The felt sensation intensity and position reported by the user after touching the virtual cactus.
Supplementary information
Supplementary Information
Supplementary Notes 1–4, captions for Videos 1–7, Figs. 1–22, and Tables 1 and 2.
Supplementary Video 1
Real-time tactile feedback for simulating the pressure distribution during grasping a tennis ball, with the ball first dropping to parts near the wrist and then rolling to fingers, where pressing force becomes higher. Left: pressure distribution map; Right: stimulation current distribution map.
Supplementary Video 2
Real-time tactile feedback for simulating the pressure (same strength) and feeling (gentle touch) when a virtual mouse was running on the hand, from index fingertip to the wrist side of hypothenar eminence. Left: screen recording of the GUI of customized application on smartphone for automatically controlling WeTac system. Centre: the AR scenario where the virtual mouse runs. Right bottom: the real-time dynamic map of actual current output intensity,
Supplementary Video 3
Demonstration of WeTac in applications of human–machine interaction, for providing different pressure strength information when the user controlled the robotic hand to knock an object.
Supplementary Video 4
Demonstration of WeTac in applications of human–machine interaction, for providing feedback when the user controlled the robotic hand to grasp a plastic cup, and stimulations were applied on his five fingers.
Supplementary Video 5
Demonstration of WeTac in applications of VR interactions, where electrotactile feedback were applied to five fingers of the user when he grasped the virtual tennis ball.
Supplementary Video 6
Demonstration of WeTac in applications of VR interactions, where an strong but short electrotactile feedback were applied on the index fingertip at the moment the user touched the virtual cactus, and the user showed instinctive hand-withdrawal reflex.
Supplementary Video 7
Overall voice-over introduction.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Yao, K., Zhou, J., Huang, Q. et al. Encoding of tactile information in hand via skin-integrated wireless haptic interface. Nat Mach Intell 4, 893–903 (2022). https://doi.org/10.1038/s42256-022-00543-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s42256-022-00543-y
