skip to main content
10.1145/3411764.3445099acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Touch&Fold: A Foldable Haptic Actuator for Rendering Touch in Mixed Reality

Published: 07 May 2021 Publication History

Abstract

We propose a nail-mounted foldable haptic device that provides tactile feedback to mixed reality (MR) environments by pressing against the user's fingerpad when a user touches a virtual object. What is novel in our device is that it quickly tucks away when the user interacts with real-world objects. Its design allows it to fold back on top of the user's nail when not in use, keeping the user's fingerpad free to, for instance, manipulate handheld tools and other objects while in MR. To achieve this, we engineered a wireless and self-contained haptic device, which measures 24×24×41 mm and weighs 9.5 g. Furthermore, our foldable end-effector also features a linear resonant actuator, allowing it to render not only touch contacts (i.e., pressure) but also textures (i.e., vibrations). We demonstrate how our device renders contacts with MR surfaces, buttons, low- and high-frequency textures. In our first user study, we found that participants perceived our device to be more realistic than a previous haptic device that also leaves the fingerpad free (i.e., fingernail vibration). In our second user study, we investigated the participants’ experience while using our device in a real-world task that involved physical objects. We found that our device allowed participants to use the same finger to manipulate handheld tools, small objects, and even feel textures and liquids, without much hindrance to their dexterity, while feeling haptic feedback when touching MR interfaces.

Supplementary Material

MP4 File (3411764.3445099_videofigure.mp4)
Supplemental video
MP4 File (3411764.3445099_videopreview.mp4)
Preview video

References

[1]
Merwan Achibet, Benoît Le Gouis, Maud Marchal, Pierre-Alexandre Léziart, Ferran Argelaguet, Adrien Girard, Anatole Lécuyer, and Hiroyuki Kajimoto. 2017. FlexiFingers: Multi-finger interaction in VR combining passive haptics and pseudo-haptics. In 2017 IEEE Symposium on 3D User Interfaces (3DUI), 103–106. https://doi.org/10.1109/3DUI.2017.7893325
[2]
Hideyuki Ando, Eisuke Kusachi, and Junji Watanabe. 2007. Nail-mounted Tactile Display for Boundary/Texture Augmentation. In Proceedings of the International Conference on Advances in Computer Entertainment Technology (ACE ’07), 292–293. https://doi.org/10.1145/1255047.1255131
[3]
Hideyuki Ando, Takeshi Miki, Masahiko Inami, and Taro Maeda. 2002. The Nail-mounted Tactile Display for the Behavior Modeling. In ACM SIGGRAPH 2002 Conference Abstracts and Applications (SIGGRAPH ’02), 264–264. https://doi.org/10.1145/1242073.1242277
[4]
Takafumi Aoki, Hironori Mitake, Shoichi Hasegawa, and Makoto Sato. 2009. Haptic ring: touching virtual creatures in mixed reality environments. In SIGGRAPH ’09: Posters (SIGGRAPH ’09), 1. https://doi.org/10.1145/1599301.1599401
[5]
Takumi Azai, Syunsuke Ushiro, Junlin Li, Mai Otsuki, Fumihisa Shibata, and Asako Kimura. 2018. Tap-tap menu: body touching for virtual interactive menus. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (VRST ’18), 1–2. https://doi.org/10.1145/3281505.3281561
[6]
Olivier Bau and Ivan Poupyrev. 2012. REVEL: tactile feedback technology for augmented reality. ACM Transactions on Graphics 31, 4: 89:1-89:11. https://doi.org/10.1145/2185520.2185585
[7]
M. Bouzit, G. Burdea, G. Popescu, and R. Boian. 2002. The Rutgers Master II-new design force-feedback glove. IEEE/ASME Transactions on Mechatronics 7, 2: 256–263. https://doi.org/10.1109/TMECH.2002.1011262
[8]
Inrak Choi, Heather Culbertson, Mark R. Miller, Alex Olwal, and Sean Follmer. 2017. Grabity: A Wearable Haptic Interface for Simulating Weight and Grasping in Virtual Reality. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST ’17), 119–130. https://doi.org/10.1145/3126594.3126599
[9]
Inrak Choi and Sean Follmer. 2016. Wolverine: A Wearable Haptic Interface for Grasping in VR. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST ’16 Adjunct), 117–119. https://doi.org/10.1145/2984751.2985725
[10]
Kiran Dandekar, Balasundar I. Raju, and Mandayam A. Srinivasan. 2003. 3-D Finite-Element Models of Human and Monkey Fingertips to Investigate the Mechanics of Tactile Sense. Journal of Biomechanical Engineering 125, 5: 682–691. https://doi.org/10.1115/1.1613673
[11]
Artem Dementyev, Alex Olwal, and Richard F. Lyon. 2020. Haptics with Input: Back-EMF in Linear Resonant Actuators to Enable Touch, Pressure and Environmental Awareness. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST ’20), 420–429. https://doi.org/10.1145/3379337.3415823
[12]
Cathy Fang, Yang Zhang, Matthew Dworman, and Chris Harrison. 2020. Wireality: Enabling Complex Tangible Geometries in Virtual Reality with Worn Multi-String Haptics. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), 1–10. https://doi.org/10.1145/3313831.3376470
[13]
Yuan-Ling Feng, Charith Lasantha Fernando, Jan Rod, and Kouta Minamizawa. 2017. Submerged haptics: a 3-DOF fingertip haptic display using miniature 3D printed airbags. 1–2. https://doi.org/10.1145/3084822.3084835
[14]
Xiaochi Gu, Yifei Zhang, Weize Sun, Yuanzhe Bian, Dao Zhou, and Per Ola Kristensson. 2016. Dexmo: An Inexpensive and Lightweight Mechanical Exoskeleton for Motion Capture and Force Feedback in VR. 1991–1995. https://doi.org/10.1145/2858036.2858487
[15]
Teng Han, Fraser Anderson, Pourang Irani, and Tovi Grossman. 2018. HydroRing: Supporting Mixed Reality Haptics Using Liquid Flow. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST ’18), 913–925. https://doi.org/10.1145/3242587.3242667
[16]
Steven C. Hauser and Gregory J. Gerling. 2016. Measuring tactile cues at the fingerpad for object compliances harder and softer than the skin. IEEE Haptics Symposium: [proceedings]. IEEE Haptics Symposium 2016: 247–252. https://doi.org/10.1109/HAPTICS.2016.7463185
[17]
Steven J. Henderson and Steven Feiner. 2009. Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In 2009 8th IEEE International Symposium on Mixed and Augmented Reality, 135–144. https://doi.org/10.1109/ISMAR.2009.5336486
[18]
Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality. 1957–1967. https://doi.org/10.1145/2858036.2858134
[19]
Ronan Hinchet, Velko Vechev, Herbert Shea, and Otmar Hilliges. 2018. DextrES: Wearable Haptic Feedback for Grasping in VR via a Thin Form-Factor Electrostatic Brake. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST ’18), 901–912. https://doi.org/10.1145/3242587.3242657
[20]
Seungwoo Je, Myung Jin Kim, Woojin Lee, Byungjoo Lee, Xing-Dong Yang, Pedro Lopes, and Andrea Bianchi. 2019. Aero-plane: A Handheld Force-Feedback Device that Renders Weight Motion Illusion on a Virtual 2D Plane. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST ’19), 763–775. https://doi.org/10.1145/3332165.3347926
[21]
Hiroyuki Kajimoto. 2012. Electrotactile Display with Real-Time Impedance Feedback Using Pulse Width Modulation. IEEE Transactions on Haptics 5, 2: 184–188. https://doi.org/10.1109/TOH.2011.39
[22]
Hwan Kim, Minhwan Kim, and Woohun Lee. 2016. HapThimble: A Wearable Haptic Device Towards Usable Virtual Touch Screen. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), 3694–3705. https://doi.org/10.1145/2858036.2858196
[23]
Hwan Kim, HyeonBeom Yi, Hyein Lee, and Woohun Lee. 2018. HapCube: A Wearable Tactile Device to Provide Tangential and Normal Pseudo-Force Feedback on a Fingertip. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), 1–13. https://doi.org/10.1145/3173574.3174075
[24]
Sunjun Kim and Geehyuk Lee. 2013. Haptic feedback design for a virtual button along force-displacement curves. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST ’13), 91–96. https://doi.org/10.1145/2501988.2502041
[25]
Robert Kovacs, Eyal Ofek, Mar Gonzalez Franco, Alexa Fay Siu, Sebastian Marwecki, Christian Holz, and Mike Sinclair. 2020. Haptic PIVOT: On-Demand Handhelds in VR. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST ’20), 1046–1059. https://doi.org/10.1145/3379337.3415854
[26]
James F. Kramer. 1993. Force feedback and textures simulating interface device. Retrieved May 4, 2020 from https://patents.google.com/patent/US5184319A/en
[27]
Shinobu Kuroki, Hiroyuki Kajimoto, Hideaki Nii, Naoki Kawakami, and Susumu Tachi. 2007. Proposal for tactile sense presentation that combines electrical and mechanical stimulus. In Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC’07), 121–126. https://doi.org/10.1109/WHC.2007.92
[28]
Susan J. Lederman and Lynette A. Jones. 2011. Tactile and Haptic Illusions. IEEE Transactions on Haptics 4, 4: 273–294. https://doi.org/10.1109/TOH.2011.2
[29]
David Lindlbauer and Andy D. Wilson. 2018. Remixed Reality: Manipulating Space and Time in Augmented Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), 129:1-129:13. https://doi.org/10.1145/3173574.3173703
[30]
Pedro Lopes and Patrick Baudisch. 2013. Muscle-propelled Force Feedback: Bringing Force Feedback to Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13), 2577–2580. https://doi.org/10.1145/2470654.2481355
[31]
Pedro Lopes, Alexandra Ion, and Patrick Baudisch. 2015. Impacto: Simulating Physical Impact by Combining Tactile Stimulation with Electrical Muscle Stimulation. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST ’15), 11–19. https://doi.org/10.1145/2807442.2807443
[32]
Pedro Lopes, Sijing You, Lung-Pan Cheng, Sebastian Marwecki, and Patrick Baudisch. 2017. Providing Haptics to Walls & Heavy Objects in Virtual Reality by Means of Electrical Muscle Stimulation. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 1471–1482. https://doi.org/10.1145/3025453.3025600
[33]
Pedro Lopes, Sijing You, Alexandra Ion, and Patrick Baudisch. 2018. Adding Force Feedback to Mixed Reality Experiences and Games Using Electrical Muscle Stimulation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), 446:1-446:13. https://doi.org/10.1145/3173574.3174020
[34]
Aditya Shekhar Nittala, Klaus Kruttwig, Jaeyeon Lee, Roland Bennewitz, Eduard Arzt, and Jürgen Steimle. 2019. Like A Second Skin: Understanding How Epidermal Devices Affect Human Tactile Perception. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 380:1-380:16. https://doi.org/10.1145/3290605.3300610
[35]
Claudio Pacchierotti, Stephen Sinclair, Massimiliano Solazzi, Antonio Frisoli, Vincent Hayward, and Domenico Prattichizzo. 2017. Wearable Haptic Systems for the Fingertip and the Hand: Taxonomy, Review, and Perspectives. IEEE Transactions on Haptics 10, 4: 580–600. https://doi.org/10.1109/TOH.2017.2689006
[36]
Jun Rekimoto. 2013. Traxion: a tactile interaction device with virtual force sensation. 427–432. https://doi.org/10.1145/2501988.2502044
[37]
Stefano Scheggi, Gionata Salvietti, and Domenico Prattichizzo. 2010. Shape and Weight Rendering for Haptic Augmented Reality. 44–49. https://doi.org/10.1109/ROMAN.2010.5598632
[38]
Dieter Schmalstieg and Daniel Wagner. 2007. Experiences with Handheld Augmented Reality. 3–18. https://doi.org/10.1109/ISMAR.2007.4538819
[39]
Samuel B. Schorr and Allison M. Okamura. 2017. Fingertip Tactile Devices for Virtual Object Manipulation and Exploration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 3115–3119. https://doi.org/10.1145/3025453.3025744
[40]
Mike Sinclair, Michel Pahud, and Hrvoje Benko. 2014. TouchMover 2.0 - 3D touchscreen with force feedback and haptic texture. In 2014 IEEE Haptics Symposium (HAPTICS), 1–6. https://doi.org/10.1109/HAPTICS.2014.6775425
[41]
Anthony Webster Steven, Steven Feiner, Blair Macintyre, William Massie, and Theodore Krueger. 1996. Augmented Reality in Architectural Construction, Inspection, and Renovation.
[42]
Paul Strohmeier and Kasper Hornb . 2017. Generating Haptic Textures with a Vibrotactile Actuator. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 4994–5005. https://doi.org/10.1145/3025453.3025812
[43]
Shan-Yuan Teng, Tzu-Sheng Kuo, Chi Wang, Chi-huan Chiang, Da-Yuan Huang, Liwei Chan, and Bing-Yu Chen. 2018. PuPoP: Pop-up Prop on Palm for Virtual Reality. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST ’18), 5–17. https://doi.org/10.1145/3242587.3242628
[44]
Dzmitry Tsetserukou, Katsunari Sato, and Susumu Tachi. 2010. FlexTorque: Exoskeleton Interface for Haptic Interaction with the Digital World. In Proceedings of the 2010 International Conference on Haptics - Generating and Perceiving Tangible Sensations: Part II (EuroHaptics’10), 166–171. Retrieved July 14, 2019 from http://dl.acm.org/citation.cfm?id=1893760.1893786
[45]
Scott Varga. HaptX | Haptic gloves for VR training, simulation, and design. HaptX. Retrieved May 4, 2020 from https://haptx.com/
[46]
Chiu-Hsuan Wang, Chen-Yuan Hsieh, Neng-Hao Yu, Andrea Bianchi, and Liwei Chan. 2019. HapticSphere: Physical Support To Enable Precision Touch Interaction in Mobile Mixed-Reality. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 331–339. https://doi.org/10.1109/VR.2019.8798255
[47]
Graham Wilson, Tom Carter, Sriram Subramanian, and Stephen Brewster. 2014. Perception of Ultrasonic Haptic Feedback on the Hand: Localisation and Apparent Motion. https://doi.org/10.1145/2556288.2557033
[48]
S. H. Winter and M. Bouzit. 2007. Use of Magnetorheological Fluid in a Force Feedback Glove. IEEE Transactions on Neural Systems and Rehabilitation Engineering 15, 1: 2–8. https://doi.org/10.1109/TNSRE.2007.891401
[49]
Anusha Withana, Daniel Groeger, and Jürgen Steimle. 2018. Tacttoo: A Thin and Feel-Through Tattoo for On-Skin Tactile Output. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST ’18), 365–378. https://doi.org/10.1145/3242587.3242645
[50]
Vibol Yem, Ryuta Okazaki, and Hiroyuki Kajimoto. 2016. FinGAR: combination of electrical and mechanical stimulation for high-fidelity tactile presentation. In ACM SIGGRAPH 2016 Emerging Technologies (SIGGRAPH ’16), 1–2. https://doi.org/10.1145/2929464.2929474
[51]
2020. aestheticinteractive/Hover-UI-Kit. Aesthetic Interactive. Retrieved May 4, 2020 from https://github.com/aestheticinteractive/Hover-UI-Kit
[52]
CyberGrasp. CyberGlove Systems LLC. Retrieved March 8, 2020 from http://www.cyberglovesystems.com/cybergrasp
[53]
VIVETM | Discover Virtual Reality Beyond Imagination. Retrieved May 4, 2020 from https://www.vive.com/us/
[54]
Microsoft HoloLens | Mixed Reality Technology for Business. Retrieved May 4, 2020 from https://www.microsoft.com/en-us/hololens
[55]
Magic Leap 1. Retrieved May 4, 2020 from https://www.magicleap.com/magic-leap-1
[56]
(PDF) Towards multimodal haptics for teleoperation: Design of a tactile thermal display. ResearchGate. http://dx.doi.org/10.1109/AMC.2012.6197145
[57]
CyberTouch. CyberGlove Systems LLC. Retrieved March 8, 2020 from http://www.cyberglovesystems.com/cybertouch
[58]
Hand interaction examples scene | Mixed Reality Toolkit Documentation. Retrieved September 16, 2020 from https://microsoft.github.io/MixedRealityToolkit-Unity/Documentation/README_HandInteractionExamples.html
[59]
Project North Star. Leap Motion Developer. Retrieved March 30, 2020 from http://developer.leapmotion.com/northstar

Cited By

View all
  • (2025)Eliciting Design Guidelines of Paper-Based Tactile Interfaces for Eyes-Free ScenariosProceedings of the Nineteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3689050.3704930(1-15)Online publication date: 4-Mar-2025
  • (2025)A Pneumatic-Actuated Feel-Through Wearable Haptic Display for Multi-Cue DeliveryIEEE Robotics and Automation Letters10.1109/LRA.2024.351989410:2(1225-1232)Online publication date: Feb-2025
  • (2024)EStatiG: Wearable Haptic Feedback with Multi-Phalanx Electrostatic Brake for Enhanced Object Perception in VRProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785678:3(1-29)Online publication date: 9-Sep-2024
  • Show More Cited By

Index Terms

  1. Touch&Fold: A Foldable Haptic Actuator for Rendering Touch in Mixed Reality
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
        May 2021
        10862 pages
        ISBN:9781450380966
        DOI:10.1145/3411764
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 07 May 2021

        Permissions

        Request permissions for this article.

        Check for updates

        Badges

        • Honorable Mention

        Author Tags

        1. Haptics
        2. Mixed Reality
        3. Wearable

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Conference

        CHI '21
        Sponsor:

        Acceptance Rates

        Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

        Upcoming Conference

        CHI 2025
        ACM CHI Conference on Human Factors in Computing Systems
        April 26 - May 1, 2025
        Yokohama , Japan

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)322
        • Downloads (Last 6 weeks)33
        Reflects downloads up to 22 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2025)Eliciting Design Guidelines of Paper-Based Tactile Interfaces for Eyes-Free ScenariosProceedings of the Nineteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3689050.3704930(1-15)Online publication date: 4-Mar-2025
        • (2025)A Pneumatic-Actuated Feel-Through Wearable Haptic Display for Multi-Cue DeliveryIEEE Robotics and Automation Letters10.1109/LRA.2024.351989410:2(1225-1232)Online publication date: Feb-2025
        • (2024)EStatiG: Wearable Haptic Feedback with Multi-Phalanx Electrostatic Brake for Enhanced Object Perception in VRProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785678:3(1-29)Online publication date: 9-Sep-2024
        • (2024)Demonstration of JetUnit: Rendering Diverse Force Feedback in Virtual Reality Using Water JetsAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686742(1-3)Online publication date: 13-Oct-2024
        • (2024)Understanding the Effects of Restraining Finger Coactivation in Mid-Air Typing: from a Neuromechanical PerspectiveProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676441(1-18)Online publication date: 13-Oct-2024
        • (2024)JetUnit: Rendering Diverse Force Feedback in Virtual Reality Using Water JetsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676440(1-15)Online publication date: 13-Oct-2024
        • (2024)AirPush: A Pneumatic Wearable Haptic Device Providing Multi-Dimensional Force Feedback on a FingertipProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642536(1-13)Online publication date: 11-May-2024
        • (2024)Haptic Source-Effector: Full-Body Haptics via Non-Invasive Brain StimulationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642483(1-15)Online publication date: 11-May-2024
        • (2024)Stick&Slip: Altering Fingerpad Friction via Liquid CoatingsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642299(1-14)Online publication date: 11-May-2024
        • (2024)Haptic Permeability: Adding Holes to Tactile Devices Improves DexterityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642156(1-12)Online publication date: 11-May-2024
        • Show More Cited By

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media