skip to main content
10.1145/3526114.3558669acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
demonstration
Public Access

Hands-On: Using Gestures to Control Descriptions of a Virtual Environment for People with Visual Impairments

Published: 28 October 2022 Publication History

Abstract

Virtual reality (VR) uses three main senses to relay information: sight, sound, and touch. People with visual impairments (PVI) rely primarily on auditory and haptic feedback to receive information in VR. While researchers have explored several approaches to make navigation and perception of objects more accessible in VR, none of them offer a natural way to request descriptions of objects, nor control of the flow of auditory information. In this demonstration, we present a haptic glove that PVI can use to request object descriptions in VR with their hands through familiar hand gestures. We contribute designs for a set of hand gestures that allow PVI to interactively get descriptions of the VR environment. We plan to conduct an user study where we will have PVI interact with a VR environment using these gestures to request audio descriptions.

References

[1]
2022. OrCam MyEye 2.0 - For People Who Are Blind or Visually Impaired. OrCam. Retrieved July 12, 2022 from https://www.orcam.com/en/myeye2/
[2]
2022. Seeing AI App from Microsoft. Microsoft. Retrieved July 12, 2022 from https://www.microsoft.com/en-us/ai/seeing-ai
[3]
Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. 2010. VizWiz: nearly real-time answers to visual questions. Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) - W4A(2010). https://doi.org/10.1145/1805986.1806020
[4]
Chetz Colwell, Helen Petrie, Diana Kornbrot, Andrew Hardwick, and Stephen Furner. 1998. Haptic Virtual Reality for Blind Computer Users. Proceedings of the third international ACM conference on Assistive technologies - Assets (1998). https://doi.org/10.1145/274497.274515
[5]
Cathy Fang, Yang Zhang, Matthew Dworman, and Chris Harrison. 2020. Wireality: Enabling Complex Tangible Geometries in Virtual Reality with Worn Multi-String Haptics. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (2020). https://doi.org/10.1145/3313831.3376470
[6]
Robert Kovacs, Eyal Ofek, Mar Gonzalez Franco, Alexa Fay Siu, Sebastian Marwecki, Christian Holz, and Mike Sinclair. 2020. Haptic PIVOT : On-Demand Handhelds in VR. Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (2020). https://doi.org/10.1145/3379337.3415854
[7]
Orly Lahav, Hadas Gedalevitz, Steven Battersby, David Brown, Lindsay Evett, and Patrick Merritt. 2017. Virtual environment navigation with look-around mode to explore new real spaces by people who are blind. Disability and Rehabilitation 40, 9 (2017), 1072–1084. https://doi.org/10.1080/09638288.2017.1286391
[8]
Orly Lahav and David Mioduser. 2004. Exploration of Unknown Spaces by People Who are Blind Using a Multi-Sensory Virtual Environment. Journal of Special Education Technology 40, 19 (2004), 15–23. https://doi.org/10.1177/016264340401900302
[9]
Orly Lahav, David Schloerb, Siddarth Kumar, and Mandyam Srinivasan. 2012. A virtual environment for people who are blind – a usability study. Journal of Assistive Technologies 6, 1 (2012), 38–52. https://doi.org/10.1108/17549451211214346
[10]
Lucas De Bonet. 2021. LucidGloves. https://github.com/LucidVR/lucidgloves.git
[11]
Tony Morelli, John Foley, Luis Columna, Lauren Lieberman, and Eelke Folmer. 2010. VI- Tennis: a Vibrotactile/ Audio Exergame for Players who are Visually Impaired. Proceedings of the Fifth International Conference on the Foundations of Digital Games - FDG(2010). https://doi.org/10.1145/1822348.1822368
[12]
Helen L. Petrie, Paul Robert Penn, Diana Kornbrot, Stephen Furner, and Andrew John Hardwick. 2000. Haptic virtual environments for blind people: further explorations with the Phantom device. Proc. ICDVRAT (2000).
[13]
Ludek Pokluda and Jiri Sochor. 2003. Spatial Haptic Orientation for Visually Impaired People. EG 3(2003), 29–34.
[14]
Mike Sinclair, Eyal Ofek, Mar Gonzalez-Franco, and Christian Holz. 2019. CapstanCrunch: A Haptic VR Controller with User- supplied Force Feedback. Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (2019). https://doi.org/10.1145/3332165.3347891
[15]
Alexa F. Siu, Mike Sinclair, Robert Kovacs, Eyal Ofek, Christian Holz, and Edward Cutrell. 2020. Virtual Reality Without Vision: A Haptic and Auditory White Cane to Navigate Complex Virtual Worlds. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (2020). https://doi.org/10.1145/3313831.3376353
[16]
Unity Technologies. 2022. Unity. Unity Technologies.
[17]
Ryan Wedoff, Lindsay Ball, Amelia Wang, Yi Xuan Khoo, Lauren Lieberman, and Kyle Rector. 2019. Virtual Showdown: An Accessible Virtual Reality Game with Scaffolds for Youth with Visual Impairments. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2019). https://doi.org/10.1145/3290605.3300371
[18]
Tzu-Yun Wei, Hsin-Ruey Tsai, Yu-So Liao, Chieh Tsai, Yi-Shan Chen, Chi Wang, and Bing-Yu Chen. 2020. ElastiLinks: Force Feedback between VR Controllers with Dynamic Points of Application of Force. Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (2020). https://doi.org/10.1145/3379337.3415836
[19]
Yuhang Zhao, Cynthia L. Bennett, Hrvoje Benko, Edward Cutrell, Christian Holz, Meredith Ringel Morris, and Mike Sinclair. 2018. Enabling People with Visual Impairments to Navigate Virtual Reality with a Haptic and Auditory Cane Simulation. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (April 2018), 1–14. https://doi.org/10.1145/3173574.3173690
[20]
Yuhang Zhao, Edward Cutrell, Christian Holz, Meredith Ringel Morris, Eyal Ofek, and Andrew D. Wilson. 2019. SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2019). https://doi.org/10.1145/3290605.3300341

Cited By

View all
  • (2024)Hapstick-Figure: Investigating the Design of a Haptic Representation of Human Gestures from Theater Performances for Blind and Visually-Impaired PeopleProceedings of the ACM on Human-Computer Interaction10.1145/36981528:ISS(649-673)Online publication date: 24-Oct-2024
  • (2024)Accessible Nonverbal Cues to Support Conversations in VR for Blind and Low Vision PeopleProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675663(1-13)Online publication date: 27-Oct-2024
  • (2023)Opportunities for Accessible Virtual Reality Design for Immersive Musical Performances for Blind and Low-Vision PeopleProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614540(1-21)Online publication date: 13-Oct-2023
  • Show More Cited By

Index Terms

  1. Hands-On: Using Gestures to Control Descriptions of a Virtual Environment for People with Visual Impairments

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        UIST '22 Adjunct: Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology
        October 2022
        413 pages
        ISBN:9781450393218
        DOI:10.1145/3526114
        Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 28 October 2022

        Check for updates

        Author Tags

        1. Accessibility
        2. Audio Feedback
        3. Hand Gestures
        4. Haptic Feedback
        5. Virtual Reality
        6. Visual Impairments

        Qualifiers

        • Demonstration
        • Research
        • Refereed limited

        Funding Sources

        Conference

        UIST '22

        Acceptance Rates

        Overall Acceptance Rate 355 of 1,733 submissions, 20%

        Upcoming Conference

        UIST '25
        The 38th Annual ACM Symposium on User Interface Software and Technology
        September 28 - October 1, 2025
        Busan , Republic of Korea

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)195
        • Downloads (Last 6 weeks)28
        Reflects downloads up to 14 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)Hapstick-Figure: Investigating the Design of a Haptic Representation of Human Gestures from Theater Performances for Blind and Visually-Impaired PeopleProceedings of the ACM on Human-Computer Interaction10.1145/36981528:ISS(649-673)Online publication date: 24-Oct-2024
        • (2024)Accessible Nonverbal Cues to Support Conversations in VR for Blind and Low Vision PeopleProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675663(1-13)Online publication date: 27-Oct-2024
        • (2023)Opportunities for Accessible Virtual Reality Design for Immersive Musical Performances for Blind and Low-Vision PeopleProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614540(1-21)Online publication date: 13-Oct-2023
        • (2023)“The Guide Has Your Back”: Exploring How Sighted Guides Can Enhance Accessibility in Social Virtual Reality for Blind and Low Vision PeopleProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608386(1-14)Online publication date: 22-Oct-2023
        • (2023)Beyond Audio Description: Exploring 360° Video Accessibility with Blind and Low Vision Users Through Collaborative CreationProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608381(1-17)Online publication date: 22-Oct-2023

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Login options

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media