skip to main content
10.1145/3334480.3383142acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
abstract

A Dynamic AI System for Extending the Capabilities of Blind People

Published: 25 April 2020 Publication History

Abstract

We introduce an advanced computer vision-based AI system that offers people with vision impairments (VI) dynamic, in-situ access to information about the location, identity and gaze-direction of other people nearby. Our AI system utilizes the camera technology of a head-worn HoloLens device, which captures a near 180° field-of-view surrounding the person who is wearing it. Captured images are then processed by multiple state-of-the-art perception algorithms whose outputs are integrated into a real-time tracking model of all people that the system detected. Users can receive information about those people acoustically (via spatialized audio) using a wrist-worn input controller. Having such dynamic access to information, through AI system interactions, enables people with VI to: develop their communication skills; more easily focus on others; and be more confident in their social interactions. Thus, our work explores how AI systems can serve as a useful resource for humans, helping expand their agency to develop new or extend existing skills.

Supplemental Material

MP4 File
Preview video

References

[1]
Jeffrey P Bigham and Patrick Carrington. 2018. Learning from the Front: People with Disabilities as Early Adopters of AI. HCIC.
[2]
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. Proc. IEEE CVRP, 7291--7299.
[3]
Daniel Coelho de Castro and Sebastian Nowozin. 2018. From Face Recognition to Models of Identity: A Bayesian Approach to Learning about Unknown Identities from Unsupervised Data. Proc. ECCV, 745--761.
[4]
Kori Inkpen, Stevie Chancellor, Munmun De Choudhury, Michael Veale, and Eric P. S. Baumer. 2019. Where is the Human?: Bridging the Gap Between AI and HCI. Ext. Abstr. CHI 2019. ACM, Paper W09, 9 pages.
[5]
Cecily Morrison, Edward Cutrell, Anupama Dhareshwar, Kevin Doherty, Anja Thieme, and Alex Taylor. 2017. Imagining Artificial Intelligence Applications with People with Visual Disabilities using Tactile Ideation. Proc. ASSETS 2017. 81--90.
[6]
Sergey Prokudin, Peter Gehler, and Sebastian Nowozin. 2018. Deep directional statistics: Pose estimation with uncertainty quantification. Proc. ECCV, 534--551.
[7]
Lee Stearns and Anja Thieme. 2018. Automated Person Detection in Dynamic Scenes to Assist People with Vision Impairments: An Initial Investigation. Proc. ASSETS 2018. 391--394.
[8]
Anja Thieme, Cynthia L. Bennett, Cecily Morrison, Edward Cutrell, and Alex S. Taylor. 2018. "I can do everything but see!" -- How People with Vision Impairments Negotiate their Abilities in Social Contexts. Proc. CHI 2018. ACM, Paper 203.

Cited By

View all
  • (2024)EchoSee: An Assistive Mobile Application for Real-Time 3D Environment Reconstruction and Sonification Supporting Enhanced Navigation for People with Vision ImpairmentsBioengineering10.3390/bioengineering1108083111:8(831)Online publication date: 14-Aug-2024
  • (2024)Help and The Social Construction of Access: A Case-Study from IndiaProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675606(1-12)Online publication date: 27-Oct-2024
  • (2024)Context matters: Investigating information sharing in mixed-visual ability social interactionsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651121(1-8)Online publication date: 11-May-2024
  • Show More Cited By

Index Terms

  1. A Dynamic AI System for Extending the Capabilities of Blind People

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems
    April 2020
    4474 pages
    ISBN:9781450368193
    DOI:10.1145/3334480
    Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 25 April 2020

    Check for updates

    Author Tags

    1. AI system
    2. accessibility
    3. blindness
    4. design + AI
    5. disability
    6. inclusive design
    7. innovation
    8. visual impairment

    Qualifiers

    • Abstract

    Conference

    CHI '20
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)36
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)EchoSee: An Assistive Mobile Application for Real-Time 3D Environment Reconstruction and Sonification Supporting Enhanced Navigation for People with Vision ImpairmentsBioengineering10.3390/bioengineering1108083111:8(831)Online publication date: 14-Aug-2024
    • (2024)Help and The Social Construction of Access: A Case-Study from IndiaProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675606(1-12)Online publication date: 27-Oct-2024
    • (2024)Context matters: Investigating information sharing in mixed-visual ability social interactionsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651121(1-8)Online publication date: 11-May-2024
    • (2024)What Factors Motivate Culturally Deaf People to Want Assistive Technologies?Extended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650871(1-7)Online publication date: 11-May-2024
    • (2024)Help Supporters: Exploring the Design Space of Assistive Technologies to Support Face-to-Face Help Between Blind and Sighted StrangersProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642816(1-24)Online publication date: 11-May-2024
    • (2023)Understanding Curators' Practices and Challenge of Making Exhibitions More Accessible for People with Visual ImpairmentsProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608384(1-18)Online publication date: 22-Oct-2023
    • (2023)Computer Vision, Human Likeness, and Problematic BehaviorsAdjunct Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization10.1145/3563359.3597381(206-214)Online publication date: 26-Jun-2023
    • (2023)Using Self-Driving Techniques for Blind Assistance2023 International Conference in Advances in Power, Signal, and Information Technology (APSIT)10.1109/APSIT58554.2023.10201728(520-525)Online publication date: 9-Jun-2023
    • (2023)Six Human-Centered Artificial Intelligence Grand ChallengesInternational Journal of Human–Computer Interaction10.1080/10447318.2022.215332039:3(391-437)Online publication date: 2-Jan-2023
    • (2022)A Survey on Recent Advances in AI and Vision-Based Methods for Helping and Guiding Visually Impaired PeopleApplied Sciences10.3390/app1205230812:5(2308)Online publication date: 23-Feb-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media