skip to main content
10.1145/3597638.3608431acmconferencesArticle/Chapter ViewAbstractPublication PagesassetsConference Proceedingsconference-collections
research-article

“Not There Yet”: Feasibility and Challenges of Mobile Sound Recognition to Support Deaf and Hard-of-Hearing People

Published: 22 October 2023 Publication History

Abstract

While recent advances have enabled mobile sound recognition tools for deaf and hard of hearing (DHH) people, these tools have only been studied in the lab or through short, controlled experiments. To assess the real-world feasibility and guide the future designs of mobile sound awareness systems, we conducted a three-week field study of SoundWatch, a smartwatch-based sound recognition app, with 10 DHH participants. Our findings suggest the app's utility in increasing environmental awareness and facilitating everyday tasks for DHH users. However, several challenges, such as background noises, variability of real-world sounds, and confusion among similar sounding sounds, indicated that mobile sound recognition solutions are “not there yet” for adoption and use in daily life. We close by presenting HCI design opportunities to improve model reliability by increasing contextual awareness, supporting end-user customization, and fostering the collective improvement of sound recognition models.

References

[1]
Taslima Akter, Tousif Ahmed, Apu Kapadia, and Swami Manohar Swaminathan. 2020. Privacy Considerations of the Visually Impaired with Camera Based Assistive Technologies: Misrepresentation, Impropriety, and Fairness. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Virtual Event Greece, 1–14.
[2]
Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland Uk, 1–13.
[3]
Mirza Mansoor Baig, Shereen Afifi, Hamid GholamHosseini, and Farhaan Mirza. 2019. A Systematic Review of Wearable Sensors and IoT-Based Monitoring Applications for Older Adults – a Focus on Ageing Population and Independent Living. J. Med. Syst. 43, 8 (August 2019), 233.
[4]
Cynthia L. Bennett, Erin Brady, and Stacy M. Branham. 2018. Interdependence as a Frame for Assistive Technology Research and Design. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Galway Ireland, 161–173.
[5]
Danielle Bragg, Nicholas Huynh, and Richard E. Ladner. 2016. A Personalizable Mobile Sound Detector App Design for Deaf and Hard-of-Hearing Users. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Reno Nevada USA, 3–13.
[6]
Stacy M. Branham and Shaun K. Kane. 2015. The Invisible Work of Accessibility: How Blind Employees Manage Accessibility in Mixed-Ability Workplaces. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility - ASSETS ’15, ACM Press, Lisbon, Portugal, 163–171.
[7]
Virginia Braun and Victoria Clarke. 2021. Thematic Analysis: A Practical Guide. SAGE Publications.
[8]
Anna Cavender and Richard E. Ladner. 2008. Hearing Impairments. In Web Accessibility, Simon Harper and Yeliz Yesilada (eds.). Springer London, London, 25–35.
[9]
Leah Findlater, Bonnie Chinh, Dhruv Jain, Jon Froehlich, Raja Kushalnagar, and Angela Carey Lin. 2019. Deaf and Hard-of-hearing Individuals’ Preferences for Wearable and Mobile Sound Awareness Technologies. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland Uk, 1–13.
[10]
Steven Goodman, Susanne Kirchner, Rose Guttman, Dhruv Jain, Jon Froehlich, and Leah Findlater. 2020. Evaluating Smartwatch-based Sound Feedback for Deaf and Hard-of-hearing Users Across Contexts. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 1–13.
[11]
Steven M. Goodman, Ping Liu, Dhruv Jain, Emma J. McDonnell, Jon E. Froehlich, and Leah Findlater. 2021. Toward User-Driven Sound Recognizer Personalization with People Who Are d/Deaf or Hard of Hearing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 2 (June 2021), 1–23.
[12]
Benjamin M. Gorman. 2014. VisAural:: a wearable sound-localisation device for people with impaired hearing. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility - ASSETS ’14, ACM Press, Rochester, New York, USA, 337–338.
[13]
Fabien Gouyon, François Pachet, and Olivier Delerue. 2000. ON THE USE OF ZERO-CROSSING RATE FOR AN APPLICATION OF CLASSIFICATION OF PERCUSSIVE SOUNDS. (2000).
[14]
Ru Guo, Yiru Yang, Johnson Kuang, Xue Bin, Dhruv Jain, Steven Goodman, Leah Findlater, and Jon Froehlich. 2020. HoloSound: Combining Speech and Sound Identification for Deaf or Hard of Hearing Users on a Head-mounted Display. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Virtual Event Greece, 1–4.
[15]
Guojun Lu and T. Hankinson. 2000. An investigation of automatic audio classification and segmentation. In WCC 2000 - ICSP 2000. 2000 5th International Conference on Signal Processing Proceedings. 16th World Computer Congress 2000, IEEE, Beijing, China, 776–781.
[16]
Foad Hamidi, Kellie Poneres, Aaron Massey, and Amy Hurst. 2018. Who Should Have Access to my Pointing Data?: Privacy Tradeoffs of Adaptive Assistive Technologies. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Galway Ireland, 203–216.
[17]
Foad Hamidi, Kellie Poneres, Aaron Massey, and Amy Hurst. 2020. Using a participatory activities toolkit to elicit privacy expectations of adaptive assistive technologies. In Proceedings of the 17th International Web for All Conference, ACM, Taipei Taiwan, 1–12.
[18]
F Wai-ling Ho-Ching, Jennifer Mankoff, and James A Landay. Can you see what I hear? The Design and Evaluation of a Peripheral Sound Display for the Deaf.
[19]
Yasha Iravantchi, Karan Ahuja, Mayank Goel, Chris Harrison, and Alanson Sample. 2021. PrivacyMic: Utilizing Inaudible Frequencies for Privacy Preserving Daily Activity Recognition. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, ACM, Yokohama Japan, 1–13.
[20]
Dhruv Jain, Khoa Huynh Anh Nguyen, Steven M. Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon E. Froehlich. 2022. ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users. In CHI Conference on Human Factors in Computing Systems, ACM, New Orleans LA USA, 1–16.
[21]
Dhruv Jain, Kelly Mack, Akli Amrous, Matt Wright, Steven Goodman, Leah Findlater, and Jon E. Froehlich. 2020. HomeSound: An Iterative Field Deployment of an In-Home Sound Awareness System for Deaf or Hard of Hearing Users. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 1–12.
[22]
Dhruv Jain, Hung Ngo, Pratyush Patel, Steven Goodman, Leah Findlater, and Jon Froehlich. 2020. SoundWatch: Exploring Smartwatch-based Deep Learning Approaches to Support Sound Awareness for Deaf and Hard of Hearing Users. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Virtual Event Greece, 1–13.
[23]
W. Bradley Knox and Peter Stone. 2015. Framing reinforcement learning from human reward: Reward positivity, temporal discounting, episodicity, and performance. Artif. Intell. 225, (August 2015), 24–50.
[24]
Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of Explanatory Debugging to Personalize Interactive Machine Learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces, ACM, Atlanta Georgia USA, 126–137.
[25]
R. Shantha Selva Kumari, D. Sugumar, and V. Sadasivam. 2007. Audio Signal Classification Based on Optimal Wavelet and Support Vector Machine. In International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007), IEEE, Sivakasi, Tamil Nadu, India, 544–548.
[26]
Gierad Laput, Karan Ahuja, Mayank Goel, and Chris Harrison. 2018. Ubicoustics: Plug-and-Play Acoustic Activity Recognition. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, ACM, Berlin Germany, 213–224.
[27]
Jaewook Lee, Jaylin Herskovitz, Yi-Hao Peng, and Anhong Guo. 2022. ImageExplorer: Multi-Layered Touch Exploration to Encourage Skepticism Towards Imperfect AI-Generated Image Captions. In CHI Conference on Human Factors in Computing Systems, ACM, New Orleans LA USA, 1–15.
[28]
Hong Lu, Wei Pan, Nicholas D. Lane, Tanzeem Choudhury, and Andrew T. Campbell. 2009. SoundSense: scalable sound sensing for people-centric applications on mobile phones. In Proceedings of the 7th international conference on Mobile systems, applications, and services, ACM, Kraków Poland, 165–178.
[29]
Tara Matthews, Janette Fong, F. Wai-Ling Ho-Ching, and Jennifer Mankoff. 2006. Evaluating non-speech sound visualizations for the deaf. Behav. Inf. Technol. 25, 4 (July 2006), 333–351.
[30]
Tara Matthews, Janette Fong, and Jennifer Mankoff. 2005. Visualizing non-speech sounds for the deaf. In Proceedings of the 7th international ACM SIGACCESS conference on Computers and accessibility, ACM, Baltimore MD USA, 52–59.
[31]
Matthew S. Moore and Linda Levitan. 1992. For Hearing People Only: Answers to some of the most commonly asked questions about the deaf community, its culture, and the" deaf reality". Deaf Life Press.
[32]
Yuri Nakao and Yusuke Sugano. 2020. Use of Machine Learning by Non-Expert DHH People: Technological Understanding and Sound Perception. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, ACM, Tallinn Estonia, 1–12.
[33]
Halley Profita, Reem Albaghli, Leah Findlater, Paul Jaeger, and Shaun K. Kane. 2016. The AT Effect: How Disability Affects the Perceived Social Acceptability of Head-Mounted Display Use. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, San Jose California USA, 4884–4895.
[34]
Halley P. Profita, Abigale Stangl, Laura Matuszewska, Sigrunn Sky, and Shaun K. Kane. 2016. Nothing to Hide: Aesthetic Customization of Hearing Aids and Cochlear Implants in an Online Community. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Reno Nevada USA, 219–227.
[35]
Kristen Shinohara and Josh Tenenberg. 2009. A blind person's interactions with technology. Commun. ACM 52, 8 (August 2009), 58–66.
[36]
Kristen Shinohara and Jacob O. Wobbrock. 2011. In the shadow of misperception: assistive technology use and social interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, Vancouver BC Canada, 705–714.
[37]
Liu Sicong, Zhou Zimu, Du Junzhao, Shangguan Longfei, Jun Han, and Xin Wang. 2017. UbiEar: Bringing Location-independent Sound Awareness to the Hard-of-hearing People with Smartphones. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 2 (June 2017), 1–21.
[38]
M Tomitsch and T Grechenig. DESIGN IMPLICATIONS FOR A UBIQUITOUS AMBIENT SOUND DISPLAY FOR THE DEAF.
[39]
Joe Tullio, Anind K. Dey, Jason Chalecki, and James Fogarty. 2007. How It Works: A Field Study of Non-Technical Users Interacting with an Intelligent System. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07), Association for Computing Machinery, New York, NY, USA, 31–40.
[40]
Beatrice Vincenzi, Alex S. Taylor, and Simone Stumpf. 2021. Interdependence in Action: People with Visual Impairments and their Guides Co-constituting Common Spaces. Proc. ACM Hum.-Comput. Interact. 5, CSCW1 (April 2021), 1–33.
[41]
Jacob O. Wobbrock, Krzysztof Z. Gajos, Shaun K. Kane, and Gregg C. Vanderheiden. 2018. Ability-based design. Commun. ACM 61, 6 (May 2018), 62–71.
[42]
Jason Wu, Chris Harrison, Jeffrey P. Bigham, and Gierad Laput. 2020. Automated Class Discovery and One-Shot Interactions for Acoustic Activity Recognition. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 1–14.
[43]
Alina Zajadacz. 2015. Evolution of models of disability as a basis for further policy changes in accessible tourism. J. Tour. Futur. 1, 3 (September 2015), 189–202.
[44]
2011. Access Intimacy: The Missing Link. Leaving Evidence. Retrieved July 28, 2023 from https://leavingevidence.wordpress.com/2011/05/05/access-intimacy-the-missing-link/
[45]
2017. Access Intimacy, Interdependence and Disability Justice. Leaving Evidence. Retrieved July 28, 2023 from https://leavingevidence.wordpress.com/2017/04/12/access-intimacy-interdependence-and-disability-justice/
[46]
2020. Important household sounds become more accessible. Google. Retrieved May 3, 2023 from https://blog.google/products/android/new-sound-notifications-on-android/
[47]
People + AI Guidebook. Retrieved July 27, 2023 from https://design.google/ai-guidebook
[48]
Accessibility - Hearing. Apple. Retrieved May 3, 2023 from https://www.apple.com/accessibility/hearing/
[49]
TensorFlow Hub. Retrieved April 30, 2023 from https://tfhub.dev/google/lite-model/yamnet/tflite/1
[50]
Live Transcribe | Speech to Text App. Android. Retrieved May 3, 2023 from https://www.android.com/accessibility/live-transcribe/
[51]
Audio transcription for cloud recordings. Zoom Support. Retrieved May 3, 2023 from https://support.zoom.us/hc/en-us/articles/115004794983-Audio-transcription-for-cloud-recordings
[52]
ReSound Smart 3D hearing aid app | ReSound. Retrieved May 3, 2023 from https://www.resound.com/en-us/hearing-aids/apps/smart-3d
[53]
Real-time Call Caption App | Android & Iphone. InnoCaption. Retrieved May 3, 2023 from https://www.innocaption.com
[54]
Nest Aware. Google Store. Retrieved May 3, 2023 from https://store.google.com/us/product/nest_aware?hl=en-US
[55]
ReCal2: Reliability for 2 Coders – Deen Freelon, Ph.D. Retrieved July 31, 2023 from http://dfreelon.org/utils/recalfront/recal2/

Cited By

View all
  • (2024)Facilitating Joint Awareness Within Care Networks for Noise Sensitivity Management and RegulationCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3682042(17-20)Online publication date: 11-Nov-2024
  • (2024)Scaffolding Digital Literacy Through Digital Skills Training for Disabled People in the Global SouthProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675666(1-14)Online publication date: 27-Oct-2024
  • (2024)Misfitting With AI: How Blind People Verify and Contest AI ErrorsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675659(1-17)Online publication date: 27-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ASSETS '23: Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility
October 2023
1163 pages
ISBN:9798400702204
DOI:10.1145/3597638
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2023

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ASSETS '23
Sponsor:

Acceptance Rates

ASSETS '23 Paper Acceptance Rate 55 of 182 submissions, 30%;
Overall Acceptance Rate 436 of 1,556 submissions, 28%

Upcoming Conference

ASSETS '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)204
  • Downloads (Last 6 weeks)20
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Facilitating Joint Awareness Within Care Networks for Noise Sensitivity Management and RegulationCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3682042(17-20)Online publication date: 11-Nov-2024
  • (2024)Scaffolding Digital Literacy Through Digital Skills Training for Disabled People in the Global SouthProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675666(1-14)Online publication date: 27-Oct-2024
  • (2024)Misfitting With AI: How Blind People Verify and Contest AI ErrorsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675659(1-17)Online publication date: 27-Oct-2024
  • (2024)A Human-AI Collaborative Approach for Designing Sound Awareness SystemsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642062(1-11)Online publication date: 11-May-2024
  • (2024)Advancing Inclusive Beauty Experiences: A System for Communication Support for the Hearing Impaired in Hair Salon EnvironmentsComputers Helping People with Special Needs10.1007/978-3-031-62849-8_2(10-17)Online publication date: 8-Jul-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media