Jeremy Zhengqi Huang
ABSTRACT
While recent advances have enabled mobile sound recognition tools for deaf and hard of hearing (DHH) people, these tools have only been studied in the lab or through short, controlled experiments. To assess the real-world feasibility and guide the future designs of mobile sound awareness systems, we conducted a three-week field study of SoundWatch, a smartwatch-based sound recognition app, with 10 DHH participants. Our findings suggest the app's utility in increasing environmental awareness and facilitating everyday tasks for DHH users. However, several challenges, such as background noises, variability of real-world sounds, and confusion among similar sounding sounds, indicated that mobile sound recognition solutions are “not there yet” for adoption and use in daily life. We close by presenting HCI design opportunities to improve model reliability by increasing contextual awareness, supporting end-user customization, and fostering the collective improvement of sound recognition models.
- Taslima Akter, Tousif Ahmed, Apu Kapadia, and Swami Manohar Swaminathan. 2020. Privacy Considerations of the Visually Impaired with Camera Based Assistive Technologies: Misrepresentation, Impropriety, and Fairness. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Virtual Event Greece, 1–14. DOI:https://doi.org/10.1145/3373625.3417003Google Scholar
Digital Library
- Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland Uk, 1–13. DOI:https://doi.org/10.1145/3290605.3300233Google ScholarDigital Library
- Mirza Mansoor Baig, Shereen Afifi, Hamid GholamHosseini, and Farhaan Mirza. 2019. A Systematic Review of Wearable Sensors and IoT-Based Monitoring Applications for Older Adults – a Focus on Ageing Population and Independent Living. J. Med. Syst. 43, 8 (August 2019), 233. DOI:https://doi.org/10.1007/s10916-019-1365-7Google ScholarDigital Library
- Cynthia L. Bennett, Erin Brady, and Stacy M. Branham. 2018. Interdependence as a Frame for Assistive Technology Research and Design. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Galway Ireland, 161–173. DOI:https://doi.org/10.1145/3234695.3236348Google ScholarDigital Library
- Danielle Bragg, Nicholas Huynh, and Richard E. Ladner. 2016. A Personalizable Mobile Sound Detector App Design for Deaf and Hard-of-Hearing Users. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Reno Nevada USA, 3–13. DOI:https://doi.org/10.1145/2982142.2982171Google ScholarDigital Library
- Stacy M. Branham and Shaun K. Kane. 2015. The Invisible Work of Accessibility: How Blind Employees Manage Accessibility in Mixed-Ability Workplaces. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility - ASSETS ’15, ACM Press, Lisbon, Portugal, 163–171. DOI:https://doi.org/10.1145/2700648.2809864Google ScholarDigital Library
- Virginia Braun and Victoria Clarke. 2021. Thematic Analysis: A Practical Guide. SAGE Publications.Google Scholar
- Anna Cavender and Richard E. Ladner. 2008. Hearing Impairments. In Web Accessibility, Simon Harper and Yeliz Yesilada (eds.). Springer London, London, 25–35. DOI:https://doi.org/10.1007/978-1-84800-050-6_3Google ScholarCross Ref
- Leah Findlater, Bonnie Chinh, Dhruv Jain, Jon Froehlich, Raja Kushalnagar, and Angela Carey Lin. 2019. Deaf and Hard-of-hearing Individuals’ Preferences for Wearable and Mobile Sound Awareness Technologies. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland Uk, 1–13. DOI:https://doi.org/10.1145/3290605.3300276Google ScholarDigital Library
- Steven Goodman, Susanne Kirchner, Rose Guttman, Dhruv Jain, Jon Froehlich, and Leah Findlater. 2020. Evaluating Smartwatch-based Sound Feedback for Deaf and Hard-of-hearing Users Across Contexts. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 1–13. DOI:https://doi.org/10.1145/3313831.3376406Google ScholarDigital Library
- Steven M. Goodman, Ping Liu, Dhruv Jain, Emma J. McDonnell, Jon E. Froehlich, and Leah Findlater. 2021. Toward User-Driven Sound Recognizer Personalization with People Who Are d/Deaf or Hard of Hearing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 2 (June 2021), 1–23. DOI:https://doi.org/10.1145/3463501Google ScholarDigital Library
- Benjamin M. Gorman. 2014. VisAural:: a wearable sound-localisation device for people with impaired hearing. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility - ASSETS ’14, ACM Press, Rochester, New York, USA, 337–338. DOI:https://doi.org/10.1145/2661334.2661410Google ScholarDigital Library
- Fabien Gouyon, François Pachet, and Olivier Delerue. 2000. ON THE USE OF ZERO-CROSSING RATE FOR AN APPLICATION OF CLASSIFICATION OF PERCUSSIVE SOUNDS. (2000).Google Scholar
- Ru Guo, Yiru Yang, Johnson Kuang, Xue Bin, Dhruv Jain, Steven Goodman, Leah Findlater, and Jon Froehlich. 2020. HoloSound: Combining Speech and Sound Identification for Deaf or Hard of Hearing Users on a Head-mounted Display. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Virtual Event Greece, 1–4. DOI:https://doi.org/10.1145/3373625.3418031Google ScholarDigital Library
- Guojun Lu and T. Hankinson. 2000. An investigation of automatic audio classification and segmentation. In WCC 2000 - ICSP 2000. 2000 5th International Conference on Signal Processing Proceedings. 16th World Computer Congress 2000, IEEE, Beijing, China, 776–781. DOI:https://doi.org/10.1109/ICOSP.2000.891627Google ScholarCross Ref
- Foad Hamidi, Kellie Poneres, Aaron Massey, and Amy Hurst. 2018. Who Should Have Access to my Pointing Data?: Privacy Tradeoffs of Adaptive Assistive Technologies. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Galway Ireland, 203–216. DOI:https://doi.org/10.1145/3234695.3239331Google ScholarDigital Library
- Foad Hamidi, Kellie Poneres, Aaron Massey, and Amy Hurst. 2020. Using a participatory activities toolkit to elicit privacy expectations of adaptive assistive technologies. In Proceedings of the 17th International Web for All Conference, ACM, Taipei Taiwan, 1–12. DOI:https://doi.org/10.1145/3371300.3383336Google ScholarDigital Library
- F Wai-ling Ho-Ching, Jennifer Mankoff, and James A Landay. Can you see what I hear? The Design and Evaluation of a Peripheral Sound Display for the Deaf.Google Scholar
- Yasha Iravantchi, Karan Ahuja, Mayank Goel, Chris Harrison, and Alanson Sample. 2021. PrivacyMic: Utilizing Inaudible Frequencies for Privacy Preserving Daily Activity Recognition. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, ACM, Yokohama Japan, 1–13. DOI:https://doi.org/10.1145/3411764.3445169Google ScholarDigital Library
- Dhruv Jain, Khoa Huynh Anh Nguyen, Steven M. Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon E. Froehlich. 2022. ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users. In CHI Conference on Human Factors in Computing Systems, ACM, New Orleans LA USA, 1–16. DOI:https://doi.org/10.1145/3491102.3502020Google ScholarDigital Library
- Dhruv Jain, Kelly Mack, Akli Amrous, Matt Wright, Steven Goodman, Leah Findlater, and Jon E. Froehlich. 2020. HomeSound: An Iterative Field Deployment of an In-Home Sound Awareness System for Deaf or Hard of Hearing Users. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 1–12. DOI:https://doi.org/10.1145/3313831.3376758Google ScholarDigital Library
- Dhruv Jain, Hung Ngo, Pratyush Patel, Steven Goodman, Leah Findlater, and Jon Froehlich. 2020. SoundWatch: Exploring Smartwatch-based Deep Learning Approaches to Support Sound Awareness for Deaf and Hard of Hearing Users. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Virtual Event Greece, 1–13. DOI:https://doi.org/10.1145/3373625.3416991Google ScholarDigital Library
- W. Bradley Knox and Peter Stone. 2015. Framing reinforcement learning from human reward: Reward positivity, temporal discounting, episodicity, and performance. Artif. Intell. 225, (August 2015), 24–50. DOI:https://doi.org/10.1016/j.artint.2015.03.009Google ScholarDigital Library
- Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of Explanatory Debugging to Personalize Interactive Machine Learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces, ACM, Atlanta Georgia USA, 126–137. DOI:https://doi.org/10.1145/2678025.2701399Google ScholarDigital Library
- R. Shantha Selva Kumari, D. Sugumar, and V. Sadasivam. 2007. Audio Signal Classification Based on Optimal Wavelet and Support Vector Machine. In International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007), IEEE, Sivakasi, Tamil Nadu, India, 544–548. DOI:https://doi.org/10.1109/ICCIMA.2007.370Google ScholarDigital Library
- Gierad Laput, Karan Ahuja, Mayank Goel, and Chris Harrison. 2018. Ubicoustics: Plug-and-Play Acoustic Activity Recognition. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, ACM, Berlin Germany, 213–224. DOI:https://doi.org/10.1145/3242587.3242609Google ScholarDigital Library
- Jaewook Lee, Jaylin Herskovitz, Yi-Hao Peng, and Anhong Guo. 2022. ImageExplorer: Multi-Layered Touch Exploration to Encourage Skepticism Towards Imperfect AI-Generated Image Captions. In CHI Conference on Human Factors in Computing Systems, ACM, New Orleans LA USA, 1–15. DOI:https://doi.org/10.1145/3491102.3501966Google ScholarDigital Library
- Hong Lu, Wei Pan, Nicholas D. Lane, Tanzeem Choudhury, and Andrew T. Campbell. 2009. SoundSense: scalable sound sensing for people-centric applications on mobile phones. In Proceedings of the 7th international conference on Mobile systems, applications, and services, ACM, Kraków Poland, 165–178. DOI:https://doi.org/10.1145/1555816.1555834Google ScholarDigital Library
- Tara Matthews, Janette Fong, F. Wai-Ling Ho-Ching, and Jennifer Mankoff. 2006. Evaluating non-speech sound visualizations for the deaf. Behav. Inf. Technol. 25, 4 (July 2006), 333–351. DOI:https://doi.org/10.1080/01449290600636488Google ScholarCross Ref
- Tara Matthews, Janette Fong, and Jennifer Mankoff. 2005. Visualizing non-speech sounds for the deaf. In Proceedings of the 7th international ACM SIGACCESS conference on Computers and accessibility, ACM, Baltimore MD USA, 52–59. DOI:https://doi.org/10.1145/1090785.1090797Google ScholarDigital Library
- Matthew S. Moore and Linda Levitan. 1992. For Hearing People Only: Answers to some of the most commonly asked questions about the deaf community, its culture, and the" deaf reality". Deaf Life Press.Google Scholar
- Yuri Nakao and Yusuke Sugano. 2020. Use of Machine Learning by Non-Expert DHH People: Technological Understanding and Sound Perception. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, ACM, Tallinn Estonia, 1–12. DOI:https://doi.org/10.1145/3419249.3420157Google ScholarDigital Library
- Halley Profita, Reem Albaghli, Leah Findlater, Paul Jaeger, and Shaun K. Kane. 2016. The AT Effect: How Disability Affects the Perceived Social Acceptability of Head-Mounted Display Use. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, San Jose California USA, 4884–4895. DOI:https://doi.org/10.1145/2858036.2858130Google ScholarDigital Library
- Halley P. Profita, Abigale Stangl, Laura Matuszewska, Sigrunn Sky, and Shaun K. Kane. 2016. Nothing to Hide: Aesthetic Customization of Hearing Aids and Cochlear Implants in an Online Community. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Reno Nevada USA, 219–227. DOI:https://doi.org/10.1145/2982142.2982159Google ScholarDigital Library
- Kristen Shinohara and Josh Tenenberg. 2009. A blind person's interactions with technology. Commun. ACM 52, 8 (August 2009), 58–66. DOI:https://doi.org/10.1145/1536616.1536636Google ScholarDigital Library
- Kristen Shinohara and Jacob O. Wobbrock. 2011. In the shadow of misperception: assistive technology use and social interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, Vancouver BC Canada, 705–714. DOI:https://doi.org/10.1145/1978942.1979044Google ScholarDigital Library
- Liu Sicong, Zhou Zimu, Du Junzhao, Shangguan Longfei, Jun Han, and Xin Wang. 2017. UbiEar: Bringing Location-independent Sound Awareness to the Hard-of-hearing People with Smartphones. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 2 (June 2017), 1–21. DOI:https://doi.org/10.1145/3090082Google ScholarDigital Library
- M Tomitsch and T Grechenig. DESIGN IMPLICATIONS FOR A UBIQUITOUS AMBIENT SOUND DISPLAY FOR THE DEAF.Google Scholar
- Joe Tullio, Anind K. Dey, Jason Chalecki, and James Fogarty. 2007. How It Works: A Field Study of Non-Technical Users Interacting with an Intelligent System. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07), Association for Computing Machinery, New York, NY, USA, 31–40. DOI:https://doi.org/10.1145/1240624.1240630Google ScholarDigital Library
- Beatrice Vincenzi, Alex S. Taylor, and Simone Stumpf. 2021. Interdependence in Action: People with Visual Impairments and their Guides Co-constituting Common Spaces. Proc. ACM Hum.-Comput. Interact. 5, CSCW1 (April 2021), 1–33. DOI:https://doi.org/10.1145/3449143Google ScholarDigital Library
- Jacob O. Wobbrock, Krzysztof Z. Gajos, Shaun K. Kane, and Gregg C. Vanderheiden. 2018. Ability-based design. Commun. ACM 61, 6 (May 2018), 62–71. DOI:https://doi.org/10.1145/3148051Google ScholarDigital Library
- Jason Wu, Chris Harrison, Jeffrey P. Bigham, and Gierad Laput. 2020. Automated Class Discovery and One-Shot Interactions for Acoustic Activity Recognition. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 1–14. DOI:https://doi.org/10.1145/3313831.3376875Google ScholarDigital Library
- Alina Zajadacz. 2015. Evolution of models of disability as a basis for further policy changes in accessible tourism. J. Tour. Futur. 1, 3 (September 2015), 189–202. DOI:https://doi.org/10.1108/JTF-04-2015-0015Google ScholarCross Ref
- 2011. Access Intimacy: The Missing Link. Leaving Evidence. Retrieved July 28, 2023 from https://leavingevidence.wordpress.com/2011/05/05/access-intimacy-the-missing-link/Google Scholar
- 2017. Access Intimacy, Interdependence and Disability Justice. Leaving Evidence. Retrieved July 28, 2023 from https://leavingevidence.wordpress.com/2017/04/12/access-intimacy-interdependence-and-disability-justice/Google Scholar
- 2020. Important household sounds become more accessible. Google. Retrieved May 3, 2023 from https://blog.google/products/android/new-sound-notifications-on-android/Google Scholar
- People + AI Guidebook. Retrieved July 27, 2023 from https://design.google/ai-guidebookGoogle Scholar
- Accessibility - Hearing. Apple. Retrieved May 3, 2023 from https://www.apple.com/accessibility/hearing/Google Scholar
- TensorFlow Hub. Retrieved April 30, 2023 from https://tfhub.dev/google/lite-model/yamnet/tflite/1Google Scholar
- Live Transcribe | Speech to Text App. Android. Retrieved May 3, 2023 from https://www.android.com/accessibility/live-transcribe/Google Scholar
- Audio transcription for cloud recordings. Zoom Support. Retrieved May 3, 2023 from https://support.zoom.us/hc/en-us/articles/115004794983-Audio-transcription-for-cloud-recordingsGoogle Scholar
- ReSound Smart 3D hearing aid app | ReSound. Retrieved May 3, 2023 from https://www.resound.com/en-us/hearing-aids/apps/smart-3dGoogle Scholar
- Real-time Call Caption App | Android & Iphone. InnoCaption. Retrieved May 3, 2023 from https://www.innocaption.comGoogle Scholar
- Nest Aware. Google Store. Retrieved May 3, 2023 from https://store.google.com/us/product/nest_aware?hl=en-USGoogle Scholar
- ReCal2: Reliability for 2 Coders – Deen Freelon, Ph.D. Retrieved July 31, 2023 from http://dfreelon.org/utils/recalfront/recal2/Google Scholar
Index Terms
- “Not There Yet”: Feasibility and Challenges of Mobile Sound Recognition to Support Deaf and Hard-of-Hearing People
Index terms have been assigned to the content through auto-classification.
Recommendations
ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing SystemsRecent advances have enabled automatic sound recognition systems for deaf and hard of hearing (DHH) users on mobile devices. However, these tools use pre-trained, generic sound recognition models, which do not meet the diverse needs of DHH users. We ...
Deaf and Hard-of-hearing Individuals' Preferences for Wearable and Mobile Sound Awareness Technologies
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsTo investigate preferences for mobile and wearable sound awareness systems, we conducted an online survey with 201 DHH participants. The survey explores how demographic factors affect perceptions of sound awareness technologies, gauges interest in ...
How people who are deaf, Deaf, and hard of hearing use technology in creative sound activities
ASSETS '22: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and AccessibilityCreative sound activities, such as music playing and audio engineering, are said to have been democratized with the development of technology. Yet, the use of technology in creative sound activities by people who are deaf, Deaf, and hard of hearing (DHH)...
Comments