skip to main content
10.1145/3517428.3544817acmconferencesArticle/Chapter ViewAbstractPublication PagesassetsConference Proceedingsconference-collections
research-article

SoundVizVR: Sound Indicators for Accessible Sounds in Virtual Reality for Deaf or Hard-of-Hearing Users

Published: 22 October 2022 Publication History

Abstract

Sounds provide vital information such as spatial and interaction cues in virtual reality (VR) applications to convey more immersive experiences to VR users. However, it may be a challenge for deaf or hard-of-hearing (DHH) VR users to access the information given by sounds, which could limit their VR experience. To address this limitation, we present “SoundVizVR”, which explores visualizing sound characteristics and sound types for several types of sounds in VR experience. SoundVizVR uses Sound-Characteristic Indicators to visualize loudness, duration, and location of sound sources in VR and Sound-Type Indicators to present more information about the type of the sound. First, we examined three types of Sound-Characteristic Indicators (On-Object Indicators, Full Mini-Maps and Partial Mini-Maps) and their combinations in a study with 11 DHH participants. We identified that the combination of Full Mini-Map technique and On-Object Indicator was the most preferred visualization and performed best at locating sound sources in VR. Next, we explored presenting more information about the sounds using text and icons as Sound-Type Indicators. A second study with 14 DHH participants found that all Sound-Type Indicator combinations were successful at locating sound sources.

Supplementary Material

MP4 File (soundViZVR-ASSETS.mp4)
Supplementary material contains the video figure of the SoundVizVR submission.

References

[1]
Belén Agulló, Mario Montagud, and Isaac Fraile. 2019. Making interaction with virtual reality accessible: rendering and guiding methods for subtitles. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 33 (11 2019), 1–13. https://doi.org/10.1017/S0890060419000362
[2]
Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. J. Usability Studies 4, 3 (may 2009), 114–123.
[3]
Fl’vio Coutinho, Raquel O. Prates, and Luiz Chaimowicz. 2011. An Analysis of Information Conveyed through Audio in an FPS Game and Its Impact on Deaf Players Experience. In 2011 Brazilian Symposium on Games and Digital Entertainment. 53–62. https://doi.org/10.1109/SBGAMES.2011.16
[4]
Sigal Eden and Sara Ingber. 2014. Enhancing Storytelling Ability with Virtual Environment among Deaf and Hard-of-Hearing Children. In Computers Helping People with Special Needs, Klaus Miesenberger, Deborah Fels, Dominique Archambault, Petr Peňáz, and Wolfgang Zagler (Eds.). Springer International Publishing, Cham, 386–392.
[5]
3D Everything. 2021. Kids Room | 3D Interior | Unity Asset Store. Retrieved April 13, 2022 from https://assetstore.unity.com/packages/3d/props/interior/kids-room-48596
[6]
Facebook. 2022. Oculus Rift S: PC-Powered VR Gaming Headset | Oculus. Retrieved April 13, 2022 from https://www.oculus.com/rift-s/
[7]
Epic Games. 2022. Fortnite – a free-to-play Battle Royale game and more. Retrieved April 12, 2022 from https://www.epicgames.com/fortnite/en-US/home
[8]
William W. Gaver. 1987. Auditory Icons: Using Sound in Computer Interfaces. SIGCHI Bull. 19, 1 (jul 1987), 74. https://doi.org/10.1145/28189.1044809
[9]
Kathrin Gerling, Patrick Dickinson, Kieran Hicks, Liam Mason, Adalberto L. Simeone, and Katta Spiel. 2020. Virtual Reality Games for People Using Wheelchairs. Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3313831.3376265
[10]
Kathrin Gerling and Katta Spiel. 2021. A Critical Examination of Virtual Reality Technology in the Context of the Minority Body. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 599, 14 pages. https://doi.org/10.1145/3411764.3445196
[11]
Ru Guo, Yiru Yang, Johnson Kuang, Xue Bin, Dhruv Jain, Steven Goodman, Leah Findlater, and Jon Froehlich. 2020. HoloSound: Combining Speech and Sound Identification for Deaf or Hard of Hearing Users on a Head-Mounted Display. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery, New York, NY, USA, Article 71, 4 pages. https://doi.org/10.1145/3373625.3418031
[12]
S. G. Hart. 2006. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50(2006), 904 – 908.
[13]
Chris J. Hughes, Marta B. Zapata, Matthew Johnston, and Pilar Orero. 2020. Immersive Captioning: Developing a framework for evaluating user needs. In 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). 313–318. https://doi.org/10.1109/AIVR50618.2020.00063
[14]
Dhruv Jain, Bonnie Chinh, Leah Findlater, Raja Kushalnagar, and Jon Froehlich. 2018. Exploring Augmented Reality Approaches to Real-Time Captioning: A Preliminary Autoethnographic Study. In Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems (Hong Kong, China) (DIS ’18 Companion). Association for Computing Machinery, New York, NY, USA, 7–11. https://doi.org/10.1145/3197391.3205404
[15]
Dhruv Jain, Brendon Chiu, Steven Goodman, Chris Schmandt, Leah Findlater, and Jon E. Froehlich. 2020. Field Study of a Tactile Sound Awareness Device for Deaf Users. In Proceedings of the 2020 International Symposium on Wearable Computers (Virtual Event, Mexico) (ISWC ’20). Association for Computing Machinery, New York, NY, USA, 55–57. https://doi.org/10.1145/3410531.3414291
[16]
Dhruv Jain, Leah Findlater, Jamie Gilkeson, Benjamin Holland, Ramani Duraiswami, Dmitry Zotkin, Christian Vogler, and Jon E. Froehlich. 2015. Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 241–250. https://doi.org/10.1145/2702123.2702393
[17]
Dhruv Jain, Khoa Huynh Anh Nguyen, Steven M. Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon E. Froehlich. 2022. ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 305, 16 pages. https://doi.org/10.1145/3491102.3502020
[18]
Dhruv Jain, Sasa Junuzovic, Eyal Ofek, Mike Sinclair, John Porter, Chris Yoon, Swetha Machanavajhala, and Meredith Ringel Morris. 2021. A Taxonomy of Sounds in Virtual Reality. In Designing Interactive Systems Conference 2021(Virtual Event, USA) (DIS ’21). Association for Computing Machinery, New York, NY, USA, 160–170. https://doi.org/10.1145/3461778.3462106
[19]
Dhruv Jain, Sasa Junuzovic, Eyal Ofek, Mike Sinclair, John R. Porter, Chris Yoon, Swetha Machanavajhala, and Meredith Ringel Morris. 2021. Towards Sound Accessibility in Virtual Reality. In Proceedings of the 2021 International Conference on Multimodal Interaction (Montréal, QC, Canada) (ICMI ’21). Association for Computing Machinery, New York, NY, USA, 80–91. https://doi.org/10.1145/3462244.3479946
[20]
Dhruv Jain, Kelly Mack, Akli Amrous, Matt Wright, Steven Goodman, Leah Findlater, and Jon E. Froehlich. 2020. HomeSound: An Iterative Field Deployment of an In-Home Sound Awareness System for Deaf or Hard of Hearing Users. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376758
[21]
Peter Keller and Catherine Stevens. 2004. Meaning From Environmental Sounds: Types of Signal-Referent Relations and Their Effect on Recognizing Auditory Icons. Journal of experimental psychology. Applied 10 (04 2004), 3–12. https://doi.org/10.1037/1076-898X.10.1.3
[22]
Firesprite Limited. 2021. The Persistence - Games | PlayStation®. Retrieved April 12, 2022 from https://www.playstation.com/en-us/games/the-persistence/
[23]
Carme Mangiron. 2016. Reception of game subtitles: an empirical study. The Translator 22, 1 (2016), 72–93. https://doi.org/10.1080/13556509.2015.1110000 arXiv:https://doi.org/10.1080/13556509.2015.1110000
[24]
Tara Matthews, Janette Fong, F Wai-Ling Ho-Ching, and Jennifer Mankoff. 2006. Evaluating non-speech sound visualizations for the deaf. Behaviour & Information Technology 25, 4 (2006), 333–351.
[25]
Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321–1329.
[26]
Mohammadreza Mirzaei, Peter Kán, and Hannes Kaufmann. 2020. EarVR: Using Ear Haptics in Virtual Reality for Deaf and Hard-of-Hearing People. IEEE Transactions on Visualization and Computer Graphics 26, 5(2020), 2084–2093. https://doi.org/10.1109/TVCG.2020.2973441
[27]
Mojang. 2022. Minecraft Official Site | Minecraft. Retrieved April 13, 2022 from https://www.minecraft.net/en-us
[28]
Jorge Mori and Deborah I. Fels. 2009. Seeing the music can animated lyrics provide access to the emotional content in music for people who are deaf or hard of hearing?. In 2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC-STH). 951–956. https://doi.org/10.1109/TIC-STH.2009.5444362
[29]
Suranga Nanayakkara, Elizabeth Taylor, Lonce Wyse, and S H. Ong. 2009. An Enhanced Musical Experience for the Deaf: Design and Evaluation of a Music Display and a Haptic Chair. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for Computing Machinery, New York, NY, USA, 337–346. https://doi.org/10.1145/1518701.1518756
[30]
Suranga Chandima Nanayakkara, Lonce Wyse, S. H. Ong, and Elizabeth A. Taylor. 2013. Enhancing Musical Experience for the Hearing-Impaired Using Visual and Haptic Displays. Human–Computer Interaction 28, 2 (2013), 115–160. https://doi.org/10.1080/07370024.2012.697006 arXiv:https://www.tandfonline.com/doi/pdf/10.1080/07370024.2012.697006
[31]
De Xing Ong, Kai Xiang Chia, Yi Yi Huang, Jasper Teck Siong Teo, Jezamine Tan, Melissa Lim, Dongyu Qiu, Xinxing Xia, and Frank Yunqing Guan. 2021. Smart Captions: A Novel Solution for Closed Captioning in Theatre Settings with AR Glasses. In 2021 IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI). 1–5. https://doi.org/10.1109/SOLI54607.2021.9672391
[32]
Shanmugam Muruga Palaniappan, Ting Zhang, and Bradley S. Duerstock. 2019. Identifying Comfort Areas in 3D Space for Persons with Upper Extremity Mobility Impairments Using Virtual Reality. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 495–499. https://doi.org/10.1145/3308561.3353810
[33]
David Passig and Sigal Eden. 2001. Virtual reality as a tool for improving spatial rotation among deaf and hard-of-hearing children. CyberPsychology & Behavior 4, 6 (2001), 681–686.
[34]
Yi-Hao Peng, Ming-Wei Hsi, Paul Taele, Ting-Yu Lin, Po-En Lai, Leon Hsu, Tzu-chuan Chen, Te-Yen Wu, Yu-An Chen, Hsien-Hui Tang, and Mike Y. Chen. 2018. SpeechBubbles: Enhancing Captioning Experiences for Deaf and Hard-of-Hearing People in Group Conversations. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3173574.3173867
[35]
Sandra Poeschl, Konstantin Wall, and Nicola Doering. 2013. Integration of spatial sound in immersive virtual environments an experimental study on effects of spatial sound on presence. In 2013 IEEE Virtual Reality (VR). 129–130. https://doi.org/10.1109/VR.2013.6549396
[36]
Sylvia Rothe, Kim Tran, and Heinrich Hussmann. 2018. Positioning of Subtitles in Cinematic Virtual Reality. https://doi.org/10.2312/egve.20181307
[37]
Zhanna Sarsenbayeva, Niels van Berkel, Chu Luo, Vassilis Kostakos, and Jorge Goncalves. 2017. Challenges of Situational Impairments during Interaction with Mobile Devices. In Proceedings of the 29th Australian Conference on Computer-Human Interaction (Brisbane, Queensland, Australia) (OZCHI ’17). Association for Computing Machinery, New York, NY, USA, 477–481. https://doi.org/10.1145/3152771.3156161
[38]
Stefania Serafin and Giovanni Serafin. 2004. Sound Design to Enhance Presence in Photorealistic Virtual Reality. In ICAD.
[39]
Lichao Shen, MHD Yamen Saraiji, Kai Kunze, Kouta Minamizawa, and Roshan Lalintha Peiris. 2020. Visuomotor Influence of Attached Robotic Neck Augmentation. In Symposium on Spatial User Interaction (Virtual Event, Canada) (SUI ’20). Association for Computing Machinery, New York, NY, USA, Article 14, 10 pages. https://doi.org/10.1145/3385959.3418460
[40]
Alexa F. Siu, Mike Sinclair, Robert Kovacs, Eyal Ofek, Christian Holz, and Edward Cutrell. 2020. Virtual Reality Without Vision: A Haptic and Auditory White Cane to Navigate Complex Virtual Worlds. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376353
[41]
Sean M. Smith and Glen N. Williams. 1997. A Visualization of Music. In Proceedings of the 8th Conference on Visualization ’97 (Phoenix, Arizona, USA) (VIS ’97). IEEE Computer Society Press, Washington, DC, USA, 499–ff.
[42]
Axel Stockburger. 2003. The game environment from an auditive perspective. Level Up (2003), 4–6.
[43]
Valday Team. 2016. Vintage 50s Radio and Phone | 3D Electronics | Unity Asset Store. Retrieved April 13, 2022 from https://assetstore.unity.com/packages/3d/props/electronics/vintage-50s-radio-and-phone-71657
[44]
Mauro Teófilo, Alvaro Lourenço, Juliana Postal, and Vicente Lucena Jr. 2018. Exploring Virtual Reality to Enable Deaf or Hard of Hearing Accessibility in Live Theaters: A Case Study. 132–148. https://doi.org/10.1007/978-3-319-92052-8_11
[45]
vUv. 2021. Speakers PBR | 3D Electronics | Unity Asset Store. Retrieved April 13, 2022 from https://assetstore.unity.com/packages/3d/props/electronics/speakers-pbr-111606
[46]
Ryan Wedoff, Lindsay Ball, Amelia Wang, Yi Xuan Khoo, Lauren Lieberman, and Kyle Rector. 2019. Virtual Showdown: An Accessible Virtual Reality Game with Scaffolds for Youth with Visual Impairments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3290605.3300371
[47]
Minecraft Wiki. 2022. Subtitles – Minecraft Wiki. Retrieved April 12, 2022 from https://minecraft.fandom.com/wiki/Subtitles
[48]
Bei Yuan, Eelke Folmer, and Frederick C. Harris. 2011. Game Accessibility: A Survey. Univers. Access Inf. Soc. 10, 1 (mar 2011), 81–100. https://doi.org/10.1007/s10209-010-0189-5
[49]
Veronica Zammitto. 2008. VISUALIZATION TECHNIQUES IN VIDEO GAMES. https://doi.org/10.14236/ewic/EVA2008.30
[50]
Yuhang Zhao, Edward Cutrell, Christian Holz, Meredith Ringel Morris, Eyal Ofek, and Andrew D. Wilson. 2019. SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300341

Cited By

View all
  • (2024)Exploring the SoundVizVR Plugin in the Development of Sound-Accessible Virtual Reality Games: Insights from Game Developers and PlayersACM Transactions on Accessible Computing10.1145/369888217:4(1-20)Online publication date: 5-Oct-2024
  • (2024)SoundModVR: Sound Modifications in Virtual Reality for Sound AccessibilityAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686754(1-4)Online publication date: 13-Oct-2024
  • (2024)Supporting Sound Accessibility by Exploring Sound Augmentations in Virtual RealityProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688525(1-5)Online publication date: 27-Oct-2024
  • Show More Cited By

Index Terms

  1. SoundVizVR: Sound Indicators for Accessible Sounds in Virtual Reality for Deaf or Hard-of-Hearing Users

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ASSETS '22: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility
      October 2022
      902 pages
      ISBN:9781450392587
      DOI:10.1145/3517428
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 22 October 2022

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. accessibility
      2. audio visualization
      3. deaf and hard-of-hearing
      4. virtual reality

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Conference

      ASSETS '22
      Sponsor:

      Acceptance Rates

      ASSETS '22 Paper Acceptance Rate 35 of 132 submissions, 27%;
      Overall Acceptance Rate 436 of 1,556 submissions, 28%

      Upcoming Conference

      ASSETS '25

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)365
      • Downloads (Last 6 weeks)24
      Reflects downloads up to 15 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Exploring the SoundVizVR Plugin in the Development of Sound-Accessible Virtual Reality Games: Insights from Game Developers and PlayersACM Transactions on Accessible Computing10.1145/369888217:4(1-20)Online publication date: 5-Oct-2024
      • (2024)SoundModVR: Sound Modifications in Virtual Reality for Sound AccessibilityAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686754(1-4)Online publication date: 13-Oct-2024
      • (2024)Supporting Sound Accessibility by Exploring Sound Augmentations in Virtual RealityProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688525(1-5)Online publication date: 27-Oct-2024
      • (2024)SoundModVR: Sound Modifications in Virtual Reality to Support People who are Deaf and Hard of HearingProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675653(1-15)Online publication date: 27-Oct-2024
      • (2024)SoundHapticVR: Head-Based Spatial Haptic Feedback for Accessible Sounds in Virtual Reality for Deaf and Hard of Hearing UsersProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675639(1-17)Online publication date: 27-Oct-2024
      • (2024)SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality AwarenessProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661556(116-132)Online publication date: 1-Jul-2024
      • (2024)SocialCueSwitch: Towards Customizable Accessibility by Representing Social Cues in Multiple SensesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651109(1-7)Online publication date: 11-May-2024
      • (2024)Accessibility Feature Implementation Within Free VR ExperiencesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650935(1-9)Online publication date: 11-May-2024
      • (2024)Communication, Collaboration, and Coordination in a Co-located Shared Augmented Reality Game: Perspectives From Deaf and Hard of Hearing PeopleProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642953(1-14)Online publication date: 11-May-2024
      • (2024)Barriers to Photosensitive Accessibility in Virtual RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642635(1-13)Online publication date: 11-May-2024
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media