Skip to main content

A Personalized Captioning Strategy for the Deaf and Hard-of-Hearing Users in an Augmented Reality Environment

  • Conference paper
  • First Online:
Extended Reality (XR Salento 2024)

Abstract

Integrating Augmented Reality (AR) in learning environments has revolutionized traditional learning approaches by providing immersive and interactive experiences beyond conventional constraints. While several studies have used AR to enhance learning for individuals with disabilities, there is a significant gap in research focusing on generating captions within AR environments to optimize the experience for users who are deaf or hard of hearing (DHH). This study aims to investigate a personalized captioning strategy tailored to the unique needs of the DHH community. To achieve this, a scoping review of published articles in three databases was conducted to discover how captions are generated, where they are placed, and what attributes of captions are personalizable in AR spaces. Using the PRISMA methodology followed by an interview, the study identified nomenclature discrepancies in AR spaces for hand-held devices, proposing the terms “world-lock-view” and “screen-lock-view”. While analyzing captioning technology, the study identified a lack of customization options in Automatic Speech Recognition implementations. This is particularly significant for DHH users in AR spaces, who require control over font styles, background settings, and caption placement to enhance accessibility for DHH users within AR environments, contributing to a more inclusive and enriched learning experience. The study also discusses the limitations of the captioning strategy and proposes solutions to these limitations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Avila-Garzon, C., Bacca-Acosta, J., Duarte, J., Betancourt, J.: Augmented reality in education: an overview of twenty-five years of research. Contemp. Educ. Technol. 13 (2021)

    Google Scholar 

  2. Garzón, J., Acevedo, J.: Meta-analysis of the impact of Augmented Reality on students’ learning gains. Educ. Res. Rev. 27, 244–260 (2019)

    Google Scholar 

  3. Marrahí-Gómez, V., Belda-Medina, J.: The integration of augmented reality (AR) in education (2022). https://doi.org/10.14738/assrj.912.13689

  4. Garzón, J., Pavón, J., Baldiris, S.: Augmented reality applications for education: five directions for future research. In: De Paolis, L.T., Bourdot, P., and Mongelli, A. (eds.) Augmented Reality, Virtual Reality, and Computer Graphics, pp. 402–414. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60922-5_31

  5. Sungkur, R.K., Panchoo, A., Bhoyroo, N.K.: Augmented reality, the future of contextual mobile learning. Interact. Technol. Smart Educ. 13, 123–146 (2016). https://doi.org/10.1108/ITSE-07-2015-0017

    Google Scholar 

  6. Liono, R.A., Amanda, N., Pratiwi, A., Gunawan, A.A.S.: A systematic literature review: learning with visual by the help of augmented reality helps students learn better. Procedia Comput. Sci. 179, 144–152 (2021). https://doi.org/10.1016/j.procs.2020.12.019

    Google Scholar 

  7. Altmeyer, K., Kapp, S., Thees, M., Malone, S., Kuhn, J., Brünken, R.: The use of augmented reality to foster conceptual knowledge acquisition in STEM laboratory courses—theoretical background and empirical results. Br. J. Educ. Technol. 51, 611–628 (2020). https://doi.org/10.1111/bjet.12900

    Google Scholar 

  8. Guntur, M.I.S., Setyaningrum, W., Retnawati, H., Marsigit, M.: Assessing the potential of augmented reality in education. In: Proceedings of the 2020 11th International Conference on E-Education, E-Business, E-Management, and E-Learning, pp. 93–97. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3377571.3377621

  9. Samson, F., Shidende, D., Moebs, S.: Accessible augmented reality chemistry lab for students in developing countries. In: 2024 IST-Africa Conference (IST-Africa), pp. 1–10 (2024). https://doi.org/10.23919/IST-Africa63983.2024.10569711

  10. Guevara, C., Coronel, D.M.V.: Multisensory learning system applying augmented reality. In: Nazir, S., Ahram, T., and Karwowski, W. (eds.) Advances in Human Factors in Training, Education, and Learning Sciences, pp. 336–342. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50896-8_48

  11. Maboe, M.J., Eloff, M., Schoeman, M.: The role of accessibility and usability in bridging the digital divide for students with disabilities in an e-learning environment. In: Proceedings of the Annual Conference of the South African Institute of Computer Scientists and Information Technologists, pp. 222–228. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3278681.3278708

  12. Internation Standard Organisation: ISO 9241-210:2019 Ergonomics of human-system interaction Part 210: Human-centred design for interactive systems (2019). https://www.iso.org/standard/77520.html

  13. Rusu, C., Rusu, V., Roncagliolo, S., González, C.: Usability and user experience: what should we care about? Int. J. Inf. Technol. Syst. Approach IJITSA. 8, 1–12 (2015). https://doi.org/10.4018/IJITSA.2015070101

    Google Scholar 

  14. Howard, S., et al.: Visual inspection with augmented reality head-mounted display: an Australian usability case study. Hum. Factors Ergon. Manuf. Serv. Ind. 33, 272–296 (2023). https://doi.org/10.1002/hfm.20986

    Google Scholar 

  15. Ghazwani, Y., Smith, S.: Interaction in augmented reality: challenges to enhance user experience. In: Proceedings of the 2020 4th International Conference on Virtual and Augmented Reality Simulations, pp. 39–44. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3385378.3385384

  16. Ioannou, A., Constantinou, V.: Augmented reality supporting deaf students in mainstream schools: two case studies of practical utility of the technology. In: Auer, M.E. and Tsiatsos, T. (eds.) Interactive Mobile Communication Technologies and Learning, pp. 387–396. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75175-7_39

  17. Li, J.: Real-time augmented reality visual-captions for deaf and hard-of-hearing children in classrooms. In: 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 641–642 (2023). https://doi.org/10.1109/VRW58643.2023.00163

  18. Atanan, Y., Sombunsukho, S., Boonlue, S.: E-future classroom: a study mixed reality learning environment for deaf learners in thailand. Int. J. Environ. Sci. Educ. 12, 2291–2315 (2017)

    Google Scholar 

  19. Lee, G.-B., Jang, H., Jeong, H., Woo, W.: Designing a multi-modal communication system for the deaf and hard-of-hearing users. In: 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 429–434 (2021). https://doi.org/10.1109/ISMAR-Adjunct54149.2021.00097

  20. Luo, L., Weng, D., Songrui, G., Hao, J., Tu, Z.: Avatar interpreter: improving classroom experiences for deaf and hard-of-hearing people based on augmented reality. In: CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1–5. ACM, New Orleans (2022). https://doi.org/10.1145/3491101.3519799

  21. Peng, Y.-H., et al.: SpeechBubbles: enhancing captioning experiences for deaf and hard-of-hearing people in group conversations. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–10. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3173574.3173867

  22. Ridha, A.M., Shehieb, W.: Assistive technology for hearing-impaired and deaf students utilizing augmented reality. In: 2021 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), pp. 1–5. IEEE (2021)

    Google Scholar 

  23. McDonnell, E.: Understanding social and environmental factors to enable collective access approaches to the design of captioning technology. In: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–8. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3517428.3550417

  24. Simpson, E., Dalal, S., Semaan, B.: “Hey, can you add captions?”: the critical infrastructuring practices of neurodiverse people on tiktok. Proc. ACM Hum.-Comput. Interact. 7, 57:1–57:27 (2023). https://doi.org/10.1145/3579490

  25. Mahajan, D., Bhosale, S., Nighot, Y., Tayal, M.: A review of video captioning methods. Int. J. -Gener. Comput. 12 (2021)

    Google Scholar 

  26. Hrga, I., Ivašić-Kos, M.: Deep image captioning: an overview. In: 2019 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 995–1000 (2019). https://doi.org/10.23919/MIPRO.2019.8756821

  27. Westin, T., Neves, J., Mozelius, P., Sousa, C., Mantovan, L.: Inclusive AR-games for education of deaf children: challenges and opportunities. Eur. Conf. Games Based Learn. 16, 597–604 (2022). https://doi.org/10.34190/ecgbl.16.1.588

  28. Garzón, J.: An overview of twenty-five years of augmented reality in education. Multimodal Technol. Interact. 5, 37 (2021). https://doi.org/10.3390/mti5070037

    Google Scholar 

  29. Jain, D., Chinh, B., Findlater, L., Kushalnagar, R., Froehlich, J.: Exploring augmented reality approaches to real-time captioning: a preliminary autoethnographic study. In: Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems, pp. 7–11. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3197391.3205404

  30. Fernandes, N., Leite Junior, A.J.M., Marçal, E., Viana, W.: Augmented reality in education for people who are deaf or hard of hearing: a systematic literature review. Univ. Access Inf. Soc. (2023). https://doi.org/10.1007/s10209-023-00994-z

  31. Kawas, S., Karalis, G., Wen, T., Ladner, R.E.: Improving real-time captioning experiences for deaf and hard of hearing students. In: Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 15–23. Association for Computing Machinery, New York (2016). https://doi.org/10.1145/2982142.2982164

  32. Jain, D., Franz, R., Findlater, L., Cannon, J., Kushalnagar, R., Froehlich, J.: Towards accessible conversations in a mobile context for people who are deaf and hard of hearing. In: Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 81–92. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3234695.3236362

  33. Peters, M.D.J., Godfrey, C.M., Khalil, H., McInerney, P., Parker, D., Soares, C.B.: Guidance for conducting systematic scoping reviews. JBI Evid. Implement. 13, 141 (2015). https://doi.org/10.1097/XEB.0000000000000050

    Google Scholar 

  34. Chen, F., Li, X., Tang, J., Li, S., Wang, T.: A survey on recent advances in image captioning. J. Phys. Conf. Ser. 1914, 012053 (2021). https://doi.org/10.1088/1742-6596/1914/1/012053

    Google Scholar 

  35. Downey, G.J.: Closed Captioning: Subtitling, Stenography, and the Digital Convergence of Text with Television. JHU Press (2008)

    Google Scholar 

  36. Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., Cucchiara, R.: From show to tell: a survey on deep learning-based image captioning. IEEE Trans. Pattern Anal. Mach. Intell. 45, 539–559 (2022). https://doi.org/10.1109/TPAMI.2022.3148210

    Google Scholar 

  37. Eddin Za’ter, M., Talafha, B.: Bench-marking and improving Arabic automatic image captioning through the use of multi-task learning paradigm (2022). https://ui.adsabs.harvard.edu/abs/2022arXiv220205474E. https://doi.org/10.48550/arXiv.2202.05474

  38. Chen, S., Yao, T., Jiang, Y.-G.: Deep learning for video captioning: a review, 6283–6290 (2019)

    Google Scholar 

  39. Li, S., Tao, Z., Li, K., Fu, Y.: Visual to text: survey of image and video captioning. IEEE Trans. Emerg. Top. Comput. Intell. 3, 297–312 (2019). https://doi.org/10.1109/TETCI.2019.2892755

    Google Scholar 

  40. Wang, X., Chen, W., Wu, J., Wang, Y.-F., Wang, W.Y.: Video captioning via hierarchical reinforcement learning. In: Presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Google Scholar 

  41. Gupta, N., Jalal, A.S.: Integration of textual cues for fine-grained image captioning using deep CNN and LSTM. Neural Comput. Appl. 32, 17899–17908 (2020). https://doi.org/10.1007/s00521-019-04515-z

    Google Scholar 

  42. Yang, B., Zhang, T., Zou, Y.: CLIP meets video captioning: concept-aware representation learning does matter (2022). http://arxiv.org/abs/2111.15162. https://doi.org/10.48550/arXiv.2111.15162

  43. Chen, C., et al.: Towards better caption supervision for object detection. IEEE Trans. Vis. Comput. Graph. 28, 1941–1954 (2022). https://doi.org/10.1109/TVCG.2021.3138933

    Google Scholar 

  44. Berger, A., Kostak, M., Maly, F.: Mobile AR solution for deaf people. In: Awan, I., Younas, M., Ünal, P., and Aleksy, M. (eds.) Mobile Web and Intelligent Information Systems, pp. 243–254. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27192-3_19

  45. Mathew, R., Mak, B., Dannels, W.: Access on demand: real-time, multi-modal accessibility for the deaf and hard-of-hearing based on augmented reality. In: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–6. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3517428.3551352

  46. Eksvärd, S., Falk, J.: Evaluating Speech-to-Text Systems and AR-glasses : a study to develop a potential assistive device for people with hearing impairments (2021). https://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-437608

  47. Mirzaei, M.R., Ghorshi, S., Mortazavi, M.: Combining augmented reality and speech technologies to help deaf and hard of hearing people. In: 2012 14th Symposium on Virtual and Augmented Reality, pp. 174–181 (2012). https://doi.org/10.1109/SVR.2012.10

  48. Ghasemi, Y., Singh, A., Kim, M., Johnson, A., Jeong, H.: Effects of head-locked augmented reality on user’s performance and perceived workload. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 65, 1094–1098 (2021). https://doi.org/10.1177/1071181321651169

  49. Brown, A., Turner, J., Patterson, J., Schmitz, A., Armstrong, M., Glancy, M.: Exploring subtitle behaviour for 360 video. White Pap. WHP. 330 (2018)

    Google Scholar 

  50. Brescia-Zapata, M., Krejtz, K., Duchowski, A.T., Hughes, C.J., Orero, P.: Subtitles in VR 360° video. results from an eye-tracking experiment. In: Perspectives (Montclair), pp. 1–23 (2023). https://doi.org/10.1080/0907676X.2023.2268122

  51. Hughes, C.J., Zapata, M.B., Johnston, M., Orero, P.: Immersive captioning: developing a framework for evaluating user needs. In: 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), pp. 313–318 (2020). https://doi.org/10.1109/AIVR50618.2020.00063

  52. Munn, Z., Peters, M.D.J., Stern, C., Tufanaru, C., McArthur, A., Aromataris, E.: Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med. Res. Methodol. 18, 143 (2018). https://doi.org/10.1186/s12874-018-0611-x

    Google Scholar 

  53. Mak, S., Thomas, A.: Steps for conducting a scoping review. J. Grad. Med. Educ. 14, 565–567 (2022). https://doi.org/10.4300/JGME-D-22-00621.1

    Google Scholar 

  54. Page, M.J., et al.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372, n71 (2021). https://doi.org/10.1136/bmj.n71

    Google Scholar 

  55. Mathew, R., Dannels, W.A., Parker, A.J.: An augmented reality based approach for optimization of language access services in healthcare for deaf patients. In: Antona, M., Stephanidis, C. (eds.) Universal Access in Human-Computer Interaction, pp. 29–52. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-35897-5_3

  56. Chen, H., et al.: Augmented reality, deep learning and vision-language query system for construction worker safety. Autom. Constr. 157, 105158 (2024). https://doi.org/10.1016/j.autcon.2023.105158

    Google Scholar 

  57. Kurahashi, T., Sakuma, R., Zempo, K., Mizutani, K., Wakatsuki, N.: Retrospective speech balloons on speech-visible AR via head-mounted display. In: 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 423–424 (2018). https://doi.org/10.1109/ISMAR-Adjunct.2018.00127

  58. Schipper, C., Brinkman, B.: Caption placement on an augmented reality head worn device. In: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 365–366. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3132525.3134786

  59. Ong, D.X., et al.: Smart captions: a novel solution for closed captioning in theatre settings with AR glasses. In: 2021 IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI), pp. 1–5 (2021). https://doi.org/10.1109/SOLI54607.2021.9672391

  60. Dabran, I., Avny, T., Singher, E., Ben Danan, H.: Augmented reality speech recognition for the hearing impaired. In: 2017 IEEE International Conference on Microwaves, Antennas, Communications and Electronic Systems (COMCAS), pp. 1–4 (2017). https://doi.org/10.1109/COMCAS.2017.8244731

  61. Aljowaysir, N., Ozdemir, T.O., Kim, T.: Differentiated learning patterns with mixed reality. In: 2019 IEEE Games, Entertainment, Media Conference (GEM), pp. 1–4 (2019). https://doi.org/10.1109/GEM.2019.8811558

  62. Sabie, D., Sheta, H., Ferdous, H.S., Kopalakrishnan, V., Ahmed, S.I.: Be our guest: intercultural heritage exchange through augmented reality (AR). In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–15. Association for Computing Machinery, New York (2023). https://doi.org/10.1145/3544548.3581005

  63. Wang, Y., Lualdi, C.P., Angrave, L., Purushotam, G.N.: Using deep learning and augmented reality to improve accessibility: inclusive conversations using diarization, captions, and visualization. In: Presented at the 2023 ASEE Annual Conference & Exposition, 25 June 2023 (2023)

    Google Scholar 

  64. InformedHealth.org: Hearing loss and deafness: Normal hearing and impaired hearing. In: InformedHealth.org Independence, Evidence-based. Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany (2017)

    Google Scholar 

  65. Diaz, C.L., Arellano, M.R.M., Rosillo, V.M.L., Ochoa, A.: Augmented reality system to promote the inclusion of deaf people in smart cities. Res Comput Sci. 147, 49–64 (2018)

    Google Scholar 

  66. Shidende, D., Kessel, T., Moebs, S.: Towards accessible augmented reality learning authoring tool: a case of MirageXR. In: 2023 IST-Africa Conference (IST-Africa), pp. 1–13 (2023). https://doi.org/10.23919/IST-Africa60249.2023.10187746

  67. Tyler, M.D., Jones, C., Grebennikov, L., Leigh, G., Noble, W., Burnham, D.: Effect of caption rate on the comprehension of educational television programmes by deaf school students. Deaf. Educ. Int. 11, 152–162 (2009). https://doi.org/10.1002/dei.262

    Google Scholar 

  68. Berke, L., Caulfield, C., Huenerfauth, M.: Deaf and hard-of-hearing perspectives on imperfect automatic speech recognition for captioning one-on-one meetings. In: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 155–164. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3132525.3132541

  69. Kushalnagar, R.S., Lasecki, W.S., Bigham, J.P.: Accessibility evaluation of classroom captions. ACM Trans. Access. Comput. 5, 7:1–7:24 (2014). https://doi.org/10.1145/2543578

  70. Prud’hommeaux, E., Jimerson, R., Hatcher, R., Michelson, K.: Automatic speech recognition for supporting endangered language documentation (2021)

    Google Scholar 

  71. Li, F.M., Lu, C., Lu, Z., Carrington, P., Truong, K.N.: An exploration of captioning practices and challenges of individual content creators on youtube for people with hearing impairments. Proc. ACM Hum.-Comput. Interact. 6, 75:1–75:26 (2022). https://doi.org/10.1145/3512922

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Deogratias Shidende .

Editor information

Editors and Affiliations

Appendix

Appendix

Appendix I. List of Articles Included in the Analysis

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shidende, D., Kessel, T., Treydte, A., Moebs, S. (2024). A Personalized Captioning Strategy for the Deaf and Hard-of-Hearing Users in an Augmented Reality Environment. In: De Paolis, L.T., Arpaia, P., Sacco, M. (eds) Extended Reality. XR Salento 2024. Lecture Notes in Computer Science, vol 15028. Springer, Cham. https://doi.org/10.1007/978-3-031-71704-8_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-71704-8_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-71703-1

  • Online ISBN: 978-3-031-71704-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics