Abstract
Integrating Augmented Reality (AR) in learning environments has revolutionized traditional learning approaches by providing immersive and interactive experiences beyond conventional constraints. While several studies have used AR to enhance learning for individuals with disabilities, there is a significant gap in research focusing on generating captions within AR environments to optimize the experience for users who are deaf or hard of hearing (DHH). This study aims to investigate a personalized captioning strategy tailored to the unique needs of the DHH community. To achieve this, a scoping review of published articles in three databases was conducted to discover how captions are generated, where they are placed, and what attributes of captions are personalizable in AR spaces. Using the PRISMA methodology followed by an interview, the study identified nomenclature discrepancies in AR spaces for hand-held devices, proposing the terms “world-lock-view” and “screen-lock-view”. While analyzing captioning technology, the study identified a lack of customization options in Automatic Speech Recognition implementations. This is particularly significant for DHH users in AR spaces, who require control over font styles, background settings, and caption placement to enhance accessibility for DHH users within AR environments, contributing to a more inclusive and enriched learning experience. The study also discusses the limitations of the captioning strategy and proposes solutions to these limitations.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Avila-Garzon, C., Bacca-Acosta, J., Duarte, J., Betancourt, J.: Augmented reality in education: an overview of twenty-five years of research. Contemp. Educ. Technol. 13 (2021)
Garzón, J., Acevedo, J.: Meta-analysis of the impact of Augmented Reality on students’ learning gains. Educ. Res. Rev. 27, 244–260 (2019)
Marrahí-Gómez, V., Belda-Medina, J.: The integration of augmented reality (AR) in education (2022). https://doi.org/10.14738/assrj.912.13689
Garzón, J., Pavón, J., Baldiris, S.: Augmented reality applications for education: five directions for future research. In: De Paolis, L.T., Bourdot, P., and Mongelli, A. (eds.) Augmented Reality, Virtual Reality, and Computer Graphics, pp. 402–414. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60922-5_31
Sungkur, R.K., Panchoo, A., Bhoyroo, N.K.: Augmented reality, the future of contextual mobile learning. Interact. Technol. Smart Educ. 13, 123–146 (2016). https://doi.org/10.1108/ITSE-07-2015-0017
Liono, R.A., Amanda, N., Pratiwi, A., Gunawan, A.A.S.: A systematic literature review: learning with visual by the help of augmented reality helps students learn better. Procedia Comput. Sci. 179, 144–152 (2021). https://doi.org/10.1016/j.procs.2020.12.019
Altmeyer, K., Kapp, S., Thees, M., Malone, S., Kuhn, J., Brünken, R.: The use of augmented reality to foster conceptual knowledge acquisition in STEM laboratory courses—theoretical background and empirical results. Br. J. Educ. Technol. 51, 611–628 (2020). https://doi.org/10.1111/bjet.12900
Guntur, M.I.S., Setyaningrum, W., Retnawati, H., Marsigit, M.: Assessing the potential of augmented reality in education. In: Proceedings of the 2020 11th International Conference on E-Education, E-Business, E-Management, and E-Learning, pp. 93–97. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3377571.3377621
Samson, F., Shidende, D., Moebs, S.: Accessible augmented reality chemistry lab for students in developing countries. In: 2024 IST-Africa Conference (IST-Africa), pp. 1–10 (2024). https://doi.org/10.23919/IST-Africa63983.2024.10569711
Guevara, C., Coronel, D.M.V.: Multisensory learning system applying augmented reality. In: Nazir, S., Ahram, T., and Karwowski, W. (eds.) Advances in Human Factors in Training, Education, and Learning Sciences, pp. 336–342. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50896-8_48
Maboe, M.J., Eloff, M., Schoeman, M.: The role of accessibility and usability in bridging the digital divide for students with disabilities in an e-learning environment. In: Proceedings of the Annual Conference of the South African Institute of Computer Scientists and Information Technologists, pp. 222–228. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3278681.3278708
Internation Standard Organisation: ISO 9241-210:2019 Ergonomics of human-system interaction Part 210: Human-centred design for interactive systems (2019). https://www.iso.org/standard/77520.html
Rusu, C., Rusu, V., Roncagliolo, S., González, C.: Usability and user experience: what should we care about? Int. J. Inf. Technol. Syst. Approach IJITSA. 8, 1–12 (2015). https://doi.org/10.4018/IJITSA.2015070101
Howard, S., et al.: Visual inspection with augmented reality head-mounted display: an Australian usability case study. Hum. Factors Ergon. Manuf. Serv. Ind. 33, 272–296 (2023). https://doi.org/10.1002/hfm.20986
Ghazwani, Y., Smith, S.: Interaction in augmented reality: challenges to enhance user experience. In: Proceedings of the 2020 4th International Conference on Virtual and Augmented Reality Simulations, pp. 39–44. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3385378.3385384
Ioannou, A., Constantinou, V.: Augmented reality supporting deaf students in mainstream schools: two case studies of practical utility of the technology. In: Auer, M.E. and Tsiatsos, T. (eds.) Interactive Mobile Communication Technologies and Learning, pp. 387–396. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75175-7_39
Li, J.: Real-time augmented reality visual-captions for deaf and hard-of-hearing children in classrooms. In: 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 641–642 (2023). https://doi.org/10.1109/VRW58643.2023.00163
Atanan, Y., Sombunsukho, S., Boonlue, S.: E-future classroom: a study mixed reality learning environment for deaf learners in thailand. Int. J. Environ. Sci. Educ. 12, 2291–2315 (2017)
Lee, G.-B., Jang, H., Jeong, H., Woo, W.: Designing a multi-modal communication system for the deaf and hard-of-hearing users. In: 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 429–434 (2021). https://doi.org/10.1109/ISMAR-Adjunct54149.2021.00097
Luo, L., Weng, D., Songrui, G., Hao, J., Tu, Z.: Avatar interpreter: improving classroom experiences for deaf and hard-of-hearing people based on augmented reality. In: CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1–5. ACM, New Orleans (2022). https://doi.org/10.1145/3491101.3519799
Peng, Y.-H., et al.: SpeechBubbles: enhancing captioning experiences for deaf and hard-of-hearing people in group conversations. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–10. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3173574.3173867
Ridha, A.M., Shehieb, W.: Assistive technology for hearing-impaired and deaf students utilizing augmented reality. In: 2021 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), pp. 1–5. IEEE (2021)
McDonnell, E.: Understanding social and environmental factors to enable collective access approaches to the design of captioning technology. In: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–8. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3517428.3550417
Simpson, E., Dalal, S., Semaan, B.: “Hey, can you add captions?”: the critical infrastructuring practices of neurodiverse people on tiktok. Proc. ACM Hum.-Comput. Interact. 7, 57:1–57:27 (2023). https://doi.org/10.1145/3579490
Mahajan, D., Bhosale, S., Nighot, Y., Tayal, M.: A review of video captioning methods. Int. J. -Gener. Comput. 12 (2021)
Hrga, I., Ivašić-Kos, M.: Deep image captioning: an overview. In: 2019 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 995–1000 (2019). https://doi.org/10.23919/MIPRO.2019.8756821
Westin, T., Neves, J., Mozelius, P., Sousa, C., Mantovan, L.: Inclusive AR-games for education of deaf children: challenges and opportunities. Eur. Conf. Games Based Learn. 16, 597–604 (2022). https://doi.org/10.34190/ecgbl.16.1.588
Garzón, J.: An overview of twenty-five years of augmented reality in education. Multimodal Technol. Interact. 5, 37 (2021). https://doi.org/10.3390/mti5070037
Jain, D., Chinh, B., Findlater, L., Kushalnagar, R., Froehlich, J.: Exploring augmented reality approaches to real-time captioning: a preliminary autoethnographic study. In: Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems, pp. 7–11. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3197391.3205404
Fernandes, N., Leite Junior, A.J.M., Marçal, E., Viana, W.: Augmented reality in education for people who are deaf or hard of hearing: a systematic literature review. Univ. Access Inf. Soc. (2023). https://doi.org/10.1007/s10209-023-00994-z
Kawas, S., Karalis, G., Wen, T., Ladner, R.E.: Improving real-time captioning experiences for deaf and hard of hearing students. In: Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 15–23. Association for Computing Machinery, New York (2016). https://doi.org/10.1145/2982142.2982164
Jain, D., Franz, R., Findlater, L., Cannon, J., Kushalnagar, R., Froehlich, J.: Towards accessible conversations in a mobile context for people who are deaf and hard of hearing. In: Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 81–92. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3234695.3236362
Peters, M.D.J., Godfrey, C.M., Khalil, H., McInerney, P., Parker, D., Soares, C.B.: Guidance for conducting systematic scoping reviews. JBI Evid. Implement. 13, 141 (2015). https://doi.org/10.1097/XEB.0000000000000050
Chen, F., Li, X., Tang, J., Li, S., Wang, T.: A survey on recent advances in image captioning. J. Phys. Conf. Ser. 1914, 012053 (2021). https://doi.org/10.1088/1742-6596/1914/1/012053
Downey, G.J.: Closed Captioning: Subtitling, Stenography, and the Digital Convergence of Text with Television. JHU Press (2008)
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., Cucchiara, R.: From show to tell: a survey on deep learning-based image captioning. IEEE Trans. Pattern Anal. Mach. Intell. 45, 539–559 (2022). https://doi.org/10.1109/TPAMI.2022.3148210
Eddin Za’ter, M., Talafha, B.: Bench-marking and improving Arabic automatic image captioning through the use of multi-task learning paradigm (2022). https://ui.adsabs.harvard.edu/abs/2022arXiv220205474E. https://doi.org/10.48550/arXiv.2202.05474
Chen, S., Yao, T., Jiang, Y.-G.: Deep learning for video captioning: a review, 6283–6290 (2019)
Li, S., Tao, Z., Li, K., Fu, Y.: Visual to text: survey of image and video captioning. IEEE Trans. Emerg. Top. Comput. Intell. 3, 297–312 (2019). https://doi.org/10.1109/TETCI.2019.2892755
Wang, X., Chen, W., Wu, J., Wang, Y.-F., Wang, W.Y.: Video captioning via hierarchical reinforcement learning. In: Presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
Gupta, N., Jalal, A.S.: Integration of textual cues for fine-grained image captioning using deep CNN and LSTM. Neural Comput. Appl. 32, 17899–17908 (2020). https://doi.org/10.1007/s00521-019-04515-z
Yang, B., Zhang, T., Zou, Y.: CLIP meets video captioning: concept-aware representation learning does matter (2022). http://arxiv.org/abs/2111.15162. https://doi.org/10.48550/arXiv.2111.15162
Chen, C., et al.: Towards better caption supervision for object detection. IEEE Trans. Vis. Comput. Graph. 28, 1941–1954 (2022). https://doi.org/10.1109/TVCG.2021.3138933
Berger, A., Kostak, M., Maly, F.: Mobile AR solution for deaf people. In: Awan, I., Younas, M., Ünal, P., and Aleksy, M. (eds.) Mobile Web and Intelligent Information Systems, pp. 243–254. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27192-3_19
Mathew, R., Mak, B., Dannels, W.: Access on demand: real-time, multi-modal accessibility for the deaf and hard-of-hearing based on augmented reality. In: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–6. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3517428.3551352
Eksvärd, S., Falk, J.: Evaluating Speech-to-Text Systems and AR-glasses : a study to develop a potential assistive device for people with hearing impairments (2021). https://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-437608
Mirzaei, M.R., Ghorshi, S., Mortazavi, M.: Combining augmented reality and speech technologies to help deaf and hard of hearing people. In: 2012 14th Symposium on Virtual and Augmented Reality, pp. 174–181 (2012). https://doi.org/10.1109/SVR.2012.10
Ghasemi, Y., Singh, A., Kim, M., Johnson, A., Jeong, H.: Effects of head-locked augmented reality on user’s performance and perceived workload. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 65, 1094–1098 (2021). https://doi.org/10.1177/1071181321651169
Brown, A., Turner, J., Patterson, J., Schmitz, A., Armstrong, M., Glancy, M.: Exploring subtitle behaviour for 360 video. White Pap. WHP. 330 (2018)
Brescia-Zapata, M., Krejtz, K., Duchowski, A.T., Hughes, C.J., Orero, P.: Subtitles in VR 360° video. results from an eye-tracking experiment. In: Perspectives (Montclair), pp. 1–23 (2023). https://doi.org/10.1080/0907676X.2023.2268122
Hughes, C.J., Zapata, M.B., Johnston, M., Orero, P.: Immersive captioning: developing a framework for evaluating user needs. In: 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), pp. 313–318 (2020). https://doi.org/10.1109/AIVR50618.2020.00063
Munn, Z., Peters, M.D.J., Stern, C., Tufanaru, C., McArthur, A., Aromataris, E.: Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med. Res. Methodol. 18, 143 (2018). https://doi.org/10.1186/s12874-018-0611-x
Mak, S., Thomas, A.: Steps for conducting a scoping review. J. Grad. Med. Educ. 14, 565–567 (2022). https://doi.org/10.4300/JGME-D-22-00621.1
Page, M.J., et al.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372, n71 (2021). https://doi.org/10.1136/bmj.n71
Mathew, R., Dannels, W.A., Parker, A.J.: An augmented reality based approach for optimization of language access services in healthcare for deaf patients. In: Antona, M., Stephanidis, C. (eds.) Universal Access in Human-Computer Interaction, pp. 29–52. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-35897-5_3
Chen, H., et al.: Augmented reality, deep learning and vision-language query system for construction worker safety. Autom. Constr. 157, 105158 (2024). https://doi.org/10.1016/j.autcon.2023.105158
Kurahashi, T., Sakuma, R., Zempo, K., Mizutani, K., Wakatsuki, N.: Retrospective speech balloons on speech-visible AR via head-mounted display. In: 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 423–424 (2018). https://doi.org/10.1109/ISMAR-Adjunct.2018.00127
Schipper, C., Brinkman, B.: Caption placement on an augmented reality head worn device. In: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 365–366. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3132525.3134786
Ong, D.X., et al.: Smart captions: a novel solution for closed captioning in theatre settings with AR glasses. In: 2021 IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI), pp. 1–5 (2021). https://doi.org/10.1109/SOLI54607.2021.9672391
Dabran, I., Avny, T., Singher, E., Ben Danan, H.: Augmented reality speech recognition for the hearing impaired. In: 2017 IEEE International Conference on Microwaves, Antennas, Communications and Electronic Systems (COMCAS), pp. 1–4 (2017). https://doi.org/10.1109/COMCAS.2017.8244731
Aljowaysir, N., Ozdemir, T.O., Kim, T.: Differentiated learning patterns with mixed reality. In: 2019 IEEE Games, Entertainment, Media Conference (GEM), pp. 1–4 (2019). https://doi.org/10.1109/GEM.2019.8811558
Sabie, D., Sheta, H., Ferdous, H.S., Kopalakrishnan, V., Ahmed, S.I.: Be our guest: intercultural heritage exchange through augmented reality (AR). In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–15. Association for Computing Machinery, New York (2023). https://doi.org/10.1145/3544548.3581005
Wang, Y., Lualdi, C.P., Angrave, L., Purushotam, G.N.: Using deep learning and augmented reality to improve accessibility: inclusive conversations using diarization, captions, and visualization. In: Presented at the 2023 ASEE Annual Conference & Exposition, 25 June 2023 (2023)
InformedHealth.org: Hearing loss and deafness: Normal hearing and impaired hearing. In: InformedHealth.org Independence, Evidence-based. Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany (2017)
Diaz, C.L., Arellano, M.R.M., Rosillo, V.M.L., Ochoa, A.: Augmented reality system to promote the inclusion of deaf people in smart cities. Res Comput Sci. 147, 49–64 (2018)
Shidende, D., Kessel, T., Moebs, S.: Towards accessible augmented reality learning authoring tool: a case of MirageXR. In: 2023 IST-Africa Conference (IST-Africa), pp. 1–13 (2023). https://doi.org/10.23919/IST-Africa60249.2023.10187746
Tyler, M.D., Jones, C., Grebennikov, L., Leigh, G., Noble, W., Burnham, D.: Effect of caption rate on the comprehension of educational television programmes by deaf school students. Deaf. Educ. Int. 11, 152–162 (2009). https://doi.org/10.1002/dei.262
Berke, L., Caulfield, C., Huenerfauth, M.: Deaf and hard-of-hearing perspectives on imperfect automatic speech recognition for captioning one-on-one meetings. In: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 155–164. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3132525.3132541
Kushalnagar, R.S., Lasecki, W.S., Bigham, J.P.: Accessibility evaluation of classroom captions. ACM Trans. Access. Comput. 5, 7:1–7:24 (2014). https://doi.org/10.1145/2543578
Prud’hommeaux, E., Jimerson, R., Hatcher, R., Michelson, K.: Automatic speech recognition for supporting endangered language documentation (2021)
Li, F.M., Lu, C., Lu, Z., Carrington, P., Truong, K.N.: An exploration of captioning practices and challenges of individual content creators on youtube for people with hearing impairments. Proc. ACM Hum.-Comput. Interact. 6, 75:1–75:26 (2022). https://doi.org/10.1145/3512922
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix
Appendix
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Shidende, D., Kessel, T., Treydte, A., Moebs, S. (2024). A Personalized Captioning Strategy for the Deaf and Hard-of-Hearing Users in an Augmented Reality Environment. In: De Paolis, L.T., Arpaia, P., Sacco, M. (eds) Extended Reality. XR Salento 2024. Lecture Notes in Computer Science, vol 15028. Springer, Cham. https://doi.org/10.1007/978-3-031-71704-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-71704-8_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-71703-1
Online ISBN: 978-3-031-71704-8
eBook Packages: Computer ScienceComputer Science (R0)