survey

Evaluation of XR Applications: A Tertiary Review

Authors:

Carla M. Dal Sasso FreitasAuthors Info & Claims

ACM Computing Surveys, Volume 56, Issue 5

Article No.: 110, Pages 1 - 35

https://doi.org/10.1145/3626517

Published: 25 November 2023 Publication History

Abstract

Extended reality (XR) applications—encompassing virtual reality, augmented reality, and mixed reality—are finding their way into multiple domains. Each area has different motivations for employing and different criteria for evaluating XR. Multiple surveys describe XR and its evaluation in particular fields. However, these surveys do not always agree on the definition of XR. This lack of consensus makes it hard to compare and use learnings from XR research across areas. Through a tertiary systematic literature review, we analyzed 81 surveys from several fields to provide a comprehensive summary of the state of XR research regarding the evaluation of XR applications. We seek to understand (i) how is XR defined? (ii) why is XR employed? (iii) how is XR evaluated? (iv) what are the main criticisms and future research paths outlined by the surveys? and (v) how good are the surveys? We present our findings describing XR research in 10 categories. Given our findings, we propose that future research should build upon a solid XR taxonomy and depart from effectiveness into efficiency research—to understand not only if but also how XR achieves the desired outcomes.

Supplementary Material

ZIP File (csur-2021-0759-supp.zip)

Download
29.25 MB

3626517.supp (3626517.supp.pdf)

Supplementary material

Download
2.25 MB

References

[1]

Kamran Ahmed, Muhammed Jawad, May Abboudi, Andrea Gavazzi, Ara Darzi, Thanos Athanasiou, Justin Vale, Mohammad Shamim Khan, and Prokar Dasgupta. 2011. Effectiveness of procedural simulation in urology: A systematic review. Journal of Urology 186, 1 (July2011), 26–34. DOI:

[2]

Murat Akçayır and Gökçe Akçayır. 2017. Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review 20 (Feb.2017), 1–11. DOI:

[3]

Medhat Alaker, Greg R. Wynn, and Tan Arulampalam. 2016. Virtual reality training in laparoscopic surgery: A systematic review & meta-analysis. International Journal of Surgery 29 (May2016), 85–94. DOI:

[4]

Anas R. Alashram, Giuseppe Annino, Elvira Padua, Cristian Romagnoli, and Nicola Biagio Mercuri. 2019. Cognitive rehabilitation post traumatic brain injury: A systematic review for emerging use of virtual reality technology. Journal of Clinical Neuroscience 66 (Aug.2019), 209–219. DOI:

[5]

Asit Arora, Loretta Y. M. Lau, Zaid Awad, Ara Darzi, Arvind Singh, and Neil Tolley. 2014. Virtual reality simulation training in otolaryngology. International Journal of Surgery 12, 2 (Feb.2014), 87–94. DOI:

[6]

R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre. 2001. Recent advances in augmented reality. IEEE Computer Graphics and Applications 21, 6 (Dec.2001), 34–47. DOI:

Digital Library

[7]

Ronald T. Azuma. 1997. A survey of augmented reality. Presence: Teleoperators and Virtual Environments 6, 4 (Aug.1997), 355–385. DOI:

Digital Library

[8]

Florence Aïm, Guillaume Lonjon, Didier Hannouche, and Rémy Nizard. 2016. Effectiveness of virtual reality training in orthopaedic surgery. Arthroscopy: The Journal of Arthroscopic & Related Surgery 32, 1 (Jan.2016), 224–232. DOI:

[9]

Amanda A. Benbow and Page L. Anderson. 2019. A meta-analytic examination of attrition in virtual reality exposure therapy for anxiety disorders. Journal of Anxiety Disorders 61 (Jan.2019), 18–26. DOI:

[10]

Sylvain Bernhardt, Stéphane A. Nicolau, Luc Soler, and Christophe Doignon. 2017. The status of augmented reality in laparoscopic surgery as of 2016. Medical Image Analysis 37 (April2017), 66–90. DOI:

[11]

Jonas Blattgerste, Patrick Renner, and Thies Pfeiffer. 2019. Augmented reality action assistance and learning for cognitively impaired people: A systematic literature review. In Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA ’19). ACM, New York, NY, 270–279. DOI:

Digital Library

[12]

Brent Bluett, Ece Bayram, and Irene Litvan. 2019. The virtual reality of Parkinson’s disease freezing of gait: A systematic review. Parkinsonism & Related Disorders 61 (April2019), 26–33. DOI:

[13]

Simone Borsci, Glyn Lawson, and Simon Broome. 2015. Empirical evidence, evaluation criteria and challenges for the effectiveness of virtual and mixed reality tools for training operators of car service maintenance. Computers in Industry 67 (Feb.2015), 17–26. DOI:

Digital Library

[14]

R. Bosc, A. Fitoussi, B. Hersant, T. H. Dao, and J. P. Meningaud. 2019. Intraoperative augmented reality with heads-up displays in maxillofacial surgery: A systematic review of the literature and a classification of relevant technologies. International Journal of Oral and Maxillofacial Surgery 48, 1 (Jan.2019), 132–139. DOI:

[15]

Lal Bozgeyikli, Andrew Raij, Srinivas Katkoori, and Redwan Alqasemi. 2018. A survey on virtual reality for individuals with autism spectrum disorder: Design considerations. IEEE Transactions on Learning Technologies 11, 2 (April2018), 133–151. DOI:

[16]

Oliver Brunckhorst, Abdullatif Aydin, Hamid Abboudi, Arun Sahai, Muhammad Shamim Khan, Prokar Dasgupta, and Kamran Ahmed. 2015. Simulation-based ureteroscopy training: A systematic review. Journal of Surgical Education 72, 1 (Jan.2015), 135–143. DOI:

[17]

David Budgen, Pearl Brereton, Sarah Drummond, and Nikki Williams. 2018. Reporting systematic reviews: Some lessons from a tertiary study. Information and Software Technology 95 (March2018), 62–74. DOI:

[18]

Marysol Cacciata, Anna Stromberg, Jung-Ah Lee, Dara Sorkin, Dawn Lombardo, Steve Clancy, Adeline Nyamathi, and Lorraine S. Evangelista. 2019. Effect of exergaming on health-related quality of life in older adults: A systematic review. International Journal of Nursing Studies 93 (May2019), 30–40. DOI:

[19]

Roxana A. I. Cardoş, Oana A. David, and Daniel O. David. 2017. Virtual reality exposure therapy in flight anxiety: A quantitative meta-analysis. Computers in Human Behavior 72 (July2017), 371–380. DOI:

Digital Library

[20]

Emily Carl, Aliza T. Stein, Andrew Levihn-Coon, Jamie R. Pogue, Barbara Rothbaum, Paul Emmelkamp, Gordon J. G. Asmundson, Per Carlbring, and Mark B. Powers. 2019. Virtual reality exposure therapy for anxiety and related disorders: A meta-analysis of randomized controlled trials. Journal of Anxiety Disorders 61 (Jan.2019), 27–36. DOI:

[21]

T. P. Caudell and D. W. Mizell. 1992. Augmented reality: An application of heads-up display technology to manual manufacturing processes. In Proceedings of the 25th Hawaii International Conference on System Sciences, Vol. 2. IEEE, Los Alamitos, CA, 659–669. DOI:

[22]

Yu Chen, Kingsley Travis Abel, John T. Janecek, Yunan Chen, Kai Zheng, and Steven C. Cramer. 2019. Home-based technologies for stroke rehabilitation: A systematic review. International Journal of Medical Informatics 123 (March2019), 11–22. DOI:

[23]

B. Chi, B. Chau, E. Yeo, and P. Ta. 2019. Virtual reality for spinal cord injury-associated neuropathic pain: Systematic review. Annals of Physical and Rehabilitation Medicine 62, 1 (Jan.2019), 49–57. DOI:

[24]

Anna D. Clark, Damiano G. Barone, Nicholas Candy, Mathew Guilfoyle, Karol Budohoski, Riikka Hofmann, Thomas Santarius, Ramez Kirollos, and Rikin A. Trivedi. 2017. The effect of 3-dimensional simulation on neurosurgical skill acquisition and surgical performance: A review of the literature. Journal of Surgical Education 74, 5 (Sept.2017), 828–836. DOI:

[25]

M. Cogné, M. Taillade, B. N’Kaoua, A. Tarruella, E. Klinger, F. Larrue, H. Sauzéon, P. A. Joseph, and E. Sorita. 2017. The contribution of virtual reality to the diagnosis of spatial navigation disorders and to the study of the role of navigational aids: A systematic literature review. Annals of Physical and Rehabilitation Medicine 60, 3 (June2017), 164–176. DOI:

[26]

Davide Corbetta, Federico Imeri, and Roberto Gatti. 2015. Rehabilitation that incorporates virtual reality is more effective than standard rehabilitation for improving walking speed, balance and mobility after stroke: A systematic review. Journal of Physiotherapy 61, 3 (July2015), 117–124. DOI:

[27]

Cléber G. Corrêa, Fátima L. S. Nunes, Edith Ranzini, Ricardo Nakamura, and Romero Tori. 2019. Haptic interaction for needle insertion training in medical applications: The state-of-the-art. Medical Engineering & Physics 63 (Jan.2019), 6–25. DOI:

[28]

Hannah Coyle, Victoria Traynor, and Nadia Solowij. 2015. Computerized and virtual reality cognitive training for individuals at high risk of cognitive decline: Systematic review of the literature. American Journal of Geriatric Psychiatry 23, 4 (April2015), 335–359. DOI:

[29]

Carolina Cruz-Neira, Daniel J. Sandin, and Thomas A. DeFanti. 1993. Surround-screen projection-based virtual reality: the design and implementation of the CAVE. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’93). ACM, New York, NY, 135–142. DOI:

Digital Library

[30]

Katherine Cárdenas and Mariana Aranda. 2017. Psychotherapies for the treatment of phantom limb pain. Revista Colombiana de Psiquiatría (English ed.) 46, 3 (July2017), 178–186. DOI:

[31]

Simon Davis, Keith Nesbitt, and Eugene Nalivaiko. 2014. A systematic review of cybersickness. In Proceedings of the 2014 Conference on Interactive Entertainment (IE ’14). ACM, New York, NY, 1–9. DOI:

Digital Library

[32]

Wenrui Deng, Die Hu, Sheng Xu, Xiaoyu Liu, Jingwen Zhao, Qian Chen, Jiayuan Liu, Zheng Zhang, Wenxiu Jiang, Lijun Ma, Xinyi Hong, Shengrong Cheng, Boya Liu, and Xiaoming Li. 2019. The efficacy of virtual reality exposure therapy for PTSD symptoms: A systematic review and meta-analysis. Journal of Affective Disorders 257 (Oct.2019), 698–709. DOI:

[33]

Felicitas J. Detmer, Julian Hettig, Daniel Schindele, Martin Schostak, and Christian Hansen. 2017. Virtual and augmented reality systems for renal interventions: A systematic review. IEEE Reviews in Biomedical Engineering 10 (2017), 78–94. DOI:

[34]

Lucas Pfeiffer Salomão Dias, Jorge Luis Victória Barbosa, and Henrique Damasceno Vianna. 2018. Gamification and serious games in depression care: A systematic mapping study. Telematics and Informatics 35, 1 (April2018), 213–224. DOI:

[35]

World Leaders in Research-Based User Experience. 2012. Usability 101: Introduction to Usability. Retrieved October 17, 2023 from https://www.nngroup.com/articles/usability-101-introduction-to-usability/

[36]

Zhenan Feng, Vicente A. González, Robert Amor, Ruggiero Lovreglio, and Guillermo Cabrera-Guerrero. 2018. Immersive virtual reality serious games for evacuation training and research: A systematic literature review. Computers & Education 127 (Dec.2018), 252–266. DOI:

[37]

Javier Fernández-Álvarez, Alexander Rozental, Per Carlbring, Desirée Colombo, Giuseppe Riva, Page L. Anderson, Rosa María Baños, Amanda A. Benbow, Stéphane Bouchard, Juana María Bretón-López, Georgina Cárdenas, JoAnn Difede, Paul Emmelkamp, Azucena García-Palacios, Verónica Guillén, Hunter Hoffman, Isabel Kampann, Ramona Moldovan, Andreas Mühlberger, Max North, Paul Pauli, Wenceslao Peñate Castro, Soledad Quero, Miquel Tortella-Feliu, Kataryzna Wyka, and Cristina Botella. 2019. Deterioration rates in virtual reality therapy: An individual patient data level meta-analysis. Journal of Anxiety Disorders 61 (Jan.2019), 3–17. DOI:

[38]

Pierre Fite-Georgel. 2011. Is there a reality in industrial augmented reality? In Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality. IEEE, Los Alamitos, CA, 201–210. DOI:

Digital Library

[39]

María Antonia Fuentes Calderón, Ainhoa Navarro Miralles, Mauricio Jaramillo Pimienta, Jesús María Gonçalves Estella, and María José Sánchez Ledesma. 2019. Analysis of the factors related to the effectiveness of transcranial current stimulation in upper limb motor function recovery after stroke: A systematic review. Journal of Medical Systems 43, 3 (Feb.2019), 69. DOI:

Digital Library

[40]

Yifan Gao, Vicente A. Gonzalez, and Tak Wing Yiu. 2019. The effectiveness of traditional tools and computer-aided technologies for health and safety training in the construction sector: A systematic review. Computers & Education 138 (Sept.2019), 101–115. DOI:

Digital Library

[41]

Juan Garzón and Juan Acevedo. 2019. Meta-analysis of the impact of augmented reality on students’ learning gains. Educational Research Review 27 (June2019), 244–260. DOI:

[42]

Alexandra Ghiţă and José Gutiérrez-Maldonado. 2018. Applications of virtual reality in individuals with alcohol misuse: A systematic review. Addictive Behaviors 81 (June2018), 1–11. DOI:

[43]

Francesco Greco, Jeffrey A. Cadeddu, Inderbir S. Gill, Jihad H. Kaouk, Mesut Remzi, R. Houston Thompson, Fijs W. B. van Leeuwen, Henk G. van der Poel, Paolo Fornara, and Jens Rassweiler. 2014. Current perspectives in the use of molecular imaging to target surgical treatments for genitourinary cancers. European Urology 65, 5 (May2014), 947–964. DOI:

[44]

Hugo Gonçalo Guedes, Zêmia Maria Câmara Costa Ferreira, Layra Ribeiro de Sousa Leão, Edna Frasson Souza Montero, José Pinhata Otoch, and Everson Luiz de Almeida Artifon. 2019. Virtual reality simulator versus box-trainer to teach minimally invasive procedures: A meta-analysis. International Journal of Surgery 61 (Jan.2019), 60–68. DOI:

[45]

Kumar Raghav Gujjar, Arjen van Wijk, Ratika Kumar, and Ad de Jongh. 2019. Are technology-based interventions effective in reducing dental anxiety in children and adults? A systematic review. Journal of Evidence Based Dental Practice 19, 2 (June2019), 140–155. DOI:

[46]

Jieun Han, Hyo-Jin Kang, Minjung Kim, and Gyu Hyun Kwon. 2020. Mapping the intellectual structure of research on surgery with mixed reality: Bibliometric network analysis (2000–2019). Journal of Biomedical Informatics 109 (Sept.2020), 103516. DOI:

Digital Library

[47]

Matt C. Howard. 2017. A meta-analysis and systematic literature review of virtual reality rehabilitation programs. Computers in Human Behavior 70 (May2017), 317–327. DOI:

Digital Library

[48]

Youliang Huang, Qian Huang, Sajid Ali, Xing Zhai, Xiaoming Bi, and Renquan Liu. 2016. Rehabilitation using virtual reality technology: A bibliometric analysis, 1996–2015. Scientometrics 109, 3 (Dec.2016), 1547–1559. DOI:

Digital Library

[49]

María-Blanca Ibáñez and Carlos Delgado-Kloos. 2018. Augmented reality for STEM learning: A systematic review. Computers & Education 123 (Aug.2018), 109–123. DOI:

Digital Library

[50]

ISO. 2018. ISO 9241-11:2018. Retrieved October 17, 2023 from https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/06/35/63500.html

[51]

ISO/IEC. 2016. ISO/IEC 25066:2016. Retrieved October 17, 2023 from https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/06/38/63831.html

[52]

T. Joda, G. O. Gallucci, D. Wismeijer, and N. U. Zitzmann. 2019. Augmented and virtual reality in dental medicine: A systematic review. Computers in Biology and Medicine 108 (May2019), 93–100. DOI:

Digital Library

[53]

Isabel L. Kampmann, Paul M. G. Emmelkamp, and Nexhmedin Morina. 2016. Meta-analysis of technology-assisted interventions for social anxiety disorder. Journal of Anxiety Disorders 42 (Aug.2016), 71–84. DOI:

[54]

Mehmet Karakus, Alpay Ersozlu, and Aaron C. Clark. 2019. Augmented reality research in education: A bibliometric study. Eurasia Journal of Mathematics, Science and Technology Education 15, 10 (May2019), em1755. DOI:

[55]

Cassie C. Kennedy, Fabien Maldonado, and David A. Cook. 2013. Simulation-based bronchoscopy training: Systematic review and meta-analysis. Chest 144, 1 (July2013), 183–192. DOI:

[56]

Marta Kersten-Oertel, Pierre Jannin, and D. Louis Collins. 2013. The state of the art of visualization in mixed reality image guided surgery. Computerized Medical Imaging and Graphics 37, 2 (March2013), 98–112. DOI:

[57]

Kangsoo Kim, Mark Billinghurst, Gerd Bruder, Henry Been-Lirn Duh, and Gregory F. Welch. 2018. Revisiting trends in augmented reality research: A review of the 2nd decade of ISMAR (2008–2017). IEEE Transactions on Visualization and Computer Graphics 24, 11 (Nov.2018), 2947–2962. DOI:

[58]

Barbara Kitchenham and Stuart Charters. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering. Technical Report 2007-01. EBSE.

[59]

Marianne E. Klinke, Thóra B. Hafsteinsdóttir, Haukur Hjaltason, and Helga Jónsdóttir. 2015. Ward-based interventions for patients with hemispatial neglect in stroke rehabilitation: A systematic literature review. International Journal of Nursing Studies 52, 8 (Aug.2015), 1375–1403. DOI:

[60]

Antonia Koskina, Iain C. Campbell, and Ulrike Schmidt. 2013. Exposure therapy in eating disorders revisited. Neuroscience & Biobehavioral Reviews 37, 2 (Feb.2013), 193–208. DOI:

[61]

Eugenijus Kurilovas. 2016. Evaluation of quality and personalisation of VR/AR/MR learning systems. Behaviour & Information Technology 35, 11 (Nov.2016), 998–1007. DOI:

Digital Library

[62]

Steven LaValle. 2015. Virtual Reality. Retrieved October 17, 2023 from http://lavalle.pl/vr/

[63]

Joseph J. LaViola, Ernst Kruijff, Ryan P. McMahan, Doug A. Bowman, and Ivan Poupyrev. 2017. 3D User Interfaces: Theory and Practice (2nd ed.). Addison-Wesley, Boston, MA.

[64]

I.-Hsien Lin, Han-Ting Tsai, Chien-Yung Wang, Chih-Yang Hsu, Tsan-Hon Liou, and Yen-Nung Lin. 2019. Effectiveness and superiority of rehabilitative treatments in enhancing motor recovery within 6 months poststroke: A systemic review. Archives of Physical Medicine and Rehabilitation 100, 2 (Feb.2019), 366–378. DOI:

[65]

Kevin M. Malloy and Leonard S. Milling. 2010. The effectiveness of virtual reality distraction for pain reduction: A systematic review. Clinical Psychology Review 30, 8 (Dec.2010), 1011–1018. DOI:

[66]

Steven Martin, Paul Sutcliffe, Frances Griffiths, Jackie Sturt, John Powell, Ann Adams, and Jeremy Dale. 2011. Effectiveness and impact of networked communication interventions in young people with mental health conditions: A systematic review. Patient Education and Counseling 85, 2 (Nov.2011), e108–e119. DOI:

[67]

Anabela Correia Martins, Cláudia Santos, Catarina Silva, Daniela Baltazar, Juliana Moreira, and Nuno Tavares. 2018. Does modified Otago Exercise Program improves balance in older people? A systematic review. Preventive Medicine Reports 11 (Sept.2018), 231–239. DOI:

[68]

Travis Mazur, Tarek R. Mansour, Luke Mugge, and Azedine Medhkour. 2018. Virtual reality–based simulators for cranial tumor surgery: A systematic review. World Neurosurgery 110 (Feb.2018), 414–422. DOI:

[69]

Zahira Merchant, Ernest T. Goetz, Lauren Cifuentes, Wendy Keeney-Kennicutt, and Trina J. Davis. 2014. Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Computers & Education 70 (Jan.2014), 29–40. DOI:

[70]

P. Milgram and F. Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems E77-D, 12 (1994), 1–15.

[71]

Andrea Moglia, Vincenzo Ferrari, Luca Morelli, Mauro Ferrari, Franco Mosca, and Alfred Cuschieri. 2016. A systematic review of virtual reality simulators for robot-assisted surgery. European Urology 69, 6 (June2016), 1065–1080. DOI:

[72]

Roghayeh Mohammadi, Alireza Vaezpour Semnani, Majid Mirmohammadkhani, and Namrata Grampurohit. 2019. Effects of virtual reality compared to conventional therapy on balance poststroke: A systematic review and meta-analysis. Journal of Stroke and Cerebrovascular Diseases 28, 7 (July2019), 1787–1798. DOI:

[73]

Alexander Moreno, Kylie Janine Wall, Karthick Thangavelu, Lucas Craven, Emma Ward, and Nadeeka N. Dissanayaka. 2019. A systematic review of the use of virtual reality and its effects on cognition in individuals with neurocognitive disorders. Alzheimer’s & Dementia: Translational Research & Clinical Interventions 5, 1 (Jan.2019), 834–850. DOI:

[74]

Tamara Munzner. 2015. Visualization Analysis and Design. Boca Raton, FL.

[75]

Alexandra Neguţ, Silviu-Andrei Matu, Florin Alin Sava, and Daniel David. 2016. Task difficulty of virtual reality-based assessment tools compared to classical paper-and-pencil or computerized measures: A meta-analytic approach. Computers in Human Behavior 54 (Jan.2016), 414–424. DOI:

Digital Library

[76]

David L. Neumann, Robyn L. Moffitt, Patrick R. Thomas, Kylie Loveday, David P. Watling, Chantal L. Lombard, Simona Antonova, and Michael A. Tremeer. 2018. A systematic review of the application of interactive virtual reality to sport. Virtual Reality 22, 3 (Sept.2018), 183–198. DOI:

Digital Library

[77]

Yu-Leung Ng, Flora Ma, Frederick K. Ho, Patrick Ip, and King-Wa Fu. 2019. Effectiveness of virtual and augmented reality-enhanced exercise on physical activity, psychological outcomes, and physical performance: A systematic review and meta-analysis of randomized controlled trials. Computers in Human Behavior 99 (Oct.2019), 278–291. DOI:

Digital Library

[78]

Annalaura Nocentini, Valentina Zambuto, and Ersilia Menesini. 2015. Anti-bullying programs and information and communication technologies (ICTs): A systematic review. Aggression and Violent Behavior 23 (July2015), 52–60. DOI:

[79]

Monica Nour, Sin Hang Yeung, Stephanie Partridge, and Margaret Allman-Farinelli. 2017. A narrative review of social media and game-based nutrition interventions targeted at young adults. Journal of the Academy of Nutrition and Dietetics 117, 5 (May2017), 735–752.e10. DOI:

[80]

Clayton R. Pereira, Danilo R. Pereira, Silke A. T. Weber, Christian Hook, Victor Hugo C. de Albuquerque, and João P. Papa. 2019. A survey on computer-assisted Parkinson’s disease diagnosis. Artificial Intelligence in Medicine 95 (April2019), 48–63. DOI:

Digital Library

[81]

Anaick Perrochon, Benoit Borel, Dan Istrate, Maxence Compagnat, and Jean-Christophe Daviet. 2019. Exercise-based games interventions at home in individuals with a neurological disease: A systematic review and meta-analysis. Annals of Physical and Rehabilitation Medicine 62, 5 (Sept.2019), 366–378. DOI:

[82]

Michael Pfandler, Marc Lazarovici, Philipp Stefan, Patrick Wucherer, and Matthias Weigl. 2017. Virtual reality-based simulators for spine surgery: A systematic review. Spine Journal 17, 9 (Sept.2017), 1352–1363. DOI:

[83]

D. K. Ravi, N. Kumar, and P. Singhi. 2017. Effectiveness of virtual reality rehabilitation for children and adolescents with cerebral palsy: An updated evidence-based systematic review. Physiotherapy 103, 3 (Sept.2017), 245–258. DOI:

[84]

Michael K. Rooney, Fan Zhu, Erin F. Gillespie, Jillian R. Gunther, Ryan P. McKillip, Matthew Lineberry, Ara Tekian, and Daniel W. Golden. 2018. Simulation as more than a treatment-planning tool: A systematic review of the literature on radiation oncology simulation-based medical education. International Journal of Radiation Oncology *Biology* Physics 102, 2 (October 2018), 257–283. DOI:

[85]

Marc Ericson C. Santos, Angie Chen, Takafumi Taketomi, Goshiro Yamamoto, Jun Miyazaki, and Hirokazu Kato. 2014. Augmented reality learning experiences: Survey of prototype design and evaluation. IEEE Transactions on Learning Technologies 7, 1 (Jan.2014), 38–56. DOI:

[86]

Mona Meral Savran, Stine Maya Dreier Sørensen, Lars Konge, Martin G. Tolsgaard, and Flemming Bjerrum. 2016. Training and assessment of hysteroscopic skills: A systematic review. Journal of Surgical Education 73, 5 (Sept.2016), 906–918. DOI:

[87]

Soliane Scapin, Maria Elena Echevarría-Guanilo, Paulo Roberto Boeira Fuculo Junior, Natália Gonçalves, Patrícia Kuerten Rocha, and Rebeca Coimbra. 2018. Virtual reality in the treatment of burn patients: A systematic review. Burns 44, 6 (Sept.2018), 1403–1416. DOI:

[88]

Michael Scheffler, Susan Koranyi, Winfried Meissner, Bernhard Strauß, and Jenny Rosendahl. 2018. Efficacy of non-pharmacological interventions for procedural pain relief in adults undergoing burn wound care: A systematic review and meta-analysis of randomized controlled trials. Burns 44, 7 (Nov.2018), 1709–1720. DOI:

[89]

K. W. M. See, K. H. Chui, W. H. Chan, K. C. Wong, and Y. C. Chan. 2016. Evidence for endovascular simulation training: A systematic review. European Journal of Vascular and Endovascular Surgery 51, 3 (March2016), 441–451. DOI:

[90]

Helen Sharp. 2019. Interaction Design 5e. John Wiley & Sons, Indianapolis, IN.

[91]

Mel Slater and Sylvia Wilbur. 1997. A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence: Teleoperators and Virtual Environments 6, 6 (Dec.1997), 603–616. DOI:

Digital Library

[92]

Sandeep K. Subramanian and Shreya S. Prasanna. 2018. Virtual reality and noninvasive brain stimulation in stroke: How effective is their combination for upper limb motor improvement?—A meta-analysis. PM&R 10, 11 (2018), 1261–1270. DOI:

[93]

Ayoung Suh and Jane Prophet. 2018. The state of immersive technology research: A literature analysis. Computers in Human Behavior 86 (Sept.2018), 77–90. DOI:

[94]

Bhing-Leet Tan, Sara-Ann Lee, and Jimmy Lee. 2018. Social cognitive interventions for people with schizophrenia: A systematic review. Asian Journal of Psychiatry 35 (June2018), 115–131. DOI:

[95]

Charison Tay, Ankur Khajuria, and Chinmay Gupte. 2014. Simulation training: A systematic review of simulation in arthroscopy and proposal of a new competency-based training framework. International Journal of Surgery 12, 6 (June2014), 626–633. DOI:

[96]

Ann Sofia S. Thomsen, Yousif Subhi, Jens Folke Kiilgaard, Morten la Cour, and Lars Konge. 2015. Update on simulation-based surgical training and assessment in ophthalmology: A systematic review. Ophthalmology 122, 6 (June2015), 1111–1130.e1. DOI:

[97]

Wesley A. Turner and Leanne M. Casey. 2014. Outcomes associated with virtual reality in psychological interventions: Where are we now? Clinical Psychology Review 34, 8 (Dec.2014), 634–644. DOI:

[98]

S. Viñas-Diz and M. Sobrido-Prieto. 2016. Virtual reality for therapeutic purposes in stroke: A systematic review. Neurología (English Ed.) 31, 4 (May2016), 255–277. DOI:

[99]

Sarah Vogt, Nina Skjæret-Maroni, Dorothee Neuhaus, and Jochen Baumeister. 2019. Virtual reality interventions for balance prevention and rehabilitation after musculoskeletal lower limb impairments in young up to middle-aged adults: A comprehensive review on used technology, balance outcome measures and observed effects. International Journal of Medical Informatics 126 (June2019), 46–58. DOI:

[100]

Patrice L. Weiss, Rachel Kizony, Uri Feintuch, and Noomi Katz. 2006. Virtual reality in neurorehabilitation. In Textbook of Neural Repair and Rehabilitation, Michael Selzer, Stephanie Clarke, Leonardo Cohen, Pamela Duncan, and Fred Gage (Eds.). Cambridge University Press, Cambridge, UK, 182–197. DOI:

Cited By

Kim J(2024)Action Recognition System Using Full-body XR Devices for Sports Metaverse Games2024 15th International Conference on Information and Communication Technology Convergence (ICTC)10.1109/ICTC62082.2024.10827474(1962-1965)Online publication date: 16-Oct-2024
https://doi.org/10.1109/ICTC62082.2024.10827474
Siddiqui ARimal BReisslein MWang Y(2024)Survey on Unified Threat Management (UTM) Systems for Home NetworksIEEE Communications Surveys & Tutorials10.1109/COMST.2024.338247026:4(2459-2509)Online publication date: Dec-2025
https://doi.org/10.1109/COMST.2024.3382470
Magee PIenca MFarahany N(2024)Beyond neural data: Cognitive biometrics and mental privacyNeuron10.1016/j.neuron.2024.09.004112:18(3017-3028)Online publication date: Sep-2024
https://doi.org/10.1016/j.neuron.2024.09.004

Index Terms

Evaluation of XR Applications: A Tertiary Review
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
    2. Interaction paradigms
      1. Mixed / augmented reality

Recommendations

What is XR? Towards a Framework for Augmented and Virtual Reality
Abstract
Augmented Reality (AR), Virtual Reality (VR), Mixed Reality, and Extended Reality (often – misleadingly – abbreviated as XR) are commonly used terms to describe how technologies generate or modify reality. However, academics and ...
Highlights
- XR should not be used as an abbreviation for “Extended Reality”; X should be the placeholder for “all” new reality formats.
Virtual, mixed, and augmented reality: a systematic review for immersive systems research
Abstract
Immersive systems can be used to capture new data, create new experiences, and provide new insights by generating virtual elements of physical and imagined worlds. Immersive systems are seeing increased application across a broad array of fields. ...
Unified Representation for XR Content and its Rendering Method
Web3D '20: Proceedings of the 25th International Conference on 3D Web Technology

Virtual Reality (VR) and Augmented Reality (AR) have become familiar technologies with related markets growing rapidly every year. Moreover, the idea of considering VR and AR as one eXtended reality (XR) has broken the border between virtual space and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 56, Issue 5

May 2024

1019 pages

EISSN:1557-7341

DOI:10.1145/3613598

Editor:
Albert Zomaya
University of Sydney, Australia

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 November 2023

Online AM: 09 October 2023

Accepted: 18 September 2023

Revised: 14 May 2023

Received: 15 July 2022

Published in CSUR Volume 56, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Survey

Funding Sources

Brazilian funding agencies CAPES
National Research Council for Research and Development

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
1,573
Total Downloads

Downloads (Last 12 months)990
Downloads (Last 6 weeks)98

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kim J(2024)Action Recognition System Using Full-body XR Devices for Sports Metaverse Games2024 15th International Conference on Information and Communication Technology Convergence (ICTC)10.1109/ICTC62082.2024.10827474(1962-1965)Online publication date: 16-Oct-2024
https://doi.org/10.1109/ICTC62082.2024.10827474
Siddiqui ARimal BReisslein MWang Y(2024)Survey on Unified Threat Management (UTM) Systems for Home NetworksIEEE Communications Surveys & Tutorials10.1109/COMST.2024.338247026:4(2459-2509)Online publication date: Dec-2025
https://doi.org/10.1109/COMST.2024.3382470
Magee PIenca MFarahany N(2024)Beyond neural data: Cognitive biometrics and mental privacyNeuron10.1016/j.neuron.2024.09.004112:18(3017-3028)Online publication date: Sep-2024
https://doi.org/10.1016/j.neuron.2024.09.004

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents