Skip to main content
SpringerLink
Log in

Evaluation and Validation Process of Extended Reality Applications Developed in an Industrial Context: A Systematic Review

  • Review Article
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Extended reality (XR) technologies are achieving greater diffusion thanks to the recent digital breakthroughs enabling users to interact and accomplish tasks within immersive environments (IEs). Applications are notably diversifying toward the industrial field, where the use of XR technologies is at the core of the transition toward Industry 4.0. However, despite their high potential, it seems the use of XR is not yet widespread in the industry. Because of the amount of effort and time required to develop a new immersive environment (IE), evaluating its relevance and effectiveness appears to be a paramount factor for a wider adoption of these technologies in the industry. Previous works have already investigated the roots of this slow adoption, but to the best of our knowledge none of them give a detailed overview of the specific and decisive stage of the evaluation of new IEs before a possible implementation. Through a detailed study of 101 scientific papers published between 2000 and early 2021, this work therefore intends to provide an overview of existing studies involving XR experience design, evaluation approaches, and materials to help identify the areas that could require more targeted research to improve the adoption of immersive technologies in the industry. To do so, we propose an analytical framework combining key factors to assess user-centered evaluation and perform a systematic literature review to establish the current status of these factors. Results show the variety of assessment methods used and the emergence of new physiological technologies over recent years. Constraints related to the user-centered evaluation methodology are also identified and discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The data generated by the literature analysis performed in this work are available in the Dataverse repository: https://doi.org/10.12763/H3BDFH.

References

  1. Lu Y. Industry 4.0: a survey on technologies, applications and open research issues. J Ind Inf Integr. 2017;6:1–10. https://doi.org/10.1016/j.jii.2017.04.005.

    Article  Google Scholar 

  2. Rojko A. Industry 4.0 concept: background and overview. Int J Interact Mob Technol. 2017;11:77. https://doi.org/10.3991/ijim.v11i5.7072.

    Article  Google Scholar 

  3. Rüßmann M, Lorenz M, Gerbert P, et al (2015) Industry 4.0: The Future of Productivity and Growth in Manufacturing. The boston consulting group 0–20

  4. Milgram P, Kishino F (1994) A Taxonomy of Mixed Reality Visual Displays. In: undefined. /paper/A-Taxonomy-of-Mixed-Reality-Visual-Displays-Milgram-Kishino/f78a31be8874eda176a5244c645289be9f1d4317. Accessed 14 Feb 2021

  5. Ong SK, Yuan ML, Nee AYC. Augmented reality applications in manufacturing: a survey. Int J Prod Res. 2008;46:2707–42. https://doi.org/10.1080/00207540601064773.

    Article  MATH  Google Scholar 

  6. Perkins Coie (2019) XR/AR/VR/MR technology and content investment focus worldwide from 2016 to 2019 [Graph]. In: Statista. https://www.statista.com/statistics/829729/investments-focus-vr-augmented-reality-worldwide/. Accessed 19 Oct 2021

  7. Cipresso P, Giglioli IAC, Raya MA, Riva G. The past, present, and future of virtual and augmented reality research: a network and cluster analysis of the literature. Front Psychol. 2018. https://doi.org/10.3389/fpsyg.2018.02086.

    Article  Google Scholar 

  8. Elia V, Gnoni MG, Lanzilotto A. Evaluating the application of augmented reality devices in manufacturing from a process point of view: An AHP based model. Expert Syst Appl. 2016;63:187–97. https://doi.org/10.1016/j.eswa.2016.07.006.

    Article  Google Scholar 

  9. Frank AG, Dalenogare LS, Ayala NF. Industry 4.0 technologies: implementation patterns in manufacturing companies. Int J Prod Econ. 2019;210:15–26. https://doi.org/10.1016/j.ijpe.2019.01.004.

    Article  Google Scholar 

  10. Roldán JJ, Crespo E, Martín-Barrio A, et al. A training system for Industry 4.0 operators in complex assemblies based on virtual reality and process mining. Robot Comp-Integ Manuf. 2019;59:305–16. https://doi.org/10.1016/j.rcim.2019.05.004.

    Article  Google Scholar 

  11. Coburn JQ, Freeman I, Salmon JL. A review of the capabilities of current low-cost virtual reality technology and its potential to enhance the design process. J Comput Inf Sci Eng. 2017;17:031013. https://doi.org/10.1115/1.4036921.

    Article  Google Scholar 

  12. Nee AYC, Ong SK, Chryssolouris G, Mourtzis D. Augmented reality applications in design and manufacturing. CIRP Ann. 2012;61:657–79. https://doi.org/10.1016/j.cirp.2012.05.010.

    Article  Google Scholar 

  13. Berg LP, Vance JM. Industry use of virtual reality in product design and manufacturing: a survey. Virtual Reality. 2017;21:1–17. https://doi.org/10.1007/s10055-016-0293-9.

    Article  Google Scholar 

  14. Hamid NSS, Aziz FA, Azizi A (2014) Virtual reality applications in manufacturing system. In: 2014 Science and Information Conference. IEEE, London, UK, pp 1034–1037

  15. Perkins Coie, XRA, Boost VC (2020) Leading obstacles to mass adoption of augmented reality (AR) and virtual reality (VR) technologies according to XR/AR/VR/MR industry experts in the United States in 2020 [Graph]. In: Statista. https://www.statista.com/statistics/1098558/obstacles-to-mass-adoption-of-ar-technologies/. Accessed 19 Oct 2021

  16. Davis S, Nesbitt K, Nalivaiko E (2014) A Systematic Review of Cybersickness 9

  17. McCauley M, Sharkey T. Cybersickness: perception of self-motion in virtual environment. Presence. 1992;1:311–8. https://doi.org/10.1162/pres.1992.1.3.311.

    Article  Google Scholar 

  18. Mourtzis D. Simulation in the design and operation of manufacturing systems: state of the art and new trends. Int J Prod Res. 2020;58:1927–49. https://doi.org/10.1080/00207543.2019.1636321.

    Article  Google Scholar 

  19. Chuah SH-W. Why and who will adopt extended reality technology? Literature review, synthesis, and future research agenda. SSRN J. 2018. https://doi.org/10.2139/ssrn.3300469.

    Article  Google Scholar 

  20. Kim YM, Rhiu I, Yun MH. A systematic review of a virtual reality system from the perspective of user experience. Int J Human-Comput Int. 2020;36:893–910. https://doi.org/10.1080/10447318.2019.1699746.

    Article  Google Scholar 

  21. Suh A, Prophet J. The state of immersive technology research: a literature analysis. Comput Hum Behav. 2018;86:77–90. https://doi.org/10.1016/j.chb.2018.04.019.

    Article  Google Scholar 

  22. Gong L, Fast-Berglund A, Johansson B. A framework for extended reality system development in manufacturing. IEEE Access. 2021;9:24796–813. https://doi.org/10.1109/ACCESS.2021.3056752.

    Article  Google Scholar 

  23. Shafer DM, Carbonara CP, Korpi MF. Factors affecting enjoyment of virtual reality games: a comparison involving consumer-grade virtual reality technology. Games Health J. 2018;8:15–23. https://doi.org/10.1089/g4h.2017.0190.

    Article  Google Scholar 

  24. Won AS, Bailey J, Bailenson J, et al. Immersive virtual reality for pediatric pain. Children. 2017;4:52. https://doi.org/10.3390/children4070052.

    Article  Google Scholar 

  25. Gavish N, Gutiérrez T, Webel S, et al. Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interact Learn Environ. 2015;23:778–98. https://doi.org/10.1080/10494820.2013.815221.

    Article  Google Scholar 

  26. Sermet Y, Demir I (2019) Flood action VR: a virtual reality framework for disaster awareness and emergency response training. In: ACM SIGGRAPH 2019 Posters. Association for Computing Machinery, New York, NY, USA, pp 1–2

  27. Bordegoni M, Caruso G. Mixed reality distributed platform for collaborative design review of automotive interiors. Virt Phys Prototyping. 2012;7:243–59. https://doi.org/10.1080/17452759.2012.721605.

    Article  Google Scholar 

  28. Wolfartsberger J. Analyzing the potential of Virtual Reality for engineering design review. Autom Constr. 2019;104:27–37. https://doi.org/10.1016/j.autcon.2019.03.018.

    Article  Google Scholar 

  29. Andrée K, Nilsson D, Eriksson J. Evacuation experiments in a virtual reality high-rise building: exit choice and waiting time for evacuation elevators. Fire Mater. 2016;40:554–67. https://doi.org/10.1002/fam.2310.

    Article  Google Scholar 

  30. Gillath O, McCall C, Shaver P, Blascovich J. What can virtual reality teach us about prosocial tendencies in real and virtual environments? Media Psychol Media Psychol. 2008;11:259–82. https://doi.org/10.1080/15213260801906489.

    Article  Google Scholar 

  31. Hily A, Jacob C, Dupont L, et al (2020) Reliability of virtual reality for user experience in spatial cognition: an exploratory approach. In: IJVR (ed) ConVRgence (VRIC) Virtual Reality International Conference 2020. IJVR, Laval, France, pp 68–80

  32. Slater M. Grand challenges in virtual environments. Front Robot AI. 2014. https://doi.org/10.3389/frobt.2014.00003.

    Article  Google Scholar 

  33. Baños RM, Botella C, Rubió I, et al. Presence and emotions in virtual environments: the influence of stereoscopy. Cyberpsychol Behav. 2008;11:1–8. https://doi.org/10.1089/cpb.2007.9936.

    Article  Google Scholar 

  34. Vermeeren APOS, Law EL-C, Roto V, et al (2010) User experience evaluation methods: current state and development needs. In: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries. Association for Computing Machinery, New York, NY, USA, pp 521–530

  35. Usoh M, Catena E, Arman S, Slater M. Using presence questionnaires in reality. Presence Teleoperat Virtual Environ. 2000. https://doi.org/10.1162/105474600566989.

    Article  Google Scholar 

  36. Witmer BG, Singer MJ. Measuring presence in virtual environments: a presence questionnaire. Pres Teleoperat Virtual Environ. 1998;7:225–40. https://doi.org/10.1162/105474698565686.

    Article  Google Scholar 

  37. Grassini S, Laumann K. Questionnaire measures and physiological correlates of presence: a systematic review. Front Psychol. 2020;11:349. https://doi.org/10.3389/fpsyg.2020.00349.

    Article  Google Scholar 

  38. Maia CLB, Furtado ES. A systematic review about user experience evaluation. In: Marcus A, editor. Design, user experience, and usability: design thinking and methods. Cham: Springer International Publishing; 2016. p. 445–55.

    Chapter  Google Scholar 

  39. Slater M. How colorful was your day? Why questionnaires cannot assess presence in virtual environments. Presence. 2004;13:484–93. https://doi.org/10.1162/1054746041944849.

    Article  Google Scholar 

  40. Insko BE (2003) Measuring Presence: Subjective, Behavioral and Physiological Methods. 11

  41. Martinez M, Arbelaez Garces G, Dupont L, et al (2020) Physiological assessment of User eXprience supported by Immersive Environments: First input from a literature review. In: Virtual L (ed) ConVRgence (VRIC) Virtual Reality International Conference. Laval Virtual, Laval, France, pp 89–108

  42. Meehan M, Razzaque S, Insko B, et al. Review of four studies on the use of physiological reaction as a measure of presence in stressfulvirtual environments. Appl Psychophysiol Biofeedback. 2005;30:239–58. https://doi.org/10.1007/s10484-005-6381-3.

    Article  Google Scholar 

  43. Kroes MCW, Dunsmoor JE, Mackey WE, et al. Context conditioning in humans using commercially available immersive Virtual Reality. Sci Rep. 2017. https://doi.org/10.1038/s41598-017-08184-7.

    Article  Google Scholar 

  44. Alemdag E, Cagiltay K. A systematic review of eye tracking research on multimedia learning. Comput Educ. 2018;125:413–28. https://doi.org/10.1016/j.compedu.2018.06.023.

    Article  Google Scholar 

  45. Kiefer P, Giannopoulos I, Raubal M, Duchowski A. Eye tracking for spatial research: cognition, computation, challenges. Spat Cogn Comput. 2017;17:1–19. https://doi.org/10.1080/13875868.2016.1254634.

    Article  Google Scholar 

  46. Ganglbauer E, Schrammel J, Deutsch S, Tscheligi M. Applying psychophysiological methods for measuring user experience: possibilities, challenges and feasibility. In: Workshop on user experience evaluation methods in product development. Uppsala (2009)

  47. Sutherland IE (1965) The ultimate display. In: Proceedings of the IFIP Congress, vol. 2, No. 506–508, pp. 506–508

  48. Fenn J, Blosch M (2018) Understanding Gartner’s Hype Cycles. Gartner. https://www.gartner.com/en/documents/3887767/understanding-gartner-s-hype-cycles

  49. Richir S, Fuchs P, Lourdeaux D, et al. How to design compelling virtual reality or augmented reality experience? Int J Virtual Reality. 2015;15(1):35–47.

    Article  Google Scholar 

  50. Vi S, da Silva TS, Maurer F, et al. User experience guidelines for designing hmd extended reality applications. In: Lamas D, Loizides F, Nacke L, et al., editors. Human-Computer Interaction – INTERACT 2019. Cham: Springer International Publishing; 2019. p. 319–41.

    Chapter  Google Scholar 

  51. Budgen D, Brereton P (2006) Performing systematic literature reviews in software engineering. In: Proceedings of the 28th international conference on Software engineering. ACM, Shanghai China, pp 1051–1052

  52. Webster J, Watson RT (2002) Analyzing the Past to Prepare for the Future: Writing a Literature Review. MIS Quarterly 26: xiii–xxiii

  53. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989;13:319. https://doi.org/10.2307/249008.

    Article  Google Scholar 

  54. Sagnier C, Loup-Escande É, Valléry G. Technology acceptance of virtual reality: a review. Le travail humain. 2019;82:183–212.

    Article  Google Scholar 

  55. Webel S, Bockholt U, Engelke T, et al. An augmented reality training platform for assembly and maintenance skills. Robot Auton Syst. 2013;61:398–403. https://doi.org/10.1016/j.robot.2012.09.013.

    Article  Google Scholar 

  56. Werrlich S, Lorber C, Nguyen P-A, et al (2018) Assembly Training: Comparing the Effects of Head-Mounted Displays and Face-to-Face Training. In: HCI

  57. Wiedenmaier S, Oehme O, Schmidt L, Luczak H. Augmented reality (AR) for assembly processes design and experimental evaluation. Int J Human-Comput Int. 2003;16:497–514. https://doi.org/10.1207/S15327590IJHC1603_7.

    Article  Google Scholar 

  58. Alves J, Marques B, Oliveira M, et al (2019) Comparing Spatial and Mobile Augmented Reality for Guiding Assembling Procedures with Task Validation. In: 2019 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). IEEE, Porto, Portugal, pp 1–6

  59. de Souza Cardoso LF, Mariano FCMQ, Zorzal ER. Mobile augmented reality to support fuselage assembly. Comput Ind Eng. 2020;148:106712. https://doi.org/10.1016/j.cie.2020.106712.

    Article  Google Scholar 

  60. Konstantinidis FK, Kansizoglou I, Santavas N, et al. MARMA: a mobile augmented reality maintenance assistant for fast-track repair procedures in the context of industry 4.0. Machines. 2020;8:88. https://doi.org/10.3390/machines8040088.

    Article  Google Scholar 

  61. Schwerdtfeger B, Reif R, Gunthner WA, et al (2009) Pick-by-Vision: A first stress test. In: 2009 8th IEEE International Symposium on Mixed and Augmented Reality. IEEE, Orlando, FL, USA, pp 115–124

  62. Loch F, Quint F, Brishtel I (2016) Comparing Video and Augmented Reality Assistance in Manual Assembly. In: 2016 12th International Conference on Intelligent Environments (IE). IEEE, London, United Kingdom, pp 147–150

  63. Aschenbrenner D, Rojkov M, Leutert F, et al (2018) Comparing Different Augmented Reality Support Applications for Cooperative Repair of an Industrial Robot. In: 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, Munich, Germany, pp 69–74

  64. Bayona, S, Fernandez-Arroyo, JM, Bayona, P, & Martin, I. "A Global Approach to the Design and Evaluation of Virtual Reality Medical Simulators." Proceedings of the ASME 2011 World Conference on Innovative Virtual Reality. ASME 2011 World Conference on Innovative Virtual Reality. Milan, Italy. June 27–29, 2011. pp. 367–376. ASME. https://doi.org/10.1115/WINVR2011-5554

  65. Harrington CM, Kavanagh DO, Quinlan JF, et al. Development and evaluation of a trauma decision-making simulator in Oculus virtual reality. Am J Surg. 2018;215:42–7. https://doi.org/10.1016/j.amjsurg.2017.02.011.

    Article  Google Scholar 

  66. Bordegoni M. Product virtualization: an effective method for the evaluation of concept design of new products. In: Bordegoni M, Rizzi C, editors. Innovation in product design: from CAD to virtual prototyping. London: Springer; 2011. p. 117–41.

    Chapter  Google Scholar 

  67. Rieuf V, Bouchard C, Meyrueis V, Omhover J-F. Emotional activity in early immersive design: Sketches and moodboards in virtual reality. Des Stud. 2017;48:43–75. https://doi.org/10.1016/j.destud.2016.11.001.

    Article  Google Scholar 

  68. Brioso X, Calderon-Hernandez C, Irizarry J, Paes D. Using immersive virtual reality to improve choosing by advantages system for the selection of fall protection measures. In: Computing in civil engineering 2019. Atlanta, Georgia: American Society of Civil Engineers; 2019. p. 146–53.

    Chapter  Google Scholar 

  69. Gabajová G, Furmannová B, Medvecká I, et al. Virtual training application by use of augmented and virtual reality under university technology enhanced learning in slovakia. Sustainability. 2019;11:6677. https://doi.org/10.3390/su11236677.

    Article  Google Scholar 

  70. Fiorentino M, de Amicis R, Monno G, Stork A (2002) Spacedesign: a mixed reality workspace for aesthetic industrial design. In: Proceedings. International Symposium on Mixed and Augmented Reality. IEEE Comput. Soc, Darmstadt, Germany, pp 86–318

  71. Gong L, Berglund J, Saluäär D, Johansson B. A novel VR tool for collaborative planning of manufacturing process change using point cloud data. Procedia CIRP. 2017;63:336–41. https://doi.org/10.1016/j.procir.2017.03.089.

    Article  Google Scholar 

  72. Ramos Mota RC, Cartwright S, Sharlin E, et al (2016) Exploring Immersive Interfaces for Well Placement Optimization in Reservoir Models. In: Proceedings of the 2016 Symposium on Spatial User Interaction. ACM, Tokyo Japan, pp 121–130

  73. Lawson G, Herriotts P, Malcolm L, et al. The use of virtual reality and physical tools in the development and validation of ease of entry and exit in passenger vehicles. Appl Ergon. 2015;48:240–51. https://doi.org/10.1016/j.apergo.2014.12.007.

    Article  Google Scholar 

  74. Bruno F, Cosco F, Angilica A, Muzzupappa M (2010) Mixed Prototyping for Products Usability Evaluation. In: Volume 3: 30th Computers and Information in Engineering Conference, Parts A and B. ASMEDC, Montreal, Quebec, Canada, pp 1381–1390

  75. Khan MA, Israr S, Almogren S, A, et al. Using augmented reality and deep learning to enhance Taxila Museum experience. J Real-Time Image Proc. 2021;18:321–32. https://doi.org/10.1007/s11554-020-01038-y.

    Article  Google Scholar 

  76. Halabi O. Immersive virtual reality to enforce teaching in engineering education. Multimed Tools Appl. 2020;79:2987–3004. https://doi.org/10.1007/s11042-019-08214-8.

    Article  Google Scholar 

  77. Mastrolembo Ventura S, Castronovo F, Ciribini ALC. A design review session protocol for the implementation of immersive virtual reality in usability-focused analysis. ITcon. 2020;25:233–53. https://doi.org/10.36680/j.itcon.2020.014.

    Article  Google Scholar 

  78. Salzmann H, Froehlich B (2008) The Two-User Seating Buck: Enabling Face-to-Face Discussions of Novel Car Interface Concepts. 2008 IEEE Virtual Reality Conference, 75–82. https://doi.org/10.1109/VR.2008.4480754

  79. Otto M, Prieur M, Agethen P, Rukzio E. Dual reality for production verification workshops: a comprehensive set of virtual methods. Procedia CIRP. 2016;44:38–43. https://doi.org/10.1016/j.procir.2016.02.140.

    Article  Google Scholar 

  80. Luo X, Mojica Cabico CD. Development and evaluation of an augmented reality learning tool for construction engineering education. In: Construction research congress 2018. New Orleans, Louisiana: American Society of Civil Engineers; 2018. p. 149–59.

    Chapter  Google Scholar 

  81. Saunders J, Davey S, Bayerl PS, Lohrmann P (2019) Validating Virtual Reality as an Effective Training Medium in the Security Domain. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Osaka, Japan, pp 1908–1911

  82. Grandi F, Zanni L, Peruzzini M, et al. A Transdisciplinary digital approach for tractor’s human-centred design. Int J Comput Integr Manuf. 2020;33:377–97. https://doi.org/10.1080/0951192X.2019.1599441.

    Article  Google Scholar 

  83. Peruzzini M, Pellicciari M, Grandi F, Oreste Andrisano A. A multimodal virtual reality set-up for human- centered design of industrial workstations. DYNAII. 2019;94:182–8. https://doi.org/10.6036/8889.

    Article  Google Scholar 

  84. Shi Y, Zhu Y, Mehta RK, Du J. A neurophysiological approach to assess training outcome under stress: a virtual reality experiment of industrial shutdown maintenance using functional near-infrared spectroscopy (fNIRS). Adv Eng Inf. 2020;46:101153. https://doi.org/10.1016/j.aei.2020.101153.

    Article  Google Scholar 

  85. Torres-Guerrero F, Neira-Tovar L, Torres-Treviño L. An introductive training for welding workshop: a biometric evaluation using virtual reality scenes to improve practice. In: Trojanowska J, Ciszak O, Machado JM, Pavlenko I, editors. Advances in manufacturing II. Cham: Springer International Publishing; 2019. p. 319–31.

    Chapter  Google Scholar 

  86. Louison C, Ferlay F, Keller D, Mestre DR. Operators’ accessibility studies for assembly and maintenance scenarios using virtual reality. Fusion Eng Des. 2017;124:610–4. https://doi.org/10.1016/j.fusengdes.2017.03.017.

    Article  Google Scholar 

  87. Mehrfard A, Fotouhi J, Forster T, et al. On the effectiveness of virtual reality-based training for surgical robot setup. Comp Methods Biomech Biomed Eng Imag Visualizat. 2021;9:243–52. https://doi.org/10.1080/21681163.2020.1835558.

    Article  Google Scholar 

  88. Helin K, Kuula T, Vizzi C, et al. User experience of augmented reality system for astronaut’s manual work support. Front Robot AI. 2018;5:106. https://doi.org/10.3389/frobt.2018.00106.

    Article  Google Scholar 

  89. Hettig J, Engelhardt S, Hansen C, Mistelbauer G. AR in VR: assessing surgical augmented reality visualizations in a steerable virtual reality environment. Int J CARS. 2018;13:1717–25. https://doi.org/10.1007/s11548-018-1825-4.

    Article  Google Scholar 

  90. Rhijn JV, Bosch T, Looze MD (2011) Mixed reality system and objective ergonomics evaluation for designing work stations in manufacturing industry. undefined

  91. Bruno F, Muzzupappa M. Product interface design: A participatory approach based on virtual reality. Int J Hum Comput Stud. 2010;68:254–69. https://doi.org/10.1016/j.ijhcs.2009.12.004.

    Article  Google Scholar 

  92. Kloiber S, Schinko C, Settgast V, et al. Integrating assembly process design and VR-based evaluation using the unreal engine. GRAPP. 2020. https://doi.org/10.5220/0008965002710278.

    Article  Google Scholar 

  93. Kim C, Lee C, Lehto M, Yun M. Evaluation of customer impressions using virtual prototypes in the internet environment. Int J Ind Ergon. 2011. https://doi.org/10.1016/J.ERGON.2010.12.006.

    Article  Google Scholar 

  94. Abidi M, Al-Ahmari A, El-Tamimi A, et al. Development and evaluation of the virtual prototype of the first saudi arabian-designed car. Computers. 2016;5:26. https://doi.org/10.3390/computers5040026.

    Article  Google Scholar 

  95. Heydarian A, Carneiro J, Gerber D, et al. Immersive virtual environments versus physical built environments: a benchmarking study for building design and user-built environment explorations. Autom Constr. 2015. https://doi.org/10.1016/J.AUTCON.2015.03.020.

    Article  Google Scholar 

  96. Dietrich T, Rundle-Thiele S, Kubacki K, et al. Virtual reality in social marketing: a process evaluation. MIP. 2019;37:806–20. https://doi.org/10.1108/MIP-11-2018-0537.

    Article  Google Scholar 

  97. Sutcliffe AG, Poullis C, Gregoriades A, et al. Reflecting on the design process for virtual reality applications. Int J Human-Comput Interact. 2019;35:168–79. https://doi.org/10.1080/10447318.2018.1443898.

    Article  Google Scholar 

  98. Dongas R (2018) Special Session—Designing Virtual Learning Experiences: Iterating Immersive Environments. In: 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE). pp 1225–1227

  99. Ghisleni CV, Frankenberg-Berger AV, de Oliveira MF, et al. Data collection using virtual reality contributions of human-centered design for research practices. In: Ahram T, Taiar R, editors., et al., Human interaction emerging technologies and future systems V. Cham: Springer International Publishing; 2022. p. 346–54.

    Chapter  Google Scholar 

Download references

Funding

Authors received partial funding for this research from the ANR under the N-Hum-Inno labcom program (No. ANR-18-LCV3-0007-01) and the company TEA under the CIFRE (No. 2019/0657).

Author information

Authors and Affiliations

  1. Université de Lorraine, ERPI, F-54000, Nancy, France

    Anaëlle Hily, Laurent Dupont, Giovanny Arbelaez-Garces & Mauricio Camargo

  2. Université de Lorraine, 2LPN, F-54000, Nancy, France

    Anaëlle Hily & Jérôme Dinet

Authors
  1. Anaëlle Hily
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Laurent Dupont
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Giovanny Arbelaez-Garces
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mauricio Camargo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jérôme Dinet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anaëlle Hily.

Ethics declarations

Conflict of interest

The authors have no conflict of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hily, A., Dupont, L., Arbelaez-Garces, G. et al. Evaluation and Validation Process of Extended Reality Applications Developed in an Industrial Context: A Systematic Review. SN COMPUT. SCI. 4, 637 (2023). https://doi.org/10.1007/s42979-023-02089-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-023-02089-1

Keywords

Search

Navigation