Skip to main content
SpringerLink
Log in

Evaluation Methods Applied to Virtual Reality Educational Applications: A Systematic Review

  • Conference paper
  • First Online:
Computational Science and Its Applications – ICCSA 2021 (ICCSA 2021)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12958))

Included in the following conference series:

Abstract

Evaluation is an important activity in software development since it allows detecting issues like design flaws under different analysis perspectives. However, conducting a satisfactory assessment is challenging, especially considering new interactive technologies such as virtual reality (VR). This paper presents a systematic review revealing the most used evaluation methods in this context to deepen the understanding of the evaluation of educational applications employing VR. Considering a search in the ACM database with a filter for the last six years, we selected 1351 initial studies and 81 for analysis. The results show that the tests of usability, flow, and technology acceptance model are the most common, as well as the methods used in many articles present weaknesses due to the lack of an adequate theoretical foundation. Accordingly, we strengthen the relevance of using well-defined support theories in all stages of an evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://ieeexplore.ieee.org/Xplore/home.jsp.

  2. 2.

    https://dl.acm.org/.

  3. 3.

    https://www.scopus.com/search/form.uri?display=basic.

  4. 4.

    https://apps.webofknowledge.com/.

  5. 5.

    https://www.google.com/forms/about/.

  6. 6.

    http://www.igroup.org/index.php.

  7. 7.

    http://attrakdiff.de/index-en.html.

  8. 8.

    https://www.ibm.com/br-pt/analytics/spss-statistics-software.

  9. 9.

    https://www.oculus.com.

  10. 10.

    https://arvr.google.com/cardboard/.

References

  1. American Educational Research Association, American Psychological Association, N.C.o.M.i.E.: Standards for Educational and Psychological Testing. Amer Educational Research Assn (2014)

    Google Scholar 

  2. Bach, C., Scapin, D.L.: Obstacles and perspectives for evaluating mixed reality systems usability. In: Acte du Workshop MIXER, IUI-CADUI, vol. 4. Citeseer (2004)

    Google Scholar 

  3. Bian, Y., Yang, C., Gao, F., Li, H., Zhou, S., Li, H., Sun, X., Meng, X.: A framework for physiological indicators of flow in vr games: construction and preliminary evaluation. Pers. Ubiq. Comput. 20(5), 821–832 (2016)

    Article  Google Scholar 

  4. Brooke, J., et al.: Sus-a quick and dirty usability scale. Usab. Eval. Ind. 189(194), 4–7 (1996)

    Google Scholar 

  5. Csikszentmihalyi, M., Larson, R.: Validity and reliability of the experience-sampling method. In: Flow and the Foundations of Positive Psychology, pp. 35–54. Springer, Dordrecht (2014). https://doi.org/10.1007/978-94-017-9088-8_3

    Chapter  Google Scholar 

  6. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q., 319–340 (1989)

    Google Scholar 

  7. Dünser, A., Grasset, R., Billinghurst, M.: A survey of evaluation techniques used in augmented reality studies. Human Interface Technology Laboratory New Zealand (2008)

    Google Scholar 

  8. Elmore, P.B.: Reporting standards for research publications. Couns. Outcome Res. Eval. 1(2), 19–29 (2010)

    Article  Google Scholar 

  9. Engeser, S., Rheinberg, F.: Flow, performance and moderators of challenge-skill balance. Motiv. Emot. 32, 158–172 (2008). https://doi.org/10.1007/s11031-008-9102-4

    Article  Google Scholar 

  10. Greene, J.C., Caracelli, V.J., Graham, W.F.: Toward a conceptual framework for mixed-method evaluation designs. Educ. Eval. Policy Anal. 11(3), 255–274 (1989)

    Article  Google Scholar 

  11. Hart, S.G.: Nasa-task load index (nasa-tlx); 20 years later. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 50, pp. 904–908. Sage Publications Sage CA, Los Angeles (2006)

    Google Scholar 

  12. Hevner, A.R., March, S.T., Park, J., Ram, S.: Design science in information systems research. Manag. Inf. Syst. Q. 28, 75–105 (2004)

    Article  Google Scholar 

  13. Ho, C.C., MacDorman, K.F.: Measuring the uncanny valley effect. Int. J. Soc. Rob. 9(1), 129–139 (2017)

    Article  Google Scholar 

  14. Kennedy, R.S., Lane, N.E., Berbaum, K.S., Lilienthal, M.G.: Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 3(3), 203–220 (1993)

    Article  Google Scholar 

  15. Kitchenham, B.: Procedures for performing systematic reviews. Keele, UK, Keele Univ. 33(2004), 1–26 (2004)

    Google Scholar 

  16. Kitcher, P.: The Advancement of Science: Science without Legend, Objectivity without Illusions. Oxford University Press, Oxford (1995)

    Book  Google Scholar 

  17. Kuhn, D., Pearsall, S.: Developmental origins of scientific thinking. J. Cogn. Dev. 1(1), 113–129 (2000)

    Article  Google Scholar 

  18. Laugwitz, B., Held, T., Schrepp, M.: Construction and evaluation of a user experience questionnaire. In: Holzinger, A. (ed.) USAB 2008. LNCS, vol. 5298, pp. 63–76. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89350-9_6

    Chapter  Google Scholar 

  19. Lessiter, J., Freeman, J., Keogh, E., Davidoff, J.: A cross-media presence questionnaire: the itc-sense of presence inventory. Pres. Teleoper. Virtual Environ. 10(3), 282–297 (2001)

    Article  Google Scholar 

  20. Lund, A.M.: Measuring usability with the use questionnaire12. Usab. Interface 8(2), 3–6 (2001)

    Google Scholar 

  21. Makransky, G., Lilleholt, L., Aaby, A.: Development and validation of the multimodal presence scale for virtual reality environments: a confirmatory factor analysis and item response theory approach. Comput. Human Behav. 72 (2017). https://doi.org/10.1016/j.chb.2017.02.066

  22. Nielsen, J.: Thinking aloud: The# 1 usability tool. Nielsen Norman Group 16 (2012)

    Google Scholar 

  23. Rauschenberger, M., Schrepp, M., Pérez Cota, M., Olschner, S., Thomaschewski, J.: Efficient measurement of the user experience of interactive products. How to use the user experience questionnaire (ueq). example: Spanish language version. Int. J. Interact. Multimedia Artif. Intell. (2013)

    Google Scholar 

  24. Roberts, P.G., Guyver, P., Baldwin, M., Akhtar, K., Alvand, A., Price, A.J., Rees, J.L.: Validation of the updated arthros simulator: face and construct validity of a passive haptic virtual reality simulator with novel performance metrics. Knee Surg. Sports Traumatol. Arthrosc. 25(2), 616–625 (2017)

    Article  Google Scholar 

  25. Ruthenbeck, G.S., Reynolds, K.J.: Virtual reality for medical training: the state-of-the-art. J. Simul. 9(1), 16–26 (2015)

    Article  Google Scholar 

  26. Rutten, N., Van Joolingen, W.R., Van Der Veen, J.T.: The learning effects of computer simulations in science education. Comput. Educ. 58(1), 136–153 (2012)

    Article  Google Scholar 

  27. Schubert, T., Friedmann, F., Regenbrecht, H.: The experience of presence: factor analytic insights. Pres. Teleoper. Virt. Environ. 10(3), 266–281 (2001)

    Article  Google Scholar 

  28. Shadish, W.R., Cook, T.D., Campbell, D.T., et al.: Experimental and quasi-experimental designs for generalized causal inference/William R. Shedish, Thomas D. Cook, Donald T. Campbell. Houghton Mifflin, Boston (2002)

    Google Scholar 

  29. Sharp, H., Rogers, Y.J.P.: Design de Interação: Além da interação humano computador. Bookman (2005)

    Google Scholar 

  30. Slater, M., Usoh, M., Steed, A.: Depth of presence in virtual environments. Pres. Teleoper. Virt. Environ. 3(2), 130–144 (1994)

    Article  Google Scholar 

  31. Sommerville, I.: Engenharia de software. PEARSON BRASIL (2011). https://books.google.com.br/books?id=H4u5ygAACAAJ

  32. Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D.: User acceptance of information technology: toward a unified view. MIS Q., 425–478 (2003)

    Google Scholar 

  33. Villagrasa, S., Fonseca, D., Durán, J.: Teaching case: applying gamification techniques and virtual reality for learning building engineering 3d arts. In: Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality, pp. 171–177 (2014)

    Google Scholar 

  34. Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: a presence questionnaire. Presence 7(3), 225–240 (1998)

    Article  Google Scholar 

  35. Zeri, F., Livi, S.: Visual discomfort while watching stereoscopic three-dimensional movies at the cinema. Ophthal. Physiol. Opt. 35 (2015). https://doi.org/10.1111/opo.12194

Download references

Acknowledgments

This project has partially supported by Huawei do Brasil Telecomunicações Ltda (Fundunesp Process # 3123/2020), and CAPES.

Author information

Authors and Affiliations

  1. Universidade Federal de São João del-Rei, São João del-Rei, Brazil

    Daniel Bueno Domingueti, Diego Roberto Colombo Dias & Dárlinton Barbosa Feres Carvalho

  2. Universidade Federal de São Paulo/UNIFACCAMP, Campo Limpo Paulista, Brazil

    Marcelo de Paiva Guimarães

Authors
  1. Daniel Bueno Domingueti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Diego Roberto Colombo Dias
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marcelo de Paiva Guimarães
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dárlinton Barbosa Feres Carvalho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Bueno Domingueti .

Editor information

Editors and Affiliations

  1. University of Perugia, Perugia, Italy

    Osvaldo Gervasi

  2. University of Basilicata, Potenza, Potenza, Italy

    Beniamino Murgante

  3. Covenant University, Ota, Nigeria

    Sanjay Misra

  4. University of Cagliari, Cagliari, Italy

    Chiara Garau

  5. University of Cagliari, Cagliari, Italy

    Ivan Blečić

  6. Monash University, Clayton, VIC, Australia

    David Taniar

  7. Kyushu Sangyo University, Fukuoka, Japan

    Bernady O. Apduhan

  8. University of Minho, Braga, Portugal

    Ana Maria A. C. Rocha

  9. Polytechnic University of Bari, Bari, Italy

    Eufemia Tarantino

  10. Polytechnic University of Bari, Bari, Italy

    Carmelo Maria Torre

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Domingueti, D.B., Dias, D.R.C., Guimarães, M.d.P., Carvalho, D.B.F. (2021). Evaluation Methods Applied to Virtual Reality Educational Applications: A Systematic Review. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2021. ICCSA 2021. Lecture Notes in Computer Science(), vol 12958. Springer, Cham. https://doi.org/10.1007/978-3-030-87016-4_46

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87016-4_46

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87015-7

  • Online ISBN: 978-3-030-87016-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation