Skip to main content
Springer Nature Link
Log in

Optimizing AR Application Testing: Integrating Metamorphic Testing to Address Developer and End-User Challenges

  • Conference paper
  • First Online:
HCI International 2024 – Late Breaking Papers (HCII 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15377))

Included in the following conference series:

  • 250 Accesses

Abstract

The rapid proliferation of Augmented Reality (AR) applications in gaming, education, and healthcare underscores the urgent need for effective testing methodologies. As AR applications become more complex and widely used, ensuring their robustness and reliability is increasingly challenging. Traditional methods, like Junit and Unity tests, often fall short due to AR’s inherent complexities, including 3D interactions and immersive environments. Our research addresses these challenges by integrating Metamorphic Testing (MT) to enhance AR application testing processes. MT overcomes traditional testing limitations by focusing on Metamorphic Relations (MRs) - expected relationships between inputs and outputs without a reliable test oracle. We define specific MRs tailored for mobile AR to facilitate fault identification and rectification. Our study presents a comprehensive framework for applying MT in mobile AR, detailing the MR identification and implementation process. We identified a set of MRs for mobile AR. These MRs, including Object Scaling with Distance, Raycast within Boundary, Visibility and Occlusion, Overlapping with Game Objects, and Orientation Consistency Relative to Gravity, address crucial aspects of AR functionality. They ensure realistic object scaling, consistent placement, correct occlusion handling, prevention of object overlap, and maintained orientation consistency. Adopting Metamorphic Testing significantly advances AR application testing. We are developing a tool to automatically incorporate these MRs into mobile AR applications, streamlining testing processes and enhancing system reliability. Future work will refine these relations and expand their scope to cover a broader range of AR functionalities.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/brintodibyendu/metamorphic_test_ar.

References

  1. 10 usability heuristics for user interface design. https://www.nngroup.com/articles/ten-usability-heuristics/

  2. Microsoft hololens. https://www.microsoft.com/en-us/hololens/

  3. Oculus quest. https://www.oculus.com/experiences/quest/

  4. Ardiny, H., Khanmirza, E.: The role of ar and vr technologies in education developments: opportunities and challenges. In: 2018 6th RSI International Conference on Robotics and Mechatronics (IcRoM), pp. 482–487 (2018)

    Google Scholar 

  5. Ashtari, N., Bunt, A., McGrenere, J., Nebeling, M., Chilana, P.K.: Creating augmented and virtual reality applications: current practices, challenges, and opportunities. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI 2020, pp. 1–13. Association for Computing Machinery, New York (2020)

    Google Scholar 

  6. Burnett, M., Cook, C., Rothermel, G.: End-user software engineering. Commun. ACM 47(9), 53–58 (2004)

    Google Scholar 

  7. Chan, W., Ho, J., Tse, T.: Piping classification to metamorphic testing: an empirical study towards better effectiveness for the identification of failures in mesh simplification programs, vol. 1, pp. 397–404 (2007)

    Google Scholar 

  8. Chen, T., Cheung, S.-C., Yiu, S.: Metamorphic testing: a new approach for generating next test cases (2020)

    Google Scholar 

  9. Chen, T., Huang, D., Tse, T., Zhou, Z.Q.: Case studies on the selection of useful relations in metamorphic testing (2004)

    Google Scholar 

  10. Chen, T.Y., Cheung, S.C., Yiu, S.M.: Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Department of Computer Science The Hong Kong University of Science and Technology (1998)

    Google Scholar 

  11. Chen, T.Y., et al.: Metamorphic testing: a review of challenges and opportunities. ACM Comput. Surv. (CSUR) 51(1), 1–27 (2018)

    Article  MATH  Google Scholar 

  12. Chuah, S.: Why and who will adopt extended reality technology? literature review, synthesis, and future research agenda (2018)

    Google Scholar 

  13. Doer, K.-U., Schiefel, J., Kubbat, W.: Virtual cockpit simulation for pilot training, p. 8 (2001)

    Google Scholar 

  14. Eswaran, M., Bahubalendruni, M.R.: Challenges and opportunities on AR/VR technologies for manufacturing systems in the context of industry 4.0: a state of the art review. J. Manuf. Syst. 65, 260–278 (2022)

    Article  Google Scholar 

  15. Gandy, M., MacIntyre, B.: Designer’s augmented reality toolkit, ten years later: implications for new media authoring tools. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST 2014, pp. 627–636. Association for Computing Machinery, New York (2014)

    Google Scholar 

  16. Ikeda, B., Szafir, D.: An ar debugging tool for robotics programmers. In: 4th International Workshop on Virtual, Augmented, and Mixed Reality for HRI (2021)

    Google Scholar 

  17. Jameel, T., Lin, M., Chao, L.: Test oracles based on metamorphic relations for image processing applications. In: 2015 IEEE/ACIS 16th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), pp. 1–6 (2015)

    Google Scholar 

  18. Kamińska, D., et al.: Virtual reality and its applications in education: survey. Inf. 10, 318 (2019)

    MATH  Google Scholar 

  19. Kanewala, U., Bieman, J.M.: Using machine learning techniques to detect metamorphic relations for programs without test oracles. In: 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), pp. 1–10 (2013)

    Google Scholar 

  20. Kaur, G., Jyotsna, S.: An efficient metamorphic testing technique using genetic algorithm, vol. 141, pp. 180–188 (2011)

    Google Scholar 

  21. Ko, A.J., et al.: The state of the art in end-user software engineering. ACM Comput. Surv. 43(3) (2011)

    Google Scholar 

  22. Krauß, V., Boden, A., Oppermann, L., Reiners, R.: Current practices, challenges, and design implications for collaborative ar/vr application development. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI 2021. Association for Computing Machinery, New York (2021)

    Google Scholar 

  23. Kuechenmeister, C.A., Linton, P.H., Mueller, T.V., White, H.B.: Eye tracking in relation to age, sex, and illness. Arch. General Psychiat. 34(5), 578–579 (1977)

    Article  Google Scholar 

  24. Lee, W.-S., Kim, J.-H., Cho, J.-H.: A driving simulator as a virtual reality tool. In: Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No. 98CH36146), vol. 1, pp. 71–76 (1998)

    Google Scholar 

  25. Lehman, S.M., Ling, H., Tan, C.C.: Archie: a user-focused framework for testing augmented reality applications in the wild. In: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 903–912 (2020)

    Google Scholar 

  26. Lieberman, H., Paternò, F., Klann, M., Wulf, V.: End-user development: an emerging paradigm, vol. 9, pp. 1–8 (2006)

    Google Scholar 

  27. Liu, H., Kuo, F.-C., Towey, D., Chen, T.Y.: How effectively does metamorphic testing alleviate the oracle problem? IEEE Trans. Softw. Eng. 40(1), 4–22 (2013)

    Article  MATH  Google Scholar 

  28. Mayer, J., Guderlei, R.: An empirical study on the selection of good metamorphic relations. In: 30th Annual International Computer Software and Applications Conference (COMPSAC’06), vol. 1, pp. 475–484 (2006)

    Google Scholar 

  29. Mileva, G.: How is augmented reality transforming special education? AR Post (2021). https://arpost.co/2021/06/25/augmented-reality-special-education/

  30. Murphy, C., Kaiser, G., Hu, L., Wu, L.: Properties of machine learning applications for use in metamorphic testing, pp. 867–872 (2008)

    Google Scholar 

  31. Nicholson, D., Chalk, C., Funnell, W., Daniel, S.: Can virtual reality improve anatomy education? a randomised controlled study of a computer-generated three-dimensional anatomical ear model. Med. Educ. 40, 1081–1087 (2006)

    Article  Google Scholar 

  32. Parkin, S.: How vr is training the perfect soldier (2015). https://www.wareable.com/vr/how-vr-is-training-the-perfect-soldier-1757/

  33. Rzig, D.E., Iqbal, N., Attisano, I., Qin, X., Hassan, F.: Virtual reality (vr) automated testing in the wild: a case study on unity-based vr applications. In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 1269–1281 (2023)

    Google Scholar 

  34. Samadbeik, M., Yaaghobi, D., Bastani, P., Abhari, S., Rezaee, R., Garavand, A.: The applications of virtual reality technology in medical groups teaching. J. Adv. Med. Educ. Professional. 6(3), 123 (2018)

    Google Scholar 

  35. Segura, S., Fraser, G., Sanchez, A.B., Ruiz-Cortés, A.: A survey on metamorphic testing. IEEE Trans. Softw. Eng. 42(9), 805–824 (2016)

    Article  MATH  Google Scholar 

  36. Correa Souza, A.C., Nunes, F.L., Delamaro, M.E.: An automated functional testing approach for virtual reality applications. Softw. Test. Verificat. Reliabil. 28(8), e1690 (2018)

    Article  Google Scholar 

  37. Speicher, M., Hall, B.D., Yu, A., Zhang, B., Zhang, H., Nebeling, J., Nebeling, M.: XD-AR: challenges and opportunities in cross-device augmented reality application development. Proc. ACM Hum.-Comput. Interact. 2(EICS), 1–24 (2018)

    Google Scholar 

  38. Wu, P., Xiao-Chun, S., Jiang-Jun, T., Hui-Min, L.: Metamorphic testing and special case testing: a case study. J. Softw. 16, 07 (2005)

    Article  MATH  Google Scholar 

  39. Yoo, S.: Metamorphic testing of stochastic optimisation. In: 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, pp. 192–201. IEEE (2010)

    Google Scholar 

  40. Zhang, J., et al.: Search-based inference of polynomial metamorphic relations. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE 2014, pp. 701–712. Association for Computing Machinery, New York (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Virginia Tech, Blacksburg, VA, USA

    Dibyendu Brinto Bose, Brendan David-John & Chris Brown

Authors
  1. Dibyendu Brinto Bose
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brendan David-John
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chris Brown
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chris Brown .

Editor information

Editors and Affiliations

  1. U.S. Army Research Laboratory, Adelphi, MD, USA

    Jessie Y. C. Chen

  2. U.S. Army Combat Capabilities, Orlando, FL, USA

    Gino Fragomeni

  3. Smart Future Initiative, Frankfurt, Germany

    Norbert A. Streitz

  4. Kyushu University, Fukuoka, Japan

    Shin'ichi Konomi

  5. DePaul University, Chicago, IL, USA

    Xiaowen Fang

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bose, D.B., David-John, B., Brown, C. (2025). Optimizing AR Application Testing: Integrating Metamorphic Testing to Address Developer and End-User Challenges. In: Chen, J.Y.C., Fragomeni, G., Streitz, N.A., Konomi, S., Fang, X. (eds) HCI International 2024 – Late Breaking Papers. HCII 2024. Lecture Notes in Computer Science, vol 15377. Springer, Cham. https://doi.org/10.1007/978-3-031-76812-5_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-76812-5_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-76811-8

  • Online ISBN: 978-3-031-76812-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics