Skip to main content
Springer Nature Link
Log in

The Development and Validation of an Augmented and Mixed Reality Usability Heuristic Checklist

  • Conference paper
  • First Online:
Virtual, Augmented and Mixed Reality: Design and Development (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13317))

Included in the following conference series:

  • 2871 Accesses

Abstract

Augmented Reality (AR) and Mixed Reality (MR) are new emerging technologies that are becoming increasingly popular. Because these technologies are so new, there is a lack of standards and consistency in application and hardware design. This can make it difficult to learn and frustrating for users. One way to standardized design and enhance the usability of a product is through the use of heuristic evaluations. General heuristics, such as Nielsen’s 10 or Schneiderman’s 8 usability heuristics have been used to evaluate these technologies. These heuristics are a useful starting point because they bring attention to many crucial aspects of the usability of a product. However, additional aspects that could alter the users’ experience may not be assessed due to the uniqueness of AR and MR. There are very few validated AR and MR heuristics in the literature for practitioners to use. The purpose of this study was to create and validate a heuristic checklist that can be used to assess and inform design changes that influence the user experience of an AR or MR application and/or device. We followed an established and comprehensive 8-stage methodology developed by Quiñones, Rusu, & Rusu to create and validate our AR and MR usability heuristic checklist [4]. This included a search and summary of the current literature, formally defining heuristics based on this literature search, and validating through heuristic evaluations, expert reviews, and user testing. Our final revised heuristic checklist included 11 heuristics and 94 checklist items that encompasses usability aspects for AR and MR technologies.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Milgram, P., Kishino, F.: A taxonomy of mixed reality displays. IEICE Trans. Inf. Syst. 77(12), 1321–1329 (1994)

    Google Scholar 

  2. Azuma, R.T.: A survey of augmented reality. Presence Teleoperators Virtual Environ. 6(4), 355–385 (1997)

    Article  Google Scholar 

  3. Microsoft Docs: Mixed Reality Design Start designing and prototyping, https://docs.microsoft.com/en-us/windows/mixed-reality/design/design, Accessed 10 Feb 2022

  4. Quiñones, D., Rusu, C., Rusu, V.: A methodology to develop usability/user experience heuristics. Comput. Stan. interfaces 59, 109–129 (2018)

    Article  Google Scholar 

  5. Nielsen, J., Molich, R.: Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 249–256, Association for computer machinery, New York, NY (1990)

    Google Scholar 

  6. Nielsen, J.: Ten Usability Heuristics (2005)

    Google Scholar 

  7. Schneiderman, B., Plaisant, C., Cohen, M.S., Jacobs, S., Elmqvist, N., Diakopoulous, N.: Designing the User Interface: Strategies for Effective Human-Computer Interaction. Pearson, Hoboken, NJ (2016)

    Google Scholar 

  8. Gonzalez-Holland, E., Whitmer, D., Moralez, L., Mouloua, M.: Examination of the user of Nielsen’s 10 usability heuristics & outlooks for the future. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, pp. 1472–1475 (2017)

    Google Scholar 

  9. De Paiva Guimarães, M., Martins, V.F., Corrêa, A.G.D.: Usability Evaluation of Learning Objectives with Augmented Reality for Smartphones: A Reinterpretation of Nielsen heuristics. Human-Computer Interaction, pp. 214–228 (2019)

    Google Scholar 

  10. Usability Heuristics Applied to Virtual Reality: https://www.nngroup.com/articles/usability-heuristics-virtual-reality/, Accessed 11 Feb 2022

  11. Endsley, T., Sprehn, K., Brill, R., Ryan, K., Vincent, E., Martin, J.: Augmented reality design heuristics: designing for dynamic interactions. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, pp. 2100–2104 (2017)

    Google Scholar 

  12. Kalalahti, J., Porras, J.: Developing Usability Evaluation Heuristics for Augmented Reality Applications (2015)

    Google Scholar 

  13. De Paiva Guimarães, M., Martins, V.F.: A checklist to evaluate augmented reality applications. In: 2014 XVI Symposium of Virtual and Augmented Reality, pp. 45–52. IEEE (2014)

    Google Scholar 

  14. Aultman, A., Dowie, S., Hamid, N.: Design heuristics for mobile augmented game user interfaces. In: CHI EA ’18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–5. Association for Computing Machinery, New York, NY (2018)

    Google Scholar 

  15. Wetzel, R., McCall, R., Braun, A., Broll, W.: Guidelines for designing augmented reality games. In: Proceedings of the 2008 Conference on Future Play: Research, Play, Share, pp. 173–180. Association for Computing Machinery, New York, NY (2008)

    Google Scholar 

  16. Santos, C., Miranda, B., Araujo, T., Carneiro, N., Marques, A., Mota, M., Morais, J., Meiguins, B.: Guidelines for graphical user interface design in mobile augmented reality applications. In: Lackey, S., Shumaker, R. (eds.) VAMR 2016, LNCS 9740, pp. 71–80 (2016)

    Google Scholar 

  17. Ko, S.M., Chang, W.S., Ji, Y.G.: Usability principles for augmented reality applications in a smartphone environment. In: International Journal of Human-Computer Interaction 29(8), 501–515 (2013)

    Google Scholar 

  18. Ganapathy, S.: Human Factors in Augmented Reality Environments. Springer Science + Business Media, New York, NY (2013)

    Google Scholar 

  19. Gale, N., Mirza-Babaei, P., Pedersen, I.: Heuristic guidelines for wearable augmented reality applications. In: Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, CHI PLAY, Association for Computing Machinery, New York, NY (2015)

    Google Scholar 

  20. Franklin, F., Breyer, F., Kelner, J.: Usability heuristics for collaborative augmented reality remote systems. In: Proceedings of the 16th Symposium on Virtual and Augmented Reality, SVR, pp. 53–62. IEEE, Piata Salvador, Brazil (2014)

    Google Scholar 

  21. Tuli, N., Mantri, A.: Usability principles for augmented reality based kindergarten applications. Procedia Comput. Sci. 172, 679–687 (2020)

    Article  Google Scholar 

  22. Masmuzidin, M., Aziz, N.: The adaptation of Shneiderman’s golden rules and Nielsen’s heuristics on motivational augmented reality technology design for young children. In: 2019 IEEE 9th International Conference on System Engineering and Technology, ICSET, pp. 62–67, IEEE, Shah Alam, Malaysia (2019)

    Google Scholar 

  23. Liang, S.: Design principles of augmented reality focusing on the aging population. In: Pr4oceedings of the 30th International BCS Human Computer Interaction Conference, pp. 1–7. BCS Learning and Development Ltd, Swindon, England (2016)

    Google Scholar 

  24. Gotsis, M., Lympouridis, V., Turpin, D., Tasse, A., Poulos, I.C., Tucker, D., Swider, M., Thin, A.G., Jordan-Marsh, M.: Mixed reality game prototypes for upper body exercise and rehabilitation. In: Coquillart, S., Feiner, S., Kiyokawa, K. (eds.) 2012 IEEE, Virtual Reality Workshops (VRW), pp. 181–182 (2012)

    Google Scholar 

  25. de Belen, R.A., Bednarz, T.: Mixed reality and the internet of things (MRIoT) Interface design guidelines for elderly people. In: Wyeld, T.G., Banissi, E., Ursyn, A., Bannatyne, M., Datia, N., Sarfraz, M. (eds.) International Conference In Information Visualization, Imaging and Visualization, CGIV, pp. 82–85. Adelaide, SA, Australia (2019)

    Google Scholar 

  26. Helle, S., Lehtonen, T., Woodward, C., Turunen, M., Salmi, H. (eds.): Miracle Handbook: Guidelines for Mixed Reality Applications for Culture and Learning Experiences. University of Turku, Turku, Finland (2017)

    Google Scholar 

  27. XR Association: XR primer 2.0: a starter guide for developers. https://xra.org/research/xr-primer-1-0-a-starter-guide-for-developers/, Accessed 11 Feb 2022

  28. Apple: getting started with AR: https://developer.apple.com/news/?id=c6vr1ag2, Accessed 11 Feb 2022

  29. Magic leap: design principles: https://developer.magicleap.com/en-us/learn/guides/design-principles, Accessed 11 Feb 2022

  30. ISO 9241-11:2018: https://www.iso.org/standard/63500.html#:~:text=ISO%209241%2D11%3A2018%20provides,services%20(including%20technical%20and%20personal, Accessed 11 Feb 2022

  31. Usability 101: Introduction to usability: https://www.nngroup.com/articles/usability-101-introduction-to-usability/, Accessed 11 Feb 2022

  32. User Experience Basics: https://www.usability.gov/what-and-why/user-experience.html, Accessed 11 Feb 2022

  33. Principles for Usable Design: https://www.usabilitybok.org/principles-for-usable-design, Accessed 11 Feb 2022

  34. Bangor, A., Kortum, P.T., Miller, J.T.: An empirical evaluation of the system usability scale. Intl. J. Human-Comput. Interaction 24(6), 574–594 (2008)

    Article  Google Scholar 

  35. Bangor, A., Kortum, P., Miller, J.: Determining what individual SUS scores mean: adding an adjective rating scale. J. Usability Stud. 4(3), 114–123 (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Embry-Riddle Aeronautical University, Daytona Beach, FL, USA

    Jessyca L. Derby & Barbara S. Chaparro

Authors
  1. Jessyca L. Derby
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barbara S. Chaparro
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jessyca L. Derby .

Editor information

Editors and Affiliations

  1. U.S. Army Research Laboratory, Aberdeen Proving Ground, MD, USA

    Jessie Y. C. Chen

  2. U.S. Army Combat Capabilities Development Command Soldier Center, Orlando, FL, USA

    Gino Fragomeni

Appendix

Appendix

Derby & Chaparro AR and MR Usability Heuristic Checklist

Unboxing & Set-Up

  1. 1.

    Is the unboxing process a positive experience?

  2. 2.

    When the user interacts with the device for the first time, are they introduced to the user interface, basic interaction methods, and basic features/content?

  3. 3.

    Is a quick start guide available with the device?

  4. 4.

    Is a call to action (QR code, instructions to use AR, etc.) clearly marked in the physical space?

Help & Documentation

  1. 5.

    Is there the option of a tutorial upon first use of the device and/or application?

  2. 6.

    Does the tutorial explain all of the necessary actions/mechanics to use the device and/or application?

  3. 7.

    Is the tutorial easy to understand?

  4. 8.

    Are required interactions easy to learn?

  5. 9.

    Is help or documentation easily accessible for the application?

  6. 10.

    Are instructions easy to understand?

  7. 11.

    Do instructions provide actionable feedback?

  8. 12.

    If auditory instructions are given, do these instructions match what the user is seeing in the application?

  9. 13.

    Are error messages easy to understand?

  10. 14.

    Do error messages provide actionable feedback?

  11. 15.

    Does the device’s user interface and/or application avoid irreversible errors?

  12. 16.

    Is there a way for the user to report errors or crashes to the developer?

Cognitive Overload

  1. 17.

    Is the user eased into the virtual environment?

  2. 18.

    Does the device’s user interface and/or application avoid clutter, as appropriate?

  3. 19.

    Does the device’s user interface and/or application avoid large amounts of text?

  4. 20.

    Does the screen space focus on the virtual elements rather than controls or other non-AR features, as appropriate?

  5. 21.

    Is information organized in an understandable manner?

  6. 22.

    If the quantity of information is large, is it organized in a layered or hierarchical manner so it is easy to understand?

  7. 23.

    Does the device’s user interface and/or application avoid tasks that involve a large amount of steps to complete?

  8. 24.

    Does the application make use of all of its AR functions (including information that is visual, auditory, and involved other sensory modalities)?

Integration of Physical & Virtual Worlds

  1. 25.

    Are physical (real-world) elements easily distinguishable from virtual elements?

  2. 26.

    Is it clear which virtual elements can be interacted with and which cannot?

  3. 27.

    Does the device’s user interface and/or application avoid obstructing physical or virtual elements that are necessary for the users’ goals?

  4. 28.

    Does the device’s user interface and/or application avoid obstructing virtual navigation elements?

Consistency & Standards

  1. 29.

    Are virtual elements easy to delete or close out of?

  2. 30.

    Can the user pause the application at any point?

  3. 31.

    Are all aspects of the device’s user interface and/or application (virtual elements, controls, text, etc.) clear and readable?

  4. 32.

    Are virtual elements sized appropriately?

  5. 33.

    Are virtual elements rendered a reasonable distance away from the user’s targeted point?

  6. 34.

    For mobile devices, are the controls based on known interactions for mobile devices?

  7. 35.

    For mobile devices, are landscape and portrait mode supported?

  8. 36.

    For mobile devices, is the application responsive?

  9. 37.

    Do virtual elements act as the user would expect them to in the real world?

  10. 38.

    Are virtual elements accurately placed on the real environment?

  11. 39.

    Does the device and/or application avoid lag, delays, jitter, drift, and other forms of virtual element malfunctions?

  12. 40.

    Is the navigation consistent throughout the device and/or application?

  13. 41.

    Can the user navigate freely throughout aspects of the device and/or application?

  14. 42.

    Does the device adjust to the environment it is used in?

  15. 43.

    Are environmental requirements clearly defined?

  16. 44.

    Does the device and/or application remind users to be aware of their surroundings?

  17. 45.

    Does the device and/or application avoid jargon?

  18. 46.

    Are sans serif font types used, as appropriate, throughout the device and/or application?

  19. 47.

    Is the contrast between the background and text sufficient enough that the text can be read easily under a range of normal lighting conditions?

  20. 48.

    If the text background is transparent, is the text visible across different backgrounds and under a range of normal lighting conditions?

  21. 49.

    Is the volume adjustable so the user can hear audio, even in noisy environments?

  22. 50.

    Are auditory features understandable?

  23. 51.

    Are captions available for auditory features as appropriate?

Collaboration

  1. 52.

    Does the device and/or application allow for the user to control privacy-related content?

  2. 53.

    Is it clear what information is private or public content?

  3. 54.

    Does the application allow users to preserve virtual elements from others users’ changes?

  4. 55.

    Is it clear which virtual elements can be interacted with and which cannot for each person?

  5. 56.

    Is content consistent across all users, as it is appropriate?

Comfort

  1. 57.

    Can the user experience the device and/or application without pain, discomfort, nausea, etc. DURING use?

  2. 58.

    Can the user experience the device and/or application without pain, discomfort, nausea, etc. AFTER use?

  3. 59.

    Is the device’s weight light enough to feel comfortable?

  4. 60.

    Does the device avoid overheating to the point that it is uncomfortable to use?

  5. 61.

    Are physical interactions with the application safe and comfortable?

  6. 62.

    Does the application avoid making the user walk backwards, pull their head back, or push their head downwards to see virtual elements?

  7. 63.

    Do interactions with the device and/or application avoid tiring the user?

  8. 64.

    Does the device and/or application avoid causing the user eye strain?

  9. 65.

    Are users reminded to take breaks to prevent eye strain and fatigue?

  10. 66.

    Does the device easily adjust its size for different users?

  11. 67.

    Does the device accommodate users with eyeglasses?

  12. 68.

    Does the device accommodate for personal protective equipment?

Feedback

  1. 69.

    Does the device and/or application provide feedback on its status?

  2. 70.

    Does the device and its accessories provide feedback about battery levels and charging state?

  3. 71.

    Does the device and/or application provide feedback for user input?

  4. 72.

    Does the device and/or application respond quickly to user input?

  5. 73.

    Does the device and/or application provide the user feedback after automatic selections?

  6. 74.

    If an automatic selection occurs, does the device and/or application suggest what to do next?

User Interaction

  1. 75.

    Does the user feel in control?

  2. 76.

    Are user interactions simple and easy to understand?

  3. 77.

    Does the device and/or application include multiple forms of interaction so users can choose based on ability, preference, & skill?

  4. 78.

    Are the forms of interaction direct when it is appropriate to use this form of interaction?

  5. 79.

    Does the device and/or application avoid interactions that force the user to make large or sudden movements?

  6. 80.

    Does the device and/or application accommodate for the user to complete other necessary real-world tasks?

  7. 81.

    Does object manipulation work well in all instances?

  8. 82.

    Do virtual elements adapt to the users’ position appropriately?

  9. 83.

    Does the device and/or application avoid input overloading by assigning distinct functions to buttons or gestures?

Recognition Rather than Recall

  1. 84.

    Are virtual elements and icons self-explanatory (does their form communicate function)?

  2. 85.

    Are virtual elements and controls placed near objects they reference?

  3. 86.

    If a virtual element is related to an object that is in motion, is the virtual element tightly coupled with object in motion appropriately?

  4. 87.

    Are virtual elements that are outside of the field of view easy to find?

  5. 88.

    Are available user actions identifiable?

  6. 89.

    If voice commands are included, are text labels for voice commands given?

Device Maintainability

  1. 90.

    Is the device sturdy enough to withstand multiple uses?

  2. 91.

    Does the device have a sturdy storage case?

  3. 92.

    Is it easy to clean the lenses, cameras, and other components on the device?

  4. 93.

    Are device parts fixable and replaceable as needed?

  5. 94.

    Does the device’s battery life last long enough to perform necessary tasks of the application?

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Derby, J.L., Chaparro, B.S. (2022). The Development and Validation of an Augmented and Mixed Reality Usability Heuristic Checklist. In: Chen, J.Y.C., Fragomeni, G. (eds) Virtual, Augmented and Mixed Reality: Design and Development. HCII 2022. Lecture Notes in Computer Science, vol 13317. Springer, Cham. https://doi.org/10.1007/978-3-031-05939-1_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-05939-1_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-05938-4

  • Online ISBN: 978-3-031-05939-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics