Abstract
This Chapter discusses issues with Augmented Reality (AR) systems evaluations. First the role of evaluation and various challenges for evaluating novel AR interfaces and interaction techniques are reviewed. A special focus is then provided on user-based and non-user-based evaluation techniques currently used for evaluating AR systems. The practical application of these methods is demonstrated through different examples from the scientific literature. Various points raised in this chapter provide arguments for the development of more specific frameworks and models for AR-based interfaces and interaction techniques. These will provide researchers with a better basis for developing and applying more suitable evaluation methods that address the specific requirements of evaluating AR-based systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
I. Sutherland, “A Head-Mounted Three-Dimensional Display,” presented at the Fall Joint Computer Conf., Am. Federation of Information Processing Soc. (AFIPS), Washington, D.C., USA, 1968.
A. Dünser, R. Grasset, H. Seichter, and M. Billinghurst, “Applying HCI principles to AR systems design,” Charlotte, NC, USA, 2007.
J. E. Swan and J. L. Gabbard, “Survey of User-Based Experimentation in Augmented Reality,” presented at the 1st International Conference on Virtual Reality, HCI International 2005, Las Vegas, USA, 2005.
A. Dünser, R. Grasset, and M. Billinghurst, “A survey of evaluation techniques used in augmented reality studies,” presented at the ACM SIGGRAPH ASIA 2008 courses, Singapore, 2008.
J. Nielsen, Usability Engineering. San Francisco: Morgan Kaufmann, 1993.
H. Sharp, Y. Rogers, and J. Preece, Interaction Design: Beyond Human-Computer Interaction: Wiley, 2007.
P. M. Fitts, “The information capacity of the human motor system in controlling the amplitude of movement,” Journal of Experimental Psychology, vol. 47, pp. 381–391, 1954.
R. S. Dicks, “Mis-usability: on the uses and misuses of usability testing,” presented at the Proceedings of the 20th annual international conference on Computer documentation, Toronto, Ontario, Canada, 2002.
J. Nielsen, Designing Web Usability. Indianapolis, IN, USA: New Rivers, 2000.
S. Greenberg and B. Buxton, “Usability evaluation considered harmful (some of the time),” CHI ’08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, 2008.
D. A. Bowman, J. L. Gabbard, and D. Hix, “A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods,” Presence – Teleoperators and Virtual Environments, vol. 11, pp. 404–424, 2002.
C. Bach and D. L. Scapin, “Adaptation of Ergonomic Criteria to Human-Virtual Environments Interactions,” presented at the INTERACT 2003.
A. Sutcliffe and B. Gault, “Heuristic evaluation of virtual reality applications,” Interacting with Computers, vol. 16, pp. 831–849, 2004.
K. M. Stanney, M. Mollaghasemi, L. Reeves, R. Breaux, and D. A. Graeber, ”Usability engineering of virtual environments (VEs): identifying multiple criteria that drive effective VE system design,” Int. J. Hum.-Comput. Stud., vol. 58, pp. 447–481, 2003.
A. Sutcliffe and K. Kaur, “Evaluating the usability of virtual reality user interfaces,” Behaviour and Information Technology, vol. 19, 2001.
J. Nielsen and R. Molich, “Heuristic evaluation of user interfaces,” presented at the Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, Seattle, Washington, United States, 1990.
W. Newman, “A Preliminary Analysis of the Products of HCI Research, Using Pro Forma Abstracts,” presented at the CHI, Boston, MA, 1994.
M. Fjeld, “Introduction: Augmented Reality-Usability and Collaborative Aspects,” International Journal of Human-Computer Interaction, vol. 16, p. 387–393, 2003.
R. T. Azuma, “A Survey of Augmented Reality,” Presence – Teleoperators and Virtual Environments, vol. 6, pp. 355–385, 1997.
A. Sutcliffe and K. Kaur, “Evaluating the usability of virtual reality user interfaces,” Behaviour and Information Technology, vol. 19, 2000.
D. Hix, J. L. Gabbard, J. E. S. II, M. A. Livingston, T. H. Höllerer, S. J. Julier, Y. Baillot, and D. Brown, “A Cost-Effective Usability Evaluation Progression for Novel Interactive Systems,” presented at the Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS’04) - Track 9 - Volume 9, 2004.
S. W. Gilroy, M. Cavazza, and M. Benayoun, “Using affective trajectories to describe states of flow in interactive art,” presented at the Proceedings of the International Conference on Advances in Computer Enterntainment Technology, Athens, Greece, 2009.
A. Morrison, A. Oulasvirta, P. Peltonen, S. Lemmela, G. Jacucci, G. Reitmayr, J. Näsänen, and A. Juustila, “Like bees around the hive: a comparative study of a mobile augmented reality map,” presented at the Proceedings of the 27th international conference on Human factors in computing systems, Boston, MA, USA, 2009.
E. Hughes, E. Smith, C. B. Stapleton, and D. E. Hughes, “Augmenting Museum Experiences with Mixed Reality,” presented at the Knowledge Sharing and Collaborative Engineering, St. Thomas, US Virgin Islands, 2004.
L. Barkhuus and J. A. Rode, “From Mice to Men – 24 years of Evaluation in CHI,” presented at the CHI, San Jose, USA, 2007.
J. L. Gabbard and J. E. Swan, “Usability Engineering for Augmented Reality: Employing User-Based Studies to Inform Design,” IEEE Transactions on Visualization and Computer Graphics, vol. 14, pp. 513–525, 2008.
B. Knörlein, M. D. Luca, and M. Harders, “Influence of visual and haptic delays on stiffness perception in augmented reality,” presented at the Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality, 2009.
A. Dünser, K. Steinbügl, H. Kaufmann, and J. Glück, “Virtual and augmented reality as spatial ability training tools,” presented at the Proceedings of the 7th ACM SIGCHI New Zealand chapter’s international conference on Computer-human interaction: design centered HCI, Christchurch, New Zealand, 2006.
H. Kaufmann, K. Steinbügl, A. Dünser, and J. Glück, “General Training of Spatial Abilities by Geometry Education in Augmented Reality,” Annual Review of CyberTherapy and Telemedicine: A Decade of VR, vol. 3, pp. 65–76, 2005.
CEEB College Entrance Examination Board, Special Aptitude Test in Spatial Relations MCT: CEEB, 1939.
G. K. Bennett, H. G. Seashore, and A. G. Wesman, Differential Aptitude Tests, Forms S and T. New York: The Psychological Corporation, 1973.
M. Peters, B. Laeng, K. Latham, M. Jackson, R. Zaiyouna, and C. Richardson, “A redrawn Vandenberg and Kuse mental rotations test: Different versions and factors that affect performance,” Brain and Cognition, vol. 28, pp. 39–58, 1995.
M. Hegarty and D. Waller, “A dissociation between mental rotation and perspective-taking spatial abilities,” Intelligence, vol. 32, pp. 175–191, 2004.
M. Billinghurst, H. Kato, K. Kiyokawa, D. Belcher, and I. Poupyrev, “Experiments with Face-To-Face Collaborative AR Interfaces,” Virtual Reality, vol. 6, pp. 107–121, 2002.
S. Nilsson and B. Johansson, “Acceptance of augmented reality instructions in a real work setting,” presented at the CHI ’08 extended abstracts on Human factors in computing systems, Florence, Italy, 2008.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Dünser, A., Billinghurst, M. (2011). Evaluating Augmented Reality Systems. In: Furht, B. (eds) Handbook of Augmented Reality. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-0064-6_13
Download citation
DOI: https://doi.org/10.1007/978-1-4614-0064-6_13
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-0063-9
Online ISBN: 978-1-4614-0064-6
eBook Packages: Computer ScienceComputer Science (R0)