research-article

Usability benchmarks for motion tracking systems

Authors:

Jean-Luc Lugrin,

Dennis Wiebusch,

Marc Erich Latoschik,

Alexander StrehlerAuthors Info & Claims

VRST '13: Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology

Pages 49 - 58

https://doi.org/10.1145/2503713.2503730

Published: 06 October 2013 Publication History

Abstract

Precise, accurate, fast, and low-latency motion tracking is a core requirement of real-time human-computer interfaces. The choices of tracking systems for a particular set of 3D interaction techniques are manifold. Hence, guidance in this task is greatly beneficial. In this paper, we propose to establish a set of canonical and simple game-based benchmarks for a potentially standardised comparison of tracking systems. The benchmarks focus on usability scores given reoccurring interaction tasks without requiring potentially missing, incomplete, or complex latency or accuracy raw measurements. Our first two benchmarks evaluate three tracking systems regarding motion-parallax and 3D object manipulation techniques. Our usability comparisons confirmed an expected advantage of low-latency/high-accuracy systems, while they also demonstrated that certain tracking systems perform better than suggested by previous measurements of their raw performances. This indicates that our approach provides an adequate replacement and improvement over the pure comparison of technical specifications. We believe our benchmarks could benefit the research community by facilitating a usability-based comparison of motion tracking systems.

References

[1]

Adelstein, B. D., Johnston, E. R., and Ellis, S. R. 1996. Dynamic response of electromagnetic spatial displacement trackers. Presence (Cambridge, Mass.) 5, 3, 302.

[2]

Allen, B. D., and Welch, G. 2005. A general method for comparing the expected performance of tracking and motion capture systems. In Proceedings of the ACM symposium on Virtual reality software and technology, VRST '05, ACM, 201--210.

Digital Library

[3]

Argelaguet, F., Andujar, C, and Trueba, R. 2008. Overcoming eye-hand visibility mismatch in 3D pointing selection. In Proceedings of the ACM symposium on Virtual reality software and technology, VRST '08, ACM, 43--46.

Digital Library

[4]

Arsenault, R., and Ware, C. 2004. The importance of stereo and eye-coupled perspective for eye-hand coordination in fish tank VR. Presence: Teleoperators and Virtual Environments 13, 5, 549--559.

Digital Library

[5]

Arthur, K. W., Booth, K. S., and Ware, C. 1993. Evaluating 3D task performance for fish tank virtual worlds. ACM Transactions on Information Systems (TOIS) 11, 3, 239--265.

Digital Library

[6]

Barfield, W., and Hendrix, C. 1995. The effect of update rate on the sense of presence within virtual environments. Virtual Reality 1, 1, 3--15.

Digital Library

[7]

Barfield, W., Hendrix, C, and Bystrom, K.-E. 1999. Effects of stereopsis and head tracking on performance using desktop virtual environment displays. Presence: Teleoperators and Virtual Environments 8, 2, 237--240.

Digital Library

[8]

Bowman, D. A., and Hodges, L. F. 1997. An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments. In Proceedings of the ACM symposium on Interactive 3D graphics, ACM, 35--38.

Digital Library

[9]

Bowman, D. A., Johnson, D. B., and Hodges, L. F. 1999. Testbed evaluation of virtual environment interaction techniques. In Proceedings of the ACM symposium on Virtual reality software and technology, VRST '99, ACM, 26--33.

Digital Library

[10]

Bowman, D. A., Kruijff, E., LaViola Jr, J. J., and Poupyrev, I. 2004. 3D user interfaces: theory and practice. Addison-Wesley Professional.

Digital Library

[11]

Bryson, S. T. 1993. Effects of lag and frame rate on various tracking tasks. In IS&T/SPIE's Symposium on Electronic Imaging: Science and Technology, International Society for Optics and Photonics, 155--166.

[12]

Foxlin, E. M., Harrington, M., and Altshuler, Y. 1998. Miniature six-dof inertial system for tracking hmds. In Aerospace/Defense Sensing and Controls, International Society for Optics and Photonics, 214--228.

[13]

Greuter, S., and Roberts, D. 2011. Controlling viewpoint from markerless head tracking in an immersive ball game using a commodity depth based camera. In Proceedings of the IEEE/ACM 15th International Symposium on Distributed Simulation and Real Time Applications (DS-RT), IEEE, 64--71.

Digital Library

[14]

Hale, K. S., and Stanney, K. M. 2006. Effects of low stereo acuity on performance, presence and sickness within a virtual environment. Applied Ergonomics 37, 3, 329--339.

[15]

HE, D., Fuhu, D. H., Pape, D., Dawe, G., and S, D. 2000. Video-based measurement of system latency. In International Immersive Projection Technology Workshop.

[16]

Hendrix, C., and Barfield, W. 1996. Presence within virtual environments as a function of visual display parameters. Presence: Teleoperators and Virtual Environments 5, 3, 274--289.

Digital Library

[17]

Hoffman, D. M., Girshick, A. R., Akeley, K., and Banks, M. S. 2008. Vergence--accommodation conflicts hinder visual performance and cause visual fatigue. Journal of vision 8, 3.

[18]

Hülss, J.-P., Müller, B., Pustka, D., Willneff, J., and Zürl, K. 2012. Tracking of manufacturing tools with cylindrical markers. In Proceedings of the ACM symposium on Virtual reality software and technology, ACM, VRST '12, ACM, 161--168.

Digital Library

[19]

Imagination, 2013. iotracker (advanced optical motion tracking). http://www.iotracker.com. © Imagination Computer Services Ges.m.b.H, {Online; last accessed 20-March-2013}.

[20]

ISO/TC 159 9241-11:1998: Ergonomics requirements for office work with visual display terminals (VDTs) Part 11: Guidance on usability.

[21]

Jerald, J. 2004. Latency compensation for head-mounted virtual reality. Tech. rep., Technical Report, Report.

[22]

Kennedy, R. S., Lane, N. E., Berbaum, K. S., and Lilienthal, M. G. 1993. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. The international journal of aviation psychology 3, 3, 203--220.

[23]

Khoshelham, K., and Elberink, S. O. 2012. Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12, 2, 1437--1454.

[24]

Kindratenko, V. 1999. Calibration of electromagnetic tracking devices. Virtual Reality 4, 2, 139--150.

Digital Library

[25]

Lambooij, M., Fortuin, M., IJsselsteijn, W., and Heynderickx, I. 2009. Measuring visual discomfort associated with 3d displays. In IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics.

[26]

Larsson, P., Västfjäll, D., and Kleiner, M. 2001. The actor-observer effect in virtual reality presentations. CyberPsychology & Behavior 4, 2, 239--246.

[27]

Latoschik, M. E., and Tramberend, H. 2011. Simulator X: A Scalable and Concurrent Software Platform for Intelligent Realtime Interactive Systems. In Proceedings of the IEEE Virtual Reality Conference (VR), IEEE, 171--174.

Digital Library

[28]

Lee, C., Bonebrake, S., Bowman, D., and Hollerer, T. 2010. The role of latency in the validity of AR simulation. In Proceedings of the IEEE Virtual Reality Conference (VR), IEEE, 11--18.

Digital Library

[29]

Liang, J., Shaw, C., and Green, M. 1991. On temporal-spatial realism in the virtual reality environment. In Proceedings of the 4th annual ACM symposium on User interface software and technology, UIST '91, ACM, 19--25.

Digital Library

[30]

Ling, Y., Brinkman, W.-P., Nefs, H. T., Qu, C., and Heynderickx, I. 2012. Effects of stereoscopic viewing on presence, anxiety, and cybersickness in a virtual reality environment for public speaking. Presence: Teleoperators and Virtual Environments 21, 3, 254--267.

Digital Library

[31]

Lugrin, J.-L., Charles, F., Cavazza, M., Le Renard, M., Freeman, J., and Lessiter, J. 2012. CaveUDK: a VR game engine middleware. In Proceedings of the ACM symposium on Virtual reality software and technology, ACM, 137--144.

Digital Library

[32]

Madan, A., and Dubey, S. K. 2012. Usability evaluation methods: A literature review. International Journal of Engineering Science and Technology 4(2), 590--599.

[33]

Mania, K., Adelstein, B. D., Ellis, S. R., and Hill, M. I. 2004. Perceptual sensitivity to head tracking latency in virtual environments with varying degrees of scene complexity. In Proceedings of the 1st Symposium on Applied perception in graphics and visualization, APGV '04, ACM, 39--47.

Digital Library

[34]

McMahan, R. P., Gorton, D., Gresock, J., McConnell, W., and Bowman, D. A. 2006. Separating the effects of level of immersion and 3D interaction techniques. In Proceedings of the ACM symposium on Virtual reality software and technology, VRST '06, ACM, 108--111.

Digital Library

[35]

McMahan, R. P., Bowman, D. A., Zielinski, D. J., and Brady, R. B. 2012. Evaluating display fidelity and interaction fidelity in a virtual reality game. IEEE Transactions on Visualization and Computer Graphics 18, 4, 626--633.

Digital Library

[36]

Meehan, M., Razzaque, S., Whitton, M., and Brooks, F. P., J. 2003. Effect of latency on presence in stressful virtual environments. In Proceedings of the IEEE Virtual Reality Conference (VR), IEEE, 141--148.

Digital Library

[37]

Miller, D., and Bishop, G. 2002. Latency meter: a device end-to-end latency of VE systems. In Electronic Imaging 2002, International Society for Optics and Photonics, 458--464.

[38]

Mine, M. 1993. Characterization of end-to-end delays in head-mounted display systems. The University of North Carolina at Chapel Hill, TR93-001.

Digital Library

[39]

Mulder, J., Jansen, J., and van Rhijn, A. 2003. An affordable optical head tracking system for desktop VR/AR systems. In Proceedings of the workshop on Virtual environments 2003, ACM, 215--223.

Digital Library

[40]

Narayan, M., Waugh, L., Zhang, X., Bafna, P., and Bowman, D. 2005. Quantifying the benefits of immersion for collaboration in virtual environments. In Proceedings of the ACM symposium on Virtual reality software and technology, ACM, VRST '05, ACM, 78--81.

Digital Library

[41]

Nawrot, M., and Joyce, L. 2006. The pursuit theory of motion parallax. Vision research 46, 28, 4709--4725.

[42]

Papadakis, G., Mania, K., and Koutroulis, E. 2011. A system to measure, control and minimize end-to-end head tracking latency in immersive simulations. In Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry, ACM, VRCAI '11, ACM, 581--584.

Digital Library

[43]

Pintaric, T., and Kaufmann, H. 2007. Affordable infrared-optical pose-tracking for virtual and augmented reality. In Proceedings of Trends and Issues in Tracking for Virtual Environments Workshop, IEEE VR, IEEE, 44--51.

[44]

Rao, S. G., Schmugge, S. J., and Hodges, L. F. 2004. Vision-based system for head pose tracking in indoor immersive environments. In Proceedings of the 14th International Conference on Artificial Reality and Telexistence (ICAT2004).

[45]

Regenbrecht, H., and Schubert, T. 2002. Real and illusory interactions enhance presence in virtual environments. Presence: Teleoperators and Virtual Environments 11, 4 (Aug.), 425--434.

Digital Library

[46]

Roerdink, M., Peper, C., and Beek, P. 2005. Effects of correct and transformed visual feedback on rhythmic visuo-motor tracking: Tracking performance and visual search behavior. Human Movement Science 24, 3 (June), 379--402.

[47]

Rogers, B., and Graham, M. 1979. Motion parallax as an independent cue for depth perception. Perception 8, 2, 125--134.

[48]

Rusdorf, S., and Brunnett, G. 2005. Real time tracking of high speed movements in the context of a table tennis application. In Proceedings of the ACM symposium on Virtual reality software and technology, ACM, New York, NY, USA, VRST '05, ACM, 192--200.

Digital Library

[49]

Schonauer, C., Pintaric, T., Kaufmann, H., Jansen-Kosterink, S., and Vollenbroek-Hutten, M. 2011. Chronic pain rehabilitation with a serious game using multi-modal input. In International Conference on Virtual Rehabilitation (ICVR), IEEE, 1--8.

[50]

Slater, M., Steed, A., McCarthy, J., and Maringelli, F. 1998. The influence of body movement on subjective presence in virtual environments. Human Factors: The Journal of the Human Factors and Ergonomics Society 40, 3 (Sept.), 469--477.

[51]

So, R. H., Lo, W., and Ho, A. T. 2000. Scene movement: an importance cause of cybersickness. Virtual reality.

[52]

Sprague, D. W., Po, B. A., and Booth, K. S. 2006. The importance of accurate VR head registration on skilled motor performance. In Proceedings of Graphics Interface 2006, Canadian Information Processing Society, GI '06, Canadian Information Processing Society, 131--137.

Digital Library

[53]

Stanney, K. M., Mourant, R. R., and Kennedy, R. S. 1998. Human factors issues in virtual environments: A review of the literature. Presence 7, 4, 327--351.

Digital Library

[54]

Steed, A. 2008. A simple method for estimating the latency of interactive, real-time graphics simulations. In Proceedings of the 2008 ACM symposium on Virtual reality software and technology, ACM, New York, NY, USA, VRST '08, ACM, 123--129.

Digital Library

[55]

Suma, E., Lange, B., Rizzo, A., Krum, D. M., and Bolas, M. 2011. FAAST: The Flexible Action and Articulated Skeleton Toolkit. In IEEE Virtual Reality, IEEE, 245--246.

Digital Library

[56]

Taylor, R. M., Hudson, T. C., Seeger, A., Weber, H., Juliano, J., and Helser, A. T. 2001. VRPN: A device-independent, network-transparent VR peripheral system. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST'01, ACM, 55--61.

Digital Library

[57]

Usoh, M., Catena, E., Arman, S., and Slater, M. 2000. Using Presence Questionnaires in Reality. Presence: Teleoper. Virtual Environ. 9, 5 (Oct.), 497--503.

Digital Library

[58]

VRKnowledgebase, 2012. VR Knowledgebase. http://knowledgebase.cs.vt.edu. {Online; last accessed 27-July-2013}.

[59]

Welch, G., Bishop, G., Vicci, L., Brumback, S., Keller, K., and Colucci, D. 1999. The HiBall tracker: high-performance wide-area tracking for virtual and augmented environments. In Proceedings of the ACM symposium on Virtual reality software and technology, ACM, New York, NY, USA, VRST '99, ACM, 1--10.

Digital Library

[60]

Wiebusch, D., Fischbach, M., Strehler, A., Latoschik, M. E., Bruder, G., and Steinicke, F. 2012. Evaluation von Headtracking in interaktiven virtuellen Umgebungen auf Basis der Kinect. In Proceedings of the GI-Workshop VR/AR.

[61]

Witmer, B. G., and Singer, M. J. 1998. Measuring presence in virtual environments: A presence questionnaire. Presence 7, 3, 225--240.

Digital Library

Cited By

Hameed APerkis A(2024)Authenticity and presence: defining perceived quality in VR experiencesFrontiers in Psychology10.3389/fpsyg.2024.129165015Online publication date: 19-Apr-2024
https://doi.org/10.3389/fpsyg.2024.1291650
Grootjen JWeingärtner HMayer S(2024)Investigating the Effects of Eye-Tracking Interpolation Methods on Model Performance of LSTMProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3656353(1-6)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3656353
Cheon EXu S(2024)Creative Precarity in Motion: Revealing the Hidden Labor Behind Animating Virtual CharactersProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661545(3471-3484)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661545
Show More Cited By

Index Terms

Usability benchmarks for motion tracking systems
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
      2. Virtual reality

Recommendations

AMTSet: a benchmark for abrupt motion tracking
Abstract
Since the OTB100 benchmark dataset is released, it has been widely used in a large number of researches on object tracking for performance evaluation. However, the existing datasets are insufficient to evaluate trackers in handling different ...
Motion Tracking with an Active Camera

This paper describes a method for real-time motion detection using an active camera mounted on a pan/tilt platform. Image mapping is used to align images of different viewpoints so that static camera motion detection can be applied. In the presence of ...
Building Virtual Entertainment Environment with Tiled Display Wall and Motion Tracking
ICVRV '11: Proceedings of the 2011 International Conference on Virtual Reality and Visualization

Presented in this paper is an immersive and interactive entertainment environment which integrates multiprojector tiled display wall and motion tracking. Calibration methods are proposed for the geometry alignment and color consistency of the tiled ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '13: Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology

October 2013

271 pages

ISBN:9781450323796

DOI:10.1145/2503713

General Chairs:
Nadia Magnenat Thalmann
NTU, Singapore and MIRALab, University of Geneva, Switzerland
,
Enhua Wu
Academia Sinica and University of Macau, China
,
Susumi Tachi
Keio University, Japan
,
Program Chairs:
Daniel Thalmann
NTU, Singapore and EPFL, Switzerland
,
Luciana Nedel
Federal University of Rio Grande Del Sul, Brazil
,
Weiwei Xu
Hangzhou Normal University, China

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 October 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

VRST'13

Sponsor:

VRST'13: The 19th ACM Symposium on Virtual Reality Software and Technology

October 6 - 9, 2013

Singapore

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
489
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hameed APerkis A(2024)Authenticity and presence: defining perceived quality in VR experiencesFrontiers in Psychology10.3389/fpsyg.2024.129165015Online publication date: 19-Apr-2024
https://doi.org/10.3389/fpsyg.2024.1291650
Grootjen JWeingärtner HMayer S(2024)Investigating the Effects of Eye-Tracking Interpolation Methods on Model Performance of LSTMProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3656353(1-6)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3656353
Cheon EXu S(2024)Creative Precarity in Motion: Revealing the Hidden Labor Behind Animating Virtual CharactersProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661545(3471-3484)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661545
Grootjen JWeingärtner HMayer S(2024)Uncovering and Addressing Blink-Related Challenges in Using Eye Tracking for Interactive SystemsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642086(1-23)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642086
Westermeier FBrübach LWienrich CLatoschik M(2024)Assessing Depth Perception in VR and Video See-Through AR: A Comparison on Distance Judgment, Performance, and PreferenceIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337206130:5(2140-2150)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372061
Glémarec YLugrin JBosser ACollins Jackson ABuche CLatoschik M(2021)Indifferent or Enthusiastic? Virtual Audiences Animation and Perception in Virtual RealityFrontiers in Virtual Reality10.3389/frvir.2021.6662322Online publication date: 1-Jun-2021
https://doi.org/10.3389/frvir.2021.666232
Souza VMaciel ANedel LKopper R(2021)Measuring Presence in Virtual Environments: A SurveyACM Computing Surveys10.1145/346681754:8(1-37)Online publication date: 4-Oct-2021
https://dl.acm.org/doi/10.1145/3466817
Serrano Vergel RMorillo Tena PCasas Yrurzum SCruz-Neira C(2020)A Comparative Evaluation of a Virtual Reality Table and a HoloLens-Based Augmented Reality System for Anatomy TrainingIEEE Transactions on Human-Machine Systems10.1109/THMS.2020.298474650:4(337-348)Online publication date: Aug-2020
https://doi.org/10.1109/THMS.2020.2984746
Glemarec YBosser ABuche CLugrin JLandeck MLatoschik MChollet M(2019)A Scalability Benchmark for a Virtual Audience Perception Model in Virtual RealityProceedings of the 25th ACM Symposium on Virtual Reality Software and Technology10.1145/3359996.3364784(1-1)Online publication date: 12-Nov-2019
https://dl.acm.org/doi/10.1145/3359996.3364784

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten