research-article

Public Access

XD-AR: Challenges and Opportunities in Cross-Device Augmented Reality Application Development

Authors:

Maximilian Speicher,

Janet Nebeling,

Michael NebelingAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 2, Issue EICS

Article No.: 7, Pages 1 - 24

https://doi.org/10.1145/3229089

Published: 19 June 2018 Publication History

Abstract

Augmented Reality (AR) developers face a proliferation of new platforms, devices, and frameworks. This often leads to applications being limited to a single platform and makes it hard to support collaborative AR scenarios involving multiple different devices. This paper presents XD-AR, a cross-device AR application development framework designed to unify input and output across hand-held, head-worn, and projective AR displays. XD-AR's design was informed by challenging scenarios for AR applications, a technical review of existing AR platforms, and a survey of 30 AR designers, developers, and users. Based on the results, we developed a taxonomy of AR system components and identified key challenges and opportunities in making them work together. We discuss how our taxonomy can guide the design of future AR platforms and applications and how cross-device interaction challenges could be addressed. We illustrate this when using XD-AR to implement two challenging AR applications from the literature in a device-agnostic way.

References

[1]

R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre . 2001. Recent advances in augmented reality. IEEE Computer Graphics and Applications Vol. 21, 6 (Nov . 2001), 34--47.

Digital Library

[2]

Ronald T Azuma . 1997. A survey of augmented reality. Presence: Teleoperators and virtual environments Vol. 6, 4 (1997), 355--385.

Digital Library

[3]

Martin Bauer, Bernd Bruegge, Gudrun Klinker, Asa MacWilliams, Thomas Reicher, Stefan Riss, Christian Sandor, and Martin Wagner . 2001. Design of a component-based augmented reality framework Proc. IEEE and ACM International Symposium on Augmented Reality. 45--54.

Digital Library

[4]

Oliver Baus and Stéphane Bouchard . 2014. Moving from Virtual Reality Exposure-Based Therapy to Augmented Reality Exposure-Based Therapy: A Review. Front. Hum. Neurosci. Vol. 8, March (2014), 1--15.

[5]

Mark Billinghurst, Adrian Clark, and Gun Lee . 2015. A Survey of Augmented Reality. Foundations and Trends in Human-Computer Interaction Vol. 8, 2--3 (2015), 73--272.

Digital Library

[6]

Julie Carmigniani, Borko Furht, Marco Anisetti, Paolo Ceravolo, Ernesto Damiani, and Misa Ivkovic . 2011. Augmented reality technologies, systems and applications. Multimedia Tools and Applications Vol. 51, 1 (Jan . 2011), 341--377.

Digital Library

[7]

Pei-Yu (Peggy) Chi and Yang Li . 2015. Weave: Scripting Cross-Device Wearable Interaction Proc. CHI. 3923--3932.

Digital Library

[8]

Francesco Clemente, Strahinja Dosen, Luca Lonini, Marko Markovic, Dario Farina, and Christian Cipriani . 2017. Humans can integrate augmented reality feedback in their sensorimotor control of a robotic hand. IEEE Transactions on Human-Machine Systems Vol. 47, 4 (2017), 583--589.

[9]

David Dearman and Jeffrey S. Pierce . 2008. “It's on my other Computer!”: Computing with Multiple Devices Proc. CHI. 767--776.

Digital Library

[10]

David Drascic and Paul Milgram . 1996. Perceptual issues in augmented reality. In SPIE The International Society For Optical Engineering. 123--134.

[11]

Mark Fiala . 2004. ARTag Revision 1. A Fiducial Marker System Using Digital Techniques. National Research Council Publication Vol. 47419 (2004), 1--47.

[12]

Luca Frosini and Fabio Paternò . 2014. User Interface Distribution in Multi-Device and Multi-User Environments with Dynamically Migrating Engines. In Proc. EICS. 55--64.

Digital Library

[13]

Peter Hamilton and Daniel J. Wigdor . 2014. Conductor: Enabling and Understanding Cross-Device Interaction Proc. CHI. 2773--2782.

Digital Library

[14]

Thuong Hoang, Martin Reinoso, Zaher Joukhadar, Frank Vetere, and David Kelly . 2017. Augmented Studio: Projection Mapping on Moving Body for Physiotherapy Education Proc. CHI. 1419--1430.

Digital Library

[15]

Steven Houben and Nicolai Marquardt . 2015. WatchConnect: A Toolkit for Prototyping Smartwatch-Centric Cross-Device Applications. In Proc. CHI. 1247--1256.

Digital Library

[16]

Tero Jokela, Jarno Ojala, and Thomas Olsson . 2015. A Diary Study on Combining Multiple Information Devices in Everyday Activities and Tasks. In Proc. CHI. 3903--3912.

Digital Library

[17]

Brett Jones, Rajinder Sodhi, Michael Murdock, Ravish Mehra, Hrvoje Benko, Andrew Wilson, Eyal Ofek, Blair MacIntyre, Nikunj Raghuvanshi, and Lior Shapira . 2014. RoomAlive: Magical Experiences Enabled by Scalable, Adaptive Projector-camera Units Proc. UIST. 637--644.

Digital Library

[18]

Seokmin Jung and Woontack Woo . 2004. UbiTrack: Infrared-based user Tracking System for indoor environment. International Conference on Artificial Reality and Telexisitence (ICAT04) (2004), 1345--1278.

[19]

Oliver Beren Kaul and Michael Rohs . 2017. HapticHead: A Spherical Vibrotactile Grid Around the Head for 3D Guidance in Virtual and Augmented Reality. In Proc. CHI. 3729--3740.

Digital Library

[20]

M. A. Livingston . 2005. Evaluating human factors in augmented reality systems. IEEE Computer Graphics and Applications Vol. 25, 6 (Nov . 2005), 6--9.

Digital Library

[21]

Blair MacIntyre, Maribeth Gandy, Steven Dow, and Jay David Bolter . 2004. DART: a toolkit for rapid design exploration of augmented reality experiences Proceedings of the 17th annual ACM symposium on User interface software and technology. ACM, 197--206.

Digital Library

[22]

Matt Miesnieks . 2017 a. How is ARCore better than ARKit? (2017). https://medium.com/super-ventures-blog/how-is-arcore-better-than-arkit-5223e6b3e79d

[23]

Matt Miesnieks . 2017 b. Why is ARKit better than the alternatives? (2017). https://medium.com/super-ventures-blog/why-is-arkit-better-than-the-alternatives-af8871889d6a

[24]

Paul Milgram and Fumio Kishino . 1994. A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems Vol. 77, 12 (1994), 1321--1329.

[25]

Michael Nebeling and Anind K. Dey . 2016. XDBrowser: User-Defined Cross-Device Web Page Designs Proc. CHI. 5494--5505.

Digital Library

[26]

Michael Nebeling, Theano Mintsi, Maria Husmann, and Moira C. Norrie . 2014 a. Interactive Development of Cross-Device User Interfaces Proc. CHI. 2793--2802.

Digital Library

[27]

Michael Nebeling, Elena Teunissen, Maria Husmann, and Moira C. Norrie . 2014 b. XDKinect: Development Framework for Cross-Device Interaction using Kinect Proc. EICS. 2793--2802.

Digital Library

[28]

Antti Oulasvirta and Lauri Sumari . 2007. Mobile Kits and Laptop Trays: Managing Multiple Devices in Mobile Information Work. In Proc. CHI. 1127--1136.

Digital Library

[29]

Jeffrey S. Pierce and Jeffrey Nichols . 2008. An infrastructure for extending applications' user experiences across multiple personal devices. In Proc. UIST. 101--110.

Digital Library

[30]

Roman R"adle, Hans-Christian Jetter, Mario Schreiner, Zhihao Lu, Harald Reiterer, and Yvonne Rogers . 2015. Spatially-aware or Spatially-agnostic?: Elicitation and Evaluation of User-Defined Cross-Device Interactions. In Proc. CHI. 3913--3922.

Digital Library

[31]

Gerhard Reitmayr and Dieter Schmalstieg . 2005. OpenTracker: A flexible software design for three-dimensional interaction. Virtual reality Vol. 9, 1 (2005), 79--92.

Digital Library

[32]

Johnny Salda na . 2015. The coding manual for qualitative researchers. Sage.

[33]

Stephanie Santosa and Daniel Wigdor . 2013. A Field Study of Multi-Device Workflows in Distributed Workspaces Proc. UbiComp. 63--72.

Digital Library

[34]

Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges . 2014. In-air gestures around unmodified mobile devices. In Proc. UIST. 319--329.

Digital Library

[35]

Sebastian Thrun and John J Leonard . 2008. Simultaneous localization and mapping. In Springer handbook of robotics. Springer, 871--889.

[36]

DWF Van Krevelen and Ronald Poelman . 2010. A survey of augmented reality technologies, applications and limitations. International Journal of Virtual Reality Vol. 9, 2 (2010), 1--20.

[37]

Maja Wrzesien, Cristina Botella, Juana Bretón-López, Eva del R'ıo González, Jean-Marie Burkhardt, Mariano Alca niz, and Mar'ıa Ángeles Pérez-Ara . 2015. Treating small animal phobias using a projective-augmented reality system: A single-case study. Computers in Human Behavior Vol. 49 (2015), 343--353.

Digital Library

[38]

Jishuo Yang and Daniel Wigdor . 2014. Panelrama: Enabling Easy Specification of Cross-Device Web Applications Proc. CHI. 2783--2792.

Digital Library

[39]

Feng Zhou, Henry Been-Lirn Duh, and Mark Billinghurst . 2008. Trends in Augmented Reality Tracking, Interaction and Display: A Review of Ten Years of ISMAR. In Proc. ISMAR. 193--202.

Digital Library

Cited By

Bose DBrown CFilkov VRay BZhou M(2024)An Empirical Study on Current Practices and Challenges of Core AR/VR DevelopersProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops10.1145/3691621.3694956(233-238)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691621.3694956
Janaka NCai RRam AZhu LZhao SYong K(2024)PilotAR: Streamlining Pilot Studies with OHMDs from Concept to InsightProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785768:3(1-35)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678576
Patel AKanishq Mandekar M(2024)Immersive Interior Design: Exploring Enhanced Visualization through Augmented Reality Technologies2024 Parul International Conference on Engineering and Technology (PICET)10.1109/PICET60765.2024.10716187(1-6)Online publication date: 3-May-2024
https://doi.org/10.1109/PICET60765.2024.10716187
Show More Cited By

Index Terms

XD-AR: Challenges and Opportunities in Cross-Device Augmented Reality Application Development
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
    2. Interactive systems and tools
      1. User interface toolkits

Recommendations

Dual Phone AR: Using a Second Phone as a Controller for Mobile Augmented Reality
UIST '20 Adjunct: Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Mobile Augmented Reality applications have become increasingly popular, however the possible interactions with AR content are largely limited to on-screen gestures and spatial movement. There has been a renewed interest in designing interaction methods ...
Exhibition approach using an AR and VR pillar
SA '17: SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications

This demonstration presents a development of an Augmented Reality (AR) and Virtual Reality (AR) pillar, a novel approach for showing AR and VR content in a public setting. A pillar in a public exhibition venue was converted to a four-sided AR and VR ...
Requirements for Virtualization of AR Displays within VR Environments
Proceedings, Part I, of the 6th International Conference on Virtual, Augmented and Mixed Reality. Designing and Developing Virtual and Augmented Environments - Volume 8525

Everybody has been talking about new emerging products in augmented reality AR and their potential to enhance our daily life and work. The AR technology has been around for quite a while and various use cases have been thought and tested. Clearly, the ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 2, Issue EICS

June 2018

293 pages

EISSN:2573-0142

DOI:10.1145/3233739

Editors:
Clifford Lampe
University of Michigan
,
Jeff Nichols
Google
,
Karrie Karahalios
University of Illinois Urbana-Champaign
,
Geraldine Fitzpatrick
Vienna University of Technology
,
Uichin Lee
KAIST
,
Andres Monroy-Hernandez
Microsoft Research
,
Wolfgang Stuerzlinger
Simon Fraser University

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 June 2018

Published in PACMHCI Volume 2, Issue EICS

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
1,741
Total Downloads

Downloads (Last 12 months)304
Downloads (Last 6 weeks)35

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bose DBrown CFilkov VRay BZhou M(2024)An Empirical Study on Current Practices and Challenges of Core AR/VR DevelopersProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops10.1145/3691621.3694956(233-238)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691621.3694956
Janaka NCai RRam AZhu LZhao SYong K(2024)PilotAR: Streamlining Pilot Studies with OHMDs from Concept to InsightProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785768:3(1-35)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678576
Patel AKanishq Mandekar M(2024)Immersive Interior Design: Exploring Enhanced Visualization through Augmented Reality Technologies2024 Parul International Conference on Engineering and Technology (PICET)10.1109/PICET60765.2024.10716187(1-6)Online publication date: 3-May-2024
https://doi.org/10.1109/PICET60765.2024.10716187
Bose DDavid-John BBrown C(2024)Optimizing AR Application Testing: Integrating Metamorphic Testing to Address Developer and End-User ChallengesHCI International 2024 – Late Breaking Papers10.1007/978-3-031-76812-5_2(20-33)Online publication date: 15-Dec-2024
https://doi.org/10.1007/978-3-031-76812-5_2
Echtler FMaierhöfer VHansen NWimmer R(2023)SurfaceCast: Ubiquitous, Cross-Device Surface SharingProceedings of the ACM on Human-Computer Interaction10.1145/36264757:ISS(286-308)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1145/3626475
Bateman SGutwin CMansoor HNacenta Mvan der Kamp MBaliesnyi MGagnon KRollheiser J(2023)WAMS: A Flexible API for Visual Workspaces Across Multiple SurfacesProceedings of the ACM on Human-Computer Interaction10.1145/35932367:EICS(1-40)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593236
Cao JLam KLee LLiu XHui PSu X(2023)Mobile Augmented Reality: User Interfaces, Frameworks, and IntelligenceACM Computing Surveys10.1145/355799955:9(1-36)Online publication date: 16-Jan-2023
https://dl.acm.org/doi/10.1145/3557999
Zhang LAgrawal AOney SGuo A(2023)VRGit: A Version Control System for Collaborative Content Creation in Virtual RealityProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581136(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581136
Bakale TPicek R(2023)Application of Augmented Reality in Creating Added Value for an ERP System2023 4th International Conference on Communications, Information, Electronic and Energy Systems (CIEES)10.1109/CIEES58940.2023.10378765(1-5)Online publication date: 23-Nov-2023
https://doi.org/10.1109/CIEES58940.2023.10378765
Wang XLee LBermejo Fernandez CHui P(2023)The Dark Side of Augmented Reality: Exploring Manipulative Designs in ARInternational Journal of Human–Computer Interaction10.1080/10447318.2023.218879940:13(3449-3464)Online publication date: 27-Mar-2023
https://doi.org/10.1080/10447318.2023.2188799
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents