research-article

Rapido: Prototyping Interactive AR Experiences through Programming by Demonstration

Authors:

Jens Emil Grønbæk,

Clemens Nylandsted Klokmose,

Rubaiat Habib Kazi,

Paul AsenteAuthors Info & Claims

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

Pages 626 - 637

https://doi.org/10.1145/3472749.3474774

Published: 12 October 2021 Publication History

Abstract

Programming by Demonstration (PbD) is a well-known technique that allows non-programmers to describe interactivity by performing examples of the expected behavior, but it has not been extensively explored for AR. We present Rapido, a novel early-stage prototyping tool to create fully interactive mobile AR prototypes from non-interactive video prototypes using PbD. In Rapido, designers use a mobile AR device to record a video prototype to capture context, sketch assets, and demonstrate interactions. They can demonstrate touch inputs, animation paths, and rules to, e.g., have a sketch follow the focus area of the device or the user’s world-space touches. Simultaneously, a live website visualizes an editable overview of all the demonstrated examples and infers a state machine of the user flow. Our key contribution is a method that enables designers to turn a video prototype into an executable state machine through PbD. The designer switches between these representations to interactively refine the final interactive prototype. We illustrate the power of Rapido’s approach by prototyping the main interactions of three popular AR mobile applications.

Supplementary Material

VTT File (p626-talk.vtt)

Download
8.32 KB

VTT File (p626-video_figure.vtt)

Download
8.85 KB

VTT File (p626-video_preview.vtt)

Download
.59 KB

MP4 File (p626-video_figure.mp4)

Video figure and captions

Download
185.46 MB

MP4 File (p626-talk.mp4)

Talk video and captions

Download
188.63 MB

MP4 File (p626-video_preview.mp4)

Video preview and captions

Download
39.73 MB

References

[1]

Adobe. 2021. Aero. Retrieved 2021-04-01 from https://www.adobe.com/products/aero.html

[2]

Apple Inc.2021. AR Tools - Augmented Reality - Apple Developer. Retrieved 2021-04-01 from https://developer.apple.com/augmented-reality/tools/

[3]

Narges Ashtari, Andrea Bunt, Joanna McGrenere, Michael Nebeling, and Parmit K Chilana. 2020. Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376722

Digital Library

[4]

Michel Beaudouin-Lafon and Wendy E. Mackay. 2000. Reification, Polymorphism and Reuse: Three Principles for Designing Visual Interfaces. Proceedings of the Working Conference on Advanced Visual Interfaces (2000), 102–109. https://doi.org/10.1145/345513.345267

Digital Library

[5]

Michel Beaudouin-Lafon and Wendy E. Mackay. 2003. Prototyping Tools and Techniques. In The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. 1017–1039. https://doi.org/10.1201/9781410615862

[6]

Bill Buxton. 2007. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann. 448 pages. https://doi.org/10.1016/B978-0-12-374037-3.X5043-3

[7]

Adam S. Carter and Christopher D. Hundhausen. 2010. How is User Interface Prototyping Really Done in Practice? A Survey of User Interface Designers. In 2010 IEEE Symposium on Visual Languages and Human-Centric Computing. IEEE, 207–211. https://doi.org/10.1109/VLHCC.2010.36

[8]

Allen Cypher and Daniel C. Halbert. 1993. Watch What I Do: Programming by Demonstration. MIT Press. 652 pages. http://acypher.com/wwid/WWIDToC.html

Digital Library

[9]

Facebook Inc.2021. Spark AR Studio - Create Augmented Reality Experiences. Retrieved 2021-04-01 from https://sparkar.facebook.com/ar-studio/

[10]

Figma Inc.2016. Figma: the Collaborative Interface Design Tool.https://www.figma.com/

[11]

Maribeth Gandy and Blair MacIntyre. 2014. Designer’s Augmented Reality Toolkit, Ten Years Later. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology - UIST ’14. ACM Press, New York, New York, USA, 627–636. https://doi.org/10.1145/2642918.2647369

Digital Library

[12]

Danilo Gasques, Janet G Johnson, Tommy Sharkey, and Nadir Weibel. 2019. PintAR: Sketching Spatial Experiences in Augmented Reality. In Companion Publication of the 2019 on Designing Interactive Systems Conference 2019 Companion(DIS ’19 Companion). ACM, New York, NY, USA, 17–20. https://doi.org/10.1145/3301019.3325158

Digital Library

[13]

Paul Green and Lisa Wei-Haas. 1985. The Rapid Development of User Interfaces: Experience with the Wizard of OZ Method. Proceedings of the Human Factors Society Annual Meeting 29, 5(1985), 470–474. https://doi.org/10.1177/154193128502900515

[14]

Saul Greenberg, Carpendale Sheelagh, Marquardt Nicolai, and Buxton Bill. 2012. Sketching User Experiences: The Workbook. Morgan Kaufmann. 272 pages. https://doi.org/10.1016/C2009-0-61147-8

[15]

Kim Halskov and Rune Nielsen. 2006. Virtual Video Prototyping. Human-Computer Interaction 21, 2 (may 2006), 199–233. https://doi.org/10.1207/s15327051hci2102_2

Digital Library

[16]

Steven Houben and Nicolai Marquardt. 2015. WatchConnect: A Toolkit for Prototyping Smartwatch-Centric Cross-Device Applications. Association for Computing Machinery, New York, NY, USA, 1247–1256. https://doi.org/10.1145/2702123.2702215

[17]

InVisionApp Inc.2011. InVision | Digital Product Design, Workflow and Collaboration. https://www.invisionapp.com/

[18]

Jiyoung Jeon, Min Hong, Manhui Yi, Jiyoon Chun, Ji Sim Kim, and Yoo-Joo Choi. 2016. Interactive Authoring Tool for Mobile Augmented Reality Content.JIPS 12, 4 (2016), 612–630.

[19]

Jun Kato, Takeo Igarashi, and Masataka Goto. 2016. Programming with Examples to Develop Data-Intensive User Interfaces. Computer 49, 7 (jul 2016), 34–42. https://doi.org/10.1109/MC.2016.217

Digital Library

[20]

J. F. Kelley. 1983. An Empirical Methodology for Writing User-friendly Natural Language Computer Applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’83). ACM, New York, NY, USA, 193–196. https://doi.org/10.1145/800045.801609

Digital Library

[21]

Khella Productions Inc. 2013. Keynotopia. https://keynotopia.com/

[22]

Scott R Klemmer, Björn Hartmann, and Leila Takayama. 2006. How Bodies Matter: Five Themes for Interaction Design. In Proceedings of the 6th Conference on Designing Interactive Systems(DIS ’06). Association for Computing Machinery, New York, NY, USA, 140–149. https://doi.org/10.1145/1142405.1142429

Digital Library

[23]

Veronika Krauß, Alexander Boden, Leif Oppermann, René Reiners, Sankt Augustin, and Sankt Augustin. 2021. Current Practices, Challenges, and Design Implications for Collaborative AR/VR Application Development. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (2021). https://doi.org/10.1145/3411764.3445335

Digital Library

[24]

David Ledo, Jo Vermeulen, Sheelagh Carpendale, Saul Greenberg, Lora Oehlberg, and Sebastian Boring. 2019. Astral: Prototyping Mobile and Smart Object Interactive Behaviours Using Familiar Applications. In Proceedings of the 2019 on Designing Interactive Systems Conference (San Diego, CA, USA) (DIS ’19). Association for Computing Machinery, New York, NY, USA, 711–724. https://doi.org/10.1145/3322276.3322329

Digital Library

[25]

Germán Leiva and Michel Beaudouin-Lafon. 2018. Montage: A Video Prototyping System to Reduce Re-Shooting and Increase Re-Usability. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology - UIST ’18. ACM Press, Berlin, Germany. https://doi.org/10.1145/3242587.3242613

Digital Library

[26]

Germán Leiva, Nolwenn Maudet, Wendy Mackay, and Michel Beaudouin-Lafon. 2019. Enact: Reducing Designer-Developer Breakdowns When Prototyping Custom Interactions. ACM Trans. Comput.-Hum. Interact. 26, 3 (may 2019), 19:1–19:48. https://doi.org/10.1145/3310276

Digital Library

[27]

Germán Leiva, Cuong Nguyen, Rubaiat Habib Kazi, and Paul Asente. 2020. Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376160

Digital Library

[28]

Henry Lieberman. 2001. Your Wish is My Command: Programming by Example. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

[29]

Danwei Tran Luciani and Peter Vistisen. 2017. Empowering Non-Designers Through Animation-based Sketching. 7 (2017). http://www.nordes.org/nordes2017/assets/full_papers/nordes17a-sub1006-cam-i26_LUCIANI_v2.pdf

[30]

Wendy E. Mackay. 1988. Video Prototyping: A Technique for Developing Hypermedia Systems. Vol. 5. ACM/SIGCHI.

[31]

Wendy E. Mackay and Anne-Laure Fayard. 1999. Video Brainstorming and Prototyping: Techniques for Participatory Design. CHI’99 Extended Abstracts on Human Factors in...May (1999), 118–119. https://doi.org/10.1145/632716.632790

[32]

Wendy E. Mackay, Anne V. Ratzer, and Paul Janecek. 2000. Video Artifacts for Design: Bridging the Gap Between Abstraction and Detail. In DIS ’00. ACM, New York, New York, USA, 72–82. https://doi.org/10.1145/347642.347666

Digital Library

[33]

Jan Meskens, Kris Luyten, and Karin Coninx. 2010. D-Macs: Building Multi-Device User Interfaces by Demonstrating, Sharing and Replaying Design Actions. Association for Computing Machinery, New York, NY, USA, 129–138. https://doi.org/10.1145/1866029.1866051

[34]

Brad A. Myers, Richard G. McDaniel, and David Wolber. 2000. Programming by Example: Intelligence in Demonstrational Interfaces. Commun. ACM 43, 3 (mar 2000), 82–89. https://doi.org/10.1145/330534.330545

Digital Library

[35]

Michael Nebeling, Katy Lewis, Yu-Cheng Chang, Lihan Zhu, Michelle Chung, Piaoyang Wang, and Janet Nebeling. 2020. XRDirector: A Role-Based Collaborative Immersive Authoring System. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376637

Digital Library

[36]

Michael Nebeling and Katy Madier. 2019. 360Proto: Making Interactive Virtual Reality and Augmented Reality Prototypes from Paper. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems(CHI ’19). ACM, New York, NY, USA, 596:1–596:13. https://doi.org/10.1145/3290605.3300826

Digital Library

[37]

Michael Nebeling, Janet Nebeling, Ao Yu, and Rob Rumble. 2018. ProtoAR: Rapid Physical-Digital Prototyping of Mobile Augmented Reality Applications. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems(CHI ’18). ACM, New York, NY, USA, 353:1–353:12. https://doi.org/10.1145/3173574.3173927

Digital Library

[38]

Michael Nebeling and Maximilian Speicher. 2018. The Trouble with Augmented Reality/Virtual Reality Authoring Tools. In 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 333–337. https://doi.org/10.1109/ISMAR-Adjunct.2018.00098

[39]

Mark W. Newman and James A. Landay. 2000. Sitemaps, Storyboards, and Specifications: A Sketch of Web Site Design Practice. In Proceedings of the Conference on Designing Interactive Systems Processes, Practices, Methods, and Techniques - DIS ’00. ACM Press, New York, New York, USA, 263–274. http://dl.acm.org/citation.cfm?id=347642.347758

[40]

Dan R. Olsen. 2007. Evaluating User Interface Systems Research. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (Newport, Rhode Island, USA) (UIST ’07). Association for Computing Machinery, New York, NY, USA, 251–258. https://doi.org/10.1145/1294211.1294256

Digital Library

[41]

Marc Rettig. 1994. Prototyping for Tiny Fingers. Commun. ACM 37, 4 (apr 1994), 21–27. https://doi.org/10.1145/175276.175288

Digital Library

[42]

Nazmus Saquib, Rubaiat Habib Kazi, Li-Yi Wei, and Wilmot Li. 2019. Interactive Body-Driven Graphics for Augmented Video Performance. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems(CHI ’19). ACM, New York, NY, USA, 622:1–622:12. https://doi.org/10.1145/3290605.3300852

Digital Library

[43]

SensorTower Inc.2018. ARKit-only Apps Surpass 13 Million Downloads in First Six Months, Nearly Half from Games. Retrieved 2021-03-30 from https://sensortower.com/blog/arkit-six-months

[44]

Ben Shneiderman. 1983. Direct Manipulation: A Step Beyond Programming Languages. Computer 16, 8 (aug 1983), 57–69. https://doi.org/10.1109/MC.1983.1654471

Digital Library

[45]

Ryo Suzuki, Rubaiat Habib Kazi, Li-yi Wei, Stephen DiVerdi, Wilmot Li, and Daniel Leithinger. 2020. RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology(UIST ’20). Association for Computing Machinery, New York, NY, USA, 166–181. https://doi.org/10.1145/3379337.3415892

Digital Library

[46]

Torch 3d Inc.2020. Torch. Retrieved 2020-05-25 from https://www.torch.app/

[47]

Unity Technologies. 2021. Unity Real-Time Development Platform | 3D, 2D VR and AR Engine. Retrieved 2021-04-01 from https://unity.com/

[48]

Verizon Media. 2018. ARKit-only Apps Top 13 Million Installs, Nearly Half From Games | TechCrunch. Retrieved 2021-03-30 from https://techcrunch.com/2018/03/28/arkit-only-apps-top-13-million-installs-nearly-half-are-games

[49]

Laurie Vertelney. 1989. Using Video to Prototype User Interfaces. ACM SIGCHI Bulletin 21, 2 (oct 1989), 57–61. https://doi.org/10.1145/70609.70615

Digital Library

[50]

Peter Vistisen. 2016. Sketching with Animation: Using Animation to Portray Fictional Realities–Aimed at Becoming Factual. Aalborg Universitetsforlag.

Cited By

Tong WShigyo KYuan LFan MPong TQu HXia M(2025)VisTellAR: Embedding Data Visualization to Short-Form Videos Using Mobile Augmented RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210431:3(1862-1874)Online publication date: Mar-2025
https://doi.org/10.1109/TVCG.2024.3372104
Hai-Jew S(2024)Using Artmaking Generative AIs to Support Augmented Reality Learning Designs With Adobe Aero AppInquiries of Pedagogical Shifts and Critical Mindsets Among Educators10.4018/979-8-3693-1078-6.ch006(132-152)Online publication date: 8-Mar-2024
https://doi.org/10.4018/979-8-3693-1078-6.ch006
Rosenberg KKazi RWei LXia HPerlin K(2024)DrawTalking: Building Interactive Worlds by Sketching and SpeakingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676334(1-25)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676334
Show More Cited By

Recommendations

Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Designers have limited tools to prototype AR experiences rapidly. Can lightweight, immediate tools let designers prototype dynamic AR interactions while capturing the nuances of a 3D experience? We interviewed three AR experts and identified several ...
ProjectAR: Rapid Prototyping of Projection Mapping with Mobile Augmented Reality
NordiCHI '22 Adjunct: Adjunct Proceedings of the 2022 Nordic Human-Computer Interaction Conference

Spatial Augmented Reality (AR) in the form of Projection Mapping is an interesting technology for several scenarios. However, there is a lack of rapid prototyping tools for Projection Mapping. We introduce ProjectAR, a rapid prototyping tool that ...
Rapid Prototyping for AR/VR Experiences
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

This course will provide an introduction to techniques for rapid prototyping for Augmented Reality (AR) and Virtual Reality (VR) experiences. With the rise of consumer head mounted displays and powerful mobile phones, AR and VR are becoming increasingly ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

October 2021

1357 pages

ISBN:9781450386357

DOI:10.1145/3472749

Editors:
Jeffrey Nichols
Apple, USA
,
Ranjitha Kumar
UIUC, USA
,
Michael Nebeling
University of Michigan, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Carlsbergfondet

Conference

UIST '21

Sponsor:

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

October 10 - 14, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

33
Total Citations
View Citations
1,079
Total Downloads

Downloads (Last 12 months)222
Downloads (Last 6 weeks)18

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tong WShigyo KYuan LFan MPong TQu HXia M(2025)VisTellAR: Embedding Data Visualization to Short-Form Videos Using Mobile Augmented RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210431:3(1862-1874)Online publication date: Mar-2025
https://doi.org/10.1109/TVCG.2024.3372104
Hai-Jew S(2024)Using Artmaking Generative AIs to Support Augmented Reality Learning Designs With Adobe Aero AppInquiries of Pedagogical Shifts and Critical Mindsets Among Educators10.4018/979-8-3693-1078-6.ch006(132-152)Online publication date: 8-Mar-2024
https://doi.org/10.4018/979-8-3693-1078-6.ch006
Rosenberg KKazi RWei LXia HPerlin K(2024)DrawTalking: Building Interactive Worlds by Sketching and SpeakingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676334(1-25)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676334
Liao JVan KXia ZSuzuki R(2024)RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual EffectsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661631(1248-1261)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661631
Rosenberg KKazi RWei LXia HPerlin K(2024)DrawTalking: Towards Building Interactive Worlds by Sketching and SpeakingExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651089(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651089
Su XKoh EXiao C(2024)SonifyAR: Context-Aware Sound Effect Generation in Augmented RealityExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650927(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650927
Chen JYoon D(2024)Exploring the Diminishing Allure of Paper and Low-Fidelity Prototyping Among Designers in the Software Industry: Impacts of Hybrid Work, Digital Tools, and Corporate CultureProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642774(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642774
Ye HLeng JXu PSingh KFu H(2024)ProInterAR: A Visual Programming Platform for Creating Immersive AR InteractionsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642527(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642527
Cho HYan YTodi KParent MSmith MJonker TBenko HLindlbauer D(2024)MineXR: Mining Personalized Extended Reality InterfacesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642394(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642394
Lunding RLunding MFeuchtner TPetersen MGrønbæk KSuzuki RGrollman DBroadbent EJu WSoh HWilliams T(2024)RoboVisAR: Immersive Authoring of Condition-based AR Robot VisualisationsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634972(462-471)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3634972
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten