skip to main content
10.1145/3313831.3376160acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction

Published: 23 April 2020 Publication History

Abstract

Designers have limited tools to prototype AR experiences rapidly. Can lightweight, immediate tools let designers prototype dynamic AR interactions while capturing the nuances of a 3D experience? We interviewed three AR experts and identified several recurring issues in AR design: creating and positioning 3D assets, handling the changing user position, and orchestrating multiple animations. We introduce PROJECT PRONTO, a tablet-based video prototyping system that combines 2D video with 3D manipulation. PRONTO supports four intertwined activities: capturing 3D spatial information alongside a video scenario, positioning and sketching 2D drawings in a 3D world, and enacting animations with physical interactions. An observational study with professional designers shows that participants can use PRONTO to prototype diverse AR experiences. All participants performed two tasks: replicating a sample non-trivial AR experience and prototyping their open-ended designs. All participants completed the replication task and found PRONTO easy to use. Most participants found that PRONTO encourages more exploration of designs than their current practices.

Supplemental Material

MP4 File
MP4 File
Preview video
MP4 File
Supplemental video
SRT File
Preview video captions
ZIP File
Video figure captions

References

[1]
Günter Alce, Mattias Wallergård, and Klas Hermodsson. 2015. WozARd: A Wizard of Oz Method for Wearable Augmented Reality Interaction -- A Pilot Study. Advances in Human-Computer Interaction 2015 (jun 2015), 1--10.
[2]
Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. SymbiosisSketch : Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '18) (2018).
[3]
Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2009. EverybodyLovesSketch: 3D Sketching for a Broader Audience. In Proceedings of the 22Nd Annual ACM Symposium on User Interface Software and Technology (UIST '09). ACM, NY, NY, USA, 59--68.
[4]
Jakob Bardram, Claus Bossen, Andreas Lykke-Olesen, Rune Nielsen, and Kim Halskov Madsen. 2002. Virtual video prototyping of pervasive healthcare systems. In Proceedings of the conference on Designing interactive systems processes, practices, methods, and techniques DIS '02. 167.
[5]
Michel Beaudouin-Lafon and Wendy E. Mackay. 2003. Prototyping tools and techniques. In The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications. 1017--1039.
[6]
Bill Buxton. 2007. Sketching User Experiences: getting the design right and the right design. Morgan Kaufmann. 448 pages.
[7]
Adam S. Carter and Christopher D. Hundhausen. 2010. How is User Interface Prototyping Really Done in Practice? A Survey of User Interface Designers. In 2010 IEEE Symposium on Visual Languages and Human-Centric Computing. IEEE, 207--211.
[8]
Kevin Chen and Haoqi Zhang. 2015. Remote Paper Prototype Testing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15. ACM Press, New York, New York, USA, 77--80.
[9]
Crunchbase. 2019. List of top Augmented Reality Companies. (2019). https://www.crunchbase.com/hub/augmented-reality-companies
[10]
Richard C. Davis, Brien Colwell, and James A Landay. 2008. K-Sketch : A "Kinetic" Sketch Pad for Novice Animators. (2008), 413--422.
[11]
Steven Dow, Jaemin Lee, Christopher Oezbek, Blair MacIntyre, Jay David Bolter, and Maribeth Gandy. 2005. Wizard of Oz interfaces for mixed reality applications. In CHI '05 extended abstracts on Human factors in computing systems - CHI '05. ACM Press, New York, New York, USA, 1339.
[12]
M Fiala. 2005. ARTag, a fiducial marker system using digital techniques. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), Vol. 2. 590--596 vol. 2.
[13]
Interaction Design Foundation. 2019. Augmented Reality -- The Past, The Present and The Future. (2019). https://www.interaction-design.org/literature/article/ augmented-reality-the-past-the-present-and-the-future
[14]
Maribeth Gandy and Blair MacIntyre. 2014. Designer's augmented reality toolkit, ten years later. In Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14. ACM Press, New York, New York, USA, 627--636.
[15]
Danilo Gasques, Janet G Johnson, Tommy Sharkey, and Nadir Weibel. 2019. What You Sketch Is What You Get: Quick and Easy Augmented Reality Prototyping with PintAR. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). ACM, NY, NY, USA, LBW1416:1--LBW1416:6.
[16]
Paul Green and Lisa Wei-Haas. 1985. The Rapid Development of User Interfaces: Experience with the Wizard of OZ Method. Proceedings of the Human Factors Society Annual Meeting 29, 5 (1985), 470--474.
[17]
S Guven and S Feiner. 2003. Authoring 3D hypermedia for wearable augmented and virtual reality. In Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings. 118--126.
[18]
Sinem Guven, Steven Feiner, and Ohan Oda. 2006. Mobile augmented reality interaction techniques for authoring situated media on-site. In 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE, 235--236.
[19]
Björn Hartmann, Leith Abdulla, Manas Mittal, and Scott R. Klemmer. 2007. Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition. In Proceedings of the SIGCHI conference on Human factors in computing systems CHI '07. ACM Press, New York, New York, USA, 145.
[20]
Steven Henderson and Steven Feiner. 2010. Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE transactions on visualization and computer graphics 17, 10 (2010), 1355--1368.
[21]
Steven J Henderson and Steven Feiner. 2009. Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In 2009 8th IEEE International Symposium on Mixed and Augmented Reality. IEEE, 135--144.
[22]
David G. Jansson and Steven M. Smith. 1991. Design fixation. Design Studies 12, 1 (1991), 3--11.
[23]
H Kato and M Billinghurst. 1999. Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99). 85--94.
[24]
Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, and George Fitzmaurice. 2014. Kitty: Sketching Dynamic and Interactive Illustrations. In Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14. ACM Press, New York, New York, USA, 395--405.
[25]
Rubaiat Habib Kazi, Tovi Grossman, Nobuyuki Umetani, and George Fitzmaurice. 2016. Motion Amplifiers: Sketching Dynamic Illustrations Using the Principles of 2D Animation. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, NY, NY, USA, 4599--4609.
[26]
Han-Jong Kim, Chang Min Kim, and Tek-Jin Nam. 2018. SketchStudio: Experience Prototyping with 2.5-Dimensional Animated Design Scenarios. Proceedings of the 2018 on Designing Interactive Systems Conference 2018 (2018), 831--843.
[27]
Han-Jong Kim, Ju-Whan Kim, and Tek-Jin Nam. 2016. miniStudio: Designers' Tool for Prototyping Ubicomp Space with Interactive Miniature. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16. ACM Press, New York, New York, USA, 213--224.
[28]
Kin Chung Kwan and Hongbo Fu. 2019. Mobi3DSketch: 3D Sketching in Mobile AR. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, 176:1--176:11.
[29]
James A. Landay and Brad A. Myers. 1995. Interactive Sketching for the Early Stages of User Interface Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '95). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 43--50.
[30]
David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg, and Saul Greenberg. 2018. Evaluation Strategies for HCI Toolkit Research. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18.
[31]
David Ledo, Jo Vermeulen, Sheelagh Carpendale, Saul Greenberg, Lora Oehlberg, and Sebastian Boring. 2019. Astral: Prototyping Mobile and Smart Object Interactive Behaviours Using Familiar Applications. In Proceedings of the 2019 on Designing Interactive Systems Conference (DIS '19). ACM, NY, NY, USA, 711--724.
[32]
Germán Leiva and Michel Beaudouin-Lafon. 2018. Montage: A Video Prototyping System to Reduce Re-Shooting and Increase Re-Usability. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology - UIST '18. ACM Press, Berlin, Germany.
[33]
Germán Leiva, Nolwenn Maudet, Wendy Mackay, and Michel Beaudouin-Lafon. 2019. Enact: Reducing Designer-Developer Breakdowns When Prototyping Custom Interactions. ACM Trans. Comput.-Hum. Interact. 26, 3 (may 2019), 19:1--19:48.
[34]
Yuwei Li, Xi Luo, Youyi Zheng, Pengfei Xu, and Hongbo Fu. 2017. SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, NY, NY, USA, 387--399.
[35]
Danwei Tran Luciani and Peter Vistisen. 2017. Empowering Non-Designers Through Animation-based Sketching. 7 (2017). http://www.nordes.org/nordes2017/assets/full
[36]
Blair MacIntyre, Maribeth Gandy, Steven Dow, and Jay David Bolter. 2004. DART: A Toolkit for Rapid Design Exploration of Augmented Reality Experiences Blair. In Proceedings of the 17th annual ACM symposium on User interface software and technology UIST '04. ACM Press, New York, New York, USA, 197.
[37]
Wendy E. Mackay. 1988. Video Prototyping: a technique for developing hypermedia systems. Vol. 5. ACM/SIGCHI.
[38]
Wendy E. Mackay. 2002. Using video to support interaction design. INRIA Multimedia Services. https://www.lri.fr/
[39]
Wendy E. Mackay and Anne-Laure Fayard. 1999. Video brainstorming and prototyping: techniques for participatory design. CHI'99 extended abstracts on Human factors in ... May (1999), 118--119.
[40]
Nolwenn Maudet, Germán Leiva, Michel Beaudouin-Lafon, and Wendy E. Mackay. 2017. Design Breakdowns: Designer-Developer Gaps in Representing and Interpreting Interactive Systems. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing - CSCW '17 (2017), 630--641.
[41]
Michael Flarup. 2018. Designing For Augmented Reality. (2018). https://www.youtube.com/watch?v=-r1VtYxvVIc
[42]
Peter Mohr, David Mandl, Markus Tatzgern, Eduardo Veas, Dieter Schmalstieg, and Denis Kalkofen. 2017. Retargeting video tutorials showing tools with surface contact to augmented reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 6547--6558.
[43]
Michael Nebeling and Katy Madier. 2019. 360Proto: Making Interactive Virtual Reality & Augmented Reality Prototypes from Paper. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, 596:1--596:13.
[44]
Michael Nebeling, Janet Nebeling, Ao Yu, and Rob Rumble. 2018. ProtoAR: Rapid Physical-Digital Prototyping of Mobile Augmented Reality Applications. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, 353:1--353:12.
[45]
Michael Nebeling and Maximilian Speicher. 2018. The Trouble with Augmented Reality/Virtual Reality Authoring Tools. In 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 333--337.
[46]
NewGenApps. 2019. 6 VR and AR Statistics: Shaping the Future of Augmented Reality with Data. (2019). https://www.newgenapps.com/blog/ 6-vr-and-ar-statistics-shaping-the-future-of-augmented-reality-with-data
[47]
Benjamin Nuernberger, Eyal Ofek, Hrvoje Benko, and Andrew D Wilson. 2016. SnapToReality: Aligning Augmented Reality to the Real World. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, NY, NY, USA, 1233--1244.
[48]
Marc Rettig. 1994. Prototyping for tiny fingers. Commun. ACM 37, 4 (apr 1994), 21--27.
[49]
Nazmus Saquib, Rubaiat Habib Kazi, Li-Yi Wei, and Wilmot Li. 2019. Interactive Body-Driven Graphics for Augmented Video Performance. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, 622:1 -- 622:12.
[50]
Hartmut Seichter, Julian Looser, and Mark Billinghurst. 2008. ComposAR: An Intuitive Tool for Authoring AR Applications. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR '08). IEEE Computer Society, Washington, DC, USA, 177--178.
[51]
Bruce Tognazzini. 1994. The "Starfire" video prototype project. In Proceedings of the SIGCHI conference on Human factors in computing systems celebrating interdependence - CHI '94. ACM Press, New York, New York, USA, 99--105.
[52]
Peter Vistisen and Soren Bolvig Poulsen. 2017. Return of the Vision Video: Can corporate vision videos serve as setting for participation?. In Nordic Design Research (NORDES). Nordes Digital Archive.
[53]
Pierre Wellner. 1993. Interacting with Paper on the DigitalDesk. Commun. ACM 36, 7 (jul 1993), 87--96.
[54]
Shihong Xia, Lin Gao, Yu-Kun Lai, Ming-Ze Yuan, and Jinxiang Chai. 2017. A Survey on Human Performance Capture and Animation. Journal of Computer Science and Technology 32, 3 (may 2017), 536--554.
[55]
Robert Xiao, Chris Harrison, and Scott E Hudson. 2013. WorldKit: rapid and easy creation of ad-hoc interactive applications on everyday surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 879--888.

Cited By

View all
  • (2025)VisTellAR: Embedding Data Visualization to Short-Form Videos Using Mobile Augmented RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210431:3(1862-1874)Online publication date: Mar-2025
  • (2024)How New Developers Approach Augmented Reality Development Using Simplified Creation Tools: An Observational StudyMultimodal Technologies and Interaction10.3390/mti80400358:4(35)Online publication date: 22-Apr-2024
  • (2024)"We Are Visual Thinkers, Not Verbal Thinkers!": A Thematic Analysis of How Professional Designers Use Generative AI Image Generation ToolsProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685370(1-14)Online publication date: 13-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
April 2020
10688 pages
ISBN:9781450367080
DOI:10.1145/3313831
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. ar
  2. design by enaction
  3. sketching
  4. video prototyping

Qualifiers

  • Research-article

Conference

CHI '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)240
  • Downloads (Last 6 weeks)22
Reflects downloads up to 05 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2025)VisTellAR: Embedding Data Visualization to Short-Form Videos Using Mobile Augmented RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210431:3(1862-1874)Online publication date: Mar-2025
  • (2024)How New Developers Approach Augmented Reality Development Using Simplified Creation Tools: An Observational StudyMultimodal Technologies and Interaction10.3390/mti80400358:4(35)Online publication date: 22-Apr-2024
  • (2024)"We Are Visual Thinkers, Not Verbal Thinkers!": A Thematic Analysis of How Professional Designers Use Generative AI Image Generation ToolsProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685370(1-14)Online publication date: 13-Oct-2024
  • (2024)PilotAR: Streamlining Pilot Studies with OHMDs from Concept to InsightProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785768:3(1-35)Online publication date: 9-Sep-2024
  • (2024)PhoneCanvas: 3D Sketching System Using a Depth Camera-Equipped Smartphone as a CanvasProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682078(1-12)Online publication date: 7-Oct-2024
  • (2024)Enabling Safer Augmented Reality Experiences: Usable Privacy Interventions for AR Creators and End-UsersAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686708(1-8)Online publication date: 13-Oct-2024
  • (2024)RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual EffectsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661631(1248-1261)Online publication date: 1-Jul-2024
  • (2024)How People Prompt Generative AI to Create Interactive VR ScenesProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661547(2319-2340)Online publication date: 1-Jul-2024
  • (2024)Integrated Calculators: Moving Calculation into the WorldProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661523(343-355)Online publication date: 1-Jul-2024
  • (2024)UXR-kit: An Ideation Kit and Method for Collaborative and User-Centered Design about Extended Reality systems.Proceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3689684(1-2)Online publication date: 9-Oct-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media