skip to main content
10.1145/3025453.3025688acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Retargeting Video Tutorials Showing Tools With Surface Contact to Augmented Reality

Published: 02 May 2017 Publication History

Abstract

A video tutorial effectively conveys complex motions, but may be hard to follow precisely because of its restriction to a predetermined viewpoint. Augmented reality (AR) tutorials have been demonstrated to be more effective. We bring the advantages of both together by interactively retargeting conventional, two-dimensional videos into three-dimensional AR tutorials. Unlike previous work, we do not simply overlay video, but synthesize 3D-registered motion from the video. Since the information in the resulting AR tutorial is registered to 3D objects, the user can freely change the viewpoint without degrading the experience. This approach applies to many styles of video tutorials. In this work, we concentrate on a class of tutorials which alter the surface of an object.

Supplementary Material

suppl.mov (pn2187-file3.mp4)
Supplemental video

References

[1]
Maneesh Agrawala, Doantam Phan, Julie Heiser, John Haymaker, Jeff Klingner, Pat Hanrahan, and Barbara Tversky. 2003. Designing effective step-by-step assembly instructions. ACM Trans. Graph. 22, 3 (July 2003), 828--837.
[2]
Fraser Anderson, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2013. YouMove: Enhancing Movement Training with an Augmented Reality Mirror. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 311--320.
[3]
Tadas Baltrušaitis, Louis-Philippe Morency, and Peter Robinson. 2013. Constrained Local Neural Fields for robust facial landmark detection in the wild. In 300 Faces in-the-wild challenge, International Conference on Computer Vision.
[4]
Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies 4, 3 (2009), 114--123.
[5]
Olivier Bau and Wendy E. Mackay. 2008. OctoPocus: A Dynamic Guide for Learning Gesture-based Command Sets. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 37--46.
[6]
P. Breedveld. 1997. Observation, Manipulation, and Eye-Hand Coordination Problems in Minimally Invasive Surgery. In in Proc XVI European Annual Conference on Human Decision Making and Manual Control. 9--11.
[7]
John Brooke and others. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4--7.
[8]
Andreas Butz. 1994. BETTY: Planning and Generating Animations for the Visualization of Movements and Spatial Relations. In Proc. of Advanced Visual Interfaces. 53--58.
[9]
Pei-Yu Chi, Sally Ahn, Amanda Ren, Mira Dontcheva, Wilmot Li, and Björn Hartmann. 2012. MixT: automatic generation of step-by-step mixed media tutorials. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 93--102.
[10]
Pei-Yu (Peggy) Chi, Mira Dontcheva, Li Li Wilmot, Daniel Vodel, and Björn Hartmann. 2016. Authoring Illustrations of Human Movements by Iterative Physical Demonstration. In Proceedings of the 29th Annual ACM Symposium on User Interface Software and Technology (UIST '16). to appear.
[11]
Yung-Yu Chuang, Aseem Agarwala, Brian Curless, David H. Salesin, and Richard Szeliski. 2002. Video Matting of Complex Scenes. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '02). ACM, New York, NY, USA, 243--248.
[12]
J P Collomosse. 2003. Cartoon-style Rendering of Motion from Video. Vision Video and Graphics 67, 6 (2003), 549--564.
[13]
Dima Damen, Teesid Leelasawassuk, Osian Haines, Andrew Calway, and Walterio Mayol-Cuevas. 2014. You-Do, I-Learn: Discovering Task Relevant Objects and their Modes of Interaction from Multi-User Egocentric Video. In British Machine Vision Conference (BMVC). BMVA.
[14]
David H. Douglas and Thomas K. Peucker. 1973. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica: The International Journal for Geographic Information and Geovisualization 10, 2 (1 Oct. 1973), 112--122.
[15]
Steven Feiner, Blair Macintyre, and Dorée Seligmann. 1993. Knowledge-based augmented reality. Commun. ACM 36 (July 1993), 53--62. Issue 7.
[16]
M. Goto, Y. Uematsu, H. Saito, S. Senda, and A. Iketani. 2010. Task support system by displaying instructional video onto AR workspace. In Mixed and Augmented Reality (ISMAR), 2010 9th IEEE International Symposium on. 83--90.
[17]
Floraine Grabler, Maneesh Agrawala, Wilmot Li, Mira Dontcheva, and Takeo Igarashi. 2009. Generating photo manipulation tutorials by demonstration. ACM Transactions on Graphics (TOG) 28, 3 (2009), 66.
[18]
Ankit Gupta, Dieter Fox, Brian Curless, and Michael Cohen. 2012. DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly. In Proceedings of ACM Symposium on User Interface Software and Technology (UIST '12). 389--402.
[19]
Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology 52 (1988), 139--183.
[20]
Steven Henderson and Steven Feiner. 2011. Exploring the Benefits of Augmented Reality Documentation for Maintenance and Repair. IEEE Trans. Vis. Comp. Graph. 17, 10 (2011), 1355--1368.
[21]
Zdenek Kalal, Krystian Mikolajczyk, and Jiri Matas. 2012. Tracking-Learning-Detection. IEEE Trans. Pattern Anal. Mach. Intell. 34, 7 (July 2012), 1409--1422.
[22]
Denis Kalkofen, Markus Tatzgern, and Dieter Schmalstieg. 2009. Explosion Diagrams in Augmented Reality. In Proc. of IEEE Virtual Reality (VR '09). IEEE, 71--78.
[23]
Denis Kalkofen, Eduardo E. Veas, Stefanie Zollmann, Markus Steinberger, and Dieter Schmalstieg. 2013. Adaptive ghosted views for Augmented Reality. In IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2013, Adelaide, Australia, October 1--4, 2013. IEEE, 1--9.
[24]
Bernhard Kerbl, Denis Kalkofen, Markus Steinberger, and Dieter Schmalstieg. 2015. Interactive Disassembly Planning for Complex Objects. Computer Graphics Forum (2015).
[25]
B. Kim and I. Essa. 2005. Video-based nonphotorealistic and expressive illustration of motion. In Proc. of the Computer Graphics International 2005. 32--35.
[26]
Vladislav Kraevoy, Alla Sheffer, and Michiel van de Panne. 2009. Modeling from Contour Drawings. In Proceedings of the 6th Eurographics Symposium on Sketch-Based Interfaces and Modeling (SBIM '09). ACM, New York, NY, USA, 37--44.
[27]
Tobias Langlotz, Mathäus Zingerle, Raphael Grasset, Hannes Kaufmann, and Gerhard Reitmayr. 2012. AR Record Replay: Situated Compositing of Video Content in Mobile Augmented Reality. In Proceedings of the 24th Australian Computer-Human Interaction Conference (OzCHI '12). 318--326.
[28]
Florian Ledermann and Dieter Schmalstieg. 2005. APRIL A High-Level Framework for Creating Augmented Reality Presentations. In Proc. of IEEE Virtual Reality. 187--194.
[29]
V. Lepetit, F.Moreno-Noguer, and P.Fua. 2009. EPnP: An Accurate O(n) Solution to the PnP Problem. International Journal Computer Vision 81, 2 (2009).
[30]
Wilmot Li, Maneesh Agrawala, Brian Curless, and David Salesin. 2008. Automated Generation of Interactive 3D Exploded View Diagrams. ACM Trans. Graph. 27, 3, Article 101 (Aug. 2008), 7 pages.
[31]
David G. Lowe. 2004. Distinctive Image Features from Scale-Invariant Keypoints. Int. J. Comput. Vision 60, 2 (Nov. 2004), 91--110.
[32]
Bruce D. Lucas and Takeo Kanade. 1981. An Iterative Image Registration Technique with an Application to Stereo Vision. In Proceedings of the 7th International Joint Conference on Artificial Intelligence - Volume 2 (IJCAI'81). 674--679.
[33]
Peter Mohr, Bernhard Kerbl, Michael Donoser, Dieter Schmalstieg, and Denis Kalkofen. 2015. Retargeting Technical Documentation to Augmented Reality. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3337--3346.
[34]
G. Nebehay. 2012. Robust Object Tracking Based on Tracking-Learning-Detection. Master's thesis. Faculty of Informatics, TU Vienna.
[35]
Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time Dense Surface Mapping and Tracking. In Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR '11). IEEE Computer Society, Washington, DC, USA, 127--136.
[36]
Marc Nienhaus and Jürgen Döllner. 2003. Dynamic glyphs Depicting dynamics in images of 3D scenes. In Proc. of the 3rd international conference on Smart graphics (SG'03). Springer-Verlag, Berlin, Heidelberg, 102--111.
[37]
Peter Ondruska, Pushmeet Kohli, and Shahram Izadi. 2015. MobileFusion: Real-time Volumetric Surface Reconstruction and Dense Tracking On Mobile Phones. In International Symposium on Mixed and Augmented Reality (ISMAR). Fukuoka, Japan.
[38]
M. Park, S. Serefoglou, L. Schmidt, K. Radermacher, C. Schlick, and H. Luczak. 2008. Hand-Eye Coordination Using a Video See-Through Augmented Reality System. The Ergonomics Open Journal 1 (2008), 46--53.
[39]
N. Petersen and D. Stricker. 2012. Learning task structure from video examples for workflow tracking and authoring. In Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on. 237--246.
[40]
Suporn Pongnumkul, Mira Dontcheva, Wilmot Li, Jue Wang, Lubomir Bourdev, Shai Avidan, and Michael F. Cohen. 2011. Pause-and-play: Automatically Linking Screencast Video Tutorials with Applications. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 135--144.
[41]
Dorée Duncan Seligmann and Steven Feiner. 1991. Automated Generation of Intent-Based 3D Illustrations. In Proc. of ACM SIGGRAPH. 123--132.
[42]
Jianchao Tan, Marek Dvorožňák, Daniel Sýkora, and Yotam Gingold. 2015. Decomposing Time-Lapse Paintings into Layers. ACM Transactions on Graphics (TOG) 34, 4, Article 61 (July 2015), 10 pages.
[43]
Richard Tang, Xing-Dong Yang, Scott Bateman, Joaquim Jorge, and Anthony Tang. 2015. Physio@Home: Exploring Visual Guidance and Feedback Techniques for Physiotherapy Exercises. In Proceedings of ACM CHI. 4123--4132.
[44]
Barbara Tversky, Julie Bauer Morrison Y, and Mireille Betrancourt. 2002. Animation: Can it facilitate. International Journal of Human-Computer Studies 57 (2002), 247--262.
[45]
Sean White, David Feng, and Steven Feiner. 2009. Interaction and presentation techniques for shake menus in tangible augmented reality. In Proc. of the IEEE IISMAR. 39--48.
[46]
Jürgen Zauner, Michael Haller, Alexander Brandl, and Werner Hartmann. 2003. Authoring of a Mixed Reality Assembly Instructor for Hierarchical Structures. In Proc. of IEEE/ACM ISMAR. 237--246.

Cited By

View all
  • (2024)Experiential Tutorials: Designing Tutorial Authoring Tools to Facilitate Tacit Knowledge Exchange in Creative PracticesProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3664624(30-34)Online publication date: 23-Jun-2024
  • (2024)Visual Cue Based Corrective Feedback for Motor Skill Training in Mixed Reality: A SurveyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.322799930:7(3121-3134)Online publication date: Jul-2024
  • (2023)Scene Responsiveness for Visuotactile Illusions in Mixed RealityProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606825(1-15)Online publication date: 29-Oct-2023
  • Show More Cited By

Index Terms

  1. Retargeting Video Tutorials Showing Tools With Surface Contact to Augmented Reality

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. augmented reality
    2. retargeting
    3. video tutorial
    4. virtual reality

    Qualifiers

    • Research-article

    Funding Sources

    • EU FP7 (MAGELLAN)
    • Competence Centers for Excellent Technologies (COMET)

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)45
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Experiential Tutorials: Designing Tutorial Authoring Tools to Facilitate Tacit Knowledge Exchange in Creative PracticesProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3664624(30-34)Online publication date: 23-Jun-2024
    • (2024)Visual Cue Based Corrective Feedback for Motor Skill Training in Mixed Reality: A SurveyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.322799930:7(3121-3134)Online publication date: Jul-2024
    • (2023)Scene Responsiveness for Visuotactile Illusions in Mixed RealityProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606825(1-15)Online publication date: 29-Oct-2023
    • (2023)Tacit Descriptions: Uncovering Ambiguity in Crowdsourced Descriptions of Motions and MaterialsProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596048(2522-2536)Online publication date: 10-Jul-2023
    • (2023)InstruMentAR: Auto-Generation of Augmented Reality Tutorials for Operating Digital Instruments Through Recording Embodied DemonstrationProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581442(1-17)Online publication date: 19-Apr-2023
    • (2023)Design Patterns for Situated Visualization in Augmented RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.3327398(1-12)Online publication date: 2023
    • (2023)State-Aware Configuration Detection for Augmented Reality Step-by-Step Tutorials2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00030(157-166)Online publication date: 16-Oct-2023
    • (2023)Augmented Reality Training in Manufacturing SectorsThe Digital Twin10.1007/978-3-031-21343-4_17(447-496)Online publication date: 3-Jun-2023
    • (2022)A Survey of Educational Augmented Reality in Academia and Practice: Effects on Cognition, Motivation, Collaboration, Pedagogy and Applications2022 8th International Conference of the Immersive Learning Research Network (iLRN)10.23919/iLRN55037.2022.9815979(1-8)Online publication date: 30-May-2022
    • (2022)“Kapow!”: Studying the Design of Visual Feedback for Representing Contacts in Extended RealityProceedings of the 28th ACM Symposium on Virtual Reality Software and Technology10.1145/3562939.3565607(1-11)Online publication date: 29-Nov-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media