abstract

RealitySketch: Embedding Responsive Graphics and Visualizations in AR with Dynamic Sketching

Authors:

Rubaiat Habib Kazi,

Stephen DiVerdi,

Daniel LeithingerAuthors Info & Claims

UIST '20 Adjunct: Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Pages 135 - 138

https://doi.org/10.1145/3379350.3416155

Published: 20 October 2020 Publication History

Abstract

We present RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations. In recent years, an increasing number of AR sketching tools enable users to draw and embed sketches in the real world. However, with the current tools, sketched contents are inherently static, floating in mid air without responding to the real world. This paper introduces a new way to embed dynamic and responsive graphics in the real world. In RealitySketch, the user draws graphical elements on a mobile AR screen and binds them with physical objects in real-time and improvisational ways, so that the sketched elements dynamically move with the corresponding physical motion. The user can also quickly visualize and analyze real-world phenomena through responsive graph plots or interactive visualizations. This paper contributes to a set of interaction techniques that enable capturing, parameterizing, and visualizing real-world motion without pre-defined programs and configurations. Finally, we demonstrate our tool with several application scenarios, including physics education, sports training, and in-situ tangible interfaces.

References

[1]

Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. Symbiosissketch: Combining 2d & 3d sketching for designing detailed 3d objects in situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 185.

Digital Library

[2]

John Brosz, Miguel A Nacenta, Richard Pusch, Sheelagh Carpendale, and Christophe Hurter. 2013. Transmogrification: causal manipulation of visualizations. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 97--106.

Digital Library

[3]

William O Chao, T Munzner, and MVD Panne. 2010. Napkinvis: Rapid pen-centric authoring of improvisational visualizations. In IEEE Infovis.

[4]

Takeo Igarashi, Satoshi Matsuoka, Sachiko Kawachiya, and Hidehiko Tanaka. 2007. Interactive beautification: a technique for rapid geometric design. In ACM SIGGRAPH 2007 courses. 18--es.

Digital Library

[5]

Google Inc. 2018. Just a Line. (2018). https://justaline.withgoogle.com/

[6]

PTC Inc. 2017. Vuforia Chalk AR. (2017). https://chalk.vuforia.com/

[7]

Hiroshi Ishii and Brygg Ullmer. 1997. Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. 234--241.

Digital Library

[8]

Robert JK Jacob, Audrey Girouard, Leanne M Hirshfield, Michael S Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-based interaction: a framework for post-WIMP interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems. 201--210.

Digital Library

[9]

Alan Kay. 2005. Squeak etoys, children & learning. online article 2006 (2005).

[10]

Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, and George Fitzmaurice. 2014a. Kitty: sketching dynamic and interactive illustrations. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 395--405.

Digital Library

[11]

Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, Shengdong Zhao, and George Fitzmaurice. 2014b. Draco: bringing life to illustrations with kinetic textures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 351--360.

Digital Library

[12]

Rubaiat Habib Kazi, Tovi Grossman, Nobuyuki Umetani, and George Fitzmaurice. 2016. Motion amplifiers: sketching dynamic illustrations using the principles of 2D animation. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4599--4609.

Digital Library

[13]

Kin Chung Kwan and Hongbo Fu. 2019. Mobi3DSketch: 3D Sketching in Mobile AR. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 176.

Digital Library

[14]

James A Landay. 1996. SILK: sketching interfaces like krazy. In Conference companion on Human factors in computing systems. 398--399.

Digital Library

[15]

James A Landay and Brad A Myers. 1995. Interactive sketching for the early stages of user interface design. In Proceedings of the SIGCHI conference on Human factors in computing systems. 43--50.

Digital Library

[16]

Joseph J LaViola Jr and Robert C Zeleznik. 2004. MathPad 2: a system for the creation and exploration of mathematical sketches. ACM Transactions on Graphics (TOG) 23, 3 (2004), 432--440.

Digital Library

[17]

Bongshin Lee, Rubaiat Habib Kazi, and Greg Smith. 2013. SketchStory: Telling more engaging stories with data through freeform sketching. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 2416--2425.

Digital Library

[18]

Zhicheng Liu, John Thompson, Alan Wilson, Mira Dontcheva, James Delorey, Sam Grigg, Bernard Kerr, and John Stasko. 2018. Data Illustrator: Augmenting vector design tools with lazy data binding for expressive visualization authoring. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 123.

Digital Library

[19]

Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321--1329.

[20]

Elizabeth D Mynatt, Takeo Igarashi, W Keith Edwards, and Anthony LaMarca. 1999. Flatland: new dimensions in office whiteboards. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 346--353.

Digital Library

[21]

Ken Perlin. 2016. Future Reality: How emerging technologies will change language itself. IEEE computer graphics and applications 36, 3 (2016), 84--89.

[22]

Ken Perlin, Zhenyi He, and Karl Rosenberg. 2018. Chalktalk: A Visualization and Communication Language--As a Tool in the Domain of Computer Science Education. arXiv preprint arXiv:1809.07166 (2018).

[23]

Jeremy Scott and Randall Davis. 2013. Physink: sketching physical behavior. In Proceedings of the adjunct publication of the 26th annual ACM symposium on User interface software and technology. ACM, 9--10.

Digital Library

[24]

Ivan E Sutherland. 1964. Sketchpad a man-machine graphical communication system. Simulation 2, 5 (1964), R--3.

[25]

Ryo Suzuki, Jun Kato, Mark D Gross, and Tom Yeh. 2018. Reactile: Programming Swarm User Interfaces through Direct Physical Manipulation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 199.

Digital Library

[26]

Ryo Suzuki, Rubaiat Habib Kazi, Li-Yi Wei, Stephen DiVerdi, Wilmot Li, and Daniel Leithinger. 2020. RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching. In Proceedings of the 33rd Annual Symposium on User Interface Software and Technology. ACM.

Digital Library

[27]

Bret Victor. 2013. Drawing dynamic visualizations. URL: worrydream. com (2013).

[28]

Bret Victor. 2014a. Humane representation of thought: a trail map for the 21st century. In Proceedings of the companion publication of the 2014 ACM SIGPLAN conference on Systems, Programming, and Applications: Software for Humanity. 5--5.

Digital Library

[29]

Bret Victor. 2014b. Seeing spaces. In Talk at EG conference.

[30]

Pierre Wellner. 1993. Interacting with paper on the DigitalDesk. Commun. ACM 36, 7 (1993), 87--96.

Digital Library

[31]

Aidan Wolf. 2019. DoodleLens. (2019). https://www.doodlelens.app/

[32]

Haijun Xia, Nathalie Henry Riche, Fanny Chevalier, Bruno De Araujo, and Daniel Wigdor. 2018. DataInk: Direct and Creative Data-Oriented Drawing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 223, 13 pages. http://dx.doi.org/10.1145/3173574.3173797

Digital Library

[33]

Jun Xing, Rubaiat Habib Kazi, Tovi Grossman, Li-Yi Wei, Jos Stam, and George Fitzmaurice. 2016. Energy-brushes: Interactive tools for illustrating stylized elemental dynamics. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 755--766.

Digital Library

Cited By

Frau VSpano LArtizzu VNebeling M(2023)XRSpotlight: Example-based Programming of XR Interactions using a Rule-based ApproachProceedings of the ACM on Human-Computer Interaction10.1145/35932377:EICS(1-28)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593237
Zhang YNguyen CKazi RYu L(2023)PoseVEC: Authoring Adaptive Pose-aware Effects using Visual Programming and DemonstrationsProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606788(1-15)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606788
Grubert J(2023)Mixed Reality Interaction TechniquesSpringer Handbook of Augmented Reality10.1007/978-3-030-67822-7_5(109-129)Online publication date: 1-Jan-2023
https://doi.org/10.1007/978-3-030-67822-7_5
Show More Cited By

Index Terms

RealitySketch: Embedding Responsive Graphics and Visualizations in AR with Dynamic Sketching
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality

Recommendations

Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI
UIST '22: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology

This paper introduces Sketched Reality, an approach that combines AR sketching and actuated tangible user interfaces (TUI) for bi-directional sketching interaction. Bi-directional sketching enables virtual sketches and physical objects to “affect” each ...
RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching
UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

We present RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations. In recent years, an increasing number of AR sketching tools enable users to draw and embed sketches in the real world. However, with the ...
RealitySketch: augmented reality sketching for real-time embedded and responsive visualizations
SA '21: SIGGRAPH Asia 2021 Real-Time Live!

In this Real-Time Live, we demonstrate RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations [Suzuki et al. 2020]. In recent years, an increasing number of AR sketching tools enable users to draw and embed ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '20 Adjunct: Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

October 2020

203 pages

ISBN:9781450375153

DOI:10.1145/3379350

General Chairs:
Shamsi Iqbal
Microsoft Research, USA
,
Karon MacLean
University of British Columbia, Canada
,
Program Chairs:
Fanny Chevalier
University of Toronto, Canada
,
Stefanie Mueller
MIT CSAIL, USA

Copyright © 2020 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2020

Check for updates

Author Tags

Qualifiers

Abstract

Conference

UIST '20

Sponsor:

UIST '20: The 33rd Annual ACM Symposium on User Interface Software and Technology

October 20 - 23, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 355 of 1,733 submissions, 20%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
493
Total Downloads

Downloads (Last 12 months)63
Downloads (Last 6 weeks)7

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Frau VSpano LArtizzu VNebeling M(2023)XRSpotlight: Example-based Programming of XR Interactions using a Rule-based ApproachProceedings of the ACM on Human-Computer Interaction10.1145/35932377:EICS(1-28)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593237
Zhang YNguyen CKazi RYu L(2023)PoseVEC: Authoring Adaptive Pose-aware Effects using Visual Programming and DemonstrationsProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606788(1-15)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606788
Grubert J(2023)Mixed Reality Interaction TechniquesSpringer Handbook of Augmented Reality10.1007/978-3-030-67822-7_5(109-129)Online publication date: 1-Jan-2023
https://doi.org/10.1007/978-3-030-67822-7_5
Zhu ZLiu ZWang TZhang YQian XRaja PVillanueva ARamani K(2022)MechARspace: An Authoring System Enabling Bidirectional Binding of Augmented Reality with Toys in Real-timeProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545668(1-16)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545668
Evangelista Belo JLystbæk MFeit APfeuffer KKán POulasvirta AGrønbæk K(2022)AUIT – the Adaptive User Interfaces Toolkit for Designing XR ApplicationsProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545651(1-16)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545651
Kaimoto HMonteiro KFaridan MLi JFarajian SKakehi YNakagaki KSuzuki R(2022)Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UIProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545626(1-12)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545626
Qian JZhou TYoung-Ng MMa JCheung ALi XGonsher IHuang J(2021)Portalware: Exploring Free-Hand AR Drawing with a Dual-Display Smartphone-Wearable ParadigmProceedings of the 2021 ACM Designing Interactive Systems Conference10.1145/3461778.3462098(205-219)Online publication date: 28-Jun-2021
https://dl.acm.org/doi/10.1145/3461778.3462098

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten