skip to main content
10.1145/3025453.3025467acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Public Access

User-Guided Synthesis of Interactive Diagrams

Published: 02 May 2017 Publication History

Abstract

Interactive diagrams are expensive to build, requiring significant programming experience. The cost of building such diagrams often prevents novice programmers or non-programmers from doing so. In this paper, we present user-guided techniques that transform a static diagram into an interactive one without requiring the user to write code. We also present a tool called EDDIE that prototypes these techniques. We evaluate EDDIE through: (1) a case study in which we use EDDIE to implement existing real-world diagrams from the literature and (2) a usability session with target users in which subjects build several diagrams in EDDIE and provide feedback on EDDIE's user experience. Our experiments demonstrate that EDDIE is usable and expressive, and that EDDIE enables real-world diagrams to be implemented without requiring programming expertise.

Supplementary Material

ZIP File (pn1043-file4.zip)
suppl.mov (pn1043p.mp4)
Supplemental video

References

[1]
2016. Apparatus: a hybrid graphics editor and programming environment for creating interactive diagrams. Website. (2016). Accessed on 2016-04--12 from http://aprt.us.
[2]
2016. FabricJS JavaScript Canvas Library. Website. (2016). Accessed on 2016-09--19 from http://fabricjs.com/.
[3]
2016. Interactive Mathematics: Learn math while you play with it. Website. (2016). Accessed on 2016-04-08 from http://www.intmath.com.
[4]
2016. Overconstrained: Cassowary projects and its community. Website. (2016). Accessed on 2016-04--11 from http://overconstrained.io.
[5]
2016. Pendulum Lab -- Motion, Pendulum, Simple Harmonic Motion -- PhET. Website. (2016). Accessed on 2016-09--19 from https://phet.colorado.edu/en/ simulation/legacy/pendulum-lab.
[6]
2016. PhET: Interactive Simulations for Science and Math. Website. (2016). Accessed on 2016-04--11 from https://PhET.colorado.edu.
[7]
2016. Resonance -- Resonance, Harmonic Motion, Oscillator -- PhET. Website. (2016). Accessed on 2016-09--19 from https://phet.colorado.edu/en/ simulation/legacy/resonance.
[8]
Wendy K. Adams, Sam Reid, Ron LeMaster, Sarah McKagan, Katherine Perkins, Michael Dubson, and Carl E. Wieman. 2008a. A Study of Educational Simulations Part II Interface Design. Journal of Interactive Learning Research 19, 4 (October 2008), 551--577. https://www.learntechlib.org/p/24364
[9]
Wendy K. Adams, Sam Reid, Ron LeMaster, Sarah B. McKagan, Katherine K. Perkins, Michael Dubson, and Carl E. Wieman. 2008b. A Study of Educational Simulations Part I - Engagement and Learning. Journal of Interactive Learning Research 19, 3 (July 2008), 397--419. https://www.learntechlib.org/p/24230
[10]
Christine Alvarado and Randall Davis. 2004. SketchREAD: A Multi-domain Sketch Recognition Engine. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, NY, NY, USA, 23--32.
[11]
Christine Alvarado and Randall Davis. 2006. Resolving ambiguities to create a natural computer-based sketching environment. In ACM SIGGRAPH 2006 Courses. ACM, 24.
[12]
Greg J. Badros, Alan Borning, and Peter J. Stuckey. 2001. The Cassowary linear arithmetic constraint solving algorithm. ACM Transactions on Computer-Human Interaction 8, 4 (dec 2001), 267--306.
[13]
Julia M. Chamberlain, Kelly Lancaster, Robert Parson, and Katherine K. Perkins. 2014. How guidance affects student engagement with an interactive simulation. Chem. Educ. Res. Pract. 15 (2014), 628--638. Issue 4.
[14]
Salman Cheema and Joseph J LaViola Jr. 2010. Applying mathematical sketching to sketch-based physics tutoring software. In International Symposium on Smart Graphics. Springer, 13--24.
[15]
Allen Cypher and Daniel Conrad Halbert. 1993. Watch what I do: programming by demonstration. MIT press.
[16]
George B Dantzig, Alex Orden, Philip Wolfe, and others. 1955. The generalized simplex method for minimizing a linear form under linear inequality restraints. Pacific J. Math. 5, 2 (1955), 183--195.
[17]
Richard C. Davis, Brien Colwell, and James A. Landay. 2008. K-sketch. In Proceeding of the twenty-sixth annual CHI conference on Human factors in computing systems CHI '08. ACM Press, NY, NY, USA, 413.
[18]
Tim Felgentreff, Alan Borning, Robert Hirschfeld, Jens Lincke, Yoshiki Ohshima, Bert Freudenberg, and Robert Krahn. 2014. ECOOP 2014 -- Object-Oriented Programming: 28th European Conference, Uppsala, Sweden, July 28 -- August 1, 2014. Proceedings. Springer Berlin Heidelberg, Berlin, Heidelberg, Chapter Babelsberg/JS, 411--436.
[19]
Bjorn N Freeman-Benson, John Maloney, and Alan Borning. 1990. An incremental constraint solver. Commun. ACM 33, 1 (1990), 54--63.
[20]
Floraine Grabler, Maneesh Agrawala, Wilmot Li, Mira Dontcheva, and Takeo Igarashi. 2009. Generating Photo Manipulation Tutorials by Demonstration. In ACM SIGGRAPH 2009 Papers (SIGGRAPH '09). ACM, NY, NY, USA, Article 66, 9 pages.
[21]
Sumit Gulwani. 2010. Dimensions in program synthesis. In Proceedings of the 12th international ACM SIGPLAN symposium on Principles and practice of declarative programming. ACM, 13--24.
[22]
Sumit Gulwani, Susmit Jha, Ashish Tiwari, and Ramarathnam Venkatesan. 2011a. Synthesis of loop-free programs. In Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation - PLDI '11. ACM Press, NY, NY, USA, 62.
[23]
Sumit Gulwani, Vijay Anand Korthikanti, and Ashish Tiwari. 2011b. Synthesizing geometry constructions. ACM SIGPLAN Notices 46, 6 (jun 2011), 50.
[24]
Markus Hohenwarter and Karl Fuchs. 2005. Combination of dynamic geometry, algebra and calculus in the software system GeoGebra. Computer algebra systems and dynamic geometry systems in mathematics teaching conference 2004 2002, July (2005), 1--6. http: //www.geogebratube.org/material/show/id/747
[25]
Thibaud Hottelier, Ras Bodik, and Kimiko Ryokai. 2014. Programming by manipulation for layout. In Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14. ACM Press, NY, NY, USA, 231--241.
[26]
Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. 2007. Teddy: a sketching interface for 3D freeform design. In Acm siggraph 2007 courses. ACM, 21.
[27]
Joaquim Jorge and Faramarz Samavati. 2010. Sketch-based interfaces and modeling. Springer Science & Business Media.
[28]
Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011. Wrangler: Interactive Visual Specification of Data Transformation Scripts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, NY, NY, USA, 3363--3372.
[29]
Alan Kay and Adele Goldberg. 1977. Personal dynamic media. Computer 10, 3 (1977), 31--41.
[30]
Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, and George Fitzmaurice. 2014a. Kitty: sketching dynamic and interactive illustrations. Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14 (2014), 395--405.
[31]
Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, Shengdong Zhao, and George Fitzmaurice. 2014b. Draco: Bringing Life to Illustrations with Kinetic Textures. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2014), 351--360.
[32]
Rubaiat Habib Kazi, Tovi Grossman, Nobuyuki Umetani, and George Fitzmaurice. 2016. Skuid: Sketching Dynamic Illustrations Using the Principles of 2D Animation. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4599--4609.
[33]
Vu Le, Sumit Gulwani, and Zhendong Su. 2013. SmartSynth. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services - MobiSys '13. ACM Press, NY, NY, USA, 193.
[34]
Gilly Leshed, Eben M Haber, Tara Matthews, and Tessa Lau. 2008. CoScripter: automating & sharing how-to knowledge in the enterprise. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1719--1728.
[35]
Hao Lü and Yang Li. 2012. Gesture Coder: A Tool for Programming Multi-touch Gestures by Demonstration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, NY, NY, USA, 2875--2884.
[36]
Emily B. Moore, Timothy A. Herzog, and Katherine K. Perkins. 2013. Interactive simulations as implicit support for guided-inquiry. Chem. Educ. Res. Pract. 14 (2013), 257--268. Issue 3.
[37]
B. A. Myers. 1986. Visual Programming, Programming by Example, and Program Visualization: A Taxonomy. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '86). ACM, NY, NY, USA, 59--66.
[38]
Brad A. Myers. 1990. Creating User Interfaces Using Programming by Example, Visual Programming, and Constraints. ACM Trans. Program. Lang. Syst. 12, 2 (April 1990), 143--177.
[39]
Dan R. Olsen, Jr. and Kirk Allan. 1990. Creating Interactive Techniques by Symbolically Solving Geometric Constraints. In Proceedings of the 3rd Annual ACM SIGGRAPH Symposium on User Interface Software and Technology (UIST '90). ACM, NY, NY, USA, 102--107.
[40]
Stephen Oney, Brad Myers, and Joel Brandt. 2012. ConstraintJS: programming interactive behaviors for the web by integrating constraints and states. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 229--238.
[41]
Eleanor O'Rourke, Erik Andersen, Sumit Gulwani, and Zoran Popović. 2015. A Framework for Automatically Generating Interactive Instructional Scaffolding. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, NY, NY, USA, 1545--1554.
[42]
Katherine Perkins, Wendy Adams, Michael Dubson, Noah Finkelstein, Sam Reid, Carl Wieman, and Ron LeMaster. 2006. PhET: Interactive Simulations for Teaching and Learning Physics. The Physics Teacher 44, 1 (dec 2006), 18.
[43]
Noah S. Podolefsky, Katherine K. Perkins, and Wendy K. Adams. 2009. Computer simulations to classrooms: tools for change. AIP Conference Proceedings 1179, 1 (2009), 233--236.
[44]
Robert D Rogers and Stephen Monsell. 1995. Costs of a predictible switch between simple cognitive tasks. Journal of experimental psychology: General 124, 2 (1995), 207.
[45]
Leonid Ryzhyk, Adam Walker, John Keys, Alexander Legg, Arun Raghunath, Michael Stumm, and Mona Vij. 2014. User-guided device driver synthesis. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14). 661--676.
[46]
Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vijay Saraswat. 2006. Combinatorial Sketching for Finite Programs. SIGOPS Oper. Syst. Rev. 40, 5 (Oct. 2006), 404--415.
[47]
Ivan E Sutherland. 1964. Sketch pad a man-machine graphical communication system. In Proceedings of the SHARE design automation workshop. ACM, 6--329.
[48]
Brad Vander Zanden. 1992. Languages for Developing User Interfaces. A. K. Peters, Ltd., Natick, MA, USA, Chapter An Active-value&Mdash;Spreadsheet Model for Interactive Languages, 183--209. http://dl.acm.org/citation.cfm?id=131302.131313
[49]
Bradley T. Vander Zanden, Richard Halterman, Brad A. Myers, Rob Miller, Pedro Szekely, Dario A. Giuse, David Kosbie, and Rich McDaniel. 2005. Lessons Learned from Programmers' Experiences with One-way Constraints: Research Articles. Softw. Pract. Exper. 35, 13 (Nov. 2005), 1275--1298.
[50]
Haijun Xia, Bruno Araujo, Tovi Grossman, and Daniel Wigdor. 2016. Object-Oriented Drawing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4610--4621.
[51]
Jun Xing, Rubaiat Habib Kazi, Tovi Grossman, Li-Yi Wei, Jos Stam, and George Fitzmaurice. 2016. Energy-Brushes: Interactive Tools for Illustrating Stylized Elemental Dynamics. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 755--766.
[52]
Jun Xing, Li-Yi Wei, Takaaki Shiratori, and Koji Yatani. 2015. Autocomplete hand-drawn animations. ACM Transactions on Graphics (TOG) 34, 6 (2015), 169.
[53]
Kuat Yessenov, Shubham Tulsiani, Aditya Menon, Robert C. Miller, Sumit Gulwani, Butler Lampson, and Adam Kalai. 2013. A Colorful Approach to Text Processing by Example. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, NY, NY, USA, 495--504.
[54]
Clemens Zeidler, Christof Lutteroth, Wolfgang Sturzlinger, and Gerald Weber. 2013. The Auckland Layout Editor: An Improved GUI Layout Specification Process. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, NY, NY, USA, 343--352.
[55]
Bo Zhu, Michiaki Iwata, Ryo Haraguchi, Takashi Ashihara, Nobuyuki Umetani, Takeo Igarashi, and Kazuo Nakazawa. 2011. Sketch-based Dynamic Illustration of Fluid Systems. ACM Transactions on Graphics 30, 6 (dec 2011), 1.

Cited By

View all
  • (2024)Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook DiagramsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676392(1-12)Online publication date: 13-Oct-2024
  • (2024)DrawTalking: Building Interactive Worlds by Sketching and SpeakingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676334(1-25)Online publication date: 13-Oct-2024
  • (2024)DrawTalking: Towards Building Interactive Worlds by Sketching and SpeakingExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651089(1-8)Online publication date: 11-May-2024
  • Show More Cited By

Index Terms

  1. User-Guided Synthesis of Interactive Diagrams

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. interactive diagrams
    2. program synthesis

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)89
    • Downloads (Last 6 weeks)18
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook DiagramsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676392(1-12)Online publication date: 13-Oct-2024
    • (2024)DrawTalking: Building Interactive Worlds by Sketching and SpeakingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676334(1-25)Online publication date: 13-Oct-2024
    • (2024)DrawTalking: Towards Building Interactive Worlds by Sketching and SpeakingExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651089(1-8)Online publication date: 11-May-2024
    • (2021)Examining interaction techniques in data visualization authoring tools from the perspective of goals and human cognition: a surveyJournal of Visualization10.1007/s12650-020-00705-324:2(397-418)Online publication date: 1-Apr-2021
    • (2020)RealitySketchProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology10.1145/3379337.3415892(166-181)Online publication date: 20-Oct-2020

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media