Yan Chen
Steve Oney
ABSTRACT
User Interface (UI) mockups are commonly used as shared context during interface development collaboration. In practice, UI designers often use screenshots and sketches to create mockups of desired UI behaviors for communication. However, in the later stages of UI development, interfaces can be arbitrarily complex, making it labor-intensive to sketch, and static screenshots are limited in the types of interactive and dynamic behaviors they can express. We introduce CoCapture, a system that allows designers to easily create UI behavior mockups on existing web interfaces by demonstrating and remixing, and to accurately describe their requests to helpers by referencing the resulting mockups using hypertext. We showed that participants could more accurately describe UI behaviors with CoCapture than with existing sketch and communication tools and that the resulting descriptions were clear and easy to follow. Our approach can help teams develop UIs efficiently by bridging communication gaps with more accurate visual context.
Supplemental Material
Available for Download
- 2020. Adobe XD. https://www.adobe.com/products/xd.html Accessed: April, 2020.Google Scholar
- 2020. Figma. https://www.figma.com/ Accessed: March, 2020.Google Scholar
- 2020. Google Inc.https://github.com/GoogleChromeLabs/ProjectVisBug Accessed: March, 2020.Google Scholar
- 2020. MutationObserver. https://developer.mozilla.org/en-US/docs/Web/API/MutationObserver Accessed: March, 2020.Google Scholar
- 2020. Scrimba. https://www.scrimba.com/ Accessed: March, 2020.Google Scholar
- Brian Burg, Richard Bailey, Amy J Ko, and Michael D Ernst. 2013. Interactive record/replay for web application debugging. In Proceedings of the 26th annual ACM symposium on User interface software and technology. 473–484.Google ScholarDigital Library
- Brian Burg, Amy J Ko, and Michael D Ernst. 2015. Explaining visual changes in web interfaces. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. 259–268.Google ScholarDigital Library
- N. Burtnyk and M. Wein. 1971. Computer-Generated Key-Frame Animation. Journal of the SMPTE 80(1971), 149–153.Google ScholarCross Ref
- Kerry Shih-Ping Chang and Brad A Myers. 2012. WebCrystal: understanding and reusing examples in web authoring. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 3205–3214.Google ScholarDigital Library
- Kathy Charmaz. 2006. Constructing grounded theory: A practical guide through qualitative analysis. sage.Google Scholar
- Sarah E Chasins, Maria Mueller, and Rastislav Bodik. 2018. Rousillon: Scraping Distributed Hierarchical Web Data. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 963–975.Google ScholarDigital Library
- Yan Chen, Sang Won Lee, Yin Xie, YiWei Yang, Walter S Lasecki, and Steve Oney. 2017. Codeon: On-demand software development assistance. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 6220–6231.Google ScholarDigital Library
- Yan Chen, Maulishree Pandey, Jean Y Song, Walter S Lasecki, and Steve Oney. 2020. Improving Crowd-Supported GUI Testing with Structural Guidance. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Pei-Yu Chi, Sen-Po Hu, and Yang Li. 2018. Doppio: Tracking ui flows and code changes for app development. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Parmit K Chilana, Nathaniel Hudson, Srinjita Bhaduri, Prashant Shashikumar, and Shaun Kane. 2018. Supporting Remote Real-Time Expert Help: Opportunities and Challenges for Novice 3D Modelers. In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 157–166.Google ScholarCross Ref
- Herbert H Clark and Susan E Brennan. 1991. Grounding in communication.(1991).Google Scholar
- Cecília Kremer Vieira da Cunha and Clarisse Sieckenius de Souza. 2003. Toward a culture of end-user programming understanding communication about extending applications.Google Scholar
- Paul Dourish and Victoria Bellotti. 1992. Awareness and coordination in shared workspaces. In Proceedings of the 1992 ACM conference on Computer-supported cooperative work. 107–114.Google ScholarDigital Library
- Susan R Fussell, Leslie D Setlock, Jie Yang, Jiazhi Ou, Elizabeth Mauer, and Adam DI Kramer. 2004. Gestures over video streams to support remote collaboration on physical tasks. Human-Computer Interaction 19, 3 (2004), 273–309.Google ScholarDigital Library
- Lars Grammel, Melanie Tory, and Margaret-Anne Storey. 2010. How information visualization novices construct visualizations. IEEE transactions on visualization and computer graphics 16, 6(2010), 943–952.Google ScholarDigital Library
- Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2010. Chronicle: capture, exploration, and playback of document workflow histories. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. 143–152.Google ScholarDigital Library
- Carl Gutwin and Saul Greenberg. 2002. A descriptive framework of workspace awareness for real-time groupware. Computer Supported Cooperative Work (CSCW) 11, 3-4 (2002), 411–446.Google ScholarDigital Library
- Carl Gutwin, Mark Roseman, and Saul Greenberg. 1996. A usability study of awareness widgets in a shared workspace groupware system. In Proceedings of the 1996 ACM conference on Computer supported cooperative work. 258–267.Google ScholarDigital Library
- Carl Gutwin, Gwen Stark, and Saul Greenberg. 1995. Support for workspace awareness in educational groupware.. In CSCL, Vol. 95. 147–156.Google Scholar
- Joshua Hibschman and Haoqi Zhang. 2015. Unravel: Rapid web application reverse engineering via interaction recording, source tracing, and library detection. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. 270–279.Google ScholarDigital Library
- Joshua Hibschman and Haoqi Zhang. 2016. Telescope: Fine-tuned discovery of interactive web UI feature implementation. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 233–245.Google ScholarDigital Library
- David Kirk and Danae Stanton Fraser. 2006. Comparing remote gesture technologies for supporting collaborative physical tasks. In Proceedings of the SIGCHI conference on Human Factors in computing systems. 1191–1200.Google ScholarDigital Library
- David S Kirk and Danaë Stanton Fraser. 2005. The effects of remote gesturing on distance instruction. (2005).Google Scholar
- Robert E Kraut, Susan R Fussell, and Jane Siegel. 2003. Visual information as a conversational resource in collaborative physical tasks. Human–computer interaction 18, 1-2 (2003), 13–49.Google Scholar
- Rebecca Krosnick, Sang Won Lee, Walter S Lasecki, and Steve Onev. 2018. Expresso: Building responsive interfaces with keyframes. In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 39–47.Google ScholarCross Ref
- James A Landay and Brad A Myers. 1995. Interactive sketching for the early stages of user interface design. In Proceedings of the SIGCHI conference on Human factors in computing systems. 43–50.Google ScholarDigital Library
- Walter S Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P Bigham, and Michael S Bernstein. 2015. Apparition: Crowdsourced User Interfaces that Come to Life as You Sketch Them. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1925–1934.Google ScholarDigital Library
- Sang Won Lee and Jason Freeman. 2013. Real-time music notation in mixed laptop–acoustic ensembles. Computer Music Journal 37, 4 (2013), 24–36.Google ScholarDigital Library
- Sang Won Lee, Yujin Zhang, Isabelle Wong, Yiwei Yang, Stephanie D. O’Keefe, and Walter S. Lasecki. 2017. SketchExpress: Remixing Animations for More Effective Crowd-Powered Prototyping of Interactive Interfaces. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 817–828. https://doi.org/10.1145/3126594.3126595Google ScholarDigital Library
- Germán Leiva and Michel Beaudouin-Lafon. 2018. Montage: A Video Prototyping System to Reduce Re-Shooting and Increase Re-Usability. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 675–682.Google ScholarDigital Library
- Germán Leiva, Cuong Nguyen, Rubaiat Habib Kazi, and Paul Asente. 2020. Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.Google Scholar
- Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. 2017. SUGILITE: creating multimodal smartphone automation by demonstration. In Proceedings of the 2017 CHI conference on human factors in computing systems. 6038–6049.Google Scholar
- Yang Li, Xiang Cao, Katherine Everitt, Morgan Dixon, and James A Landay. 2010. FrameWire: a tool for automatically extracting interaction logic from paper prototyping tests. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 503–512.Google ScholarDigital Library
- James Lin, Mark W Newman, Jason I Hong, and James A Landay. 2000. DENIM: finding a tighter fit between tools and practice for Web site design. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 510–517.Google ScholarDigital Library
- Michael McCurdy, Christopher Connors, Guy Pyrzak, Bob Kanefsky, and Alonso Vera. 2006. Breaking the fidelity barrier: an examination of our current characterization of prototypes and an example of a mixed-fidelity success. In Proceedings of the SIGCHI conference on Human Factors in computing systems. 1233–1242.Google ScholarDigital Library
- David W McDonald, Chunhua Weng, and John H Gennari. 2004. The multiple views of inter-organizational authoring. In Proceedings of the 2004 ACM conference on Computer supported cooperative work. 564–573.Google ScholarDigital Library
- Ronald Metoyer, Bongshin Lee, Nathalie Henry Riche, and Mary Czerwinski. 2012. Understanding the verbal language and structure of end-user descriptions of data visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1659–1662.Google ScholarDigital Library
- Brad Myers, Sun Young Park, Yoko Nakano, Greg Mueller, and Amy Ko. 2008. How designers design and program interactive behaviors. In 2008 IEEE Symposium on Visual Languages and Human-Centric Computing. IEEE, 177–184.Google ScholarDigital Library
- Jasper O’Leary, Holger Winnemöller, Wilmot Li, Mira Dontcheva, and Morgan Dixon. 2018. Charrette: Supporting In-Person Discussions around Iterations in User Interface Design. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–11.Google ScholarDigital Library
- Steve Oney, Christopher Brooks, and Paul Resnick. 2018. Creating guided code explanations with chat. codes. Proceedings of the ACM on Human-Computer Interaction 2, CSCW(2018), 1–20.Google ScholarDigital Library
- Steve Oney, Rebecca Krosnick, Joel Brandt, and Brad Myers. 2019. Implementing Multi-Touch Gestures with Touch Groups and Cross Events. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.Google ScholarDigital Library
- Stephen Oney and Brad Myers. 2009. FireCrystal: Understanding interactive behaviors in dynamic web pages. In 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 105–108.Google ScholarDigital Library
- Stephen Oney, Brad Myers, and John Zimmerman. 2009. Visions for Euclase: Ideas for Supporting Creativity through Better Prototyping of Behaviors. In ACM CHI 2009 Workshop on Computational Creativity Support.Google Scholar
- John F Pane, Brad A Myers, 2001. Studying the language and structure in non-programmers’ solutions to programming problems. International Journal of Human-Computer Studies 54, 2 (2001), 237–264.Google ScholarDigital Library
- Jungkook Park, Yeong Hoon Park, and Alice Oh. 2018. Non-Linear Editing of Text-Based Screencasts. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 403–410.Google ScholarDigital Library
- Sun Young Park, Brad Myers, and Amy J Ko. 2008. Designers’ natural descriptions of interactive behaviors. In 2008 IEEE Symposium on Visual Languages and Human-Centric Computing. IEEE, 185–188.Google ScholarDigital Library
- Amy Pavel, Dan B Goldman, Björn Hartmann, and Maneesh Agrawala. 2016. VidCrit: video-based asynchronous video review. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 517–528.Google ScholarDigital Library
- Zhengrui Qin, Yutao Tang, Ed Novak, and Qun Li. 2016. Mobiplay: A remote execution based record-and-replay tool for mobile applications. In Proceedings of the 38th International Conference on Software Engineering. 571–582.Google ScholarDigital Library
- William T. Reeves. 1981. Inbetweening for Computer Animation Utilizing Moving Point Constraints. In Proceedings of the 8th Annual Conference on Computer Graphics and Interactive Techniques(Dallas, Texas, USA) (SIGGRAPH ’81). Association for Computing Machinery, New York, NY, USA, 263–269.Google ScholarDigital Library
- Hugo Romat, Emmanuel Pietriga, Nathalie Henry-Riche, Ken Hinckley, and Caroline Appert. 2019. SpaceInk: Making Space for In-Context Annotations. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 871–882.Google ScholarDigital Library
- Amanda Swearngin, Mira Dontcheva, Wilmot Li, Joel Brandt, Morgan Dixon, and Amy J Ko. 2018. Rewire: Interface design assistance from examples. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.Google ScholarDigital Library
- Kesler Tanner, Naomi Johnson, and James A Landay. 2019. Poirot: A Web Inspector for Designers. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.Google ScholarDigital Library
- Edward R Tufte. 1983. The visual display of quantitative information. Vol. 2.Google Scholar
- April Yi Wang, Zihan Wu, Christopher Brooks, and Steve Oney. 2020. Callisto: Capturing the” Why” by Connecting Conversations with Computational Narratives. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Patricia G Wojahn, Christine M Neuwirth, and Barbara Bullock. 1998. Effects of interfaces for annotation on communication in a collaborative task. In Proceedings of the SIGCHI conference on Human factors in computing systems. 456–463.Google ScholarDigital Library
- Dongwook Yoon, Nicholas Chen, François Guimbretière, and Abigail Sellen. 2014. RichReview: blending ink, speech, and gesture to support collaborative document review. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 481–490.Google ScholarDigital Library
- Dongwook Yoon, Nicholas Chen, Bernie Randles, Amy Cheatle, Corinna E Löckenhoff, Steven J Jackson, Abigail Sellen, and François Guimbretière. 2016. RichReview++ Deployment of a Collaborative Multi-modal Annotation System for Instructor Feedback and Peer Discussion. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. 195–205.Google ScholarDigital Library
- Joyce Zhu, Jeremy Warner, Mitchell Gordon, Jeffery White, Renan Zanelatto, and Philip J Guo. 2015. Toward a domain-specific visual discussion forum for learning computer programming: An empirical study of a popular MOOC forum. In 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 101–109.Google ScholarCross Ref
Index Terms
- CoCapture: Effectively Communicating UI Behaviors on Existing Websites by Demonstrating and Remixing
Index terms have been assigned to the content through auto-classification.
Recommendations
UI Fin: a process-oriented interface design tool
IUI '09: Proceedings of the 14th international conference on Intelligent user interfacesEven though over the years a multitude of user interface design tools have been created, designers in practice find themselves limited to a small set of realistic options. These options include interface builders that are attached to development ...
Comments