Alok Mysore
ABSTRACT
It can be hard for tutorial creators to get fine-grained feedback about how learners are actually stepping through their tutorials and which parts lead to the most struggle. To provide such feedback for technical software tutorials, we introduce the idea of tutorial profiling, which is inspired by software code profiling. We prototyped this idea in a system called Porta that automatically tracks how users navigate through a tutorial webpage and what actions they take on their computer such as running shell commands, invoking compilers, and logging into remote servers. Porta surfaces this trace data in the form of profiling visualizations that augment the tutorial with heatmaps of activity hotspots and markers that expand to show event details, error messages, and embedded screencast videos of user actions. We found through a user study of 3 tutorial creators and 12 students who followed their tutorials that Porta enabled both the tutorial creators and the students to provide more specific, targeted, and actionable feedback about how to improve these tutorials. Porta opens up possibilities for performing user testing of technical documentation in a more systematic and scalable way.
Supplemental Material
- 2011. Accuracy and precision test method for remote eye trackers. https://www.tobiipro.com/siteassets/tobii-pro/accuracy-and-precision-tests/tobii-accuracy-and-precisiontest-method-version-2--1--1.pdf. (2011).Google Scholar
- 2018. Flask (A Python Microframework). http://flask.pocoo.org/. (2018).Google Scholar
- 2018. FullStory: Pixel-Perfect Session Replay. https://www.fullstory.com/. (2018).Google Scholar
- 2018. Hotjar: All-in-one Analytics & Feedback. https://www.hotjar.com/. (2018).Google Scholar
- 2018. MDN web docs: MIME types. https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types. (2018).Google Scholar
- 2018. Mouseflow - Session Replay, Heatmaps, Funnels, Forms & User Feedback. https://mouseflow.com/. (2018).Google Scholar
- Jason Alexander, Andy Cockburn, Stephen Fitchett, Carl Gutwin, and Saul Greenberg. 2009. Revisiting Read Wear: Analysis, Design, and Evaluation of a Footprints Scrollbar. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 1665--1674. Google ScholarDigital Library
- Andrea Bunt, Patrick Dubois, Ben Lafreniere, Michael A. Terry, and David T. Cormack. 2014. TaggedComments: Promoting and Integrating User Comments in Online Application Tutorials. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 4037--4046. Google ScholarDigital Library
- Bryan M. Cantrill, Michael W. Shapiro, and Adam H. Leventhal. 2004. Dynamic Instrumentation of Production Systems. In Proceedings of the Annual Conference on USENIX Annual Technical Conference (ATEC '04). USENIX Association, Berkeley, CA, USA. http://dl.acm.org/citation.cfm?id=1247415.1247417 Google ScholarDigital Library
- Parmit K. Chilana, Andrew J. Ko, and Jacob O. Wobbrock. 2012. LemonAid: Selection-based Crowdsourced Contextual Help for Web Applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1549--1558. Google ScholarDigital Library
- Biplab Deka, Zifeng Huang, Chad Franzen, Jeffrey Nichols, Yang Li, and Ranjitha Kumar. 2017. ZIPT: Zero-Integration Performance Testing of Mobile App Designs. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 727--736. Google ScholarDigital Library
- Biplab Deka, Zifeng Huang, and Ranjitha Kumar. 2016. ERICA: Interaction Mining Mobile Apps. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 767--776. Google ScholarDigital Library
- Adam Fourney, Ben Lafreniere, Parmit Chilana, and Michael Terry. 2014. InterTwine: Creating Interapplication Information Scent to Support Coordinated Use of Software. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 429--438. Google ScholarDigital Library
- Paul D. Fox. 2018. Linux port of DTrace. https://github.com/dtrace4linux/linux. (2018).Google Scholar
- Max Goldman and Robert C. Miller. 2009. Codetrail: Connecting Source Code and Web Resources. J. Vis. Lang. Comput. 20, 4 (Aug. 2009), 223--235. Google ScholarDigital Library
- Susan L. Graham, Peter B. Kessler, and Marshall K. Mckusick. 1982. Gprof: A Call Graph Execution Profiler. In Proceedings of the 1982 SIGPLAN Symposium on Compiler Construction (SIGPLAN '82). ACM, New York, NY, USA, 120--126. Google ScholarDigital Library
- Philip J. Guo and Margo Seltzer. 2012. BURRITO: Wrapping Your Lab Notebook in Computational Infrastructure. In Proceedings of the 4th USENIX Workshop on the Theory and Practice of Provenance (TaPP'12). USENIX Association, Berkeley, CA, USA. http://dl.acm.org/citation.cfm?id=2342875.2342882 Google ScholarDigital Library
- Björn Hartmann, Mark Dhillon, and Matthew K. Chan. 2011. HyperSource: Bridging the Gap Between Source and Code-related Web Sites. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2207--2210. Google ScholarDigital Library
- William C. Hill, James D. Hollan, Dave Wroblewski, and Tim McCandless. 1992. Edit Wear and Read Wear. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '92). ACM, New York, NY, USA, 3--9. Google ScholarDigital Library
- Juho Kim, Philip J. Guo, Carrie J. Cai, Shang-Wen (Daniel) Li, Krzysztof Z. Gajos, and Robert C. Miller. 2014. Data-driven Interaction Techniques for Improving Navigation of Educational Videos. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 563--572. Google ScholarDigital Library
- Sean Kross. 2017. The Unix Workbench. Chapter 6: Git and GitHub. https://seankross.com/the-unix-workbench/git-and-github.html. (2017).Google Scholar
- Benjamin Lafreniere, Tovi Grossman, and George Fitzmaurice. 2013. Community Enhanced Tutorials: Improving Tutorials with Multiple Demonstrations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1779--1788. Google ScholarDigital Library
- Tom Lieber, Joel R. Brandt, and Rob C. Miller. 2014. Addressing Misconceptions About Code with Always-on Programming Visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 2481--2490. Google ScholarDigital Library
- Justin Matejka, Tovi Grossman, and George Fitzmaurice. 2013. Patina: Dynamic Heatmaps for Visualizing Application Usage. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 3227--3236. Google ScholarDigital Library
- Will McGrath, Daniel Drew, Jeremy Warner, Majeed Kazemitabaar, Mitchell Karchemsky, David Mellis, and Björn Hartmann. 2017. Bifröst: Visualizing and Checking Behavior of Embedded Systems Across Hardware and Software. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 299--310. Google ScholarDigital Library
- Alok Mysore and Philip J. Guo. 2017. Torta: Generating Mixed-Media GUI and Command-Line App Tutorials Using Operating-System-Wide Activity Tracing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 703--714. Google ScholarDigital Library
- Mitchell J Nathan, Kenneth R Koedinger, and Martha W Alibali. 2001. Expert blind spot: When content knowledge eclipses pedagogical content knowledge. In Proceedings of the third international conference on cognitive science. Beijing: University of Science and Technology of China Press, 644--648.Google Scholar
- Peter L. T. Pirolli. 2007. Information Foraging Theory: Adaptive Interaction with Information (1 ed.). Oxford University Press, Inc., New York, NY, USA. Google ScholarDigital Library
- Mark Russinovich. 2018. Microsoft Sysinternals: Process Monitor v3.50. https://docs.microsoft.com/en-us/sysinternals/downloads/procmon. (2018).Google Scholar
- Victor Savkin. 2016. Angular: Why TypeScript? https://vsavkin.com/writing-angular-2-in-typescript-1fa77c78d8e8. (2016).Google Scholar
- Amitabh Srivastava and Alan Eustace. 1994. ATOM: A System for Building Customized Program Analysis Tools. In Proceedings of the ACM SIGPLAN 1994 Conference on Programming Language Design and Implementation (PLDI '94). ACM, New York, NY, USA, 196--205. Google ScholarDigital Library
Index Terms
- Porta: Profiling Software Tutorials Using Operating-System-Wide Activity Tracing
Recommendations
Docable: evaluating the executability of software tutorials
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringThe typical software tutorial includes step-by-step instructions for installing developer tools, editing files and code, and running commands. When these software tutorials are not executable, either due to missing instructions, ambiguous steps, or ...
Torta: Generating Mixed-Media GUI and Command-Line App Tutorials Using Operating-System-Wide Activity Tracing
UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and TechnologyTutorials are vital for helping people perform complex software-based tasks in domains such as programming, data science, system administration, and computational research. However, it is tedious to create detailed step-by-step tutorials for tasks that ...
MixT: automatic generation of step-by-step mixed media tutorials
UIST '12: Proceedings of the 25th annual ACM symposium on User interface software and technologyUsers of complex software applications often learn concepts and skills through step-by-step tutorials. Today, these tutorials are published in two dominant forms: static tutorials composed of images and text that are easy to scan, but cannot effectively ...
Comments