Hariharan Subramonyam
ABSTRACT
Learning from text is a constructive activity in which sentence-level information is combined by the reader to build coherent mental models. With increasingly complex texts, forming a mental model becomes challenging due to a lack of background knowledge, and limits in working memory and attention. To address this, we are taught knowledge externalization strategies such as active reading and diagramming. Unfortunately, paper-and-pencil approaches may not always be appropriate, and software solutions create friction through difficult input modalities, limited workflow support, and barriers between reading and diagramming. For all but the simplest text, building coherent diagrams can be tedious and difficult. We propose Active Diagramming, an approach extending familiar active reading strategies to the task of diagram construction. Our prototype, texSketch, combines pen-and-ink interactions with natural language processing to reduce the cost of producing diagrams while maintaining the cognitive effort necessary for comprehension. Our user study finds that readers can effectively create diagrams without disrupting reading.
Supplemental Material
Available for Download
zip
paper028vfc.zip (2.6 KB)
Video figure captions
sbv
paper028pvc.sbv (586 B)
Preview video captions
- Mortimer J Adler and Charles Van Doren. 2014. How to read a book: The classic guide to intelligent reading. Simon and Schuster.Google Scholar
- Explosion AI. 2019. spaCy. (2019). https://spacy.io/Google Scholar
- Shaaron Ainsworth. 2006. DeFT: A conceptual framework for considering learning with multiple representations. Learning and instruction 16, 3 (2006), 183--198.Google Scholar
- Lisa Anthony and Jacob O Wobbrock. 2010. A lightweight multistroke recognizer for user interface prototypes. In Proceedings of Graphics Interface 2010. Canadian Information Processing Society, 245--252.Google ScholarDigital Library
- David Bargeron and Tomer Moscovich. 2003. Reflowing digital ink annotations. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 385--393.Google ScholarDigital Library
- Aaron Bauer and Kenneth R Koedinger. 2007. Selection-based note-taking applications. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 981--990.Google ScholarDigital Library
- Mohammed Belatar and François Coldefy. 2010. Sketched menus and iconic gestures, techniques designed in the context of shareable interfaces. In ACM International Conference on Interactive Tabletops and Surfaces. ACM, 143--146.Google ScholarDigital Library
- Jared N Bott and Joseph J LaViola Jr. 2010. A pen-based tool for visualizing vector mathematics. In Proceedings of the Seventh Sketch-Based Interfaces and Modeling Symposium. Eurographics Association, 103--110.Google ScholarDigital Library
- Marius Brade, Christian Brändel, Angelika Salmen, and Rainer Groh. 2012. SketchViz: a sketching interface for domain comprehension tasks illustrated by an industrial network use case. In Proceedings of the 12th International Conference on Knowledge Management and Knowledge Technologies. ACM, 30.Google ScholarDigital Library
- Marius Brade, Florian Schneider, Angelika Salmen, and Rainer Groh. 2013. OntoSketch: Towards digital sketching as a tool for creating and extending ontologies for non-experts. In Proceedings of the 13th International Conference on Knowledge Management and Knowledge Technologies. ACM, 9.Google ScholarDigital Library
- Peter Brandl, Christoph Richter, and Michael Haller. 2010. Nicebook: supporting natural note taking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 599--608.Google ScholarDigital Library
- Ann L Brown and Jeanne D Day. 1983. Macrorules for summarizing texts: The development of expertise. Journal of verbal learning and verbal behavior 22, 1 (1983), 1--14.Google ScholarCross Ref
- Breanne J Byiers, Joe Reichle, and Frank J Symons. 2012. Single-subject experimental design for evidence-based practice. American journal of speech-language pathology (2012).Google Scholar
- Nicky Case. 2019. Loopy: a tool for thinking in systems. (2019). https://ncase.me/loopy/Google Scholar
- Nicholas Chen, Francois Guimbretiere, and Abigail Sellen. 2012. Designing a multi-slate reading environment to support active reading activities. ACM Transactions on Computer-Human Interaction (TOCHI) 19, 3 (2012), 18.Google ScholarDigital Library
- Michelene TH Chi. 2009. Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in cognitive science 1, 1 (2009), 73--105.Google Scholar
- Michelene TH Chi, Nicholas De Leeuw, Mei-Hung Chiu, and Christian LaVancher. 1994. Eliciting self-explanations improves understanding. Cognitive science 18, 3 (1994), 439--477.Google Scholar
- Travis J Cossairt and Joseph J LaViola Jr. 2012. SetPad: a sketch-based tool for exploring discrete math set problems. In Proceedings of the International Symposium on Sketch-Based Interfaces and Modeling. Eurographics Association, 47--56.Google Scholar
- Nathalie Coté, Susan R Goldman, and Elizabeth U Saul. 1998. Students making sense of informational text: Relations between processing and representation. Discourse Processes 25, 1 (1998), 1--53.Google ScholarCross Ref
- Richard Cox. 1999. Representation construction, externalised cognition and individual differences. Learning and instruction 9, 4 (1999), 343--363.Google Scholar
- Vonnie M DiCecco and Mary M Gleason. 2002. Using graphic organizers to attain relational knowledge from expository text. Journal of learning disabilities 35, 4 (2002), 306--320.Google ScholarCross Ref
- Docear. 2019. The Academic Literature Suite. (2019). http://www.docear.org/Google Scholar
- Matthew W Easterday, Jordan S Kanarek, and Maralee Harrell. 2009. Design requirements of argument mapping software for teaching deliberation. Online deliberation: Design, research, and practice (2009), 317--23.Google Scholar
- Hugging Face. 2019. Neural Coreference. (2019). https://huggingface.co/coref/Google Scholar
- Kenneth Forbus, Jeffrey Usher, Andrew Lovett, Kate Lockwood, and Jon Wetzel. 2011. CogSketch: Sketch understanding for cognitive science research and for education. Topics in Cognitive Science 3, 4 (2011), 648--666.Google ScholarCross Ref
- Janice D Gobert and John J Clement. 1999. Effects of student-generated diagrams versus student-generated summaries on conceptual understanding of causal and dynamic knowledge in plate tectonics. Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching 36, 1 (1999), 39--53.Google ScholarCross Ref
- Gene Golovchinsky, Morgan N Price, and Bill N Schilit. 1999. From reading to retrieval: freeform ink annotations as queries. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval. Citeseer, 19--25.Google ScholarDigital Library
- Ken Hinckley, Xiaojun Bi, Michel Pahud, and Bill Buxton. 2012. Informal information gathering techniques for active reading. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1893--1896.Google ScholarDigital Library
- Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Edward Cutrell, Michael Shilman, and Desney Tan. 2007. InkSeine: In Situ search for active note taking. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 251--260.Google ScholarDigital Library
- Matthew Hong, Anne Marie Piper, Nadir Weibel, Simon Olberding, and James Hollan. 2012. Microanalysis of active reading behavior to inform design of interactive desktop workspaces. In Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces. ACM, 215--224.Google ScholarDigital Library
- Walter Kintsch. 1988. The role of knowledge in discourse comprehension: A construction-integration model. Psychological review 95, 2 (1988), 163.Google Scholar
- Walter Kintsch and Teun A Van Dijk. 1978. Toward a model of text comprehension and production. Psychological review 85, 5 (1978), 363.Google Scholar
- Walter Kintsch and CBEMAFRS Walter Kintsch. 1998. Comprehension: A paradigm for cognition. Cambridge university press.Google Scholar
- David Kirsh. 2010. Thinking with external representations. Ai & Society 25, 4 (2010), 441--454.Google ScholarCross Ref
- Bongshin Lee, Timothy Dwyer, and Nathalie Henry Riche. 2017. Authoring visual representations for text-based documents. (April 20 2017). US Patent App. 14/945,869.Google Scholar
- Youngmin Lee and David W Nelson. 2004. A conceptual framework for external representations of knowledge in teaching and learning environments. Educational Technology (2004), 28--36.Google Scholar
- Jose A León and Inmaculada Escudero. 2015. Understanding causality in science discourse for middle and high school students. Summary task as a strategy for improving comprehension. In Improving reading comprehension of middle and high school students. Springer, 75--98.Google Scholar
- Jimmie Leppink, Fred Paas, Cees PM Van der Vleuten, Tamara Van Gog, and Jeroen JG Van Merriënboer. 2013. Development of an instrument for measuring different types of cognitive load. Behavior research methods 45, 4 (2013), 1058--1072.Google Scholar
- Chunyuan Liao, François Guimbretière, Ken Hinckley, and Jim Hollan. 2008. Papiercraft: A gesture-based command system for interactive paper. ACM Transactions on Computer-Human Interaction (TOCHI) 14, 4 (2008), 18.Google ScholarDigital Library
- Xiao Ling and Daniel S. Weld. 2010. Temporal Information Extraction. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI'10). AAAI Press, 1385--1390. http://dl.acm.org/citation.cfm?id=2898607.2898828Google Scholar
- Robert F Lorch. 1989. Text-signaling devices and their effects on reading and memory processes. Educational psychology review 1, 3 (1989), 209--234.Google Scholar
- Zhicong Lu, Mingming Fan, Yun Wang, Jian Zhao, Michelle Annett, and Daniel Wigdor. 2019. InkPlanner: Supporting Prewriting via Intelligent Visual Diagramming. IEEE transactions on visualization and computer graphics 25, 1 (2019), 277--287.Google Scholar
- Arnold M Lund. 2001. Measuring usability with the use questionnaire12. Usability interface 8, 2 (2001), 3--6.Google Scholar
- MarginNote. 2019. MarginNote Software. (2019). https://www.marginnote.comGoogle Scholar
- Catherine C Marshall. 1997. Annotation: from paper books to the digital library. In Proceedings of the second ACM international conference on Digital libraries. ACM, 131--140.Google ScholarDigital Library
- Richard E Mayer. 1984. Aids to text comprehension. Educational psychologist 19, 1 (1984), 30--42.Google Scholar
- Hrim Mehta, Adam Bradley, Mark Hancock, and Christopher Collins. 2017. Metatation: Annotation as implicit interaction to bridge close and distant reading. ACM Transactions on Computer-Human Interaction (TOCHI) 24, 5 (2017), 35.Google ScholarDigital Library
- MermaidJS. 2019. mermaid. (2019). https://mermaidjs.github.io/Google Scholar
- Bonnie JF Meyer and Leonard W Poon. 2001. Effects of structure strategy training and signaling on recall of text. Journal of educational psychology 93, 1 (2001), 141.Google ScholarCross Ref
- Thomas P Moran, Patrick Chiu, and William Van Melle. 1997. Pen-based interaction techniques for organizing material on an electronic whiteboard. In ACM Symposium on User Interface Software and Technology. 45--54.Google ScholarDigital Library
- Meredith Ringel Morris, AJ Bernheim Brush, and Brian R Meyers. 2007. Reading revisited: Evaluating the usability of digital display surfaces for active reading tasks. In Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP'07). IEEE, 79--86.Google ScholarCross Ref
- Judith S Olson and Wendy A Kellogg. 2014. Ways of Knowing in HCI. Vol. 2. Springer.Google Scholar
- Karin Perry, Holly Weimar, and Mary Ann Bell. 2017. Sketchnoting in school: discover the benefits (and fun) of visual note taking. Rowman & littlefield.Google Scholar
- Héctor R Ponce, Richard E Mayer, M Soledad Loyola, and Mario J López. 2019. Study Activities That Foster Generative Learning: Notetaking, Graphic Organizer, and Questioning. Journal of Educational Computing Research (2019), 0735633119865554.Google Scholar
- Morgan N Price, Gene Golovchinsky, and Bill N Schilit. 1998a. Linking by Inking: Trailblazing in a Paper-Like Hypertext.. In HyperText. 30--39.Google Scholar
- Morgan N Price, Bill N Schilit, and Gene Golovchinsky. 1998b. XLibris: The active reading machine. In Conference on Human Factors in Computing Systems: CHI 98 conference summary on Human factors in computing systems, Vol. 18. 22--23.Google ScholarDigital Library
- Noun Project. 2019. Noun Project - Icons for Everything. (2019). https://thenounproject.com/Google Scholar
- Yann Riche, Nathalie Henry Riche, Ken Hinckley, Sheri Panabaker, Sarah Fuelling, and Sarah Williams. 2017. As We May Ink?: Learning from Everyday Analog Pen Use to Improve Digital Ink Experiences.. In CHI. 3241--3253.Google Scholar
- Donald J Richgels, Lea M McGee, Richard G Lomax, and Catherine Sheard. 1987. Awareness of four text structures: Effects on recall of expository text. Reading Research Quarterly (1987), 177--196.Google Scholar
- Daniel H Robinson and Kenneth A Kiewra. 1995. Visual argument: Graphic organizers are superior to outlines in improving learning from text. Journal of educational psychology 87, 3 (1995), 455.Google ScholarCross Ref
- Hugo Romat, Nathalie Riche, Ken Hinckley, Bongshin Lee, Caroline Appert, Emmanuel Pietriga, and Christopher Collins. 2019. ActiveInk:(Th) Inking with Data. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM.Google ScholarDigital Library
- Wesley C Salmon. 1998. Causality and explanation. Oxford University Press.Google Scholar
- Mike Scaife and Yvonne Rogers. 1996. External cognition: how do graphical representations work? International journal of human-computer studies 45, 2 (1996), 185--213.Google Scholar
- Bill N Schilit, Gene Golovchinsky, and Morgan N Price. 1998. Beyond paper: supporting active reading with free form digital ink annotations. In CHI, Vol. 98. Citeseer, 249--256.Google ScholarDigital Library
- Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi, and Oren Etzioni. 2014. Diagram understanding in geometry questions. In Twenty-Eighth AAAI Conference on Artificial Intelligence.Google ScholarDigital Library
- Preet Shihn. 2019. Rough.js Create graphics with a hand-drawn, sketchy, appearance. (2019). https://roughjs.com/Google Scholar
- Frank Shipman, Morgan Price, Catherine C Marshall, and Gene Golovchinsky. 2003. Identifying useful passages in documents based on annotation patterns. In International Conference on Theory and Practice of Digital Libraries. Springer, 101--112.Google ScholarCross Ref
- Antonio Sorgente, Giuseppe Vettigli, and Francesco Mele. 2013. Automatic Extraction of Cause-Effect Relations in Natural Language Text. DART@ AI* IA 2013 (2013), 37--48.Google Scholar
- Craig J Sutherland, Andrew Luxton-Reilly, and Beryl Plimmer. 2016. Freeform digital ink annotations in electronic documents: A systematic mapping study. Computers & Graphics 55 (2016), 1--20.Google ScholarDigital Library
- Ivan E Sutherland. 1964. Sketchpad a man-machine graphical communication system. Simulation 2, 5 (1964), R--3.Google Scholar
- Craig S Tashman and W Keith Edwards. 2011. LiquidText: a flexible, multitouch environment to support active reading. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3285--3294.Google ScholarDigital Library
- P Van Meter and CM Firetto. 2013. Cognitive model of drawing construction. Learning through visual displays (2013), 247--280.Google Scholar
- Peggy Van Meter and Joanna Garner. 2005. The promise and practice of learner-generated drawing: Literature review and synthesis. Educational Psychology Review 17, 4 (2005), 285--325.Google ScholarCross Ref
- Jagoda Walny, Samuel Huron, Charles Perin, Tiffany Wun, Richard Pusch, and Sheelagh Carpendale. 2018. Active reading of visualizations. IEEE transactions on visualization and computer graphics 24, 1 (2018), 770--780.Google ScholarCross Ref
- Jo Wood, Petra Isenberg, Tobias Isenberg, Jason Dykes, Nadia Boukhelifa, and Aidan Slingsby. 2012. Sketchy rendering for information visualization. IEEE Transactions on Visualization and Computer Graphics 18, 12 (2012), 2749--2758.Google ScholarDigital Library
- Hsin-Kai Wu and Priti Shah. 2004. Exploring visuospatial thinking in chemistry learning. Science education 88, 3 (2004), 465--492.Google Scholar
- Sally PW Wu and Martina A Rau. 2019. How Students Learn Content in Science, Technology, Engineering, and Mathematics (STEM) Through Drawing Activities. Educational Psychology Review (2019), 1--34.Google Scholar
- Dongwook Yoon, Nicholas Chen, and François Guimbretière. 2013. TextTearing: opening white space for digital ink annotation. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 107--112.Google ScholarDigital Library
- Sacha Zyto, David Karger, Mark Ackerman, and Sanjoy Mahajan. 2012. Successful classroom deployment of a social document annotation system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1883--1892.Google ScholarDigital Library
Index Terms
- texSketch: Active Diagramming through Pen-and-Ink Annotations
Recommendations
Domain-specific language modelling with UML profiles by decoupling abstract and concrete syntaxes
UML profiling presents some acknowledged deficiencies, among which the lack of expressiveness of the profiled notations, together with the high coupling between abstract and concrete syntaxes outstand. These deficiencies may cause distress among UML-...
The active reading task: e-books and their readers
BooksOnline '08: Proceedings of the 2008 ACM workshop on Research advances in large digital book repositoriesThis paper describes the Active Reading task, a recent addition to the INEX Book Search track1. This task aims at exploring how people interact with e-books in different scenarios of use. Besides, it has been designed to take into account a corpus of ...
Diagrams based on structural object perception
AVI '00: Proceedings of the working conference on Advanced visual interfacesMost diagrams, particularly those used in software engineering, are line drawings consisting of nodes drawn as rectangles or circles, and edges drawn as lines linking them. In the present paper we review some of the literature on human perception to ...
Comments