Abstract
The Spatial Intelligence and Learning Center (SILC) pioneered the idea of spatial learning: improving learning about spatial concepts and using spatial concepts to facilitate learning about other domains. Spatial concepts are of significant importance in Science, Technology, Engineering and Mathematics (STEM) education including physics, chemistry, geoscience, and biology, as well as most branches of engineering. Spatial concepts are also important in the military. For example, sand tables are used to understand the terrain, since calculating Line of Sight (LOS) for cover and concealment on the battlefield can mean the difference between life or death. In addition, land navigation using topographic maps is a critical skill that all members of the military are required to obtain.
For this work in progress, the Army Research Laboratory (ARL) and Northwestern University, a member of SILC, are exploring sketching technologies to support spatial learning, as part of on-going research in adaptive training technologies. ARL’s Generalized Intelligent Framework for Tutoring (GIFT) provides a software platform for developing intelligent tutoring systems. Adaptive learning, where the state of the learner is used to help select their path through activities, has been shown to improve learning. As part of our on-going research, we are in the process of integrating sketch worksheets into GIFT as a new type of instructional media. Sketch worksheets were designed to be general-purpose and use artificial intelligence to provide immediate feedback to students performing sketching assignments. To facilitate dissemination, an authoring environment was created for domain experts and instructors, to enable them to create new worksheets. This poster describes a work in progress to bring together the two lines of research to support the use of sketching in an adaptive learning environment.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
Spatial concepts are of central importance in STEM education (Montello et al. 2014) including physics, chemistry, geoscience, and biology, as well as most branches of engineering. STEM is also foundational for many areas of Army training (e.g. how equipment works, how to maintain and troubleshoot it, how to reason about terrain, etc.). The SILC pioneered the idea of spatial learning: improving learning about spatial concepts and using spatial concepts to facilitate learning about other domains. One of the tools for spatial learning that SILC has explored is sketching. Sketching is commonly used to convey ideas between people, and for people to work out ideas on their own. It has been argued that sketching is a powerful tool for STEM education (Ainsworth et al. 2011; Jee et al. 2014). The importance of sketching can be seen in a survey of geoscience instructors, 80% of whom believe that sketching is important for geoscience education (Garnier et al. in press).
While sketching is very powerful, it is underutilized in education because sketches are difficult and time-consuming to grade. That same survey indicated that, while instructors think sketching is important, fewer than 50% assigned graded sketching exercises more than three times a semester because of the difficulty of grading them. If intelligent tutoring systems could interact with people via sketching, it could have a revolutionary impact on education and training. This Work-in-Progress paper describes on-going research in using sketching technology in adaptive learning environments for the U.S. Army.
1.1 Related Work
There are two basic approaches to sketch understanding. The first involves recognition, i.e. a student draws something (e.g. a component of an electronic circuit) and the system automatically recognizes what it is intended to be, either labelling it or redrawing it more neatly. Several educational software systems have been fielded using this approach (e.g. De Silva et al. 2007; Valentine et al. 2012). However, this approach has fundamental limitations:
-
It only works when a domain can be described in terms of highly conventional visual symbols, like analog or digital electronics. It fails when the mapping between visual information and conceptual information is many-to-many. For example, circles can indicate layers of the earth, planetary orbits, and latitude lines, even just within geoscience. When working with multiple STEM disciplines, the many to many problem becomes even worse, e.g. they can also indicate wall in a cellular or mechanical structure, among many other things.
-
Most sketches involve elements that are spatialized, i.e. a road is sketched not as an abstract symbol, but as a spatial element whose relationships with other elements is important. Thus understanding sketches requires understanding the spatial relationships that people perceive (or learn to perceive during training) in sketches (Jee et al. 2014).
-
Even for domains with conventional visual symbols, today’s recognition technologies tend to be low-accuracy, providing fluid interaction when they work, and frustration when they don’t. The domain-specificity of such systems means that, to date, a new system is needed for every category of sub-problem within a domain. Heuristics that work when dealing with professional users (i.e. interpreting a pause in drawing as the end of a visual element) fail when dealing with spatialized elements, because thinking through layout takes time, and with learners, because they stop to think about the domain.
1.2 Open-Domain Sketch Understanding
Over the last 15 years, Northwestern University has pioneered a different approach. It is based on an insight gleaned from observing human-to-human sketching. When people sketch together, they talk. They identify what something is intended to mean verbally, complemented by written annotations when needed. They use cultural conventions, but articulate them when needed (e.g. the length and shape of a road is intended to be to scale with the rest of the sketch, but the width is not, unless one is sketching some terrain in close detail). Our open-domain sketch understanding approach enables users to specify how they want their ink segmented into visual elements (called glyphs) and what those glyphs are intended to mean. Instead of using speech recognition plus natural language understanding – which is both beyond the state of the art for this task, and impractical in many classroom settings – we provide other interface modalities for communicating this information. The strength of the open-domain sketch understanding approach is that it focuses on using human-like visual representations and processing to understand what someone has drawn.
2 Cogsketch
In developing CogSketch (Forbus et al. 2011) the goal was to both accurately model aspects of human visual and spatial reasoning and by doing so, provide a new platform for sketch-based educational software. We achieved this goal, to a large degree. In modeling, CogSketch has been used to model geometric analogies (Lovett et al. 2009), a cross-cultural visual oddity task (Lovett and Forbus 2011), and Ravens’ Progressive Matrices (Lovett et al. 2010; Lovett and Forbus in press). In all three of these simulations, which use the same representations and processing, the system performs at human-level on the tasks and predicts aspects of human performance (e.g. reaction times and within-task problem difficulty).
2.1 Related Work
CogSketch has also been used to model learning of spatial prepositions (Lockwood et al. 2008a), solving conceptual physics ranking problems (Chang et al. 2014), reasoning about depiction (Lockwood et al. 2008b) and learning by reading from texts and sketches (Lockwood and Forbus 2009; Chang and Forbus 2015). These capabilities enabled us to develop two new forms of sketch-based educational software:
-
It only works when a domain can be described in terms of highly conventional visual symbols, like analog or digital electronics. It fails when the mapping between visual information and conceptual information is many-to-many. For example, circles can indicate layers of the earth, planetary orbits, and latitude lines, even just within geoscience. When working with multiple STEM disciplines, the many to many problem becomes even worse, e.g. they can also indicate wall in a cellular or mechanical structure, among many other things.
-
Most sketches involve elements that are spatialized, i.e. a road is sketched not as an abstract symbol, but as a spatial element whose relationships with other elements is important. Thus understanding sketches requires understanding the spatial relationships that people perceive (or learn to perceive during training) in sketches (Jee et al. 2014).
-
Even for domains with conventional visual symbols, today’s recognition technologies tend to be low-accuracy, providing fluid interaction when they work, and frustration when they don’t. The domain-specificity of such systems means that, to date, a new system is needed for every category of sub-problem within a domain. Heuristics that work when dealing with professional users (i.e. interpreting a pause in drawing as the end of a visual element) fail when dealing with spatialized elements, because thinking through layout takes time, and with learners, because they stop to think about the domain.
The strength of the open-domain sketch understanding approach is that it focuses on using human-like visual representations and processing to understand what someone has drawn. In developing CogSketch (Forbus et al. 2011) the goal was to both accurately model aspects of human visual and spatial reasoning and by doing so, provide a new platform for sketch-based educational software. We achieved this goal, to a large degree. In modeling, CogSketch has been used to model geometric analogies (Lovett et al. 2009), a cross-cultural visual oddity task (Lovett and Forbus 2011), and Ravens’ Progressive Matrices (Lovett et al. 2010; Lovett and Forbus in press). In all three of these simulations, which use the same representations and processing, the system performs at human-level on the tasks and predicts aspects of human performance (e.g. reaction times and within-task problem difficulty). CogSketch has also been used to model learning of spatial prepositions (Lockwood et al. 2008a), solving conceptual physics ranking problems (Chang et al. 2014), reasoning about depiction (Lockwood et al. 2008b) and learning by reading from texts and sketches (Lockwood and Forbus 2009; Chang and Forbus 2015). These capabilities enabled us to develop two new forms of sketch-based educational software:
1. Design Coach (Wetzel and Forbus 2010) helps students learn to explain their designs via sketching. Using sketching plus menus that help them craft a language-like explanation, they explain how their design is intended to work. Using qualitative mechanics (Wetzel and Forbus 2009), Design Coach looks to see if their design could actually work as intended. The use of qualitative mechanics is crucial because it provides more humanlike information, and numerical simulation is inappropriate for early stages of design, where most parameters are not yet known and shapes are roughly drawn. Design Coach was shown to significantly decrease student anxiety about communicating via sketching (Wetzel and Forbus 2015).
2. Sketch Worksheets (Yin et al. 2010; Forbus et al. in press) help students learn about concepts by creating sketches in response to a specific problem. For example, a geoscience student might be asked to identify the radiation flows in depicting the greenhouse effect, and a biology student might be asked to annotate a diagram of a human heart showing where the parts are and how blood flows through it. Sketch Worksheets are domain-independent: The same technology has been used in biology, geoscience, physics, and even knowledge representation. They work by the instructor using CogSketch to specify a problem and their desired solution(s), annotating facts that are important with feedback to give if they are not true, and grading rubrics. When students download a Sketch Worksheet, they draw their own solution, asking for feedback whenever they like. Feedback is generated by comparing the instructor’s sketch with the student sketch, using our model of analogy, presenting them with the instructor’s feedback when there are problems.
Sketch Worksheets were designed to be broadly applicable, and have been used by over 500 students in classroom and laboratory experiments with students ranging from fifth grade (Miller et al. 2014) to college (Yin et al. 2010; Garnier et al. in press). Moreover, Northwestern’s introductory geoscience course now routinely uses them for instruction, based on Sketch Worksheets authored by a geoscience graduate student at University of Wisconsin-Madison. Thus there is already evidence that this model can be effective and can scale. However, there is still work to be done. Sketch Worksheets are beneficial on their own, but could be even more powerful if integrated with other instructional media, such as provided in GIFT.
3 GIFT
To support adaptive training in the military, the Army Research Laboratory has developed the Generalized Intelligent Framework for Tutoring (GIFT). The framework allows users with no experience or expertise in computer programming to author intelligent tutors. The GIFT authoring tool provides a basic flowchart that walks the user through the process. As the author generates the ITS, various types of media can be used, e.g. PowerPoint, pictures, etc. The authoring tool also supports the Learning Tools Interoperability standard. As a result, GIFT can call other tutors and interface with various Learning Management Systems, e.g. edX. The next step in our research is to support learning content that includes sketching.
3.1 Integrating Sketch Worksheets into GIFT
Our first goal is to integrate CogSketch into GIFT, so that Sketch Worksheets can become a new type of instructional medium for GIFT. That is, an author for a new GIFT tutor can include Sketch Worksheets as one of the activities that students can do in that tutor. For example, a Simple Machines tutor could give a student a problem involving levers, implemented as a Sketch Worksheet. Based on their performance, it could either provide them with another Sketch Worksheet on the same topic, a different form of activity, or move on to the next topic. This adaptive sequencing goes beyond what CogSketch does.
To accomplish this, we will need to integrate CogSketch into GIFT. We plan to build a cloud-based version of CogSketch, based on Docker so that it is compatible with multiple cloud vendors. We have already implemented a prototype user interface for capturing digital ink and providing feedback in Sketch Worksheets being run remotely. We will integrate this user interface into the GIFT framework, as well as do the engineering necessary to make it compatible with the GIFT cloud. (We understand that the GIFT cloud is currently hosted on Amazon’s service, which is supported by Docker.) We will also need to implement transducers that translate the kinds of tutoring data that Sketch Worksheets provide into GIFT domain knowledge files and metadata, so that student performance information gathered by CogSketch is available for other tutoring components.
To test this integration, we will develop a pool of Sketch Worksheets on a STEM domain that is important in K-12 education and for Army training, i.e. Simple Machines. Basic principles of mechanics typically start showing up in early education, communicating qualitative principles of forces, connectivity, and motion. They reappear in training technicians and in engineering education. We will develop a set of at least a dozen Sketch Worksheets covering principles of levers, pulleys, and gears, to provide a pool of Sketch Worksheets that can be used by a Simple Machines Adaptive Tutor. This tutor will be authored using standard GIFT tools, to ensure that the integration works smoothly and to explore how the data gathered by Sketch Worksheets can be used by such tutors.
We will gather evaluation data about this tutor by two means. First, we will make it available through the GIFT community distribution mechanisms. Second, we will set up an online Sketch Academy at Northwestern, where anyone can log in and try them out, and encourage undergraduates to use this as a learning resource. CogSketch already incorporates a “phone home” mechanism that does anonymization locally, so that no student identification data is ever transmitted to us. We will add a similar mechanism to the adaptive tutor, so that we can monitor and analyze student performance.
We note that once this integration is accomplished, anyone can author new Sketch Worksheets by downloading the Windows version of CogSketch and using its authoring environment. This should enable others to experiment with Sketch Worksheets for their own domains of interest.
3.2 Integrating Evaluate and Extend
We will continue to evaluate the data collected from our tutoring systems, improving and extending them as necessary to support better student learning. We also plan to explore some more advanced tutoring capabilities that the Companion cognitive architecture should support. For example, a hypothesis we plan to explore is that qualitative representations can help learners understand equations more deeply. Students often treat equations as disconnected from their daily life, as something to memorize for exams. Experts have additional tacit knowledge, things that equations tell them, which novices tend to not get. Qualitative representations make such tacit knowledge explicit. For example, F = MA tells an expert that one can apply more force in a situation by using more mass, or make something go faster by pushing it harder. Qualitative representations provide a formal way to encode such tacit knowledge, making it available for tutoring systems to use in their domain models. Drawing graphs based on equations may be one way to help bridge from their spatial understanding of the world to their causal knowledge. Another example is giving students the opportunity to critique designs that have been proposed historically for perpetual motion machines. Determining why a design won’t work could provide an engaging way to get students to think about friction and conservation laws. We plan to build a prototype perpetual motion machine tutor to explore this.
References
Ainsworth, S., Prain, V., Tytler, R.: Drawing to learn in science. Science 333(6046), 1096–1097 (2011)
Chang, M.D., Forbus, K.D.: Towards interpretation strategies for multimodal instructional analogies. In: Proceedings of the 28th International Workshop on Qualitative Reasoning (QR2015), Minneapolis, MN (2015)
Chang, M.D., Wetzel, J.W., Forbus, K.D.: Spatial reasoning in comparative analyses of physics diagrams. In: Freksa, C., Nebel, B., Hegarty, M., Barkowsky, T. (eds.) Spatial Cognition 2014. LNCS (LNAI), vol. 8684, pp. 268–282. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11215-2_19
De Silva, R., Bischel, T.D., Lee, W., Peterson, E.J., Calfee, R.C., Stahovich, T.: Kirchhoff’s pen: a pen-based circuit analysis tutor. In: Proceedings of the 4th Eurographics Workshop on Sketch-Based Interfaces and Modeling (2007)
Forbus, K., Usher, J., Lovett, A., Lockwood, K., Wetzel, J.: CogSketch: Sketch understanding for cognitive science research and for education. Top. Cogn. Sci. 3(4), 648–666 (2011). https://doi.org/10.1111/cogs.12377. Scale Cognitive Modeling. Cognitive Science, 1–50
Forbus, K., Chang, M., McLure, M., Usher, M.: The cognitive science of sketch worksheets. Top. Cogn. Sci. (in press)
Garnier, B., Chang, M., Ormand, C., Matlen, B., Tikoff, B., Shipley, T.: Testing the efficacy of CogSketch geoscience worksheets as a spatial learning and sketching tool in introductory Geoscience courses. Top. Cogn. Sci. (in press)
Jee, B., Gentner, D., Uttal, D., Sageman, B., Forbus, K., Manduca, C., Ormand, C., Shipley, T., Tikoff, B.: Drawing on experience: how domain knowledge is reflected in sketches of scientific structures and processes. Res. Sci. Educ. 44(6), 859–883 (2014)
Lockwood, K., Forbus, K.: Multimodal knowledge capture from text and diagrams. In: Proceedings of KCAP-2009 (2009)
Lockwood, K., Lovett, A., Forbus, K.: Automatic classification of containment and support spatial relations in English and Dutch. In: Freksa, C., Newcombe, N.S., Gärdenfors, P., Wölfl, S. (eds.) Spatial Cognition VI. Learning, Reasoning, and Talking about Space. Spatial Cognition 2008. LNCS, vol. 5248, pp. 283–294. Springer, Heidelberg (2008a). https://doi.org/10.1007/978-3-540-87601-4_21
Lockwood, K., Lovett, A., Forbus, K., Dehghani, M., Usher, J.: A theory of depiction for sketches of physical systems. In: Proceedings of QR 2008 (2008b)
Lovett, A., Forbus, K.: Cultural commonalities and differences in spatial problem solving: a computational analysis. Cognition 121(2), 281–287 (2011)
Lovett, A., Forbus, K.: Modeling visual problem-solving as analogical reasoning. Psychol. Rev. (in press)
Lovett, A., Tomai, E., Forbus, K., Usher, J.: Solving geometric analogy problems through two-stage analogical mapping. Cogn. Sci. 33(7), 1192–1231 (2009)
Lovett, A., Forbus, K., Usher, J.: A structure-mapping model of raven’s progressive matrices. In: Proceedings of CogSci-10 (2010)
Miller, B., Cromley, J., Newcombe, N.: Using CogSketch to support student science learning through sketching with automatic Feedback. In: Proceedings of AERA (2014)
Montello, D., Grossner, K., Janelle, D.: Space in Mind: Concepts for Spatial Learning and Education. MIT Press, Cambridge (2014)
Valentine, S., Vides, F., Lucchese, G., Turner, D., Kim, H., Li, W., Linsey, J., Hammond, T.: Mechanix: a sketch-based tutoring system for statics courses. In: Proceedings of the Twenty-Fourth Innovative Applications of Artificial Intelligence Conference (IAAI) (2012)
Wetzel, J., Forbus, K.: Automated critique of sketched designs in engineering. In: Proceedings of the 23rd International Workshop on Qualitative Reasoning, Ljubljana, Slovenia (2009)
Wetzel, J., Forbus, K.: Design buddy: providing Feedback for sketched multi-modal causal explanations. In: Proceedings of the 24th International Workshop on Qualitative Reasoning, Portland, Oregon (2010)
Wetzel, J., Forbus, K.: Increasing student confidence in engineering sketching via a software coach. In: Hammond, T., Valentine, S., Adler, A., Payton, M. (eds.) The Impact of Pen and Touch Technology on Education. HIS, pp. 107–118. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-15594-4_11
Yin, P., Forbus, K., Usher, J., Sageman, B., Jee, B.: Sketch worksheets: a sketchbased educational software system. In: Proceedings of the 22nd Annual Conference on Innovative Applications of Artificial Intelligence (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 This is a U.S. government work and its text is not subject to copyright protection in the United States; however, its text may be subject to foreign copyright protection
About this paper
Cite this paper
Long, R., Forbus, K., Hinrichs, T., Hill, S. (2018). Sketching as a Modality in Intelligent Tutoring Systems. In: Stephanidis, C. (eds) HCI International 2018 – Posters' Extended Abstracts. HCI 2018. Communications in Computer and Information Science, vol 852. Springer, Cham. https://doi.org/10.1007/978-3-319-92285-0_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-92285-0_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-92284-3
Online ISBN: 978-3-319-92285-0
eBook Packages: Computer ScienceComputer Science (R0)