Keywords

1 Introduction

Spatial concepts are of central importance in STEM education (Montello et al. 2014) including physics, chemistry, geoscience, and biology, as well as most branches of engineering. STEM is also foundational for many areas of Army training (e.g. how equipment works, how to maintain and troubleshoot it, how to reason about terrain, etc.). The SILC pioneered the idea of spatial learning: improving learning about spatial concepts and using spatial concepts to facilitate learning about other domains. One of the tools for spatial learning that SILC has explored is sketching. Sketching is commonly used to convey ideas between people, and for people to work out ideas on their own. It has been argued that sketching is a powerful tool for STEM education (Ainsworth et al. 2011; Jee et al. 2014). The importance of sketching can be seen in a survey of geoscience instructors, 80% of whom believe that sketching is important for geoscience education (Garnier et al. in press).

While sketching is very powerful, it is underutilized in education because sketches are difficult and time-consuming to grade. That same survey indicated that, while instructors think sketching is important, fewer than 50% assigned graded sketching exercises more than three times a semester because of the difficulty of grading them. If intelligent tutoring systems could interact with people via sketching, it could have a revolutionary impact on education and training. This Work-in-Progress paper describes on-going research in using sketching technology in adaptive learning environments for the U.S. Army.

1.1 Related Work

There are two basic approaches to sketch understanding. The first involves recognition, i.e. a student draws something (e.g. a component of an electronic circuit) and the system automatically recognizes what it is intended to be, either labelling it or redrawing it more neatly. Several educational software systems have been fielded using this approach (e.g. De Silva et al. 2007; Valentine et al. 2012). However, this approach has fundamental limitations:

  • It only works when a domain can be described in terms of highly conventional visual symbols, like analog or digital electronics. It fails when the mapping between visual information and conceptual information is many-to-many. For example, circles can indicate layers of the earth, planetary orbits, and latitude lines, even just within geoscience. When working with multiple STEM disciplines, the many to many problem becomes even worse, e.g. they can also indicate wall in a cellular or mechanical structure, among many other things.

  • Most sketches involve elements that are spatialized, i.e. a road is sketched not as an abstract symbol, but as a spatial element whose relationships with other elements is important. Thus understanding sketches requires understanding the spatial relationships that people perceive (or learn to perceive during training) in sketches (Jee et al. 2014).

  • Even for domains with conventional visual symbols, today’s recognition technologies tend to be low-accuracy, providing fluid interaction when they work, and frustration when they don’t. The domain-specificity of such systems means that, to date, a new system is needed for every category of sub-problem within a domain. Heuristics that work when dealing with professional users (i.e. interpreting a pause in drawing as the end of a visual element) fail when dealing with spatialized elements, because thinking through layout takes time, and with learners, because they stop to think about the domain.

1.2 Open-Domain Sketch Understanding

Over the last 15 years, Northwestern University has pioneered a different approach. It is based on an insight gleaned from observing human-to-human sketching. When people sketch together, they talk. They identify what something is intended to mean verbally, complemented by written annotations when needed. They use cultural conventions, but articulate them when needed (e.g. the length and shape of a road is intended to be to scale with the rest of the sketch, but the width is not, unless one is sketching some terrain in close detail). Our open-domain sketch understanding approach enables users to specify how they want their ink segmented into visual elements (called glyphs) and what those glyphs are intended to mean. Instead of using speech recognition plus natural language understanding – which is both beyond the state of the art for this task, and impractical in many classroom settings – we provide other interface modalities for communicating this information. The strength of the open-domain sketch understanding approach is that it focuses on using human-like visual representations and processing to understand what someone has drawn.

2 Cogsketch

In developing CogSketch (Forbus et al. 2011) the goal was to both accurately model aspects of human visual and spatial reasoning and by doing so, provide a new platform for sketch-based educational software. We achieved this goal, to a large degree. In modeling, CogSketch has been used to model geometric analogies (Lovett et al. 2009), a cross-cultural visual oddity task (Lovett and Forbus 2011), and Ravens’ Progressive Matrices (Lovett et al. 2010; Lovett and Forbus in press). In all three of these simulations, which use the same representations and processing, the system performs at human-level on the tasks and predicts aspects of human performance (e.g. reaction times and within-task problem difficulty).

2.1 Related Work

CogSketch has also been used to model learning of spatial prepositions (Lockwood et al. 2008a), solving conceptual physics ranking problems (Chang et al. 2014), reasoning about depiction (Lockwood et al. 2008b) and learning by reading from texts and sketches (Lockwood and Forbus 2009; Chang and Forbus 2015). These capabilities enabled us to develop two new forms of sketch-based educational software:

  • It only works when a domain can be described in terms of highly conventional visual symbols, like analog or digital electronics. It fails when the mapping between visual information and conceptual information is many-to-many. For example, circles can indicate layers of the earth, planetary orbits, and latitude lines, even just within geoscience. When working with multiple STEM disciplines, the many to many problem becomes even worse, e.g. they can also indicate wall in a cellular or mechanical structure, among many other things.

  • Most sketches involve elements that are spatialized, i.e. a road is sketched not as an abstract symbol, but as a spatial element whose relationships with other elements is important. Thus understanding sketches requires understanding the spatial relationships that people perceive (or learn to perceive during training) in sketches (Jee et al. 2014).

  • Even for domains with conventional visual symbols, today’s recognition technologies tend to be low-accuracy, providing fluid interaction when they work, and frustration when they don’t. The domain-specificity of such systems means that, to date, a new system is needed for every category of sub-problem within a domain. Heuristics that work when dealing with professional users (i.e. interpreting a pause in drawing as the end of a visual element) fail when dealing with spatialized elements, because thinking through layout takes time, and with learners, because they stop to think about the domain.

The strength of the open-domain sketch understanding approach is that it focuses on using human-like visual representations and processing to understand what someone has drawn. In developing CogSketch (Forbus et al. 2011) the goal was to both accurately model aspects of human visual and spatial reasoning and by doing so, provide a new platform for sketch-based educational software. We achieved this goal, to a large degree. In modeling, CogSketch has been used to model geometric analogies (Lovett et al. 2009), a cross-cultural visual oddity task (Lovett and Forbus 2011), and Ravens’ Progressive Matrices (Lovett et al. 2010; Lovett and Forbus in press). In all three of these simulations, which use the same representations and processing, the system performs at human-level on the tasks and predicts aspects of human performance (e.g. reaction times and within-task problem difficulty). CogSketch has also been used to model learning of spatial prepositions (Lockwood et al. 2008a), solving conceptual physics ranking problems (Chang et al. 2014), reasoning about depiction (Lockwood et al. 2008b) and learning by reading from texts and sketches (Lockwood and Forbus 2009; Chang and Forbus 2015). These capabilities enabled us to develop two new forms of sketch-based educational software:

1. Design Coach (Wetzel and Forbus 2010) helps students learn to explain their designs via sketching. Using sketching plus menus that help them craft a language-like explanation, they explain how their design is intended to work. Using qualitative mechanics (Wetzel and Forbus 2009), Design Coach looks to see if their design could actually work as intended. The use of qualitative mechanics is crucial because it provides more humanlike information, and numerical simulation is inappropriate for early stages of design, where most parameters are not yet known and shapes are roughly drawn. Design Coach was shown to significantly decrease student anxiety about communicating via sketching (Wetzel and Forbus 2015).

2. Sketch Worksheets (Yin et al. 2010; Forbus et al. in press) help students learn about concepts by creating sketches in response to a specific problem. For example, a geoscience student might be asked to identify the radiation flows in depicting the greenhouse effect, and a biology student might be asked to annotate a diagram of a human heart showing where the parts are and how blood flows through it. Sketch Worksheets are domain-independent: The same technology has been used in biology, geoscience, physics, and even knowledge representation. They work by the instructor using CogSketch to specify a problem and their desired solution(s), annotating facts that are important with feedback to give if they are not true, and grading rubrics. When students download a Sketch Worksheet, they draw their own solution, asking for feedback whenever they like. Feedback is generated by comparing the instructor’s sketch with the student sketch, using our model of analogy, presenting them with the instructor’s feedback when there are problems.

Sketch Worksheets were designed to be broadly applicable, and have been used by over 500 students in classroom and laboratory experiments with students ranging from fifth grade (Miller et al. 2014) to college (Yin et al. 2010; Garnier et al. in press). Moreover, Northwestern’s introductory geoscience course now routinely uses them for instruction, based on Sketch Worksheets authored by a geoscience graduate student at University of Wisconsin-Madison. Thus there is already evidence that this model can be effective and can scale. However, there is still work to be done. Sketch Worksheets are beneficial on their own, but could be even more powerful if integrated with other instructional media, such as provided in GIFT.

3 GIFT

To support adaptive training in the military, the Army Research Laboratory has developed the Generalized Intelligent Framework for Tutoring (GIFT). The framework allows users with no experience or expertise in computer programming to author intelligent tutors. The GIFT authoring tool provides a basic flowchart that walks the user through the process. As the author generates the ITS, various types of media can be used, e.g. PowerPoint, pictures, etc. The authoring tool also supports the Learning Tools Interoperability standard. As a result, GIFT can call other tutors and interface with various Learning Management Systems, e.g. edX. The next step in our research is to support learning content that includes sketching.

3.1 Integrating Sketch Worksheets into GIFT

Our first goal is to integrate CogSketch into GIFT, so that Sketch Worksheets can become a new type of instructional medium for GIFT. That is, an author for a new GIFT tutor can include Sketch Worksheets as one of the activities that students can do in that tutor. For example, a Simple Machines tutor could give a student a problem involving levers, implemented as a Sketch Worksheet. Based on their performance, it could either provide them with another Sketch Worksheet on the same topic, a different form of activity, or move on to the next topic. This adaptive sequencing goes beyond what CogSketch does.

To accomplish this, we will need to integrate CogSketch into GIFT. We plan to build a cloud-based version of CogSketch, based on Docker so that it is compatible with multiple cloud vendors. We have already implemented a prototype user interface for capturing digital ink and providing feedback in Sketch Worksheets being run remotely. We will integrate this user interface into the GIFT framework, as well as do the engineering necessary to make it compatible with the GIFT cloud. (We understand that the GIFT cloud is currently hosted on Amazon’s service, which is supported by Docker.) We will also need to implement transducers that translate the kinds of tutoring data that Sketch Worksheets provide into GIFT domain knowledge files and metadata, so that student performance information gathered by CogSketch is available for other tutoring components.

To test this integration, we will develop a pool of Sketch Worksheets on a STEM domain that is important in K-12 education and for Army training, i.e. Simple Machines. Basic principles of mechanics typically start showing up in early education, communicating qualitative principles of forces, connectivity, and motion. They reappear in training technicians and in engineering education. We will develop a set of at least a dozen Sketch Worksheets covering principles of levers, pulleys, and gears, to provide a pool of Sketch Worksheets that can be used by a Simple Machines Adaptive Tutor. This tutor will be authored using standard GIFT tools, to ensure that the integration works smoothly and to explore how the data gathered by Sketch Worksheets can be used by such tutors.

We will gather evaluation data about this tutor by two means. First, we will make it available through the GIFT community distribution mechanisms. Second, we will set up an online Sketch Academy at Northwestern, where anyone can log in and try them out, and encourage undergraduates to use this as a learning resource. CogSketch already incorporates a “phone home” mechanism that does anonymization locally, so that no student identification data is ever transmitted to us. We will add a similar mechanism to the adaptive tutor, so that we can monitor and analyze student performance.

We note that once this integration is accomplished, anyone can author new Sketch Worksheets by downloading the Windows version of CogSketch and using its authoring environment. This should enable others to experiment with Sketch Worksheets for their own domains of interest.

3.2 Integrating Evaluate and Extend

We will continue to evaluate the data collected from our tutoring systems, improving and extending them as necessary to support better student learning. We also plan to explore some more advanced tutoring capabilities that the Companion cognitive architecture should support. For example, a hypothesis we plan to explore is that qualitative representations can help learners understand equations more deeply. Students often treat equations as disconnected from their daily life, as something to memorize for exams. Experts have additional tacit knowledge, things that equations tell them, which novices tend to not get. Qualitative representations make such tacit knowledge explicit. For example, F = MA tells an expert that one can apply more force in a situation by using more mass, or make something go faster by pushing it harder. Qualitative representations provide a formal way to encode such tacit knowledge, making it available for tutoring systems to use in their domain models. Drawing graphs based on equations may be one way to help bridge from their spatial understanding of the world to their causal knowledge. Another example is giving students the opportunity to critique designs that have been proposed historically for perpetual motion machines. Determining why a design won’t work could provide an engaging way to get students to think about friction and conservation laws. We plan to build a prototype perpetual motion machine tutor to explore this.