Patterns of use of an agent-based model and a system dynamics model: The application of patterns of use and the impacts on learning outcomes

Abstract

A classification system that was developed for the use of agent-based models was applied to strategies used by school-aged students to interrogate an agent-based model and a system dynamics model. These were compared, and relationships between learning outcomes and the strategies used were also analysed. It was found that the classification system could also be applied to the use of the system dynamics model, with the addition of criterion. This means that a classification system exists for both styles of models. The fact that the strategies could be identified, despite differences in the actual model, and the model type, compared to the original study, means that there are implications for training teachers or systems to also identify the strategies. Initial findings of this study identified links between prior knowledge and the strategy chosen, as well as links with learning outcomes.

Introduction

Buckley et al. (2004) suggest that use of simulation models, such as deciding which perceptual cues to attend to, deciding how to interact with the representation and monitoring and evaluating the results of those interactions, are important metacognitive processes that play an important part in model-based learning. If this is the case, learning outcomes should depend, amongst other factors, on the ways in which students interrogate simulations, although there are very few studies that investigate the use of simulation models and compare learning outcomes (Kennedy and Judd, 2007, Levy et al., 2004, Moxnes, 1998). Levy and Wilensky (2005) investigated the patterns of use of an agent-based model. They identified three distinct strategies for interrogating the model. In the study reported in this paper, video screen shots of students’ use of an agent-based model, a system dynamics model, and both combined were analysed in order to meet four aims: (1) to determine whether Levy and Wilensky’s strategies could be applied to other agent-based models, and to a system dynamics model; (2) to investigate the relationships between these strategies and general measures of use of the models to help to further define the classification system; (3) to investigate relationships between prior knowledge and the strategy chosen; and (4) to determine whether the strategy used was related to the learning outcomes.

The following literature review will outline the use of models for science and environmental education, provide details of the use of agent-based and system dynamics models, and discuss Levy and Wilensky’s strategies in more detail. The methods, including the design of the models, the experimental procedure and the instruments used to assess prior knowledge and learning outcomes, will also be explained. The results and discussion will follow.

Models are representations of ideas, objects, events, processes or systems (Gilbert & Boulter, 1998), and are generally simplifications of reality (Coyle, 2000, Jonassen, 2000). Computer-based models allow complex systems to be represented efficiently and constructed in a relatively short amount of time. Technology can be used for learning by modelling (Rohr and Reimann, 1998, Woolsey and Bellamy, 1997) and learning with models (Milrad, Spector, & Davidsen, 2003) to allow improved understanding of complex, dynamic systems. Many authors (see for example Stylianidou, Boohan, & Ogborn, 2004 refer to these types of learning as exploratory learning activities (students are able to explore pre-existing models) and expressive learning activities (students create their own models). A further step is for students to critique other students’ models (Gobert & Pallant, 2004). Understanding scientific models is quite different from the ability to reason with scientific models (Gobert & Pallant, 2004).

A system dynamics model was one of the types of model investigated in this study. “System Dynamics is a methodology for analysing complex systems and problems with the aid of computer simulation software” (Alessi, 2000, p. 1) and includes cause and effect relationships, time delays and feedback loops. Systems can be represented by causal loop diagrams and by stock and flow diagrams, and in this study, students were given access to the stock and flow diagram (see Fig. 1).

Although much is written of the value of system dynamics modelling in education in schools, very little empirical data exists to confirm this (Doyle et al., 1998, Stratford et al., 1998). Instead, case studies and anecdotal accounts from teachers who have used such models are available on the web (see for example Guthrie and Fisher, 1999, Ragan, 1999, Verona et al., 2001). The Core models project was a large scale implementation of system dynamics modelling in a number of schools, and evaluation of the project, while focusing on teacher support, did find that while students did improve understanding of the scientific concepts underlying modelling, they did not improve their ability to interpret the models (Maryland Virtual High School, 2001). Studies make recommendations as to how to teach system dynamics modelling (Schaffernicht, 2006, Stuntz, 2000), or how to incorporate it into a class (Draper, 1993).

Students do have trouble understanding complex systems using system dynamics models. One study found that the majority of participants in four separate studies (167 subjects in total) had biased views of the dynamics of the environmental system that they were examining which suggested that they were using a static (rather than a dynamic) mental model (Moxnes, 2000). Another study found that graduate students had very poor understanding of the processes involved in climate change, a common misconception was that stabilising emissions would “fix the problem”, showing a poor understanding of dynamic processes (Sterman & Booth Sweeny, 2002). While a number of studies have investigated learners’ inability to correctly model a natural resource problem (Booth Sweeney and Sterman, 2000, Diehl and Sterman, 1995, Moxnes, 2004), Kainz and Ossimitz (2002) determined that students did not have difficulties in determining between stocks and flows, and instead found it difficult to represent this information in a flow chart, and similarly that students were better able to interpret information from a table than from a graph. This suggests that perhaps the representational affordance of the system dynamics model, or the way in which the assessment is asked, needs to be examined.

An agent-based model is the other type of model that was investigated in this study. In agent-based modelling the focus is on the interaction between the agents, and their environment. An agent is defined as an object that controls its own behaviour, and could be individuals of a species, individuals at a particular stage in the life cycle (a cohort), or a group of individuals that can be considered identical (Ginot, Le Page, & Souissi, 2002). The rules that apply to the agents determine the behaviour of the whole system, called emergence. By laying down the rules for the agents and the system, behaviour may emerge that would otherwise not have been predicted (Bousquet and Le Page, 2004, Ginot et al., 2002, Parrott and Kok, 2001, Schieritz, 2002).

When used in education, agent-based models allow students to explore the relationship between the agents’ rules of behaviour and the patterns that emerge (Stieff & Wilensky, 2003). Students are able to make predictions and test them by exploring model outcomes as they manipulate variables (Stieff & Wilensky, 2003). The use of agent-based models in education “narrows the gap” between school biology and research biology (Wilensky & Reisman, 2006). The main advantage of using agent-based models is that students are able to employ their knowledge of the behaviour of individuals in the construction of theories about the behaviour of populations (Wilensky & Reisman, 2006). Agent-based models are able to be quite realistic. Using a realistic computer simulation can be motivating for students because they are entertaining and evocative (Goldstone & Son, 2005).

Levy and Wilensky (2005) investigated the patterns in how students used an agent-based model to learn about Chemistry. The four main statistics measured were successive settings in running the model, observation time, the average time between actions, and the number of runs. Students were engaged in a relatively ‘open’ activity (only able to change one setting, but to whatever value they chose), and Levy and Wilensky were able to identify three distinct exploration patterns: Straight To The Point, Homing In, and Oscillating.

The Straight To The Point strategy had a shorter overall observation time, but longer observation time per run; longer time between actions; and fewer runs. Levy and Wilensky identified this as an efficient mode to use the model, which may allow students to develop a deeper understanding of each state of the model. However, using a Straight To The Point strategy may mean that learners miss critical settings or transitions that would have been discovered by a wider range of values chosen. These critical settings are an important part of understanding a system from using a model (Lowe, 1993).

Students who used the Homing In strategy exhibited a shorter overall observation time, and shorter observation time per run; shorter time between actions; and more runs. These students were identified by the authors as “click happy”. The third strategy identified was called Oscillating, which consisted of a longer overall observation time, but shorter observation time per run; shorter time between actions; and an intermediate number of runs. Both the Homing In and Oscillating strategies involved speedy model changes and short observations, which implies that students were not able to detect and generalise complex relationships between variables. However, the many states of the model means that students examined many aspects of the model’s behaviour and students were more likely to detect a critical setting. Of the two, the Homing In strategy is more planned, whereas students would struggle to keep the previous state in mind for comparison when using the Oscillating strategy.

Strategies for interrogating models have been discussed, in some respect, in the system dynamics modelling literature; however no formal classification system has been developed. Moxnes (1998) discuses the combination of mental models and analysis. In terms of the analysis, he discusses a trial-and-error heuristic, a consistent analysis, and a gradient search, which can be loosely mapped to the Oscillating, Straight To The Point, and Homing In strategies noted by Levy and Wilensky. However, there was no further investigation into the links between the strategies and the decision-making and mental model development. Moxnes (1998) also investigated the use of explanatory features such as additional information that scaffolded students in how to model the system. Students were able to utilise the extra information given about growth, but not about the stock.

Another issue with regards to the use of the models is the role of prior knowledge in the use of models. The only study to specifically investigate the role of prior knowledge in the use of the model was the Levy and Wilensky (Levy & Wilensky, 2005) study discussed in some detail above. They suggested that prior knowledge about the domain may shorten the exploration time, resulting in a student focusing on a few key settings, such as the Straight To The Point strategy.