A review of current calibration and validation practices in land-change modeling

Highlights

• We describe a commonly used framework for the development of land-change models

• We systematically reviewed the calibration and validation of recent applications

• Calibration is mostly based on automated procedures and statistical analyses

• Validation is mostly based on location accuracy and much less on pattern accuracy

• The experimentation period is often longer than the calibration and validation period

Abstract

Land-change models are increasingly used to explore land-change dynamics, as well as for policy analyses and scenario studies. In this paper we review calibration and validation approaches adopted for recently published applications of land-change models. We found that statistical analyses and automated procedures are the two most common calibration approaches, while expert knowledge, manual calibration, and transfer of parameters from other applications are less frequently used. Validation of model results is predominantly based on locational accuracy assessment, while a small fraction of the applications assessed the accuracy of the generated land-use or land-cover patterns. Of the reviewed model applications, thirty-one percent did not report any validation. We argue that to mature as a scientific tool, and to gain credibility for scenario studies and policy assessments, the validation of land-change models requires consideration of challenges posed by uncertainty, complexity, and non-stationarity of land-change processes, and equifinality and multifinality of land-change models.

Introduction

Over the last two decades a wide range of models, so-called land-change models, have been developed and applied to simulate changes in land use and land cover. Although there are many purposes for which a modeling approach can be employed (Epstein, 2008), the vast majority of land-change models is used to project future land-use or land-cover changes (Sterk et al., 2011). For these purposes, models at several spatial scales have served as laboratories, in which experiments investigate how land use and land cover can change under alternative conditions, such as in scenario studies, ex-ante assessments, or policy analyses.

The application of models for land-change assessments is critically dependent on the quality of their output. Therefore, model applications require calibration and validation, to improve their fidelity to real-world conditions, and to assess their performance. A number of calibration methods have been proposed, each with their advantages and disadvantages. Model validation assess the quality of model results. This provides information about the usability of models for land-change assessments, and provides valuable feedback to land-change scientists about the ways in which we understand and represent the functioning of land systems. Therefore, the need for model calibration and model validation is widely acknowledged (e.g. Brown et al., 2013, Pontius et al., 2008, Silva and Clarke, 2002). However, there are no standards for the calibration and validation of land-change models, and a large number of different approaches have been adopted. These approaches differ in how models are calibrated, and what properties of model results are assessed (Brown et al., 2005, Hagen-Zanker and Martens, 2008).

Land-change processes are directly or indirectly driven by human decisions, such as farmers deciding on crop rotations, and property owners deciding on land transfers (e.g. Yu et al., 2013). Because the cognitive processes of all individual actors cannot be known, the resulting land changes will remain inherently uncertain at this level. Moreover, because land changes are at least partly influenced by earlier changes (i.e., path dependent), feedback loops can appear, which can cause small initial developments to grow over time (Verburg, 2006). As a result, land-change processes can be considered complex processes, yielding non-linear developments (Manson, 2007, Messina et al., 2008). Acknowledging the inherent uncertainty and complexity of land-change processes implies that land-change models cannot be expected to generate results that are perfectly accurate. As a result, several approaches have been proposed to account for near-hits (Costanza, 1989, Hagen, 2003, Pontius et al., 2011, van Vliet et al., 2013a). Another way to deal with the uncertainty and complexity in land-change processes is to focus on the composition and configuration of land-use patterns rather than the land use or land cover at the pixel level (Hagen-Zanker and Martens, 2008, Kocabas and Dragicevic, 2006, White, 2006).

The inherent uncertainty in land-change processes is reflected in many models by including some random variation. This random variation ensures that every single run can create a different outcome, and that some outcomes can be correct by chance (Brown et al., 2005). Model assessment therefore needs to account for these two effects. Moreover, many land-change models adopt a simulation approach, where simulations start from an initial map and subsequently make changes to that map. This implies that assessing land-change models based on the generated map alone is inadequate, because the amount of change can have as much influence on the accuracy as the model calibration itself (Hagen-Zanker and Lajoie, 2008, Pontius et al., 2004a, van Vliet et al., 2011). In a case where land hardly changes during the simulation period and a simulated map is compared with real-world observations, most simulation results will yield a high accuracy, even in the case where all changes are simulated incorrectly. Model assessment therefore requires a reference level that allows to assess the accuracy of the simulated change, instead of persistence (Diogo et al., 2014).

In this paper we review the calibration and validation approaches presented in recently published applications of land-change models. In the next section, we first explain the terminology that we have used in this paper, to avoid possible confusion. Then, we systematically review recent model applications for their calibration and validation approaches and discuss these in the context of the abovementioned insights in land-change processes and model properties. The focus of this paper is on the approaches to calibration and validation, while specific methods are only mentioned as an illustration. Approaches here indicate what properties of model results are assessed, while methods refer to how these properties are quantified. For more elaborate reviews of specific methods we refer to reviews presented in Bennett et al. (2013), and Kuhnert et al. (2005).