Scenario generation using adaptive sampling: The case of resource scarcity

https://doi.org/10.1016/j.envsoft.2015.09.014Get rights and content

Highlights

  • We introduce adaptive sampling to explore the output spectrum of simulation models.

  • We use roughness to characterize the dynamic complexity of transient scenarios.

  • We show that adaptive sampling supports scenario discovery and exemplar selection.

  • Using a resource scarcity model, we identify strengths, weaknesses/improvements.

Abstract

Static input-oriented sampling approaches are often used for generating model-based scenarios. However, for models of deeply uncertain and dynamically complex issues, there is no guarantee that such approaches reveal the total behavioral spectrum that could be generated by simulating them. In this paper, we present an adaptive output-oriented sampling approach for exploring the full behavioral spectrum that could be generated by computational models in view of generating interesting, even previously undiscovered, scenarios. In this paper, we use a resource scarcity model to illustrate the approach, show the difference between static sampling and adaptive sampling, and demonstrate the usefulness for scenario discovery of the latter combined with other methods. We show that this approach can be used for revealing the behavioral spectrum of models, identifying regions of the input space that generate particular behaviors, and selecting (sets of) scenarios that are representative in terms of output and input spaces.

Introduction

For decades, simulation models have been used to generate transient scenarios. In the early days, these model-based scenarios were generated manually, by simulating specific combinations of input values. Later, after sampling approaches started to be used in modelling and simulation, it became common practice to combine sampling and simulation to semi-automatically generate ensembles of model-based scenarios. Over the years, many sampling approaches have been developed (Santner et al., 2003, Saltelli et al., 2008), including full factorial sampling and regular grids sampling, pseudo random sampling aka Monte Carlo sampling, stratified sampling, Latin Hypercube sampling (McKay et al., 1979), and criterion based sampling (Johnson et al., 1990). These sampling methods are essentially input-oriented and static: ex-ante experimental designs focus on covering the input space without taking the resulting output space into account. Such sampling methods are nevertheless appropriate if the resulting experimental design properly covers the output space.

However, for issues that are characterized by dynamic complexity and deep uncertainty, static input-oriented sampling approaches may be troublesome. Systems or issues are dynamically complex if their behavior over time is not simple and predictable. Dynamic complexity is often caused by the interplay of non-linear causal effects, systemic feedback effects resulting in systemic non-linearities, and important time delays. Across (even slightly) different conditions, a dynamically complex issue or system may generate a plethora of intricate behaviors over time. When dealing with dynamically complex issues or systems and conditions cannot be determined with certainty, it therefore makes sense to unveil and account for the full spectrum of plausible behaviors by simulating non-linear models across large input spaces.

Deep uncertainty relates to those situations in which experts and parties do not agree upon models to represent these situations, probabilities associated to their inputs and outcomes, and the desirability of outcomes (Lempert et al., 2003). That does not mean that issues and systems characterized by deep uncertainty cannot be modelled and simulated. To the contrary. One may be able to make multiple simulation models and generate many plausible simulation runs. However, under deep uncertainty, one is not able to rank them in terms of their perceived likelihood (Kwakkel et al., 2010). As a consequence, it is not possible to focus attention on regions of more perceived likelihood. Moreover, the input space to be considered in situations characterized by deep uncertainty can be expected to be much larger than the input space to be considered in situations characterized by lower levels of uncertainty.

The trouble with traditional input-oriented sampling methods when applied to highly non-linear models simulated under deep uncertainty is that particularly interesting parts of the uncertainty space may remain undiscovered. As simplistically schematized in Fig. 1a, purely for the sake of argumentation, a uniform spread of points on an input space may, once transformed through a non-linear model, give output behaviors only in a subset of the total possible output space. Inversely, the largest variety of dynamically complex behaviors may, in case of highly non-linear models, correspond to a very small subspace of the input space. These exceptional behaviors may be of particular interest. Using uniform and non-uniform input-oriented sampling with a limited number of samples, these dynamically complex behaviors may remain undiscovered. That is, although input-oriented sampling may reveal new behaviors, in case of non-linear models, it is not directed towards discovering the most interesting behaviors, let alone the full spectrum of behaviors. One way to remedy this would be to perform an extremely large number of experiments, as is often done in automated Monte Carlo approaches. Another remedy would be to sample especially from areas of the input space that, based on ex-ante knowledge or expectations, can be assumed to generate the largest variety of behaviors, for example from the outer edges of the input space. Alternatively, one may try to exploit knowledge about the output space, gained while sampling, and adaptively search for previously undiscovered behaviors, as in Fig. 1b, in areas of the uncertainty space that show most behavioral variation. One may also combine the latter two approaches. That is, first stretch the sample to the outer edges of the input space and then sample adaptively within the stretched input space. This article focusses on the adaptive sampling part – stretching the uncertainty space is not dealt with here.

The concept of adaptive sampling is not new. It has been used before, for example by Bucher, 1988, Bishop et al., 2001, and Bergot (2001). In forecasting of meteorological events, adding observations in the predicted sensitive areas can improve the quality of the forecast of weather features (Bergot, 2001). This paper uses a variation on the concept, by analyzing the output behavior of random samples to locate new areas to sample from.

In this paper, we present a new adaptive output-oriented sampling approach, we compare it to traditional static input-oriented Latin Hypercube (LH) sampling, and we illustrate its use for behavioral exploration and the selection of representative exemplar scenarios. Although many specific methodological choices are made in our illustration, the overall approach is, on the one hand, applicable to model-based studies in general, and to dynamic simulation studies in particular. Each of the specific methodological choices made in this paper may, on the other hand, be changed to suit a specific case or model. For example, the specific measure used here to characterize the dynamic complexity of the simulation runs could be replaced by an alternative measure.

The remainder of the paper is structured as follows. The overall methodology is presented in section 2. Section 3 briefly introduces a System Dynamics resource scarcity model that is used to illustrate the use of the adaptive sampler. Section 4 consists of a detailed illustration of the approach using the scarcity model. Section 5 consists of a discussion. Finally, in Section 6, we present our conclusions.

The resource scarcity model is available as online supplementary material. The python code of the adaptive sampler and the snippets of code to create the figures in this paper are available as online supplementary materials too.

Section snippets

Methodology

In Section 2.1 – before presenting the adaptive sampling method – we will first define dynamic complexity and deep uncertainty, and discuss what would be needed for dealing with dynamic complexity and deep uncertainty. In Section 2.2 we will introduce a simple, yet useful, measure of behavioral complexity. In Section 2.3, we will present the adaptive output-oriented sampling method in words, in pseudo-code, and by means of a numerical example. Finally, in Section 2.4, we will discuss its use –

The resource scarcity model

To illustrate the adaptive sampler, we use a previously published System Dynamics model regarding potential resource scarcity as EMA scenario generator. The model was used because it allows one to generate a wide variety of dynamics. A detailed discussion of the model is available in Pruyt (2010), a case description is available in Pruyt (2013). The model was also used for explorative purposes by Kwakkel and Pruyt, 2013, Kwakkel and Pruyt, 2015.

Using a time series clustering approach that

Adaptive sampling

Fig. 5 illustrates the workings of the adaptive sampling process applied to the resource scarcity model. Fig. 5a displays a burn-in sample in terms of relative roughness and maximal value as well as the largest roughness gap. Next, additional sampling is performed in the input space spanned by the input values of the two scenarios that together form the largest gap. Fig. 5b shows that some of the additional simulations fall within the largest roughness gap, but also that other additions fall

Discussion

In the previous sections, we argued in favor of using adaptive sampling to explore the spectrum of outputs that can be generated by simulation models, presented an adaptive sampling method, and demonstrated its usefulness – when used in combination with other methods – for scenario discovery and the selection of exemplar scenarios. We will now have a critical look at the methodological choices made in this paper, discuss potential improvements, and suggest other use cases for adaptive sampling.

Conclusions

When simulating dynamically complex models under deep uncertainty, there is little guarantee that static input-oriented sampling approaches will reveal the total spectrum of scenarios or will support the selection of a subset of scenarios that spans the total output spectrum.

In this paper, we introduced an adaptive sampling approach which improves upon existing random sampling techniques for dynamically complex models, allows for building a clearer and more comprehensive picture of model

Acknowledgments

We acknowledge four anonymous reviewers for their many excellent review comments which enabled us to significantly improve our manuscript. We also acknowledge dr. Jan H. Kwakkel and his colleagues from Delft University of Technology for developing the open source EMA workbench. We greatly acknowledge Steven Bankes, Robert Lempert, Steven Popper and their co-authors for their contributions to the sfields of EMA and RDM. Finally, we would like to thank the special issue editors and editor in

References (44)

  • T. Bergot

    Influence of the assimilation scheme on the efficiency of adaptive observations

    Q. J. R. Meteorol. Soc.

    (2001)
  • C.H. Bishop et al.

    Adaptive sampling with the ensemble transform Kalman filter. Part I: theoretical aspects

    Mon. weather Rev.

    (2001)
  • E.W. Dijkstra

    A note on two problems in connexion with graphs

    Numer. Math.

    (1959)
  • J.W. Forrester

    Industrial Dynamics

    (1961)
  • J.H. Friedman et al.

    Bump-hunting in high-dimensional data

    Stat. Comput.

    (1999)
  • J.L. Gastwirth

    The estimation of the Lorenz curve and Gini index

    Rev. Econ. Stat.

    (1972)
  • C.E. Kampmann et al.

    Structural dominance analysis and theory building in system dynamics

    Syst. Res. Behav. Sci.

    (2008)
  • C.E. Kampmann et al.

    Analytical methods for structural dominance analysis in system dynamics

  • J.H. Kwakkel et al.

    Improving scenario discovery by bagging random boxes

  • J.H. Kwakkel et al.

    Using system dynamics for grand challenges: the ESDMA approach

    Syst. Res. Behav. Sci.

    (2015)
  • J.H. Kwakkel et al.

    Classifying and communicating uncertainties in model-based policy analysis

    Int. J. Technol. Policy Manag.

    (2010)
  • R.J. Lempert et al.

    Comparing Algorithms for Scenario Discovery

    (2008)
  • Cited by (16)

    • Structuring and evaluating decision support processes to enhance the robustness of complex human–natural systems

      2020, Environmental Modelling and Software
      Citation Excerpt :

      Other studies have acknowledged the challenges posed by path dependency, arguing how sequences of individual methodological choices influence the final results. For example, Herman et al. (2015) and McPhail et al. (2018) discussed ways that the choice of performance (robustness) measures influenced final conclusions, Moallemi et al. (2018b) discussed how sequences of methods create different decision insights, and Islam and Pruyt (2016) and Moallemi et al. (2018d) discussed how computational experiment design influences the final results. Decision support processes are affected by multiple human factors (e.g., intuition, heuristics, biases, or behavioural contexts).

    • Coping with uncertainties of sustainability transitions using exploratory modelling: The case of the MATISSE model and the UK's mobility sector

      2019, Environmental Innovation and Societal Transitions
      Citation Excerpt :

      Hence, there is a (computationally-driven) limit on the number of uncertainty parameters and on sampling from the uncertainty space to avoid this challenge. Methods and approaches for more efficient sampling (Islam and Pruyt, 2016) and for more effective delineation of the uncertainty space (Moallemi et al., 2018a,b) can be used to mitigate this challenge in future research. A transitions model can be subject to other forms of uncertainties which we did not study, such as uncertainty in the structure of the model and uncertainty in the conceptualisation which underpins the model.

    • An agent-monitored framework for the output-oriented design of experiments in exploratory modelling

      2018, Simulation Modelling Practice and Theory
      Citation Excerpt :

      Despite the goal-oriented process of exploratory modelling, experiments are often designed with an input-oriented approach focusing on sampling from the input uncertainty space with limited knowledge about which areas of that uncertainty space could lead to more interesting model behaviour and serve the goal of better modelling. An input-oriented approach can lead to missing some exceptional model responses in models with non-linear behaviours when only a specific area of the input uncertainty space could generate a wide spectrum of behaviours in the output space [26]. Further, the input-oriented approach could impose extra computational burden when endeavouring to cover an unnecessarily, but safely, wide uncertainty space without considering the relevance of generated outcomes.

    • Scenario techniques for energy and environmental research: An overview of recent developments to broaden the capacity to deal with complexity and uncertainty

      2017, Environmental Modelling and Software
      Citation Excerpt :

      However, in a context of deep uncertainty and complexity, there is no guarantee that such input-oriented sampling approaches reveal the full spectrum of outputs that could be generated. To address this issue, Islam and Pruyt (2016) develop an adaptive output-oriented sampling approach for exploring the full behavioral spectrum that could be generated by computational models in view of generating interesting, even previously undiscovered, scenarios. They use a resource scarcity model to illustrate the approach, show the difference with other sampling approaches, and demonstrate the usefulness for scenario discovery.

    View all citing articles on Scopus
    View full text