Elsevier

Environmental Modelling & Software

Volume 95, September 2017, Pages 115-131
Environmental Modelling & Software

Insights into sensitivity analysis of Earth and environmental systems models: On the impact of parameter perturbation scale

https://doi.org/10.1016/j.envsoft.2017.03.031Get rights and content

Highlights

  • Model sensitivity analysis is “sensitive” to parameter perturbation scale (e.g., step size).

  • VARS is a novel global sensitivity analysis (SA) technique that uniquely characterizes perturbation scale dependency of SA.

  • The sensitivity analysis strategies are preferred that take into account the impact of parameter perturbation scale.

Abstract

This paper investigates the commonly overlooked “sensitivity” of sensitivity analysis (SA) to what we refer to as parameter “perturbation scale”, which can be defined as a prescribed size of the sensitivity-related neighbourhood around any point in the parameter space (analogous to step size Δx for numerical estimation of derivatives). We discuss that perturbation scale is inherent to any (local and global) SA approach, and explain how derivative-based SA approaches (e.g., method of Morris) focus on small-scale perturbations, while variance-based approaches (e.g., method of Sobol) focus on large-scale perturbations. We employ a novel variogram-based approach, called Variogram Analysis of Response Surfaces (VARS), which bridges derivative- and variance-based approaches. Our analyses with different real-world environmental models demonstrate significant implications of subjectivity in the perturbation-scale choice and the need for strategies to address these implications. It is further shown how VARS can uniquely characterize the perturbation-scale dependency and generate sensitivity measures that encompass all sensitivity-related information across the full spectrum of perturbation scales.

Introduction

Sensitivity analysis (SA) is an important paradigm for understanding model behavior, characterizing uncertainty, improving model calibration, etc. It consists of identifying the most important factors influencing the model outcome and quantifying their importance. Methods for SA can be categorized into two general approaches: Local and Global. Local sensitivity analysis (LSA) techniques, in the form of one-factor-at-a-time (OFAT) or one-at-a-time (OAT), are widely used by modellers in all disciplines because of their simple mechanism and implementation (e.g., Murphy et al., 2004, Nolan et al., 2007). In the OAT method, to measure the variation in model output, one parameter is changed at a time from a base point in parameter space while others are kept constant. This requires specifying a “step size” for change in a parameter, which is a representation of what we refer to as “perturbation scale” or Δx. However, the adequacy of OAT methods is proven to be insufficient (and potentially misleading) mainly due to non-linear behavior of model response and interactions between parameters (Saltelli and Anoni, 2010), which is the case in nearly all environmental models. In order to gain a comprehensive assessment of sensitivity for the numerical models over the entire parameter space, a global sensitivity analysis (GSA) technique must be adopted (Saltelli et al., 2008). GSA methods are intended to measure “global” sensitivity of a model to different factors (e.g., model parameters, forcing data, boundary and initial conditions, etc.) across the entire multi-dimensional parameter space. The concept of perturbation scale is one of the inherent properties of GSA, and may be formally or informally defined depending on the GSA method used.

Various methods have been developed and used by researchers for GSA. These methods are rooted in different philosophies, resulting in a different and sometimes conflicting and/or counterintuitive assessment of sensitivity (Gupta and Razavi, 2016). Some methods rely on small-scale (derivative-like) perturbation of parameters within the parameter space (e.g., Morris, 1991, Campolongo et al., 2007) to characterize global sensitivity. Other methods avoid calculating derivatives by using larger scale (variance-like) perturbation of parameters based on the concept of analysis of variance (e.g., Sobol’, 1993). None of these methods, however, account for the fact that sensitivity is a scale-dependent concept as highlighted by Razavi and Gupta (2015). In other words, results of a global sensitivity analysis can be quite different between two different GSA techniques or even within a single technique, depending on the scale of parameter perturbation used. Razavi and Gupta (2015) raised this “scale issue” and provided some mathematical examples. To address this issue, Razavi and Gupta, 2016a, Razavi and Gupta, 2016b introduced a novel global sensitivity analysis technique based on the Variogram concept called Variogram Analysis of Response Surfaces (VARS). In addition to a high computational efficiency, a novel and unique feature of VARS is that it provides a comprehensive assessment of model response sensitivity across a range of parameter perturbations, as opposed to a single prescribed perturbation scale. This feature enables VARS to provide derivative- and variance-based sensitivity metrics such as elementary effects and total-order effects simultaneously, in addition to its own comprehensive metrics for global sensitivity, called IVARS (for Integrated Variograms Across a Range of Scales). We use this new powerful tool in this work, to further investigate this “scale issue”, show its significance, and address it using multiple physically-based environmental models.

Regardless of the method used, when conducting an SA, we need to first choose that, among various alternatives, the sensitivity of “what” quantity to input factors is to be assessed? This quantity can be either a direct simulated model output (e.g., simulated streamflow, soil moisture, or evapotranspiration), or a model performance (goodness-of-fit) metric (e.g., absolute error, or sum of squared errors). The difference between the two is that unlike a direct simulated model output, a model performance metric requires also observational records to calculate how well a model is predicting the observed values. This choice of SA criterion is typically made subjectively, and as different model outputs or performance metrics look at different aspects of the model, this choice can have a major impact on the outcome of SA. Accordingly, some studies have advocated the use of multiple criteria for a more comprehensive evaluation of the sensitivity of environmental models. For example, Bastidas et al. (1999) introduced a multi-objective generalized sensitivity analysis (MOGSA) approach for multi-criteria sensitivity analysis of a land surface scheme. Liu et al. (2004) combined MOGSA with an OAT analysis to determine influential parameters of a climate model coupled with a land surface model. Rosolem et al. (2012) extended this approach to a fully multiple-criteria implementation of the Sobol’ method (Sobol’, 1993) for identifying influential parameters of a land surface model. van Werkhoven et al. (2008) used four different metrics to analyze the sensitivity of a conceptual rainfall-runoff model across various hydroclimatic conditions. Nossent and Bauwens (2012) evaluated the sensitivity of a semi-distributed environmental model to parameters associated with both water quantity and water quality using multiple criteria. Accordingly, in this study, we conduct our SA using multiple metrics (criteria) to obtain a more comprehensive perspective of our findings with respect to the aforementioned scale issue.

In this paper, our goal is to study the effect of parameter perturbation scale on sensitivity assessment of complex environmental models. Accordingly, following the work by Razavi and Gupta, 2015, Razavi and Gupta, 2016a, Razavi and Gupta, 2016b, in this study, we further explore the scale issue in local and global SA of complex environmental models, highlight its significance, and address it using a suitable SA approach that can take into account the scale-dependency of SA. In order to make our findings more general and less dependent on a case study or a metric choice, we conduct these analyses with multiple models and multiple metrics.

For this purpose, VARS is applied to three different environmental models: HydroGeoSphere (HGS, Aquanty Inc, 2015), Soil and Water Assessment Tool (SWAT 2000; Neitsch et al., 2001), and Modélisation Environmentale–Surface et Hydrologie (MESH, Pietroniro et al., 2007). In each case, the sensitivity of model performance for simulating streamflow hydrograph is analyzed using different metrics selected based on various hydrograph characteristics such as high flows, low flows, and flow volume. Since we use multiple metrics in our study, we are also able to observe the effect of metric choice on the primary findings of this paper and on the results of the sensitivity analysis conducted for these complex environmental models.

In the next section, we first demonstrate the “scale dependency” of sensitivity analysis in environmental modelling studies using an OAT local sensitivity analysis example. Then we describe our methodology and case studies starting with an introduction to the VARS technique. Afterward, we present results and corresponding discussions followed by the concluding remarks at the end.

Section snippets

Perturbation scale dependency in sensitivity analysis

Suppose the response surface of a model is represented by function f as:y=f(x1,,xn)where x1,,xn are input parameters of interest varying within a space defined by the n-dimensional hypercube bounded between x1min,,xnmin and x1max,,xnmax. The local sensitivity of function y with respect to parameter xi (i=1, 2, …, n) at a nominal point (x1,,xn) in the parameter space is defined by the gradient concept and can be formulated as:si=yxi|x1,,xnΔyΔxi|x1,,xnwhere Δxi is used instead of

Theory

Variogram Analysis of Respotnse Surfaces (VARS; Razavi and Gupta, 2016a) is a novel global sensitivity analysis approach designed to efficiently provide a comprehensive assessment of global sensitivity over the entire factor (e.g., parameter) space. VARS uses an anisotropic variogram concept to characterize the variation of model response or a (goodness-of-fit) metric across a range of parameter perturbation scale within the multi-dimensional parameter space. In addition to its own sensitivity

Design of experiment

In order to make our study more comprehensive, we apply VARS using multiple SA criteria (metrics) to multiple physically-based environmental models. These models are HydroGeoSphere (HGS, Aquanty Inc, 2015), Soil and Water Assessment Tool (SWAT, Neitsch et al., 2001), and Modélisation Environmentale–Surface et Hydrologie (MESH, Pietroniro et al., 2007). In each case, the sensitivity of model performance in reproducing observed streamflow time series (hydrograph) is measured to model parameters

Case study 1: MESH model with 40 parameters

This case study is adopted from Haghnegahdar et al. (2015) where Modélisation Environmentale–Surface et Hydrologie (MESH, Pietroniro et al., 2007) was applied to Nottawasaga river basin in South-Western Ontario, Canada with a drainage area of nearly 2700 km2. The area is mainly covered by cropland followed by deciduous forest and grasslands.

MESH is a semi-distributed physically-based coupled land surface-hydrology modelling system developed by Environment and Climate Change Canada (ECCC) and

Results and discussion

In this section, we present the sensitivity analysis results of VARS technique including directional variograms and IVARS metrics for Hi = 10, 30 and 50% for all metrics in each case study. At the end, we also provide some brief results of VARS-TO and VARS-ABE to further support the findings of this study.

Summary and conclusions

In this paper, we used multiple models and performance metrics, to study the effect of parameter perturbation scale on sensitivity assessment of environmental models. Three different case studies were considered involving three environmental models: 1) A 40-parameter Modélisation Environmentale–Surface et Hydrologie (MESH, Pietroniro et al., 2007) case, 2) A 14-parameter Soil and Water Assessment Tool (SWAT 2000; Neitsch et al., 2001) case, and 3) A 10-parameter HydroGeoSphere (HGS, Aquanty

Acknowledgments

The first author is thankful to the University of Saskatchewan's Global Institute for Water Security for financial support. The authors would like to thank the Information & Communications Technology (ICT) at the University of Saskatchewan, for providing continuous support in using the High-Performance Computing (HPC) research cluster, Plato. They also would like to say special thanks to Dr. Steve Berg from Aquanty Inc. for continued support with HydroGeoSphere. They would also like to thank

References (43)

  • X. Xu et al.

    Assimilation of SMOS soil moisture over the great lakes basin

    Remote Sens. Environ.

    (2015)
  • H. Yen et al.

    A framework for propagation of uncertainty contributed by parameterization, input data, model structure, and calibration/validation data in watershed modeling

    Environ. Model. Softw.

    (2014)
  • A.S. Abdul

    Experimental and Numerical Studies of the Effect of the Capillary Fringe on Streamflow Generation

    (1985)
  • A.A. Ameli et al.

    Are all runoff processes the same? Numerical experiments comparing a Darcy-Richards solver to an overland flow-based approach for subsurface storm runoff simulation

    Water Resour. Res.

    (2015)
  • Aquanty Inc

    HydroGeoSphere. A Three-dimensional Numerical Model Describing Fully-integrated Subsurface and Surface Flow and Solute Transport

    (2015)
  • S. Ba et al.

    Optimal sliced Latin hypercube designs

    Technometrics

    (2015)
  • L.A. Bastidas et al.

    Sensitivity analysis of a land surface scheme using multicriteria methods

    J. Geophys. Res.

    (1999)
  • N.A.C. Cressie

    STATICS for SPATIAL DATA

    (1993)
  • M. Faramarzi et al.

    Modelling blue and green water resources availability in Iran

    Hydrol. Process.

    (2009)
  • H. Gupta et al.

    Challenges and future outlook of sensitivity analysis

  • H. Gupta et al.

    Toward improved calibration of hydrologic models: multiple and noncommensurable measures of information

    Water Resour. Res.

    (1998)
  • Cited by (34)

    • Robustness analysis framework for computations associated with building performance models and immersive virtual experiments

      2021, Advanced Engineering Informatics
      Citation Excerpt :

      In speech recognition using machine learning, perturbation techniques include adding noisy signal [43], making speech reverberated [44], and adding background noise [45]. Other examples of perturbations are inserting sentences in question answering systems [46] and using perturbation scale to alter data [47,48]. Perturbation techniques may be categorized as an additive perturbation (e.g., injecting noise to images, adding noisy signal, and adding background noise) and a structured transformation (e.g., changing information of images, transforming image geometry, and making speech reverberated) [49].

    View all citing articles on Scopus
    View full text