Challenge problems: uncertainty in system response given uncertain parameters

Abstract

The risk assessment community has begun to make a clear distinction between aleatory and epistemic uncertainty in theory and in practice. Aleatory uncertainty is also referred to in the literature as variability, irreducible uncertainty, inherent uncertainty, and stochastic uncertainty. Epistemic uncertainty is also termed reducible uncertainty, subjective uncertainty, and state-of-knowledge uncertainty. Methods to efficiently represent, aggregate, and propagate different types of uncertainty through computational models are clearly of vital importance. The most widely known and developed methods are available within the mathematics of probability theory, whether frequentist or subjectivist. Newer mathematical approaches, which extend or otherwise depart from probability theory, are also available, and are sometimes referred to as generalized information theory (GIT). For example, possibility theory, fuzzy set theory, and evidence theory are three components of GIT. To try to develop a better understanding of the relative advantages and disadvantages of traditional and newer methods and encourage a dialog between the risk assessment, reliability engineering, and GIT communities, a workshop was held. To focus discussion and debate at the workshop, a set of prototype problems, generally referred to as challenge problems, was constructed. The challenge problems concentrate on the representation, aggregation, and propagation of epistemic uncertainty and mixtures of epistemic and aleatory uncertainty through two simple model systems. This paper describes the challenge problems and gives numerical values for the different input parameters so that results from different investigators can be directly compared.

Introduction

Modeling and simulation efforts continue to expand in areas where they have been used for some time, and are also beginning to be used in new application areas. Modeling and simulation have been heavily used in risk assessment for high-consequence systems, such as nuclear power reactors, underground storage of radioactive wastes, and the nuclear weapon stockpile. For complex engineered or natural systems, computer-based modeling and simulation is required. Most of the mathematical models of these physical systems are given by systems of coupled partial differential equations (PDE). For example, typical computer codes that numerically solve these PDEs, and all of their auxiliary equations, can contain tens to hundreds of thousands of lines of code. Similarly, the inputs and outputs of these codes have high dimensionality, for example, hundreds of input and output quantities.

Computer codes of the size and complexity just indicated can be considered as ‘black boxes’ in the sense that very little is known concerning how the codes map inputs to outputs. For the present discussion, we consider this mapping to be deterministic. By deterministic we mean that when all necessary input data for the code are specified, the code produces only one value for every output quantity. For very large codes of interest here, one mapping, i.e. one execution of the code, could require hours or days on even the world's fastest computers. To be of value to decision makers using the results of the code for risk, reliability, or performance assessment, hundreds or thousands of executions of the code are usually required.

For realistic systems, one must include uncertainties of various types in the mathematical model of the system. Uncertainty could occur in parameters in the mathematical model, there could be uncertainty in the accuracy of the mathematical model to describe the system of interest, or in the sequence of possible events that could occur in a discrete event system. During the last 10 or so years, the risk assessment community has begun to make a clear distinction between aleatory and epistemic uncertainty. Aleatory uncertainty is also referred to in the literature as variability, irreducible uncertainty, inherent uncertainty, and stochastic uncertainty. We use the term aleatory uncertainty to describe the inherent variation associated with the physical system or the environment under consideration. Epistemic uncertainty is also termed reducible uncertainty, subjective uncertainty, and model form uncertainty. Epistemic uncertainty derives from some level of ignorance, or incomplete information, of the system or the surrounding environment.

Methods to efficiently represent, aggregate, and propagate different types of uncertainty through computational models are clearly of vital importance. The most widely known and developed methods are available within the mathematics of probability theory, whether frequentist or Bayesian estimation. Newer mathematics, which extend or otherwise depart from probability theory, are also available, and are known collectively by the name generalized information theory (GIT). For example, possibility theory, fuzzy set theory, and evidence theory are three components of GIT (see, for example, Refs. [1], [2], [3], [4]).

Assessing and comparing the multiple methods available from probability theory, GIT, and other methodologies has proven to be quite difficult. Moreover, the communities dedicated to risk assessment, reliability engineering, and GIT have not communicated extensively across these disciplines to try to develop a common understanding of the relative advantages and disadvantages of these available methods for problems of different types.

The purpose of this paper is to encourage a dialog between the risk assessment, reliability engineering, and GIT communities on the subject of uncertainty representation, aggregation, and propagation. Our emphasis is on epistemic uncertainty and mixtures of epistemic and aleatory uncertainty. We believe this emphasis is most appropriate because representation, aggregation, and propagation of aleatory uncertainty is well established using traditional probability theory. The chosen mechanism to encourage the dialog is two-fold:

• Two specific mathematical model systems are described in Section 4. The first is a simple algebraic system, which is, nonetheless, of sufficient complexity to engender significant unanswered questions. The second is a somewhat more complex, but still analytically solvable, dynamical system. It is intended that these two systems serve as simplified models of the kinds of systems that are our real focus.

• The workshop was sponsored by Sandia National Laboratories in the summer of 2002. National and international leaders of the three communities indicated above, as well as other interested individuals, were invited to participate in the workshop. The two problem sets are intended to serve as a common focus for experimentation, discussion, and comparison of mathematical approaches.

In Section 2, we outline our view of the role of uncertainty representation, aggregation, and propagation within the overall context of computational analysis support for decisions. In Section 3, we briefly discuss the various forms of uncertainty present in such systems, and briefly mention some of the mathematical representations available from the probabilistic, GIT, and other communities. In Section 4, we describe the two problem sets.