Sensitivity analysis for product design selection with an implicit value function

Abstract

In product design selection the decision maker (DM) often does not have enough information about the end users’ needs to state the “preferences” with precision, as is required by many of the existing selection methods. We present, for the case where the DM gives estimates of the preferences, a concept for calculating a “robustness index.” The concept can be used with any iterative selection method that chooses a trial design for each iteration, and uses the DM’s preference parameters at that trial design to eliminate some design options which have lower value than the trial design. Such methods, like our previously published method, are applicable to cases where the DM’s value function is implicit. Our robustness index is a metric of the allowed variation between the actual and estimated preferences for which the set of non-eliminated trial designs (which could be singleton) will not change. The DM, through experience, can use the robustness index and other information generated in calculating the index to determine what action to take: make a final selection from the present set of non-eliminated designs; improve the precision of the preference estimates; or otherwise cope with the imprecision. We present an algorithm for finding the robustness index, and demonstrate and verify the algorithm with an engineering example and a numerical example.

Introduction

Existing multi-attribute decision making (MADM) or selection methods (see, for example, Olson, 1996, Yu, 1985, Lootsma, 1999) generally assume that the decision maker (DM) has in mind a value function that he/she maximizes to make the selection. (Conventionally, the term “value” is used when the attributes are deterministic, and the term “utility” when the attributes are stochastic (Keeney and Raiffa, 1976). We consider here only deterministic attributes, and use the term value in the remainder of the paper.) These methods estimate the value function by obtaining from the DM information about quantitative “preferences” which reflect the value function. Various selection methods take the preferences in various forms, e.g., relative importance of attributes (Lootsma, 1999, Saaty, 1980), comparison of design alternatives (Koksalan et al., 1984, Malakooti, 1988), or “marginal rate of substitution” between attributes (Keeney and Raiffa, 1976, Maddulapalli et al., 2005, Yu, 1985).

In general, when the DM gives the preferences, in addition to design requirements (e.g., constraints on the size, price), he/she attempts to satisfy the needs of end users or customers (e.g., a professional user of cordless power tool prefers to have more operations per battery charge, whereas a casual user prefers lower cost) (Urban and Hauser, 1993). So, if the DM does not have complete information about the end users’ needs, he/she cannot state the preferences precisely (Insua and French, 1991). The DM might also have to project into future markets. In cases with such uncertainty the DM can give only estimates of the actual preferences. Since small variations in preferences could lead to a significant change in the set of preferred designs (which might be singleton) (Korhonen et al., 1992, White, 1972), it would be useful for the DM to have an idea about the “robustness” of the set of preferred designs with respect to variation in the preference estimates (Hannan, 1981, Korhonen et al., 1992). By robustness we mean the amount of change (or variation) allowed between the actual preferences and the preference estimates before the set of preferred designs is affected (i.e., gains additional member(s)).

Finding the degree of robustness of the preferred designs to preference variation is generally referred to as sensitivity analysis in the literature (Insua and French, 1991). One advantage of sensitivity analysis is that if the DM is confident that the set of preferred designs is sufficiently robust, then he/she can select from that set. Alternatively, using sensitivity analysis, the DM can get an idea of what other designs might be preferred if the preference variation increases.

Existing literature in sensitivity analysis addresses cases where the DM’s value function is presumed to be explicitly known (e.g., known polynomial function of attributes with unknown parameters (Keeney and Raiffa, 1976, Lootsma, 1996). Sage (1981) studied and formalized the allowed errors in the estimation and elicitation of probabilities and utilities before which the preferred design is affected. Barron and Schmidt (1987) proposed two procedures: entropy-based and least square (i.e., L₂-metric) to calculate the minimum variation required between the actual weights and the estimates of weights to change the most preferred design when the value function is linear. Ringuest (1997) later extended the L₂-metric of Barron and Schmidt (1987) to an L_P-metric. Mareschal (1988) proposed an approach for finding the “weight stability interval”, which consists of all possible weights that maintain the rank order obtained using the original estimates of weights.

Insua and French (1991) proposed some distance based tools to identify the possible competitors to the current most preferred design when the DM’s preferences change. Antunes and Climaco (1992) proposed a sensitivity analysis approach for their TRIMAP method. However, this approach is applicable only when the number of attributes is three or less, which is a significant limitation. Triantaphyllou and Sanchez (1997) proposed a sensitivity analysis approach and applied it to popular MADM methods like weighted sum model, weighted product model, and analytical hierarchy process (Saaty, 1980). Ma et al. (2001) presented a method for finding the “weight-set” that contains all possible ranges of weights of an additive value function when the rank order of alternatives is given. Triantaphyllou and Shu (2001) studied the number of feasible rankings that are possible, assuming an additive value function, for the given set of design alternatives, when the weights of the criteria are allowed to change.

Although MADM literature describes significant research on sensitivity analysis when the value function is presumed, it is well known that presuming a form for the value function is restrictive and applicable only to special cases (Maddulapalli et al., 2005, Malakooti, 1988, Thurston, 2001). In this paper we present a broadly applicable concept for calculating a robustness index for a set of preferred designs. The concept can be used with any iterative selection scheme that chooses a trial design for each iteration, and uses the DM’s estimates of preference parameters at that trial design to eliminate some design options which have lower value than the trial design (Maddulapalli et al., 2005, Malakooti, 1988). Such schemes are, in general, applicable to cases where the DM’s value function is implicit. The output of such schemes is called the set of non-eliminated trial designs.

Our robustness index is a metric of the allowed preference variation for which the set of non-eliminated trial designs will not change (i.e., all eliminated designs remain eliminated). The DM, through experience, can decide if the robustness index of the set is high enough to make a final selection from the present set, or can improve the precision of the preferences, or otherwise cope with the imprecision.

We present our sensitivity analysis concept and an implementation of it using our previously published (Maddulapalli et al., 2005) iterative method for design selection with an implicit value function that need be only differentiable, quasi-concave and non-decreasing with respect to the attributes. Our concept can also be used with iterative methods that use DM’s preferences to estimate the unknown parameters of the presumed value function (e.g., Cobb–Douglas function (Takayama, 1993)).

The organization of this paper is as follows. In Section 2 we provide some definitions for the main terminology used in the paper. In Section 3 we present an overview of our concept for finding a robustness index. Section 4 describes the implementation of the concept using our design selection method (Maddulapalli et al., 2005). Next, in Section 5 we discuss our algorithm for finding the robustness index. In Section 6, we demonstrate the application of our method with the help of two examples. Finally we conclude with a summary in Section 7.