Implausible alternatives in eliciting multi-attribute value functions

Highlights

• We analyze the impact of implausible alternatives in utility elicitation.

• Utility functions are evaluated by their consistency with plausible requirements.

• An innovative LP based approach is developed to measure consistency.

• We find that subjects evaluating implausible alternatives are more consistent.

• Possible explanations for these results are offered.

Abstract

Many methods to elicit preference models in multi-attribute decision making rely on evaluations of a set of sample alternatives by decision makers. Using orthogonal design methods to create this set of alternatives might require respondents to evaluate unrealistic alternatives. In this paper, we perform an empirical study to analyze whether the presence of such implausible alternatives has an effect on the quality of utility elicitation. Using a new approach to measure consistency, we find that implausible alternatives in fact, have a positive effect on consistency of intra-attribute preference information and consistency with dominance, but do not affect inter-attribute preference information.

Introduction

Many approaches to multi-attribute decision making attempt to infer a decision maker’s preferences from the direct evaluations of a set of sample alternatives presented to him or her. Several approaches of this type have been developed independently in different literature streams. Some methods use statistical regression techniques (Schoemaker & Waid, 1982) to estimate weights of additive multi-attribute utility functions from cardinal scores assigned to the sample alternatives. Most approaches, however, are based on pairwise comparisons or a ranking of alternatives. Such approaches are widely considered to be even less demanding in terms of cognitive effort than providing scores. A very early reference to such a method is Srinivasan and Shocker (1973). One of the best known techniques using this concept is the UTA method of Jacquet-Lagreze and Siskos (1982), which is the basis of several similar methods collectively known as preference disaggregation methods (Jacquet-Lagreze & Siskos, 2001). The Linmap method (Horsky & Rao, 1984) is also based on holistic comparisons of alternatives and uses a linear programming model to estimate attribute weights. Similarly, case-based methods, which infer preferences from a set of examples, have become popular in multi-attribute sorting (Chen et al., 2007, Doumpos and Zopounidis, 2004). Recently developed methods to calculate a representative value function (Greco, Kadzinski, & Slowinski, 2011) for decision models under incomplete information (Greco, Mousseau, & Slowinski, 2008) also follow a similar approach. In marketing, the method of Conjoint Analysis was developed to model consumer preferences (Green & Srinivasan, 1978). Conjoint Analysis is based on an additive utility model, and some approaches also use holistic comparisons between alternatives to infer preference parameters. Over the last decades, Conjoint Analysis has evolved into one of the most widely used techniques to investigate consumer preferences (Eggers and Sattler, 2011, Green et al., 2001).

All these techniques rely on the responses of decision makers on a set of alternatives (stimuli). Often, the fact that respondents holistically evaluate entire alternatives, rather than having to specify detailed information on single parameters, is described as a major advantage of such methods. Given the central role of sample alternatives in these methods, it is surprising that the question which alternatives to present to the decision maker is rarely addressed in literature. Usually, the selection of those alternatives is only described in rather general terms, e.g. as alternatives used in previous decisions, or even more general as “fictitious alternatives … which can easily be judged by the decision maker” (Jacquet-Lagreze & Siskos, 2001, p. 235/236).

While the decision making literature thus rarely has investigated the question which alternatives to present in decompositional preference elicitation methods, similar questions have received more attention in the literature on Conjoint Analysis. Given the strong statistical focus of marketing methods, the optimal design of stimuli sets is an important topic in that literature (e.g., DeSarbo et al., 1985, Green and Helsen, 1989).

Based on statistical considerations concerning the efficiency of parameter estimation, orthogonal designs are often recommended. Since orthogonal designs require the evaluation of a large number of alternatives, methods to reduce the required number of comparisons have been developed. These methods minimize the total number of questions to be asked (Holloway & White, 2003) or formulate comparisons which allow to “cut off” particularly large subsets of the remaining parameter space (Toubia, Hauser, & Simester, 2004). However, these methods usually assume that attributes are not correlated and determine the questions to be asked from a technical perspective, while the perspective of the decision maker is rarely addressed.

In real-world decision problems, alternatives often exhibit a correlation of attributes, which is caused by the trade-offs inherent to existing alternatives. For example, when selecting a car, higher speed or a stronger engine usually are accompanied by a higher fuel consumption. Any other attribute combination would be implausible. Another example for an implausible alternative would be a low-priced laptop computer with the fastest processor on the market. However, orthogonal designs (or fractional designs created by an optimization approach) usually ignore such real world correlations, and thus might require the subject to evaluate alternatives which are unlikely to exist in reality (Moore & Holbrook, 1990).

This problem was also recognized in the literature on Conjoint Analysis (e.g., Green, Helsen, & Shandler, 1988). Methods were developed to create optimal designs which avoid such alternatives (Hair, Black, Babin, & Anderson, 2010), and some empirical studies were conducted to investigate whether or not the presence of implausible alternatives has a negative impact on the preference models estimated. While these contributions offer only limited insights into the role of implausible alternatives in Conjoint measurement, they nevertheless highlight the importance of the topic.

Conjoint Analysis and multi-attribute decision models differ in the goals of preference elicitation. In Conjoint Analysis, preferences are elicited in order to predict consumer behavior in future purchases, and thus the predictive abilities of the model over an extended time span are a major concern. In contrast, multi-attribute decision models elicit preferences in order to support decision makers to make complex decisions in a way that is consistent with their preferences. Therefore, consistency is a major concern when preferences are elicited for this purpose.

Due to the focus on the models’ predictive ability, most existing studies on the effects of implausible alternatives in Conjoint Analysis use a two-stage design, in which choices predicted by the elicited model are compared to later choices (of similar complexity) made by consumers. Since the model is used for prediction of consumer behavior, respondents are not aware of the elicited model when they make their choice in the second stage.

In contrast, preference elicitation for multi-attribute decision models is mainly concerned with obtaining a consistent representation of the decision maker’s preference at the time the elicitation is performed. This representation is then used by decision makers themselves to solve more complex problems. Inconsistencies in responses, although they can be corrected by statistical means, are seen as an indicator that the decision maker is not sure about his or her preferences, or is confused by the questions being asked. This could impede the reliability of the elicitation. Therefore, consistency of the responses is a major concern, while the ability of the model to predict choices a considerable time later is not relevant for this application. Therefore, we focus on immediate results rather than on choices in a later stage.

Analyzing the consistency of responses requires to develop a method to determine compatibility of subjects’ choices with plausible assumptions about preference parameters. We introduce a new approach, and apply it in an experimental setting to study the impact of implausible alternatives on the elicitation process. More specifically, in our research we want to identify negative effects of implausible stimuli directly on the elicitation process and the estimated parameters.

The remainder of the paper is structured as follows: Section 2 provides a literature review, based mainly on literature in Conjoint Analysis, on possible effects of implausible alternatives in preference elicitation. In Section 3, we use these results to formulate research questions for our empirical study, which is described in Section 4. Section 5 describes the model we use to test compatibility of parameters with plausible assumptions, and Section 6 presents the empirical results. Section 7 concludes the paper by summarizing its main results and providing an outlook on future research.