User acceptance of multi-criteria decision support systems: The impact of preference elicitation techniques

Abstract

Previous research indicates that decision makers are often reluctant to use potentially beneficial multi-criteria decision support systems (MCDSS). Prior research has not examined the specific impact of preference elicitation techniques on user acceptance of MCDSS. The present research begins to fill this gap by examining the effect on users’ MCDSS evaluations of two commonly used preference elicitation techniques, absolute measurement and pairwise comparisons, while holding constant all other aspects of the MCDSS and decision making task. Experimental results (N = 153) indicate that users consider MCDSS with pairwise comparisons to be higher in decisional conflict, more effortful, less accurate, and overall less desirable to use than MCDSS with absolute measurements. Thus, any potential normative superiority of a preference elicitation technique must be balanced against its potentially adverse effects on user acceptance of the MCDSS within which it is employed. We present a research agenda for exploring the tradeoffs between objective validity and user acceptance in the design of decision analysis tools.

Introduction

Over several decades, significant technical advances have been made in the design of multi-criteria decision support systems (MCDSS) (see Siskos and Spyridakos (1999) for a comprehensive survey of the technical advances and Maxwell (2002) for a survey of commercial decision analysis software). There is conflicting evidence on how widely MCDSS are used in practice. Some argue that MCDSS are not readily accepted and used (Evans, 1984, Whyte and Latham, 1997, Limayem and DeSanctis, 2000), while others claim that users’ evaluations of MCDSS are generally quite favorable (Timmermans and Vlek, 1994). Despite these differences, observers concur that user evaluations vary from one specific MCDSS to another. Often mentioned is the observation that, given a choice, decision makers appear to prefer relatively unsophisticated MCDSS (Wallenius, 1975, Brockhoff, 1985, Buchanan and Daellenbach, 1987, Kottemann and Davis, 1991, Olson et al., 1995) and even prefer self-generated ad hoc solutions over those generated by MCDSS (Narasimhan and Vickery, 1988). These studies suggest that we do not have a complete understanding of what influences decision makers’ acceptance of MCDSS. A normatively superior MCDSS will only lead to improved decision making if the decision maker evaluates it favorably (on evaluative criteria linked to actual user adoption behavior) and is willing to use it. Dyer et al. (1992) call for researchers to bring behavioral and psychological insights to bear on the design of MCDSS. There has been very little investigation to date into how specific design features influence potential users’ evaluations on MCDSS. This significant gap in the current literature is the primary motivation for the present study.

The preference elicitation technique (e.g., absolute measurement, pairwise comparisons, ordinal judgments, rankings, matching, choice) employed within an MCDSS is a key element of that MCDSS. However, prior research has not examined the effect of the preference elicitation technique on users’ evaluations of MCDSS. Kottemann and Davis (1991) theorize that preference elicitation techniques embodied in MCDSS differ in the degree to which they require users to make explicit tradeoff judgments, which in turn cause a negative affective state in the decision maker called “decisional conflict”. Such decisional conflict is theorized to adversely influence users’ perceptions and preferences regarding the use of associated MCDSS. In this study, we will investigate the specific effects of the preference elicitation technique on decisional conflict and user acceptance criteria (i.e. perceived effort, perceived accuracy, and overall preference) for MCDSS.

To concretely illustrate our research question, consider two examples of leading commercial MCDSS software packages available today: Logical Decisions and Expert Choice. Expert Choice employs the analytical hierarchy process (AHP) (Saaty, 1980), while Logical Decisions supports AHP along with several other MCDA formalisms. Thus, it would be possible use AHP and pairwise comparisons with either MCDSS. As one of its alternate modes, Logical Decisions supports a direct absolute measurement method: after defining the attributes and alternatives for a decision situation, users assign preference values to a matrix formed by the intersection of the attributes and the alternatives, and then specify the weights for each attribute. Expert Choice uses pairwise comparisons among alternatives to elicit attribute preferences, giving the user the option of horizontal, vertical, or pie-chart shaped response scales and qualitative or quantitative response category anchors. Our examination of these two MCDSS reveals many striking differences in their user interfaces and functionality apart from the preference elicitation techniques employed. For example, Expert Choice gives the user feedback in the form of a consistency check and the option to perform sensitivity analysis on the results, whereas Logical Decisions is relatively unconstrained in that it allows the user to enter values falling outside of their own user-defined ranges. The two MCDSS require the user to navigate very different complex sets of menu options in order to accomplish a decision task. Thus, in addition to differences in the preference elicitation techniques employed, there are many differences between Logical Decisions and Expert Choice that could be responsible for any observed differences in users evaluations of the two MCDSS. Our research question concerns the specific effect of preference elicitation techniques on user evaluations of MCDSS, holding constant all of these various other kinds of differences from one MCDSS to another.

Our review of prior literature below will demonstrate that the effect of preference elicitation techniques on users’ evaluations of MCDSS has not been previously investigated. In particular, we will review two major groups of prior research:

• Research comparing different MCDSS on various criteria spanning user evaluations and normative properties. A given MCDSS consists of a bundle of design features, any of which may influence users’ evaluations of that system, including the preference elicitation technique, the appearance and format of the user interface, feedback and feedforward during the decision process, the format in which the user receives the system recommendations, the availability of sensitivity analysis, external information regarding the capabilities of the system, and others. Differences observed between two MCDSS might be due to any of several features that differ between them, but these studies were not designed to isolate the effects of specific features. Therefore the effects of the specific preference elicitation technique are confounded among all of the other features that differ between the MCDSS being compared.

• Research investigating the specific effects of preference elicitation techniques. Extensive prior research has addressed the impact of alternative preference elicitation techniques on normative properties of parameters elicited from decision makers and the resulting effects on objective decision outcomes such as consistency, reliability, and preference reversals. The research generally concludes that preference elicitation techniques requiring decision makers to make direct tradeoff judgments between attributes (e.g., pairwise comparisons) are normatively superior. However, prior research has overlooked the specific effects of preference elicitation techniques on user evaluations of the MCDSS that employ them. This is a serious omission, since any normatively superior features will fail to deliver realized improvements in decision making if users are not willing to use and rely on the advice of MCDSS.