Decision Support
Convergence properties and practical
estimation of the probability of rank
reversal in pairwise
comparisons for multi-criteria decision
making
problems
Introduction
Decision making (Bhushan and Rai, 2004, Yager, 2004) consists of choosing a specific course of action between several alternatives and is encountered in countless areas of human activity. In many circumstances, where complex decisions need to be made involving high stakes, it is desirable to proceed in a structured and methodological manner, rather than simply rely on the skills and intuition of a single decision maker. Multi-criteria decision analysis (MCDA) or multi-criteria decision making (MCDM) (Triantaphyllou, 2000) aim at facilitating decision makers in complicated situations where numerous and sometimes conflicting criteria or factors have to be taken into account.
MCDM is further classified into multi-objective decision making (MODM) and multi-attribute decision making (MADM) (Pohekar & Ramachandran, 2004). In MODM, a set of objective functions is optimized subject to constraints and hence the efficient solutions in a set of alternatives are sought. MODM typically requires the solution of a series of mathematical programming models in order to reveal implicitly defined efficient solutions. On the other hand, in MADM, a small number of pre-determined alternatives are to be evaluated under a common set of criteria and the best alternative is usually selected by making comparisons between alternatives with respect to each criterion.
A fundamental problem in decision making is to grade the importance of a set of alternatives and assign a weight to each of them. The importance of alternatives usually depends on several criteria which can be evaluated within the decision making framework in which pairwise comparisons (PWC) are an essential ingredient (Saaty & Vargas, 2001). In the context of MADM, PWC enables the ranking of alternatives by allowing the experts to compare the various criteria or alternatives in pairs instead of assigning their priorities in a single step (Saaty, 1977). This reduces the influence of subjective point of views, associated with eliciting weights directly. PWC is usually performed in MADM methods such as the Analytic Hierarchy Process (AHP) (Saaty, 2003), the Weighted Product Method (WPM) (Chang & Yeh, 2001), the preference ranking organization method for enrichment evaluation (PROMETHEE) (Brans, Vincke, & Mareschal, 1986), the Analytic Network Process (ANP) (Saaty, 2004) and so on. The aim of this paper is to consider the influence of uncertainty in PWC and how the credibility of the outcome can be assessed. Although the focus is on PWC alone, the results have some implications on the applications of MADM frameworks, particularly in the number of experts that are required and how the credibility of key parts of the framework can be ascertained.
In recent years, PWC has been used either as a stand-alone method or as part of complex MADM frameworks on several areas including government (Huanga, Chub, & Chiang, 2008), business (Lee & Kozar, 2006), industry (Chan, Lau & Ip, 2006), healthcare (Liberatore & Nydick, 2008), technology (Gerdsri & Kocaoglu, 2007), education (Zahedi, 1986), communications (Dede, Kamalakis, & Varoutas, 2011a, 2011b; Dede, Varoutas, Kamalakis, Fuentetaja, & Javaudin, 2010), agriculture (Abildtrup et al., 2006) and energy planning (Kok & Lootsma, 1985). The method itself has also been the focus of extensive research in the field of decision making. Recently in Fan and Liu (2010) , a form of uncertain preference information, called ordinal interval numbers, was used in pairwise comparisons in order to rank the alternatives. In Doumpos and Zopounidis (2004) , the issue of how pairwise comparisons can be used for the classification of alternatives in different classes of preference is discussed. The problem of deriving the weights from the pairwise comparison matrices using several alternative approaches is studied in Barzilai (1997) and Choo (2004) . In Kwiesielewicz and Van Uden (2004) several issues concerning the inconsistency of the pairwise comparison matrix and its impact on the decision making process are highlighted. In addition, fuzzy pairwise comparison for solving the decision making problems has been proposed in Boenderb, Graan, and Lootsma (1989), Deng (1999) and Mikhailov (2005) . In Marimin, Umano, Hatono, and Tamura (1998) linguistic labels are used in order to express fuzzy preference relations in pairwise decision problems. Moreover, Shiraishi, Tsuneshi, and Motomasa (1998) dealt with the properties of the principal eigenvector of PWC matrices.
The influence of uncertainty due to the imperfect and subjective expert judgments is of paramount importance when considering the credibility of the outcome of a decision making process. Several studies have attempted to shed some light on this issue in the context of PWC. For example, in Carmone, Karab, and Zanakis (1997), Monte Carlo simulations were performed to study the Incomplete Pairwise Comparisons (IPC) algorithm and investigate the effect of missing information in pairwise comparisons. Furthermore, in Aull-Hyde, Erdogan, and Duke (2006) it has been shown that given a sufficiently large group size, the consistency of the aggregate comparison matrix is guaranteed regardless of the measures used to estimate the consistency of the individual matrices, if the geometric mean method is used to estimate the weights. Moreover, in Hahn (2003) a stochastic characterization of the pairwise comparison judgments is provided, while statistical models for deriving the weights of the alternatives using Markov chain Monte Carlo are also presented. Furthermore, Farkas (2007) theoretically studied the conditions for rank reversal on perturbing the PWC matrices, while Chen and Kocaoglu (2008) also studied the rank reversal problem in this particular context, and came up with an algorithm to analyze the sensitivity of hierarchical decision models.
The main purpose of our work is to provide a suitable characterization of the impact of uncertainty in PWC. A first step in order to characterize the impact of uncertainty in PWCs, is to identify a suitable measure for quantifying its effects. Assume for instance that N different alternatives are pairwise compared by M experts, each with possibly a different view on the ranking of the alternatives. As discussed further below in Section 2.1, PWC aims at providing an average ranking, encompassing all these diverse opinions of the experts. It is of course natural to expect that the credibility of the overall process will be increased as the size of the expert group increases. Therefore, one possible way of measuring the trustworthiness of the results is to define the probability of rank reversal (PRR) (Saaty & Vargas, 1984) as follows: Let W1, …, WN be the weights calculated by the PWC in the case of a very large group of experts (M → ∞). In a practical situation where M is finite, uncertainty may undermine the PWC and the calculated weights wk may turn out different than Wk. Uncertainty can be due to the difference of opinion among the experts or inconsistent pairwise comparisons. The probability of rank reversal is formally defined as: A high PRR implies that the outcome of the PWC in question is not trustworthy and could therefore lead to incorrect decision making. There are two important issues that need to be addressed concerning PRR:
- a)
How does PRR relate to the number of experts M? An obvious way to reduce the effect of uncertainty is simply to increase M, but from a practical point of view, this is not a trivial task. It is usually difficult to locate many experts within a single organization or even in the wider public with sufficient expertise that would be willing to participate in the PWC surveys. On the other hand, there is no clear answer to the question of how many more experts need to participate in order to considerably reduce the uncertainty of the outcome. Say for example, that there are already M = 10 experts participating in the endeavor. How much is there to be gained in terms of PRR by doubling or even quadrupling the size of the group of experts?
- b)
How can PRR be estimated from actual expert judgments in practice? When applying PWC in a specific decision making problem, one has access to the elements of a limited number of pairwise comparison matrices P(m) (where 1 ≤ m ≤ M). So the question becomes whether one can extract any kind of information regarding PRR and hence the credibility of the results based on just the elements of P(m).
The present paper attempts to deal with both points above. We first discuss a model for incorporating uncertainty in PWC and consider a suitable measure for quantifying the uncertainty level. We then discuss how PRR varies with the group size M depending on the uncertainty level and extract several interesting conclusions from this variation. It is shown that there is not much sense in using more than M = 15 experts in the decision making process because the rate of decrease of PRR is already small for M > 15. We then address the issue of how PRR can be estimated from just the values of the pairwise comparison matrices P(m) obtained by the experts. Given this information, we discuss a numerical method for estimating PRR based on Monte Carlo simulation. The results indicate that for a sufficiently large group of experts, one can obtain a reasonable approximation to the actual value of PRR.
The rest of the paper is organized as follows: In Section 2, we briefly summarize the PWC method laying the theoretical foundations for incorporating uncertainty and present the model used in our simulations. In Section 3, the results obtained when applying the proposed model are presented and the convergence of the PRR is examined, considering the impact of various aspects which may affect the estimation of probability of rank reversal, such as varying uncertainty level among the experts, alternative preference scales and weight estimation methods. We also consider the case where the judgments are determined in a fuzzy manner. In Section 4 a numerical method is proposed in order to estimate the PRR from the actual user judgments. Finally, some concluding remarks are presented in Section 5.
Section snippets
Uncertainty modeling in PWC
In this section, we introduce the model used for incorporating uncertainty in the pairwise comparison matrices. Section 2.1 provides a brief overview of the pairwise comparison framework, while Section 2.2 considers the effects of uncertainty-induced perturbations under general assumptions. Section 2.3 details the uncertainty model adopted in this paper. In Section 2.4 we describe how we can estimate the probability of rank reversal through Monte Carlo simulation.
Numerical analysis and results
In this section, we present the results obtained when we apply the framework described above in Section 2. Section 3.1 presents some numerical results that further validate the first order analysis of Section 2.2, while Section 3.2 examines the convergence properties of PRR.
Estimation of probability of rank reversal from actual user data
The previous analysis dealt with the impact of uncertainty in the evaluation of the priorities of the alternatives, quantified in terms of PRR. We now consider the issue of estimating PRR from actual user data, i.e. a relatively small number of pairwise comparison matrices P(m) obtained by a single expert group. In this section, we propose a numerical method to achieve this and validate our approach with numerical simulations.
Conclusions
In this paper, we applied an uncertainty model to address two important issues concerning the probability of rank reversal in pairwise comparisons. The first issue concerned how this probability is reduced by augmenting the number of experts participating in the surveys. The results dictate that there is not much to be gained by increasing the number of experts beyond 15, even if uncertainty level is large. Using perturbation theory we have argued that the convergence of the probability of rank
References (50)
- et al.
Socio-economic scenario development for the assessment of climate change impacts on agricultural land use: A pairwise comparison approach
Environmental Science & Policy
(2006) - et al.
An experiment on the consistency of aggregated comparison matrices in AHP
European Journal of Operational Research
(2006) - et al.
How to select and how to rank projects: The Promethee method
European Journal of Operational Research
(1986) - et al.
A Monte Carlo investigation of incomplete pairwise comparison matrices in AHP
European Journal of Operational Research
(1997) - et al.
Evaluating airline competitiveness using multiattribute decision making
Omega
(2001) - et al.
A sensitivity analysis algorithm for hierarchical decision models
European Journal of Operational Research
(2008) - et al.
A common framework for deriving preference values from pairwise comparison matrices
Computers & Operations Research
(2004) - et al.
A multicriteria classification approach based on pairwise comparisons
European Journal of Operational Research
(2004) - et al.
Applying the Analytic Hierarchy Process (AHP) to build a strategic framework for technology roadmapping
Mathematical and Computer Modelling
(2007) - et al.
Extension principles of fuzzy set theory
Journal of Information Sciences
(1998)
Review of the main developments in the analytic hierarchy process
Expert Systems with Applications
Pairwise-comparison methods in multiple objective programming, with applications in a long-term energy-planning model
European Journal of Operational Research
Inconsistent and contradictory judgments in pairwise comparison method in the AHP
Computers & Operations Research
Investigating the effect of website quality on e-business success: An analytic hierarchy process (AHP) approach
Decision Support Systems
The analytic hierarchy process in medical and health care decision making: A literature review
European Journal of Operational Research
Application of multi-criteria decision making to sustainable energy planning—A review
Renewable and Sustainable Energy Reviews
A scaling method for priorities in hierarchical structures
Journal of Mathematical Psychology
Eigenvector and logarithmic least squares
European Journal of Operational Research
Decision-making with the AHP: Why is the principal eigenvector necessary?
European Journal of Operational Research
Inconsistency and rank preservation
Journal of Mathematical Psychology
Uncertainty and rank order in the analytic hierarchy process
European Journal of Operational Research
A modified fuzzy logarithmic least squares method of fuzzy analytic hierarchy process
Fuzzy Sets and Systems
Deriving weights from pairwise comparison matrices
Journal of the Operational Research Society
Strategic decision making: Applying the analytic hierarchy process
Multi-criteria decision analysis with fuzzy pairwise comparisons
Fuzzy Sets and Systems
Cited by (0)
- 1
Tel.: +302109549286.