Paired comparisons analysis: an axiomatic approach to ranking methods

Abstract

In this paper we present an axiomatic analysis of several ranking methods for general tournaments. We find that the ranking method obtained by applying maximum likelihood to the (Zermelo-)Bradley-Terry model, the most common method in statistics and psychology, is one of the ranking methods that perform best with respect to the set of properties under consideration. A less known ranking method, generalised row sum, performs well too. We also study, among others, the fair bets ranking method, widely studied in social choice, and the least squares method.

A Proofs of Propositions 6.4 and 6.5

We start with an auxiliary result that is crucial in the proof for both fair bets and least squares.

Lemma 8.1

Let $B\in \mathbb R ^{n\times n}$ be such that

• (i) $B$ is invertible,

• (ii) for all $i\ne j,\, B_{ij}\le 0$, and

• (iii) $\sum _{j=1}^n B_{ji}\ge 0$ for all $i\in \{1,\dots ,n\}$.

If $\gamma ,\uplambda \in \mathbb R ^n$ are two vectors such that $\uplambda$ is nonnegative and $B\gamma =\lambda$, then $\gamma$ is nonnegative.

Proof

It follows from the assumptions that $B$ is an M-matrix (see, for instance, Berman and Plemmons (1994)). As a result, $B^{-1}$ is non-negative, from which the result follows immediately. $\square$

Proof of Proposition 6.4

Let $i,j,k\in N$ be as in the statement. Below we explicitly characterise how the recursive Buchholz ratings vary as a function of $A_{ik}$ and $A_{ki}$, provided that $M_{ik}$ stays constant. Recall that $r^{\text{ rb}}$ is the unique solution of $\bar{M}x+\hat{s}=x$ such that $(r^{\text{ rb}})^{\top }e=0$. Hence, $(I-\bar{M})r^{\text{ rb}}=\hat{s}$. Define $B=I-\bar{M}$ and $\breve{N}=N\backslash \{i,k\}$. Then, equation $\ell$ of the system $Br^{\text{ rb}}=\hat{s}$ can be written as

$$\begin{aligned} \sum _{h\in \breve{N}}B_{\ell h}r^{\text{ rb}}_h=\hat{s}_{\ell }-B_{\ell i}r^{\text{ rb}}_i-B_{\ell k}r^{\text{ rb}}_k. \end{aligned}$$

(8.1)

with $\ell \in \breve{N}$. Define $\breve{B}\in \mathbb R ^{(n-2)\times (n-2)}$ to be the matrix obtained from $B$ by deleting the rows and columns corresponding to players $i$ and $k$.

We prove now that $\breve{B}$ is invertible. Suppose, on the contrary, that there is an $y\in \mathbb R ^{n-2},\, y\ne 0$, such that $y^{\top }\breve{B}^{\top }=0$. Let $\ell \in \breve{N}$ be such that $y_{\ell }=\max _{h\in \breve{N}}y_h$. We assume, without loss of generality, that $y_{\ell }>0$. For each $h\ne \ell ,\, B^{\top }_{h\ell }\le 0$ and, hence, $-y_hB^{\top }_{h\ell }\le -y_{\ell }B^{\top }_{h\ell }$, with equality only if $y_h=y_{\ell }$ or $B^{\top }_{h\ell }=0$. Since $y^{\top }\breve{B}^{\top }=0,\, \sum _{h\in \breve{N}\backslash \{\ell \}} -y_hB^{\top }_{h\ell }=y_{\ell }B^{\top }_{\ell \ell }$. Further, since $\sum _{h\in N}B^{\top }_{h\ell }=0$, we have $\sum _{h\in \breve{N}\backslash \{\ell \}}-B^{\top }_{h\ell }=B^{\top }_{\ell \ell }+B^{\top }_{i\ell }+B^{\top }_{k\ell }\le B^{\top }_{\ell \ell }$, with equality only if $B^{\top }_{i\ell }=B^{\top }_{k\ell }=0$. Then, we have

$$\begin{aligned} y_{\ell }B^{\top }_{\ell \ell }=\sum _{h\in \breve{N}\backslash \{\ell \}}-y_hB^{\top }_{h\ell }\le y_{\ell }\sum _{h\in \breve{N}\backslash \{\ell \}}-B^{\top }_{h\ell }\le y_{\ell }B^{\top }_{\ell \ell } \end{aligned}$$

and, hence, all the inequalities are indeed equalities. Therefore, $B^{\top }_{i\ell }=B^{\top }_{k\ell }=0$ and, for each $h\in \breve{N}\backslash \{\ell \},\, y_h=y_{\ell }$ or $B^{\top }_{h\ell }=0$. Define $\bar{N}=\{m\in \breve{N}\,|\,y_m=\max _{h\in \breve{N}}y_h\}$. Now, for each $m\in \bar{N}$, we have $B^{\top }_{im}=B^{\top }_{km}=0$ and, further, for each $h\in \breve{N}\backslash \bar{N},\, B^{\top }_{hm}=0$. That is, no player outside $\bar{N}$ has played against players inside $\bar{N}$, which contradicts the irreducibility of $A$.

Define $C=(\breve{B})^{-1},\, \breve{r}^{\text{ rb}}=(r^{\text{ rb}}_h)_{h\in \breve{N}},\, B^i=(B_{hi})_{h\in \breve{N}}$ and $B^k=(B_{hk})_{h\in \breve{N}}$. Then, using (8.1) we have $\breve{B}\breve{r}^{\text{ rb}}=\breve{s}-B^ir^{\text{ rb}}_i-B^kr^{\text{ rb}}_k$ and hence, $\breve{r}^{\text{ rb}}=C(\breve{s}-B^ir^{\text{ rb}}_i-B^kr^{\text{ rb}}_k)$. So, for all $\ell \in \breve{N}$,

$$\begin{aligned} r^{\text{ rb}}_{\ell }=\breve{r}^{\text{ rb}}_{\ell }=\sum _{h\in \breve{N}}C_{\ell h}(\breve{s}_h-B_{hi}r^{\text{ rb}}_i-B_{hk}r^{\text{ rb}}_k). \end{aligned}$$

Define $\gamma ^{s}_{\ell }=\sum _{h\in \breve{N}}C_{\ell h}\breve{s}_h,\, \gamma ^i_{\ell }=-\sum _{h\in \breve{N}}C_{\ell h} B_{hi}$ and $\gamma ^k_{\ell }=-\sum _{h\in \breve{N}}C_{\ell h}B_{hk}$. Then, for each $\ell \in \breve{N}$,

$$\begin{aligned} r^{\text{ rb}}_{\ell }=\gamma ^{s}_{\ell }+\gamma ^i_{\ell }r^{\text{ rb}}_i+\gamma ^k_{\ell }r^{\text{ rb}}_k. \end{aligned}$$

(8.2)

Furthermore, equation $i$ in $Br^{\text{ rb}}=\hat{s}$ is

$$\begin{aligned} B_{ii}r^{\text{ rb}}_i+B_{ik}r^{\text{ rb}}_k+\sum _{\ell \in \breve{N}}B_{i\ell }r^{\text{ rb}}_{\ell }=\hat{s}_i. \end{aligned}$$

(8.3)

Define $\Gamma ^{i,i}=-\sum _{\ell \in \breve{N}}B_{i\ell }\gamma ^i_{\ell }$ and $\Gamma ^{i,k}=-\sum _{\ell \in \breve{N}}B_{i\ell }\gamma ^k_{\ell }$. Then, plugging in the expression of each $r^{\text{ rb}}_{\ell }$ (8.2) into (8.3) we get

$$\begin{aligned} (B_{ii}-\Gamma ^{i,i})r^{\text{ rb}}_i+(B_{ik}-\Gamma ^{i,k})r^{\text{ rb}}_k=\hat{s}_i-\sum _{\ell \in \breve{N}}\gamma ^{s}_{\ell }. \end{aligned}$$

(8.4)

Now, adding up (8.2) over all $\ell \in \breve{N}$ and using that $\sum _{h\in N}r^{\text{ rb}}_h=0$,

$$\begin{aligned} (1+\sum _{\ell \in \breve{N}}\gamma ^i_{\ell })r^{\text{ rb}}_i+(1+\sum _{\ell \in \breve{N}}\gamma ^k_{\ell })r^{\text{ rb}}_k=-\sum _{\ell \in \breve{N}} \gamma ^{s}_{\ell }. \end{aligned}$$

(8.5)

Define $\sigma _i=\sum _{\ell \in \breve{N}}\gamma ^i_{\ell }$ and $\sigma _k=\sum _{\ell \in \breve{N}}\gamma ^k_{\ell }$. Then, solving Eqs. (8.4) and (8.5), we get

$$\begin{aligned} r^{\text{ rb}}_i=\frac{\hat{s}_i-(1-\frac{B_{ik}-\Gamma ^{i,k}}{1+\sigma _k})\sum _{\ell \in \breve{N}} \gamma ^{s}_{\ell }}{(B_{ii}-\Gamma ^{i,i})-(B_{ik}-\Gamma ^{i,k})\frac{1+\sigma _k}{1+\sigma _i}} \quad \text{ and} \quad r^{\text{ rb}}_k=\frac{-\sum _{\ell \in \breve{N}}\gamma ^{s}_{\ell }}{1+\sigma _k}-\frac{1+\sigma _i}{1+\sigma _k}r^{\text{ rb}}_i.\nonumber \\ \end{aligned}$$

(8.6)

To understand how $r^{\text{ rb}}_i$ and $r^{\text{ rb}}_k$ vary with $\hat{s}_i$, it is convenient to know the signs of $\gamma ^i$ and $\gamma ^k$. We claim that both $\gamma ^i$ and $\gamma ^k$ are nonnegative vectors. By definition, $\gamma ^i=-CB^i$ and, since $C^{-1}=\breve{B},\, \breve{B}\gamma =-B^i$. Furthermore, $-B^i\ge 0$. Since matrix $\breve{B}$ and vectors $\gamma ^i$ and $-B^i$ satisfy the conditions of Lemma 8.1, $\gamma ^i$ is nonnegative. The argument for $\gamma ^k$ is analogous using $-B^k$ instead of $-B^i$. The nonnegativity of $\gamma ^i$ and $\gamma ^k$ implies that $\sigma _i$ and $\sigma _k$ are also nonnegative. Since $\gamma ^k$ is nonnegative, also $\Gamma ^{i,k}$ is nonnegative and $B_{ik}-\Gamma ^{i,k}$ is negative. Furthermore,

$$\begin{aligned} B_{ii}-\Gamma ^{i,i}=B_{ii}+\sum _{\ell \in \breve{N}}B_{i\ell }\gamma ^i_{\ell }\ge B_{ii}+\sum _{\ell \in \breve{N}} B_{i\ell }\ge 0. \end{aligned}$$

We reexamine now Eq. (8.6). Note that $\gamma ^{s},\, \gamma ^i,\, \gamma ^k,\, \Gamma ^{i,i},\, \Gamma ^{i,k},\, B_{ii}$ and $B_{ik}$ only depend on $\breve{B}$. Then, the denominator of the expression for $r^{\text{ rb}}_i$ is positive and so $r^{\text{ rb}}_i$ is strictly increasing in $\hat{s}_i$. Further, since $r^{\text{ rb}}_k$ is strictly decreasing in $r^{\text{ rb}}_i$, it is strictly decreasing in $\hat{s}_i$.

Now, because of (8.2), $r^{\text{ rb}}_{\ell }$ is weakly increasing in $r^{\text{ rb}}_i$ and $r^{\text{ rb}}_k$. Yet, since $r^{\text{ rb}}_i$ and $r^{\text{ rb}}_k$ are strictly increasing and decreasing, respectively, in $\hat{s}_i$, some extra work is needed to understand how $r^{\text{ rb}}_{\ell }$ varies with $\hat{s}_i$. To do so, we first show that all the components of $\gamma ^i$ and $\gamma ^k$ are no larger than 1. We prove it for $\gamma ^i$, the proof for $\gamma ^k$ being analogous.

$$\begin{aligned} \breve{B}(e-\gamma ^i)=\breve{B}e-\breve{B}\gamma ^i=\breve{B}e- B^i, \end{aligned}$$

and, for each $\ell \in \breve{N}$,

$$\begin{aligned} (\breve{B}e-B^i)_{\ell }=\sum _{h\in \breve{N}}B_{\ell h}+B_{\ell i}\ge \sum _{h\in \breve{N}}B_{\ell h}+B_{\ell i}+B_{ki}=0. \end{aligned}$$

Then, since $\breve{B}e-B^i$ is a nonnegative vector, matrix $\breve{B}$ and vectors $e-\gamma ^i$ and $\breve{B}e- B^i$ are in the conditions of Lemma 8.1 and, hence, $e-\gamma ^i$ is nonnegative.

Therefore, we know that all the components of $\gamma ^i$ and $\gamma ^k$ are no larger than 1. Looking again at Eq. (8.2), we have that $r^{\text{ rb}}_{\ell }$ cannot increase with $\hat{s}_i$ faster than $r^{\text{ rb}}_i$ so $r^{\text{ rb}}_{\ell }/r^{\text{ rb}}_i$ is weakly decreasing in $\hat{s}_i$. Similarly, $r^{\text{ rb}}_{\ell }/r^{\text{ rb}}_k$ is weakly increasing in $\hat{s}_i$. From this, the statement follows. $\square$

Proof of Proposition 6.5

The proof of is analogous to the proof of Proposition 6.4, with $B=L_A-A$ instead of $B=I-\bar{M}$. $\square$