The Game Between a Biased Reviewer and His Editor

Abstract

This paper shows that, for a large range of parameters, the journal editor prefers to delegate the choice to review the manuscript to the biased referee. If the peer review process is informative and the review reports are costly for the reviewers, even biased referees with extreme scientific preferences may choose to become informed about the manuscript’s quality. On the contrary, if the review process is potentially informative but the reviewer reports are not costly for the referees, the biased reviewer has no incentive to become informed about the manuscript. Furthermore, if the reports are costly for referees but the peer review processes are not potentially informative, the biased reviewers will never become informed. In this paper, we also present a web resource that helps editors to experiment with the review process as a device for information transmission.

Appendices

Appendix 1: Basic Model

The reviewer–editor game consists of two players, the uninformed editor and the biased reviewer. Players (editor and reviewer) have diverse scientific preferences over the consequences of editorial decisions. Let ${\mathscr {A}}$ denote the set of possible editorial decisions. Following Garcia et al. (2016), each editorial decision d selects a particular course of action from a set ${\mathscr {A}}$ of a large number of possibilities, $d \in {\mathscr {A}}$: e.g., accept outright, without editorial revisions; accept, with one or more revisions; invite the manuscript authors to address specific concerns before a final decision is reached; reject, but recommending a resubmission after further work; reject on different grounds such as lack of novelty, insufficient originality, or major technical problems. A large number of possibilities for editorial decisions is represented assuming that ${\mathscr {A}} = {\mathbb {R}}$, i.e., editorial decisions are real numbers.

We suppose that the consequences of editorial decisions x are related to such decisions d by the linear function $x = d - \beta$, with $\beta \in \{ 0,1\}$ being the realization of a random variable that represents the stochastic relationship between editorial decisions d and their realized outcomes x. Here, $\beta =1$ represents mistaken editorial decisions, such as, accepting the manuscript would result in future authors with lower quality standards, or its rejection would decrease the journal’s impact; whereas $\beta =0$ represents better editorial decisions. When the reviewer–editor game begins, neither the editor nor the reviewer knows the value of parameter $\beta$, but both of them have a common prior p which represents the probability that $\beta =1$. When the author submits a manuscript for consideration to the journal, the editorial process involves one review stage preceding one decision stage. The sequence of events begins with two probabilistic choices by nature (see Fig. 2). In our model, the choices nature makes represent the source of uncertainty, i.e., how editorial decisions translate into consequences. Thus, nature chooses a value for the state of the world $\beta$ according to probability $p= \Pr [ \beta =1 ]$. Suppose that $p_0$ denotes the conditional probability that a peer review produces message $m=1$, given that $\beta =0$. On the other hand, $p_1$ denotes the conditional probability that a peer review produces message $m=1$, given that $\beta =1$. Then, nature chooses a value for the realized message of the peer review process (if it is held), $m \in \{ 0,1\}$, according to conditional probabilities $p_0$ and $p_1$. If $p_0 = p_1$ the peer review process is not informative. On the contrary, if $p_0 = 0$ and $p_1 =1$ then, after observing the peer-review message m, the editor knows the true state $\beta$. Thus, the larger the ratio $p_0/p_1$ the less informative the peer review process.

After nature makes its two choices for $\beta$ and m, the editor decides whether to delegate the choice to review the manuscript to one expert (see Fig. 2). If the editor does not delegate the choice to evaluate the paper, she will learn the peer review message m at cost $\varepsilon >0$. In this case, the editor chooses an editorial decision d without external review (e.g., special accept or special reject). Recall that given the large number of possibilities for editorial decisions, they are represented as real numbers. The subgame without external review is denoted by $H_{WER}$.

When the editor delegates the manuscript’s evaluation to the expert, the submission is sent out for external review. The resulting subgame $H_{ER}$ is a peer review game (see Fig. 2). In this game, the referee first decides whether he will read through the entire manuscript to learn the value of $\beta$. This will be the ‘reading stage’ of the peer review game. If the reviewer reads through the submission, he incurs a cost $k>0$ that includes time spent reading the manuscript and related literature to become versed in the topic. The referee’s choice is represented by s, with $s=1$ if the reviewer decides to read the manuscript to become informed about its quality, and $s=0$ otherwise. $H_{0}$ represents a peer review subgame in which the referee has chosen not to become informed about the manuscript’s quality. Moreover, $H_{1}$ denotes the peer review subgame in which the reviewer has chosen to read through the entire manuscript to become informed (see Fig. 2). In this case, she can condition her choice s on the observed value of $\beta$.

Next, the reviewer decides whether to write a review report (see Fig. 2). This will be the ‘writing stage’ of the peer review. There, the referee chooses a, with $a=1$ if he submits the review report in a timely manner, while a takes the value of 0 if the reviewer does not submit a timely report. When reviewers comment on a manuscript, they are likely to raise different issues. But reviewers incur a cost $c \ge 0$ while they are drafting a review report rather than their own papers or doing other academic work. Therefore, if a review report is submitted, the editor observes the peer-review message $m \in \{0,1 \}.$ If the review report is not submitted, the editor observes $\phi$. After the handling editor observes one of the three possible messages, she makes an editorial decision (see Fig. 2).

After the editor makes an editorial decision, the outcome x is realized. Fiske and Fogg (1990) show that, in the typical case, different evaluations of the same manuscript have no critical point in common; instead, individuals wrote about different topics, and their recommendations about editorial decisions hardly showed any agreement. As a result of these differing opinions, the editor’s decision may not match all reviewers’ recommendations, and this issue will be much more relevant when we consider a biased reviewer and his editor. In our model, to represent different scientific opinions, we assume a spatial model of preferences over the realized outcomes in which outcomes are points in the one-Euclidean space. We also assume that an individual (editor or reviewer) is characterized by her ‘ideal point’ in that same space, and that outcomes are judged as good as they are close to the ideal point. Editor and reviewer are assumed to have specific utilities over a one-dimensional outcome space. The editor’s utility function is

$$\begin{aligned} u_{e} (x) = - | x - x_{e} | \end{aligned}$$

(1)

where $x_{e}$ is the ideal point of the editor. The reviewer’s utility function is

$$\begin{aligned} u_{r} (x) = - | x - x_{r} | \end{aligned}$$

(2)

where $x_{r}$ is the ideal point of the reviewer. To represent a biased reviewer, we assume that the ideal point of the reviewer is $x_{r} \ge 1$, while the ideal point of the editor is $x_{e}=0$.

Appendix 2: Proof of Proposition 1

Let $\Delta$ represent the odds that nature chooses $\beta =1$, and thus,

$$\begin{aligned} \Delta = \frac{p}{1-p} \end{aligned}$$

with $p = \Pr [ \beta =1 ]$. Recall that the peer-review outcome is a message $m \in \{ 0,1\}$, and nature chooses a value for the message m according to conditional probabilities $p_0 = \Pr [ m = 1 / \beta =0]$ and $p_1 = \Pr [ m = 1 / \beta =1]$, with the ratio $p_0/p_1$ being the informative content of the peer review process.

Let h be a strategy profile for each peer-review subgame H. A Bayesian Nash equilibrium is defined as a strategy profile that maximizes the expected utility for each player given their beliefs. To refine the equilibria generated by the Bayesian Nash solution, we then apply the perfect Bayesian equilibrium that demands that subsequent play be optimal. Let $h^*$ be a perfect Bayesian equilibrium for H.

Let $EU_e (h_{WER}^*)$ be the expected utility to the editor of choosing a special decision without external review; $EU_e (h_{ER}^*)$ be the expected utility to the editor of delegating the manuscript evaluation to the biased expert.

We have to prove that when the review cost $\varepsilon$ at which the editor can learn the review message m is below a critical value, $0< \varepsilon < (p_1 - p_0)/2$, and the odds that nature chooses $\beta =1$, i.e. $\Delta$, is in the range $\left( 1, \frac{1- \varepsilon -p_0}{1 + \varepsilon - p_1} \right)$ then the editor prefers to review the manuscript herself because of its optimality, i.e., $EU_e (h_{ER}^*) < EU_e (h_{WER}^*)$. Otherwise, when $\Delta <1$ or $\Delta > \frac{1- \varepsilon -p_0}{1 + \varepsilon - p_1}$the editor prefers to delegate the choice to review the manuscript to the biased referee since $EU_e (h_{ER}^*) > EU_e (h_{WER}^*)$.

Following equations (19) and (21) in Diermeier and Feddersen (2000), if $\Delta \in \left( 1, \frac{1-p_0}{1 - p_1} \right)$, we have

$$\begin{aligned} EU_e (h_{ER}^*) - EU_e (h_{WER}^*) = \varepsilon + 2p - p p_1 + p_0 - pp_0 -1 \end{aligned}$$

(3)

Therefore, it follows that $EU_e (h_{ER}^*) - EU_e (h_{WER}^*) >0$ if and only if

$$\begin{aligned} \varepsilon + 2p - p p_1 + p_0 - pp_0 -1 >0 \end{aligned}$$

or equivalently

$$\begin{aligned} p > \frac{1 -\varepsilon - p_0}{2 - p_0 - p_1} \end{aligned}$$

or equivalently

$$\begin{aligned} \Delta = \frac{p}{1-p} > \frac{1 -\varepsilon - p_0}{1 + \varepsilon - p_1} \end{aligned}$$

where $\frac{1 -\varepsilon - p_0}{1 + \varepsilon - p_1} >1$ if and only if $0< \varepsilon < (p_1 - p_0)/2$.

Again, from equations (19) and (21) in Diermeier and Feddersen (2000), if $\Delta > \frac{1-p_0}{1 - p_1}$ or $\Delta <1$ and $c < p_0$, we have

$$\begin{aligned} EU_e (h_{ER}^*) - EU_e (h_{WER}^*) = \varepsilon \end{aligned}$$

(4)

Therefore, it follows that $EU_e (h_{ER}^*) - EU_e (h_{WER}^*) >0$ since $\varepsilon >0$.

If $\Delta <p_0/p_1$ and $c > p_0$, we have

$$\begin{aligned} EU_e (h_{ER}^*) - EU_e (h_{WER}^*) = \varepsilon + p\frac{p_1-p_0}{1-p_0} \end{aligned}$$

(5)

from equations (19) and (21) in Diermeier and Feddersen (2000). Therefore, it follows that $EU_e (h_{ER}^*) - EU_e (h_{WER}^*) >0$ since $\varepsilon + p\frac{p_1-p_0}{1-p_0}>0$.

If $p_0/p_1< \Delta <1$ and $c > p_0$, we have

$$\begin{aligned} EU_e (h_{ER}^*) - EU_e (h_{WER}^*) = \varepsilon + p - p p_1 + (1-p)p_0 - p\frac{1-p_1}{1-p_0} \end{aligned}$$

(6)

from equations (19) and (21) in Diermeier and Feddersen (2000). To see that

$$\begin{aligned} \varepsilon + p - p p_1 + (1-p)p_0 - p\frac{1-p_1}{1-p_0}>0 \end{aligned}$$

we divide by $(1-p)$ to obtain

$$\begin{aligned} \frac{\varepsilon }{1-p} + \Delta -\Delta p_1 + p_0 - \Delta \frac{1-p_1}{1-p_0} = \frac{\varepsilon }{1-p} + p_0 \left( 1 - \Delta \frac{1-p_1}{1-p_0} \right) >0 \end{aligned}$$

since $\Delta <1$ and $p_0/p_1 < 1$.

Appendix 3: Proof of Proposition 2

We now use these perfect equilibria to show how even biased reviewers become informed about the manuscript’s quality given that the peer review process is potentially informative ($p_1 > p_0$) and costly for the referee ($c > 0$). We simply have to compare the reviewer’s ex-ante expected utility for the peer review subgame in which the referee does not become informed, $H_{0}$, with the reviewer’s ex-ante expected utility for the subgame in which the reviewer decides to become informed about the manuscript’s quality, $H_{1}$.

Let h be a strategy profile for each peer-review subgame H. A Bayesian Nash equilibrium is defined as a strategy profile that maximizes the expected utility for each player given their beliefs. To refine the equilibria generated by the Bayesian Nash solution, we then apply the perfect Bayesian equilibrium that demands that subsequent play be optimal.

Let $h^*$ be a perfect Bayesian equilibrium for H; $EU_r (h_{0}^*)$ be the reviewer’s ex-ante expected utility from the peer review subgame in which the referee does not become informed about the manuscript’s quality; and $EU_r (h_{1}^*)$ be the reviewer’s ex-ante expected utility from the subgame in which the reviewer decides to become informed about the manuscript’s quality. If $EU_r (h_{1}^*) - EU_r (h_{0}^*) > 0$, then the biased reviewer will choose to become informed about the manuscript’s quality because of its optimality.

• 1. If the cost of writing reports is $0< c < p_0$ and the odds that nature is in state one are $\Delta <p_0/p_1$, then following Propositions 1 and 2 in Diermeier and Feddersen (2000), we have

  $$\begin{aligned} EU_r (h_{0}^*) = - x_r - p \end{aligned}$$

  (7)

  and

  $$\begin{aligned} EU_r (h_{1}^*) = \frac{p p_1 c - p_0 p - p c p_0 - x_r p_0}{p_0} - k \end{aligned}$$

  (8)

  Therefore, it follows that

  $$\begin{aligned} EU_r (h_{1}^*) - EU_r (h_{0}^*) = p c \frac{p_1-p_0}{p_0} - k > 0 \end{aligned}$$

  (9)

  since specialization cost k is low enough, $k < pc\frac{(p_1-p_0)}{p_0}$, and $p_1 > p_0$.

• 2. If reports are sufficiently costly to the reviewer, $p_0< c < p_0 + p(p_1- p_0)$, and the odds of nature being in state one $\beta =1$ are high enough, $p_0/p_1 < \Delta$, then following Propositions 1 and 2 in Diermeier and Feddersen (2000) we have

  $$\begin{aligned} EU_r (h_{0}^*) = -c -p + p p_1 + p_0 - x_r - p p_0 \end{aligned}$$

  (10)

  and

  $$\begin{aligned} EU_r (h_{1}^*) = \frac{-p - p p_1 c + p p_1 + p c p_0 - x_r - p p_0}{1- p_0} - k \end{aligned}$$

  (11)

  Therefore, it follows that

  $$\begin{aligned} EU_r (h_{1}^*) - EU_r (h_{0}^*) = (c-p_0)\left( 1 - \frac{ p(p_1-p_0)}{1-p_0}\right) - k > 0 \end{aligned}$$

  (12)

  since specialization cost is low enough, $k < (c-p_0)\left( 1 - \frac{ p(p_1-p_0)}{1-p_0}\right)$.

• 3. If reports are very costly to the reviewer, $p_0 + p(p_1- p_0) < c$, then following Propositions 1 and 2 in Diermeier and Feddersen (2000) we have

  $$\begin{aligned} EU_r (h_{1}^*) - EU_r (h_{0}^*) = p\frac{(p_1-p_0) - c(p_1-p_0) }{1-p_0}- k \end{aligned}$$

  (13)

  Therefore, it follows that $EU_r (h_{1}^*) - EU_r (h_{0}^*) >0$ since specialization cost is low enough, $k < p\frac{(p_1-p_0) - c(p_1-p_0) }{1-p_0}$.

Appendix 4: Proof of Proposition 3

We have to prove that when the cost at which the referee can write a reviewer report is zero, $c =0$, and the peer review is potentially informative, $p_1 > p_0$, then a biased reviewer decides not to become informed about the manuscript’s quality because of its optimality, i.e., $EU_r (h_{0}^*) > EU_r (h_{1}^*)$. We will distinguish two cases, the first when $\Delta <p_0/p_1$ and the second when $1> \Delta > p_0/p_1$.

If the odds that nature is in state one are $\Delta <p_0/p_1$ then, following Propositions 1 and 2 in Diermeier and Feddersen (2000), we have

$$\begin{aligned} EU_r (h_{0}^*) = - x_r - p \end{aligned}$$

(14)

and following Proposition 3 in Diermeier and Feddersen (2000), we have, for the perfect Bayesian equilibrium $h_{1}^*$, the maximum expected utility is $-p - x_r - k$. Therefore, $EU_r (h_{0}^*) > EU_r (h_{1}^*)$ since specialization cost $k >0$.

If the odds that nature is in state one are $1> \Delta > p_0/p_1$ then, following Propositions 1 and 2 in Diermeier and Feddersen (2000), we have

$$\begin{aligned} EU_r (h_{0}^*) = - x_r - p + p (p_1 -p_0) + p_0 \end{aligned}$$

(15)

and following Proposition 3 in Diermeier and Feddersen (2000), for the perfect Bayesian equilibrium $h_{1}^*$, the maximum expected utility is $-p - x_r - k$ or $- x_r - p + p (p_1 - p_0) + p_0 - k$. Therefore, in both cases, $EU_r (h_{0}^*) > EU_r (h_{1}^*)$ since specialization cost $k >0$.