Why the referees’ reports I receive as an editor are so much better than the reports I receive as an author?

Abstract

Authors tend to attribute manuscript acceptance to their own ability to write quality papers and simultaneously to blame rejections on negative bias in peer review, displaying a self-serving attributional bias. Here, a formal model provides rational explanations for this self-serving bias in a Bayesian framework. For the high-ability authors in a very active scientific field, the model predictions are: (1) Bayesian-rational authors are relatively overconfident about their likelihood of manuscript acceptance, whereas authors who play the role of referees have less confidence in manuscripts of other authors; (2) if the final disposition of his or her manuscript is acceptance, the Bayesian-rational author almost surely attributes this decision more to his or her own ability; (3) when the final disposition is rejection, the Bayesian-rational author almost surely attributes this decision more to negative bias in peer review; (4) some rational authors do not learn as much from the critical reviewers’ comments in case of rejection as they should from the journal editor’s perspective. In order to validate the model predictions, we present results from a survey of 156 authors. The participants in the experimental study are authors of articles published in Scientometrics from 2000 to 2012.

Appendices

Appendix 1: Proof of Proposition 1

Assume a set of I scholarly authors who can also play the role of reviewers for academic journals in the scientific field. Let there be N potential manuscripts $m_n \in M$, with $1 \le n \le N$, which any author i can choose to write and submit to an academic journal. For notational simplicity we assume that n represents manuscript $m_n$. We suppose that author i will choose to write a manuscript m(i) with the highest probability of acceptance:

$$\begin{aligned} m(i) = \arg _{1 \le n \le N} \max p_n^i \end{aligned}$$

where $p_n^i$ denotes author’s i’s belief regarding the probability of acceptance of manuscript n (conditional on the final decision being determined by manuscript quality). Recall that we suppose that the $p_n^i$ are i.i.d. draws from the atomless distribution F on [0, 1 ], with $\hat{p}$ being the supremum of the support of F, i.e., $p_n^i \sim F[0,\hat{p}]$.

We also have that $p_{m(i)}^j$ denotes the belief of some author j that m(i) will be accepted for publication by the journal (where author j, with $j\not = i$, can be a reviewer at the peer review process of m(i)). By definition, the probability distribution of $p_{m(i)}^j$ is F(x); while the distribution of $p_{m(i)}^i$ is $F(x)^N$ since

$$\begin{aligned} \Pr [ p_{m(i)}^i \le x] = \Pr [p_1^i\le x;p_2^i\le x; \dots ; p_N^i\le x] = \prod _{k=1}^N \Pr [p_k^i\le x] = F^N(x) \end{aligned}$$

given that $m(i) = \arg _{1 \le n \le N} \max p_n^i$, and $\Pr [ p_n^i \le x ] = F(x)$, with $1 \le n \le N$.

Note that the probability that all authors $1, 2, \dots , i-1, i+1, \dots , I$ other than i consider author i to over-estimate the probability of acceptance of his or her manuscript m(i) is

$$\begin{aligned} \Pr [ p_{m(i)}^1 \le p_{m(i)}^i; \dots ; p_{m(i)}^{i-1} \le p_{m(i)}^i;p_{m(i)}^{i+1} \le p_{m(i)}^i; \dots ; p_{m(i)}^I \le p_{m(i)}^i ] \end{aligned}$$

with $p_{m(i)}^j$ being the belief of some author j that m(i) will be accepted for publication, as above. In the following we prove that, as $N \rightarrow \infty$, this probability converges to 1, and consequently, all authors other than i (when they play the role of reviewer of manuscript m(i)) almost surely consider that author i over-estimates the probability of acceptance of m(i).

Let $f_{p_{m(i)}^i}(x)$ be the probability density function of the author’s i’s belief regarding the probability of acceptance of his or her manuscript m(i), i.e., $p_{m(i)}^i$. In this case, we have that

$$\begin{aligned} &\Pr [ p_{m(i)}^1 \le p_{m(i)}^i; \dots ; p_{m(i)}^{i-1} \le p_{m(i)}^i;p_{m(i)}^{i+1} \le p_{m(i)}^i; \dots ; p_{m(i)}^I \le p_{m(i)}^i ] \\ &\quad= \int \Pr [ p_{m(i)}^1 \le x; \dots ; p_{m(i)}^{i-1} \le x;p_{m(i)}^{i+1} \le x; \dots ; p_{m(i)}^I \le x ] \ f_{p_{m(i)}^i}(x) dx \\ &\quad=\int \prod _{j\not =i } \Pr [p_{m(i)}^j \le x] \ f_{p_{m(i)}^i}(x) dx \end{aligned}$$

Recall that the probability distribution of $p_{m(i)}^j$ is F(x), for $j\not = i$; and the distribution of $p_{m(i)}^i$ is $F^N(x)$. Then, it follows that

$$\begin{aligned} &\int \prod _{j\not =i } \Pr [p_{m(i)}^j \le x] \ f_{p_{m(i)}^i}(x) dx \\ &\quad= \int F(x)^{I-1} f_{p_{m(i)}^i}(x) dx \\ &\quad= \int F(x)^{I-1} d [ F^N (x) ] \end{aligned}$$

There, replacing $d [ F^N (x) ]$ by $N F(x)^{N-1} d [ F (x) ]$, and taking account that F is an atomless distribution on [0, 1 ], it follows that

$$\begin{aligned} &\int F(x)^{I-1} d [ F^N (x) ] \\ &\quad= \int _0^1 F(x)^{I-1} N F(x)^{N-1} d [ F (x) ] \\&\quad= N \int _0^1 F(x)^{N + I-2} d [ F (x) ] \\&\quad= \frac{N}{N+(I-1)} \left[ F(x)^{N + I-1} \right] _0^1 \\ &\quad= \frac{N}{N+(I-1)} \end{aligned}$$

where $\frac{N}{N+(I-1)}\rightarrow 1$ as N increases to infinity, which proves the first part of the proposition:

$$\begin{aligned} \Pr [ p_{m(i)}^1 \le p_{m(i)}^i; \dots ; p_{m(i)}^{i-1} \le p_{m(i)}^i;p_{m(i)}^{i+1} \le p_{m(i)}^i; \dots ; p_{m(i)}^I \le p_{m(i)}^i ]= \frac{N}{N+(I-1)} \rightarrow 1 \end{aligned}$$

as $N \rightarrow \infty$, and therefore, all authors other than i consider author i to over-estimate the probability of acceptance. Figure 11 shows the behaviour of the probability that all authors other than i consider author i to over-estimate the probability of acceptance of his or her manuscript m(i), as N and I increase.

Fig. 11

Probability that all authors other than i consider author i to over-estimate the probability of acceptance of his or her manuscript, as N and I increase

Let $\varepsilon > 0$. For the second part of the proposition, if the supremum of the support of distribution F is $\hat{p} =1$, it follows that

$$\begin{aligned} & \Pr [ p_{m(i)}^i > 1- \varepsilon ] \\ &\quad= 1- \Pr [ p_{m(i)}^i \le 1- \varepsilon ] \\ &\quad= 1- [F (1- \varepsilon )]^N \end{aligned}$$

since the distribution of $p_{m(i)}^i$ is $F^N(x)$. Therefore, as $N \rightarrow \infty$, any author i will almost surely believe that his or her manuscript m(i) will almost surely be accepted for publication if bias does not interfere with peer review, since

$$\begin{aligned} & \lim _{N \rightarrow \infty } \Pr [ p_{m(i)}^i > 1- \varepsilon ] \\ &\quad= \lim _{N \rightarrow \infty } 1- [F ( 1- \varepsilon )]^N = 1 \end{aligned}$$

since $F ( 1- \varepsilon ) = F ( \hat{p}- \varepsilon ) < F ( \hat{p}) =1$, given that $\varepsilon > 0$.

Appendix 2: Proof of Proposition 2

The reviewer (author j’s) ex-post probability that author’s ability caused the acceptance of manuscript m(i) follows from Bayesian inference:

$$\begin{aligned} {\Pr }^j( ability | acceptance) &= \frac{ q \times \Pr ^j(acceptance | ability)}{q \times \Pr ^j(acceptance | ability) + (1-q) \times \Pr (acceptance | bias) } \\ & = \frac{ q \times p_{m(i)}^j}{q \times p_{m(i)}^j + (1-q) \times p } \end{aligned}$$

where we suppose that bias intervenes in peer review with probability $1-q$; and therefore, bias does not arise with probability q in peer review; and where $p = \Pr (acceptance | bias)$ denotes the probability of manuscript acceptance when bias intervenes in peer review. Note that $\Pr ^j( ability | acceptance)$ increases in $p_{m(i)}^j$ whenever $p,q > 0$.

To show that author i almost surely attributes manuscript acceptance more to his or her ability than author j does, we have to prove that almost surely $\Pr ^j( ability | acceptance) < \Pr ^i( ability | acceptance)$, as N converges to infinity.

To this aim, since $\Pr ^j( ability | acceptance)$ increases in $p_{m(i)}^j$ whenever $p,q > 0$, it suffices to show that the probability that $p_{m(i)}^j < p_{m(i)}^i$ converges to one, as N converges to infinity. In this case, we would have that, as $N \rightarrow \infty$, almost surely

$$\begin{aligned} {\Pr }^j( ability | acceptance) < {\Pr }^i( ability | acceptance) \end{aligned}$$

which proves the desired result.

Now, to show that the probability $\Pr [p_{m(i)}^j < p_{m(i)}^i]$ converges to one, we first have to calculate this probability as follows:

$$\begin{aligned} & \Pr [p_{m(i)}^j < p_{m(i)}^i] \\ &\quad= \int \Pr [ p_{m(i)}^j \le x ] \ f_{p_{m(i)}^i}(x) dx \\ &\quad= \int F(x) f_{p_{m(i)}^i}(x) dx \\ &\quad= \int F(x) d [ F^N (x) ] \\ &\quad= N \int _0^1 F^N(x) d [ F (x) ] \\ &\quad= \frac{N}{N+1} \left[ F(x)^{N + 1} \right] _0^1 = \frac{N}{N+1} \end{aligned}$$

since the probability distribution of $p_{m(i)}^j$ is F(x), for $j\not = i$; and the distribution of $p_{m(i)}^i$ is $F^N(x)$. Therefore, we have that $\Pr [p_{m(i)}^j < p_{m(i)}^i] = \frac{N}{N+1}$ which converges to 1 as $N \rightarrow \infty$ which then proves the first part of the proposition.

The second part of the proposition (regarding rejection and bias) is completely analogous by using the author j’s ex-post probability that rejection was caused by negative bias

$$\begin{aligned} \Pr ( bias| rejection) = \frac{ (1-q) \times \Pr (rejection| bias)}{(1-q) \times \Pr (rejection | bias) + q \times \Pr (rejection | author's \ ability) } \end{aligned}$$

Appendix 3: Proof of Proposition 3

First, as proved in Appendix 1, if the supremum of the support of distribution F is $\hat{p} =1$, it follows that

$$\begin{aligned} \Pr [ p_{m(i)}^i > \hat{p} - \varepsilon ] = \\ 1- \Pr [ p_{m(i)}^i \le 1- \varepsilon ] = \\ 1- [F (\hat{p}- \varepsilon )]^N \end{aligned}$$

since the distribution of $p_{m(i)}^i$ is $F^N(x)$; and where $\varepsilon > 0$. This implies that

$$\begin{aligned} \lim _{N \rightarrow \infty } \Pr [ p_{m(i)}^i > \hat{p}- \varepsilon ] = \\ \lim _{N \rightarrow \infty } 1- [F ( \hat{p} - \varepsilon )]^N = 1 \end{aligned}$$

since $F ( \hat{p}- \varepsilon ) < F ( \hat{p}) =1$, given that $\varepsilon > 0$. This proves that $p_{m(i)}^i$ converges almost surely (with probability 1) towards $\hat{p}$, as $N \rightarrow \infty$.

Now, to prove the first part of the proposition we have to show that, in the limit as $N \rightarrow \infty$, the manuscript author i is almost sure that a final disposition of rejection of manuscript m(i) was due to negative bias in peer review as follows.

The author i’s ex-post probability that the manuscript rejection was caused by negative bias is

$$\begin{aligned} \Pr ( bias| rejection) &= \frac{ (1-q) \times \Pr (rejection| bias)}{(1-q) \times \Pr (rejection | bias) + q \times \Pr (rejection | author's \ ability) } \\ &= \frac{ (1-q) (1-p) }{ (1-q) (1-p) + q (1- p_{m(i)}^i)} \end{aligned}$$

where we suppose that bias intervenes in peer review with probability $1-q$; and therefore, bias does not arise with probability q in peer review; and where $1-p = \Pr (rejection | bias)$ denotes the probability of manuscript rejection when bias intervenes in peer review. Therefore, we have that $\Pr ( bias| rejection)$ converges almost surely (with probability 1) towards $\frac{ (1-q) (1-p) }{ (1-q) (1-p) + q (1- \hat{p} )}$, as $N \rightarrow \infty$, since $p_{m(i)}^i$ converges almost surely (with probability 1) towards $\hat{p}$.

In this case, if the supremum of the support of distribution F is $\hat{p} =1$, it follows that $\Pr ( bias| rejection)$ converges almost surely (with probability 1) towards 1, as $N \rightarrow \infty$, which proves the first part of the proposition.

To prove the second part, we have to show that, in the limit as $N \rightarrow \infty$, the author belief that the final disposition of acceptance is due to author’s ability converges almost surely towards $\frac{q}{q+p(1-q)}$, where positive or negative bias arises with probability $1-q$ and, if bias interferes with peer review, the probability of manuscript acceptance is p.

The author i’s ex-post probability that the manuscript acceptance was caused by author’s ability is

$$\begin{aligned} \Pr ( ability | acceptance) &= \frac{ q \times \Pr (acceptance | ability)}{q \times \Pr (acceptance | ability) + (1-q) \times \Pr (acceptance | bias) } \\ &= \frac{ q p_{m(i)}^i }{ q p_{m(i)}^i + p(1-q)} \end{aligned}$$

which converges almost surely towards $\frac{ q\hat{p} }{ q \hat{p} + p(1-q)}$ since $p_{m(i)}^i$ converges almost surely (with probability 1) towards $\hat{p}$. The proof is concluded by noting that, if $\hat{p} =1$, it follows that $\frac{ q\hat{p} }{ q \hat{p} + p(1-q)} = \frac{ q }{ q + p(1-q)}$.

Appendix 4: Proof of Proposition 4

Now we consider how author i updates his/her belief that his/her ability is not enough to write an article that can be accepted for publication, after he or she observed the final decision of rejection at the peer review stage.

We have to prove that, as $N \rightarrow \infty$, in the limit, the manuscript author i almost surely learns less from the peer review process in case of rejection than the author should, given the reference ex-post belief that rejection was caused by author’s ability.

The author i believes that his or her ability is not enough, given the decision of rejection, with probability ${\Pr }^i [ not \ enough/ rejection]$ as follows:

$$\begin{aligned} {\Pr }^i [ not \ enough/ rejection] &= \Pr [ not \ enough/ rejection \& ability] \ {\Pr }^i(ability| rejection) \\ &\quad + \Pr [ not \ enough/ rejection \& bias] \ {\Pr }^i(bias| rejection) \end{aligned}$$

where the manuscript author i thinks that the rejection was caused by his or her ability with probability ${\Pr }^i(ability| rejection)$; and where the first component

$$\begin{aligned} \Pr [ not \ enough/ rejection \& ability] \ {\Pr }^i(ability| rejection) \end{aligned}$$

represents the updating of author’s belief conditional on the rejection being caused by his/her (low) ability; while the second component

$$\begin{aligned} \Pr [ not \ enough/ rejection \& bias] \ {\Pr }^i(bias| rejection) \end{aligned}$$

represents the updating of author’s belief conditional on the rejection being caused by (negative) bias.

Taking account that

$$\begin{aligned} \Pr [ not \ enough/ rejection \& ability] =1 \end{aligned}$$

and

$$\begin{aligned} \Pr [ not \ enough/ rejection \& bias] = p_{m(i)}^i \end{aligned}$$

we have that

$$\begin{aligned} {\Pr }^i [ not \ enough/ rejection] &= \Pr [ not \ enough/ rejection \& ability] \ {\Pr }^i(ability| rejection) \\ &\quad+ \Pr [ not \ enough/ rejection \& bias] \ {\Pr }^i(bias| rejection) \\ &= {\Pr }^i(ability| rejection) + p_{m(i)}^i {\Pr }^i(bias| rejection) \end{aligned}$$

Next we calculate ${\Pr }^i(ability| rejection)$ and ${\Pr }^i (bias| rejection)$ as follows. Upon manuscript rejection at the peer review stage, the manuscript author i thinks that this rejection was caused by his or her ability with probability ${\Pr }^i(ability| rejection)$ as follows:

$$\begin{aligned} {\Pr }^i (ability| rejection) &= \frac{ q \times {\Pr }^i(rejection| ability)}{q \times {\Pr }^i(rejection | ability) + (1-q) \times \Pr (rejection | bias) } \\ &= \frac{ q (1-p_{m(i)}^i) }{ q (1-p_{m(i)}^i) + (1-q) (1- p)} \end{aligned}$$

where bias does not intervene in peer review with probability q; if bias intervenes in peer review, the probability of manuscript rejection is $1-p$; and where $1-p_{m(i)}^i$ is the belief of author i that manuscript m(i) will be rejected at the peer review stage.

Note that ${\Pr }^i (bias| rejection) = 1- {\Pr }^i (ability| rejection)$ since:

$$\begin{aligned} {\Pr }^i (bias| rejection) &= \frac{ (1-q) \times \Pr (rejection| bias)}{q \times {\Pr }^i(rejection | ability) + (1-q) \times \Pr (rejection | bias) } \\ &= 1- \frac{ q \times {\Pr }^i(rejection| ability)}{q \times {\Pr }^i(rejection | ability) + (1-q) \times \Pr (rejection | bias) } \\ &= 1- {\Pr }^i (ability| rejection) \end{aligned}$$

Thus, we can conclude that author i believes that his or her ability is not enough, given the decision of rejection, with probability

$$\begin{aligned} {\Pr }^i [ not \ enough/ rejection] &= {\Pr }^i(ability| rejection) + p_{m(i)}^i {\Pr }^i(bias| rejection) \\ &= p_{m(i)}^i + (1- p_{m(i)}^i ) {\Pr }^i(ability| rejection) \\ &= p_{m(i)}^i + (1- p_{m(i)}^i ) \frac{ q (1-p_{m(i)}^i) }{ q (1-p_{m(i)}^i) + (1-q) (1- p)} \end{aligned}$$

By using analogous arguments as above, we have also that, given the reference ex-post belief that rejection was caused by author’s ability, author i believes that his or her ability is not enough with probability ${\Pr }^* [ not \ enough/ rejection]$ as follows:

$$\begin{aligned} {\Pr }^* [ not \ enough/ rejection] &= \Pr [ not \ enough/ rejection \& ability] \ {\Pr }^*(ability| rejection) \\ &\quad + \Pr [ not \ enough/ rejection \& bias] \ {\Pr }^*(bias| rejection) \end{aligned}$$

where the reference ex-post belief that rejection was caused by author’s ability ${\Pr }^*( ability| rejection)$ is:

$$\begin{aligned} {\Pr }^* (ability| rejection) &= \frac{ q \times {\Pr }^*(rejection | ability) }{q \times {\Pr }^*(rejection | ability) + (1-q) \times \Pr (rejection | bias) } \\ &= \frac{ q (1-p_{m(i)}^*) }{ q (1- p_{m(i)}^*) + (1-q) (1-p)} \end{aligned}$$

where $1-p_{m(i)}^*$ is the reference belief of the journal editor that manuscript m(i) will be rejected at the peer review stage. Using similar arguments as above, it follows that ${\Pr }^* (bias| rejection) = 1- {\Pr }^* (ability| rejection)$. Therefore, in this case, we can also conclude that author i believes that his or her ability is not enough, given the decision of rejection, with probability

$$\begin{aligned} {\Pr }^* [ not \ enough/ rejection] &= {\Pr }^*(ability| rejection) + p_{m(i)}^i {\Pr }^*(bias| rejection) \\ &= p_{m(i)}^i + (1- p_{m(i)}^i ) {\Pr }^*(ability| rejection) \\ &= p_{m(i)}^i + (1- p_{m(i)}^i ) \frac{ q (1-p_{m(i)}^*) }{ q (1-p_{m(i)}^*) + (1-q) (1- p)} \end{aligned}$$

As N converges to infinity, in the limit, we have that almost surely $p_{m(i)}^i > p_{m(i)}^*$ (or equivalently, $1-p_{m(i)}^i < 1-p_{m(i)}^*$) using a proof analogous to that of Appendix 1 under the assumption that $p_n^*$ be drawn from the same distribution F on [0, 1 ], with $\hat{p}$ being the supremum of the support of F. Hence it follows that almost surely

$$\begin{aligned} \frac{ q (1-p_{m(i)}^i) }{ q (1-p_{m(i)}^i) + (1-q) (1- p)} < \frac{ q (1-p_{m(i)}^*) }{ q (1-p_{m(i)}^*) + (1-q) (1- p)} \end{aligned}$$

or equivalently

$$\begin{aligned} {\Pr }^i [ not \ enough/ rejection] < {\Pr }^* [ not \ enough/ rejection] \end{aligned}$$

which proves that, as $N \rightarrow \infty$, in the limit, author i almost surely learns less from the peer review process in case of rejection than the author should, given the reference ex-post belief that rejection was caused by author’s ability.

Appendix 5: Survey Questionnaire