Reference classes: a tool for benchmarking universities’ research

Abstract

Based on new comparison principles that take into account both the volume of scientific production and its impact, this paper proposes a method for defining reference classes of universities. Several tools are developed in order to enable university managers to define the value system according to which their university shall be compared to others. We apply this methodology to French universities and illustrate it using the reference classes of the best ranked universities according to several value systems.

Appendix: properties of reference classes: proofs of Theorem 1

The properties of reference classes essentially build upon the properties of dominance relations. To show this we first need to introduce a useful definition that establishes that a dominance relation is stronger than another if a dominance relation of the former type between two institutions implies a dominance of the latter over the former, for any pair of institutions.

Definition 4

A dominance relation $\succ$ is stronger than dominance relation $\succ^{\prime},$ noted $\succ\gg\succ^{\prime},$ if, $\forall i,j, i\succ j$ implies $i\succ^{\prime}j.$

We can now introduces some causal relations between the dominance relations.

Lemma 1

$\forall\phi,\phi^{\prime}\in\left[ 0,1\right]$ such that $\phi \ge \phi^{\prime}, \forall k\in K$ and $\forall m < \left\vert K-1\right\vert$:

• 1. $\blacktriangleright_{k}^{\phi}\gg\vartriangleright_{k}^{\phi}\gg\trianglerighteq_{k}^{\phi}$;

• 2. $\blacktriangleright_{k}^{\phi}\gg\blacktriangleright_{k}^{\phi^{\prime}}, \vartriangleright_{k}^{\phi}\gg\vartriangleright_{k}^{\phi^{\prime}}$ and $\trianglerighteq_{k}^{\phi}\gg\trianglerighteq_{k}^{\phi^{\prime}}$;

• 3. $\blacktriangleright^{\phi}\gg\vartriangleright^{\phi}\gg\trianglerighteq ^{\phi}$;

• 4. $\blacktriangleright^{\phi}\gg\blacktriangleright^{\phi^{\prime}}, \vartriangleright^{\phi}\gg\vartriangleright^{\phi^{\prime}}$ and $\trianglerighteq^{\phi}\gg\trianglerighteq^{\phi^{\prime}}$;

• 5. $\blacktriangleright_{k_{1},\ldots,k_{m}}^{\phi}\gg\blacktriangleright_{k_{1},\ldots,k_{m}}^{\phi^{\prime}}, \vartriangleright_{k_{1},\ldots,k_{m}}^{\phi}\gg\vartriangleright_{k_{1},\ldots,k_{m}}^{\phi^{\prime}}, \trianglerighteq_{k_{1},\ldots,k_{m}}^{\phi}\gg\trianglerighteq_{k_{1},\ldots,k_{m}}^{\phi^{\prime}}$.

• 6. $\blacktriangleright_{k_{1},\ldots,k_{m+1}}^{\phi}\gg\blacktriangleright_{k_{1},\ldots,k_{m}}^{\phi}, \vartriangleright_{k_{1},\ldots,k_{m+1}}^{\phi}\gg\vartriangleright_{k_{1},\ldots,k_{m}}^{\phi}, \trianglerighteq_{k_{1},\ldots,k_{m+1}}^{\phi}\gg\trianglerighteq_{k_{1},\ldots,k_{m}}^{\phi}.$

Proof

The proofs derive directly from the above definitions.$\square$

We now need to establish a correspondance between the relation between dominance relations and the relation between reference classes. This is the purpose of the following lemma.

Lemma 2

If $\succ\gg\succ^{\prime}$ then $c_{i}^{\succ^{\prime}}\subseteq c_{i}^{\succ},\forall i\in I.$

Proof

Assume $\succ\gg\succ^{\prime}.$ Then, $\forall i,k \in I, i\succ j$ implies $i\succ^{\prime}j$ which is equivalent to $i\nsucc^{\prime}j$ then $i\nsucc j.$ Now assume that $j\in c_{i}^{\succ^{\prime}},$ which requires that either $a) i\nsucc^{\prime}j$ and $j\nsucc^{\prime}i$ or $b) i\succ^{\prime}j$ and $j\succ^{\prime}i.$ If (a) holds, then the definition of the “≫” relation leads to $i\nsucc^{\prime}j$ and $j\nsucc^{\prime}i,$ which in turn implies that $j\in c_{i}^{\succ}.$ Now let us consider that (b) holds. When $i\succ^{\prime}j,$ it is impossible that both $j\succ i$ and $i\nsucc j$ which simply means that it is impossible that $j\notin c_{i}^{\succ}.$ Similarly, when $j\succ^{\prime}i,$ it is impossible that both $i\succ j$ and $j\nsucc i,$ that is, it is impossible that $j\notin c_{i}^{\succ}.$ Therefore, necessarily $j\in c_{i}^{\succ}$ (and $i\in c_{j}^{\succ}$). So, it is possible to conclude that when $j\in c_{i}^{\succ^{\prime}}$ then $j\in c_{i}^{\succ},$ which implies that $c_{i}^{\succ^{\prime}}\subseteq c_{i}^{\succ}.$

By application of the above Lemma, the five properties of reference classes in Theorem 1 now directly derive from the five properties of dominance relation in Lemma 1. $\square$