How to detect indications of potential sources of bias in peer review: A generalized latent variable modeling approach exemplified by a gender study

Abstract

The universalism norm of the ethos of science requires that contributions to science are not excluded because of the contributors’ gender, nationality, social status, or other irrelevant criteria. Here, a generalized latent variable modeling approach is presented that grant program managers at a funding organization can use in order to obtain indications of potential sources of bias in their peer review process (such as the applicants’ gender). To implement the method, the data required are the number of approved and number of rejected applicants for grants among different groups (for example, women and men or natural and social scientists). Using the generalized latent variable modeling approach indications of potential sources of bias can be examined not only for grant peer review but also for journal peer review.

Introduction

According to Merton (1942), the founder of the modern sociology of science, the functional goal of science is the expansion of “true” and secure knowledge. To fulfill this function in society, a set of ideal norms became established among scientists: the ethos of science. The universalism norm requires that contributions to science are not excluded because of the nationality, gender, social status of the contributors or other irrelevant personal or social criteria (MacCoun, 1998, Ziman, 2000). Critics of peer review argue that decisions in peer review are, nevertheless, frequently biased—that is, that they are not based solely on scientific merit but are influenced also by personal attributes of the applicants (Daniel, Mittag, & Bornmann, 2007; Marsh, Jayasinghe, & Bond, 2008). But an evaluation of a peer review process that can yield reliable and valid results on the influence of potential sources of bias on the review process is as a rule very elaborate and costly. The reasons for this are: (1) The research on peer review has identified a large number of attributes of applicants that can represent potential sources of bias in the peer review process (Wessely, 1998), (2) The study design should meet the highest requirements in order to establish unambiguously that the work from a particular group of applicants has a higher rejection rate due to biases in the peer review process and not simply as a consequence of the lesser scientific merit of the group of applications, and (3) The grant peer review process is a secret activity (Tight, 2003); reviews are secured with assurance of confidentiality.

Before a research funding organization conducts an extensive evaluation study, it should therefore seek indications of the influence of potential sources of bias in the grant peer review process, (1) in order to determine the necessity for an evaluation study, and (2) if a necessity is found, to identify the sources of bias that should be examined more closely (Ledin, Bornmann, Gannon, & Wallon, 2007). In the following, we present a statistical method that program managers at a research funding organization can use to obtain initial indications of potential sources of bias in their peer review process. The method has already been used for a meta-analysis investigating gender differences in grant award decisions (Bornmann, 2007; Bornmann, Mutz, & Daniel, 2007). To demonstrate application of the method for examining the peer review process, we utilized data from the Swiss National Science Foundation (SNSF) that are published on the Internet (http://www.snf.ch/E/aboutus/facts/Pages/statistics.aspx; Retrieved: November 23, 2007). The SNSF statistics show gender-specific figures for the research projects that were approved and rejected for funding in a total of 20 disciplines and subject areas in the years 2004–2006 (see Table 1, Table 2, Table 3).

With our statistical approach to obtaining initial indications of potential sources of bias in peer review processes, we are operating under the assumption that the odds of being approved among women applicants should be equal to the odds of being approved among men applicants. Unequal odds indicate a gender effect. If the effect is statistically significant, it is an evidence of bias and a detailed study of the peer review process should be conducted (see here also Women in Science & Engineering Leadership Institute, 2006).