Different Processes, Similar Results? A Comparison of Performance Assessment in Three Countries

Abstract

Monitoring the scientific performance of a country, region, or organization has become a high priority for research managers and government agencies. Research assessments have been implemented to provide evidence and facilitate their decisions. They differ in the methodologies applied, the disciplinary and regional breadth, and the consequences that follow. We sought to examine the extent to which quantitative, indicator-based analysis can contribute to identifying and better understanding the effects and effectiveness of the different assessment regimes. To this end, we analyzed the publications from three countries (Australia, the United Kingdom, and Germany) with contrasting systems in place, seeking to demonstrate the possibilities and limitations of using an indicator-based methodology for determining the outcomes from different approaches to assessment.

We intentionally selected three countries with different assessment regimes, expecting to see the effects of this in the bibliometric analyses we undertook. However, we found that the data alone do not allow us to conclude that any one system has a beneficial or detrimental influence on performance. Rather, the data suggest that it is not the specific system that makes a difference but the fact that performance becomes a central topic of conversation.

In order to better understand the mechanisms behind changing performance, restricting scrutiny to mere numbers is insufficient. Contextual information at various levels of aggregation—within and outside the institutions—is highly relevant.