Expert judgments versus publication-based metrics: do the two methods produce identical results in measuring academic reputation?

This study aims to assess the validity of citation metrics based on the disciplinary representative survey.

The present project compared citation rankings for individual scientists with expert judgments collected through a survey of 818 Russian sociologists. The Russian Index of Science Citation was used to construct the general population of 3,689 Russian sociologists, to whom the survey was sent by email. The regression analyses of bibliometric indicators and peer review scores for 723 names of scholars mentioned in the survey have been undertaken.

Findings suggest that scientometric indicators predict with significant accuracy the names of the most influential sociologists and those scholars who are not mentioned while they are less relevant for prediction names which received moderate attention in the survey.

This study contributes to the research on the validity of citation metrics by focusing on scientometric indicators, not limited to traditional metrics but including non-standard publication metrics and indicators of potential metric abuse. Besides, the study presents the national bibliometric data source that is especially important for non-Western higher education systems, less presented in the Web of Science or Scopus.