Abstract
Bibliometric measures based on citations are widely used in assessing the scientific publication records of authors, institutions and journals. Yet currently favored measures lack a clear theoretical foundation and are known to have counter-intuitive properties. The paper proposes a new approach that is grounded on a theoretical “influence function,” representing explicit prior beliefs about how citations reflect influence. Conditions are derived for robust qualitative comparisons of influence—conditions that can be implemented using readily-available data. Two examples are provided, one using the world’s top-10 economics department, the other using the top-10 economics journals.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The idea that measures of performance should be built on a precise formulation of the objective function is not, of course, new. The most prominent example we know of has been in a strand of the economics literature on the measurement of income inequality, in which the measure is defined as the loss of aggregate social welfare due to inequality, where social welfare is the sum of individual “utilities,” each of which is a stable function of own income; the utility function is taken to be unobserved, and so a matter for prior judgment. The properties of the inequality measure are thus derived from the prior (ethical) assumptions made about social welfare. Dalton (1920) first suggested this approach. An influential formalization and development was provided by Atkinson (1970).
The theory of stochastic dominance has been mainly used for comparing risky portfolios and in comparing income distributions in terms of social welfare or poverty. An important early contribution in the context of portfolio choice was Hadar and Russell (1971). Applications to social welfare and the measurement of poverty and inequality include Atkinson (1970, 1987), Dasgupta et al. (1973) and Shorrocks (1983). A difference to past applications of dominance theory (that we know of) is that in the present case one cares about the number of objects being compared (the number of publications) as well as their “results” (citations in our case, returns to an investment or incomes in other applications).
The analysis for a non-additive but increasing and quasi-concave (or, more generally, Schur-concave) aggregate influence function would have a number of formal similarities with the analysis in Dasgupta et al. (1973), in the context of measuring income inequality.
The inverse citation curve is obtained by flipping the ordinary citation curve—swapping the axes.
This should not be confused with the h-index per publication, which is also called the “normalized h-index” in some of the literature (Alonso et al. 2009).
The following results can be modified to allow for a multiplicative scaling factor, γ i , so that the influence function for records of type i is γ i I(c).
This can be weakened to allow \(I^{\prime}\)(c) = 0 for some c.
Condition (v) is not essential for our main results, though it simplifies things, and by allowing a sufficiently large c max it does not seem unduly restrictive to set \(I^{\prime}\)(c max) = 0.
D has some similarity to a measure of inequality, but note that (unlike an inequality measure) D does not go to zero when all publications have the same citations unless that is at the level c max.
Note that G i (0) = 0 and that \(I^{\prime}\)(c max) = 0 (under the extra assumptions). However, our second-order dominance condition also holds for \(I^{\prime}\)(c max) > 0.
Unlike Engemann and Wall we do not restrict the set of journals where citations come from (they counted citations only in the top seven top general-interest journals, with the aim of capturing the degree to which authors get cited outside their field of interest).
References
Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2009). h-Index: A review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics, 3, 273–289.
Atkinson, A. B. (1970). On the measurement of inequality. Journal of Economic Theory, 2, 244–263.
Atkinson, A. B. (1987). On the measurement of poverty. Econometrica, 55, 749–764.
Dalton, Hugh. (1920). The measurement of the inequality of incomes. Economic Journal, 30, 348–361.
Dasgupta, P., Sen, A., & Starrett, D. (1973). Notes on the measurement of inequality. Journal of Economic Theory, 6, 180–187.
Egghe, Leo. (2006). Theory and practice of the g-index. Scientometrics, 69(1), 131–152.
Engemann, K. M., & Wall, H. J. (2009). A journal ranking for the ambitious economist. Federal Reserve Bank of St. Louis Review, 91(3), 127–139.
Hadar, J., & Russell, W. R. (1971). Stochastic dominance and diversification. Journal of Economic Theory, 3, 288–305.
Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the USA, 102(46), 16569–16572.
Oswald, A. J. (2007). An examination of the reliability of prestigious scholarly journals: Evidence and implications for decision-makers. Economica, 74(293), 21–31.
Shorrocks, A. F. (1983). Ranking income distributions. Economica, 50, 3–17.
Vanclay, J. (2007). On the robustness of the h-index. Journal of the American Society for Information Science and Technology, 58(10), 1547–1550.
Woeginger, G. J. (2008a). An axiomatic characterization of the Hirsch-Index. Mathematical Social Sciences, 56, 224–232.
Woeginger, G. J. (2008b). An axiomatic analysis of Egghe’s g-index. Journal of Informetrics, 2, 364–368.
Author information
Authors and Affiliations
Corresponding authors
Rights and permissions
About this article
Cite this article
Ravallion, M., Wagstaff, A. On measuring scholarly influence by citations. Scientometrics 88, 321–337 (2011). https://doi.org/10.1007/s11192-011-0375-0
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-011-0375-0