Skip to main content
SpringerLink
Log in

Comparison of bibliometric measures for assessing relative importance of researchers

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Quantitative evaluation of citation data to support funding decisions has become widespread. For this purpose there exist many measures (indices) and while their properties were well studied there is little comprehensive experimental comparison of the ranking lists obtained when using different methods. A further problem of the existing studies is that lack of available data about net citations prevents researchers from studying the effect of measuring scientific impact by using net citations (all citations minus self-citations). In this paper we use simulated data to study factors that could potentially influence the degree of agreement between the rankings obtained when using different indices with the emphasis given to the comparison of the number of net citations per author to other more established indices. We observe that the researchers publishing papers with a large number of co-authors are systematically ranked higher when using h-index or total citations (TC) instead of the number of citations per author (TCA), that the researchers who publish a small proportion of papers which receive many citations while the rest of their papers receive only few citations are systematically ranked higher when using TCA or TC instead of h-index, and that the authors who have lower proportion of self-citations are ranked higher when considering indices which include the number of net citations in comparison with indices considering only the total citation count. Results are verified and illustrated also by analyzing a large dataset from the field of medical science in Slovenia for the period 1986–2007.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Bartneck, C., & Kokkelmans, S. (2011). Detecting h-index manipulation through self-citation analysis. Scientometrics, 87(1), 85–98.

    Article  Google Scholar 

  • Batista, P. D., Campiteli, M. G., & Kinouchi, O. (2006). Is it possible to compare researchers with different scientific interests? Scientometrics, 68(1), 179–189.

    Article  Google Scholar 

  • Bland, J. M., & Altman, D. G. (2010). Statistical methods for assessing agreement between two methods of clinical measurement. International Journal of Nursing Studies, 47(8), 931–936.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H. D. (2007). What do we know about the h index? Journal of the American Society for Information Science and Technology, 58(9), 1381–1385. doi:10.1002/asi.20609.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., & Daniel, H. D. (2008). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. Journal of the American Society for Information Science and Technology, 59(5), 830–837.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., Daniel, H. D., Wallon, G., & Ledin, A. (2009). Are there really two types of h index variants? A validation study by using molecular life sciences data. Research Evaluation, 18(3), 185–190.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H. D. (2011). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5(3), 346–359.

    Article  Google Scholar 

  • Bras-Amorós, M., Domingo-Ferrer, J., & Torra, V. (2011). A bibliometric index based on the collaboration distance between cited and citing authors. Journal of Informetrics, 5(2), 248–264.

    Article  Google Scholar 

  • Cole, J. R., & Cole, S. (1973). Social stratification in science. Chicago: The University of Chicago Press.

    Google Scholar 

  • Egghe, L. (1987). An exact calculation of price’s law for the law of lotka. Scientometrics, 11(1–2), 81–97.

    Article  Google Scholar 

  • Egghe, L. (1991). The exact place of zipf’s and pareto’s law amongst the classical informetric laws. Scientometrics, 20(1), 93–106.

    Article  Google Scholar 

  • Egghe, L. (1998). Mathematical theories of citation. Scientometrics, 43(1), 57–62.

    Article  MathSciNet  Google Scholar 

  • Egghe, L. (2005). A characterization of the law of lotka in terms of sampling. Scientometrics, 62(3), 321–328.

    Article  Google Scholar 

  • Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69(1), 131–152.

    Article  MathSciNet  Google Scholar 

  • Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.

    Article  Google Scholar 

  • Glanzel, W. (2006). On the h-index—A mathematical approach to a new measure of publication activity and citation impact. Scientometrics, 67(2), 315–321.

    Article  Google Scholar 

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16,569–16,572.

    Article  Google Scholar 

  • Hirsch, J. E. (2007). Does the h index have predictive power? Proceedings of the National Academy of Sciences, 104(49), 19,193–19,198. doi:10.1073/pnas.0707962104.

    Article  Google Scholar 

  • Hristovski, D., Rožić, A., & Adamič, Š. (1996) A decision support system for biomedical research evaluation. In Medical informatics Europe ‘96: Human facets in information technologies, pp 609–613

  • Iglesias, J. E., & Pecharroman, C. (2006). Scaling the h-index for different scientific isi fields. arXiv:physics/0607224.

  • Imperial, J., & Rodrguez-Navarro, A. (2007). Usefulness of hirsch’s h-index to evaluate scientific research in spain. Scientometrics, 71(2), 271–282. doi:10.1007/s11192-007-1665-4.

    Article  Google Scholar 

  • Nerur, S., Sikora, R., Mangalaraj, G., & Balijepally, V. (2005). Assessing the relative influence of journals in a citation network. Commun ACM, 48(11), 71–74.

    Article  Google Scholar 

  • Panaretos, J., & Malesios, C. (2009). Assessing scientific research performance and impact with single indices. Scientometrics, 81(3), 635–670.

    Article  Google Scholar 

  • Pareto, V. (1897). Course d’économie politique. Lausanne: F. Rouge.

    Google Scholar 

  • R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

    Google Scholar 

  • Roediger, H. L. (2006). The h index in science: A new measure of scholarly contribution. APS Observer 19.

  • Saam, N., & Reiter, L. (1999). Lotka’s law reconsidered: The evolution of publication and citation distributions in scientific fields. Scientometrics, 44(2), 135–155.

    Article  Google Scholar 

  • Schreiber, M. (2007). Self-citation corrections for the hirsch index. EPL (Europhysics Letters), 78(3), 30,002.

    Article  Google Scholar 

  • Schreiber, M. (2008a). A modification of the h-index: The hm-index accounts for multi-authored manuscripts. Journal of Informetrics, 2(3), 211–216.

    Article  MathSciNet  Google Scholar 

  • Schreiber, M. (2008b). To share the fame in a fair way, \(h_m\) modifies h for multi-authored manuscripts. New Journal of Physics, 10(4), 040,201.

    Article  Google Scholar 

  • Silagadze, Z. (2010). Citation entropy and research impact estimation. Acta Physica Polonica B, 41(11), 2325–2333.

    Google Scholar 

  • Tol, R. (2011). Credit where credits due: Accounting for co-authorship in citation counts. Scientometrics, 89(1), 291–299.

    Article  Google Scholar 

  • Wan, J. K., Hua, Ph, & Rousseau, R. (2007). The pure h-index: Calculating an authors h-index by taking co-authors into account. Collnet Journal of Scientometrics and Information Management, 1(2), 1–5.

    Article  Google Scholar 

Download references

Acknowledgments

Our sincere thanks of gratitude goes to Dr. Hristovski who developed a programme to automatically analyze the Science Citation Index database (and later Web of Science).

Author information

Authors and Affiliations

  1. Institute for Biostatistics and Medical Informatics, University of Ljubljana, Ljubljana, Slovenia

    Rok Blagus, Brane L. Leskošek & Janez Stare

Authors
  1. Rok Blagus
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brane L. Leskošek
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Janez Stare
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rok Blagus.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 87 KB)

Supplementary material 2 (pdf 62 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Blagus, R., Leskošek, B.L. & Stare, J. Comparison of bibliometric measures for assessing relative importance of researchers. Scientometrics 105, 1743–1762 (2015). https://doi.org/10.1007/s11192-015-1622-6

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-015-1622-6

Keywords

Search

Navigation