Abstract
Scholarly impact assessment has always been a hot issue. It has played an important role in evaluating researchers, scientific papers, scientific teams, and institutions within science of science. Scholarly impact assessment is also used to address fundamental issues, such as reward evaluation, funding allocation, promotion and recruitment decision. Scholars generally agree that it is more reasonable to use weighted citations to assess the scholarly impact. Although a great number of researchers use weighted citations to access the scholarly impact, there is a lack of a systematic summary of citation weighting methods. To fill the gap, this paper summarizes the existing classical indicators and weighting methods used in measuring scholarly impact from the perspectives of articles, authors and journals. We also summarize the focus of the indicators involved in this paper and the weighting factors that involved in the weighting methods. Finally, we discuss the open issues to try to discover the hidden trends of citation weighting. Through this paper, we can not only have a clearer understanding of the weighting methods in the scholarly impact assessment, but also think more deeply about the weighting factors to be explored.



Similar content being viewed by others
References
Abujbara, A., Ezra, J., & Radev, D. R. (2013). Purpose and polarity of citation: Towards NLP-based bibliometrics. In Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: Human language technologies (pp. 596–606).
Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2010). hg-index: A new index to characterize the scientific output of researchers based on the h- and g-indices. Scientometrics, 82(2), 391–400.
Anderson, T. R., Hankin, R. K. S., & Killworth, P. D. (2008). Beyond the Durfee square: Enhancing the h-index to score total publication output. Scientometrics, 76(3), 577–588.
Bai, X., Feng, X., Lee, I., Zhang, J., & Ning, Z. (2016a). Identifying anomalous citations for objective evaluation of scholarly article impact. PLoS ONE, 11(9), 1–15.
Bai, X., Hou, J., Du, H., Kong, X., & Xia, F. (2017a). Evaluating the impact of articles with geographical distances between institutions. In International conference on world wide web companion (pp. 1243–1244).
Bai, X., Liu, H., Zhang, F., Ning, Z., Kong, X., Lee, I., et al. (2017b). An overview on evaluating and predicting scholarly article impact. Information, 8(3), 73–86.
Bai, X., Zhang, J., Cui, H., Ning, Z., & Xia, F. (2016b). PNCOIRank: Evaluating the impact of scholarly articles with positive and negative citations. In International conference companion on world wide web (pp. 9–10).
Batista, P. D., Campiteli, M. G., & Kinouchi, O. (2006). Is it possible to compare researchers with different scientific interests. Scientometrics, 68(1), 179–189.
Bergstrom, C. (2007). Eigenfactor: Measuring the value and prestige of scholarly journals. College and Research Libraries News, 68(5), 314–316.
Bollen, J., Rodriquez, M. A., & Van de Sompel, H. (2006). Journal status. Scientometrics, 69(3), 669–687.
Bollen, J., Sompel, H. V. D., Hagberg, A., & Chute, R. (2009). A principal component analysis of 39 scientific impact measures. PLoS ONE, 4(6), e6022–e6032.
Bornmann, L., & Marx, W. (2016). The journal impact factor and alternative metrics: A variety of bibliometric measures has been developed to supplant the impact factor to better assess the impact of individual research papers. EMBO Reports, 17(8), 1094–1097.
Braun, T., Glänzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169–173.
Brown, R. J. C. (2009). A simple method for excluding self-citation from the h-index: The b-index. Online Information Review, 33(6), 1129–1136.
Callaway, E. (2016). Publishing elite turns against impact factor. Nature, 535(7611), 210–211.
Cantín, M., Muñoz, M., & Roa, I. (2015). Comparison between impact factor, eigenfactor score, and scimago journal rank indicator in anatomy and morphology journals. International Journal of Morphology, 33(3), 1183–1188.
Casadevall, A., Bertuzzi, S., Buchmeier, M. J., Davis, R. J., Drake, H., Fang, F. C., et al. (2016). Asm journals eliminate impact factor information from journal websites. Applied and Environmental Microbiology, 1(4), 2407–2408.
Casadevall, A., & Fang, F. C. (2014). Causes for the persistence of impact factor mania. MBio, 5(2), e00,064–14.
Cavalcanti, D. C., Prudncio, R. B. C., Pradhan, S. S., Shah, J. Y., & Pietrobon, R. S. (2011). Good to be bad? Distinguishing between positive and negative citations in scientific impact. In IEEE international conference on TOOLS with artificial intelligence (pp. 156–162).
Costas, R., & Bordons, M. O. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1(3), 193–203.
Costas, R., Zahedi, Z., & Wouters, P. (2015). Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019.
Davis, P. M. (2008). Eigenfactor: Does the principle of repeated improvement result in better estimates than raw citation counts? Journal of the American Society for Information Science and Technology, 59(13), 2186–2188.
Dellavalle, R. P., Schilling, L. M., Rodriguez, M. A., van de Sompel, H., & Bollen, J. (2007). Refining dermatology journal impact factors using PageRank. Journal of the American Academy of Dermatology, 57(1), 116–119.
Ding, Y., & Yan, E. (2010). Measuring scholarly impact in heterogeneous networks. Proceedings of the Association for Information Science and Technology, 47(1), 1–7.
Dong, P., Loh, M., & Mondry, A. (2005). The “impact factor” revisited. Biomedical digital libraries, 2(1), 1–8.
Eck, N. J. V., & Waltman, L. (2008). Generalizing the h- and g-indices. Journal of Informetrics, 2(4), 263–271.
Egghe, L. (2006). An improvement of the H-index: The G-index. ISSI, 2(1), 8–9.
Elliott, D. B. (2014). The impact factor: A useful indicator of journal quality or fatally flawed? Ophthalmic and Physiological Optics, 34(1), 4–7.
Fersht, A. R. (2009). The most influential journals: Impact factor and eigenfactor. Proceedings of the National Academy of Sciences of the United States of America, 106(17), 6883–6884.
Fiala, D. (2012). Time-aware pagerank for bibliographic networks. Journal of Informetrics, 6(3), 370–388.
Fiala, D. (2014). Current index: A proposal for a dynamic rating system for researchers. Journal of the American Society for Information Science and Technology, 65(4), 850–855.
Fiala, D., Rousselot, F., & Jezek, K. (2008). Pagerank for bibliographic networks. Scientometrics, 76(1), 135–158.
Franceschet, M. (2010). The difference between popularity and prestige in the sciences and in the social sciences: A bibliometric analysis. Journal of Informetrics, 4(1), 55–63.
Franceschet, M., & Colavizza, G. (2018). Timerank: A dynamic approach to rate scholars using citations. Journal of Informetrics, 11(4), 1128–1141.
Fujimagari, H., & Fujita, K. (2015). Detecting research fronts using neural network model for weighted citation network analysis. Journal of Information Processing, 23(6), 753–758.
Fujita, K., Kajikawa, Y., Mori, J., & Sakata, I. (2014). Detecting research fronts using different types of weighted citation networks. Journal of Engineering and Technology Management, 32, 129–146.
Galam, S. (2011). Tailor based allocations for multiple authorship: A fractional gh-index. Scientometrics, 89(1), 365–379.
Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA, 295(1), 90–93.
Gómez-Núñez, A. J., Batagelj, V., Vargas-Quesada, B., Moya-Anegón, F., & Chinchilla-Rodríguez, Z. (2014). Optimizing SCImago journal & country rank classification by community detection. Journal of Informetrics, 8(2), 369–383.
Gonzalezpereira, B., Guerrerobote, V. P., & De Moyaanegon, F. (2010). A new approach to the metric of journals scientific prestige: The SJR indicator. Journal of Informetrics, 4(3), 379–391.
Guerrero-Bote, V. P., & Moya-Anegón, F. (2012). A further step forward in measuring journals scientific prestige: The SJR2 indicator. Journal of Informetrics, 6(4), 674–688.
Habibzadeh, F., & Yadollahie, M. (2008). Journal weighted impact factor: A proposal. Journal of Informetrics, 2(2), 164–172.
Hagen, N. T. (2010). Harmonic publication and citation counting: Sharing authorship credit equitably not equally, geometrically or arithmetically. Scientometrics, 84(3), 785–793.
Hall, C. M., & Page, S. J. (2015). Following the impact factor: Utilitarianism or academic compliance? Tourism Management, 51, 309–312.
Harzing, A. W., & van der Wal, R. (2009). A google scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60(1), 41–46.
Ha, T. C., Tan, S. B., & Soo, K. C. (2006). The journal impact factor: Too much of an impact? Annals-Academy of Medicine Singapore, 35(12), 911–916.
Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.
Jha, R., Jbara, A. A., Qazvinian, V., & Radev, D. R. (2017). NLP-driven citation analysis for scientometrics. Natural Language Engineering, 23(1), 93–130.
Jin, B. (2006). H-Index: An evaluation indicator proposed by scientist. Science Focus, 1(1), 8–9.
Jin, B., Liang, L., Rousseau, R., & Egghe, L. (2007). The R-and AR-indices: Complementing the h-index. Chinese Science Bulletin, 52(6), 855–863.
Kianifar, H., Sadeghi, R., & Zarifmahmoudi, L. (2014). Comparison between impact factor, eigenfactor metrics, and scimago journal rank indicator of pediatric neurology journals. Acta Informatica Medica, 22(2), 103–106.
Kong, X., Mao, M., Wang, W., Liu, J., & Xu, B. (2018). Voprec: Vector representation learning of papers with text information and structural identity for recommendation. IEEE Transactions on Emerging Topics in Computing, PP(99), 1.
Kosmulski, M. (2006). A new Hirsch-type index saves time and works equally well as the original h-index. ISSI Newsletter, 2(3), 4–6.
Leydesdorff, L., & Bornmann, L. (2011). How fractional counting of citations affects the impact factor: Normalization in terms of differences in citation potentials among fields of science. Journal of the American Society for Information Science and Technology, 62(2), 217–229.
Leydesdorff, L., & Opthof, T. (2010). Normalization at the field level: Fractional counting of citations. Journal of Informetrics, 4(4), 644–646.
Leydesdorff, L., & Opthof, T. (2011). Remaining problems with the “New Crown Indicator” (MNCS) of the CWTS. Journal of Informetrics, 5(1), 224–225.
Lundberg, J. (2007). Lifting the crown—citation z-score. Journal of Informetrics, 1(2), 145–154.
Luo, D., Gong, C., Hu, R., Duan, L., & Ma, S. (2016). Ensemble enabled weighted PageRank. arXiv preprint arXiv:160405462.
Mathur, V. P., Sharma, A., et al. (2009). Impact factor and other standardized measures of journal citation: A perspective. Indian Journal of Dental Research, 20(1), 81–85.
Mingers, J., & Yang, L. (2017). Evaluating journal quality: A review of journal citation indicators and ranking in business and management. European Journal of Operational Research, 257(1), 323–337.
Moed, H. F. (2010a). CWTS crown indicator measures citation impact of a research group’s publication oeuvre. Journal of Informetrics, 4(3), 436–438.
Moed, H. F. (2010b). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.
Moustafa, K. (2015). The disaster of the impact factor. Science and Engineering Ethics, 21(1), 139–142.
Nykl, M., Campr, M., & Jezek, K. (2015). Author ranking based on personalized pagerank. Journal of Informetrics, 9(4), 777–799.
Nykl, M., Jezek, K., Fiala, D., & Dostal, M. (2014). Pagerank variants in the evaluation of citation networks. Journal of Informetrics, 8(3), 683–692.
Opthof, T., & Leydesdorff, L. (2010). Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance. Journal of Informetrics, 4(3), 423–430.
Ordunamalea, E., & Lopezcozar, E. D. (2014). Google scholar metrics evolution: An analysis according to languages. Scientometrics, 98(3), 2353–2367.
Pajić, D. (2015). On the stability of citation-based journal rankings. Journal of Informetrics, 9(4), 990–1006.
Priem, J., & Hemminger, B. H. (2010). Scientometrics 2.0: New metrics of scholarly impact on the social web. First Monday, 15(7), 1.
Ramin, S., & Shirazi, A. S. (2012). Comparison between Impact factor, SCImago journal rank indicator and eigenfactor score of nuclear medicine journals. Nuclear Medicine Review, 15(2), 132–136.
Roldan-Valadez, E., Orbe-Arteaga, U., & Rios, C. (2018). Eigenfactor score and alternative bibliometrics surpass the impact factor in a 2-years ahead annual-citation calculation: A linear mixed design model analysis of radiology, nuclear medicine and medical imaging journals. La Radiologia Medica, 123, 1–11.
Rousseau, R. (2008). Woeginger’s axiomatisation of the h-index and its relation to the g-index, the h(2)-index and the R2-index. Journal of Informetrics, 2(4), 335–340.
Ruiz-Castillo, J., & Waltman, L. (2015). Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journal of Informetrics, 9(1), 102–117.
Saad, G. (2006). Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics, 69(1), 117–120.
Sayyadi, H., & Getoor, L. (2009). Futurerank: Ranking scientific articles by predicting their future PageRank. In Siam international conference on data mining (pp. 533–544).
Schreiber, M. (2008). EDITORIAL: To share the fame in a fair way, hm modifies h for multi-authored manuscripts. New Journal of Physics, 10(4), 1131–1137.
Schreiber, M. (2010). The influence of self-citation corrections and the fractionalised counting of multi-authored manuscripts on the Hirsch index. Annalen Der Physik, 18(9), 607–621.
Schubert, A., Glänzel, W., & Thijs, B. (2006). The weight of author self-citations. A fractional approach to self-citation counting. Scientometrics, 67(3), 503–514.
Sekercioglu, C. H. (2008). Quantifying coauthor contributions. Science, 322(5900), 371–371.
Smolinsky, L. (2016). Expected number of citations and the crown indicator. Journal of Informetrics, 10(1), 43–47.
Su, C., Pan, Y. T., Zhen, Y. N., Ma, Z., Yuan, J. P., Guo, H., et al. (2011). PrestigeRank: A new evaluation method for papers and journals. Journal of Informetrics, 5(1), 1–13.
Vaccario, G., Medo, M., Wider, N., & Mariani, M. S. (2017). Quantifying and suppressing ranking bias in a large citation network. Journal of Informetrics, 11(3), 766–782.
Valenzuela, M., Ha, V., & Etzioni, O. (2015). Identifying meaningful citations. In Association for the advancement of artificial intelligence WS-15-13 (pp. 21–26).
Van, H. B. A., Phelps, J., Barnes, M., & Suk, W. A. (2000). Evaluating scientific impact. Environmental Health Perspectives, 108(9), 392–393.
Walker, D., Xie, H., Yan, K. K., & Maslov, S. (2007). Ranking scientific publications using a model of network traffic. Journal of Statistical Mechanics: Theory and Experiment, 6(6), 1–5.
Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365–391.
Waltman, L., & van Eck, N. J. (2008). Some comments on the journal weighted impact factor proposed by Habibzadeh and Yadollahie. Journal of Informetrics, 2(4), 369–372.
Waltman, L., & Van Eck, N. J. (2010). The relation between eigenfactor, audience factor, and influence weight. Journal of the Association for Information Science and Technology, 61(7), 1476–1486.
Waltman, L., & van Eck, N. J. (2012). A new methodology for constructing a publication-level classification system of science. Journal of the American Society for Information Science and Technology, 63(12), 2378–2392.
Waltman, L., & van Eck, N. J. (2013a). Source normalized indicators of citation impact: An overview of different approaches and an empirical comparison. Scientometrics, 96(3), 699–716.
Waltman, L., & Van Eck, N. J. (2013b). A systematic empirical comparison of different approaches for normalizing citation impact indicators. Journal of Informetrics, 7(4), 833–849.
Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., & Visser, M. S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7(2), 272–285.
Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., Visser, M. S., & Van Raan, A. F. J. (2011a). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5(1), 37–47.
Waltman, L., Yan, E., & van Eck, N. J. (2011b). A recursive field-normalized bibliometric performance indicator: An application to the field of library and information science. Scientometrics, 89(1), 301–314.
Wang, l. (2013). A new h-type index A+ index. Information Magazine, 1, 55–58.
Wang, Y., Tong, Y., & Zeng, M. (2013). Ranking scientific articles by exploiting citations, authors, journals, and time information. In AAAI conference on artificial intelligence (pp. 933–939).
Wang, S., Xie, S., Zhang, X., Li, Z., Yu, P. S., & Shu, X. (2016). Future influence ranking of scientific literature. Computer Science, 9, 1–9.
Wan, X., & Liu, F. (2014). Are all literature citations equally important? Automatic citation strength estimation and its applications. Journal of the Association for Information Science & Technology, 65(9), 1929–1938.
Wesley-Smith, I., Bergstrom, C. T., & West, J. D. (2016). Static ranking of scholarly papers using article-level eigenfactor (ALEF). arXiv preprint arXiv:160608534.
West, J. D., Bergstrom, T. C., & Bergstrom, C. T. (2010). The eigenfactor metrics™: A network approach to assessing scholarly journals. College and Research Libraries, 71(3), 236–244.
West, J. D., Jensen, M. C., Dandrea, R. J., Gordon, G. J., & Bergstrom, C. T. (2013). Author-level eigenfactor metrics: Evaluating the influence of authors, institutions, and countries within the social science research network community. Journal of the Association for Information Science and Technology, 64(4), 787–801.
Wu, Q. (2010). The w-index: A measure to assess scientific impact by focusing on widely cited papers. Journal of the American Society for Information Science and Technology, 61(3), 609–614.
Yan, E., & Ding, Y. (2011). Discovering author impact: A PageRank perspective. Information Processing and Management, 47(1), 125–134.
Zhang, C. T. (2009a). A proposal for calculating weighted citations based on author rank. Embo Reports, 10(5), 416–417.
Zhang, C. T. (2009b). The e-index, complementing the h-index for excess citations. PLoS ONE, 4(5), e5429–e5432.
Zhou, D., Orshanskiy, S. A., Zha, H., & Giles, C. L. (2007). Co-ranking authors and documents in a heterogeneous network. In IEEE international conference on data mining (pp. 739–744).
Zhu, X., Turney, P., Lemire, D., & Vellino, A. (2015). Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology, 66(2), 408–427.
Zitt, M. (2010). Citing-side normalization of journal impact: A robust variant of the Audience Factor. Journal of Informetrics, 4(3), 392–406.
Zitt, M., Ramanana-Rahary, S., & Bassecoulard, E. (2005). Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation. Scientometrics, 63(2), 373–401.
Zitt, M., & Small, H. (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59(11), 1856–1860.
Zyczkowski, K. (2010). Citation graph, weighted impact factors and performance indices. Scientometrics, 85(1), 301–315.
Acknowledgements
This work was partially supported by the National Natural Science Foundation of China (61872054), the Fund for Promoting the Reform of Higher Education by Using Big Data Technology, Energizing Teachers and Students to Explore the Future (2017A01002), and the Fundamental Research Funds for the Central Universities (DUT18JC09).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Cai, L., Tian, J., Liu, J. et al. Scholarly impact assessment: a survey of citation weighting solutions. Scientometrics 118, 453–478 (2019). https://doi.org/10.1007/s11192-018-2973-6
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-018-2973-6