Skip to main content
Log in

Comparison of the effect of mean-based method and z-score for field normalization of citations at the level of Web of Science subject categories

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Field normalization is a necessary step in a fair cross-field comparison of citation impact. In practice, mean-based method (m-score) is the most popular method for field normalization. However, considering that mean-based method only utilizes the central tendency of citation distribution in the normalization procedure and dispersion is also a significant characteristic, an open and important issue is whether alternative normalization methods which take both central tendency and variability into account perform better than mean-based method. With the aim of collapsing citation distributions of different fields into a universal distribution, this study compares the normalization effect of m-score and z-score based on 236 Web of Science (WoS) subject categories. The results show that both m-score and z-score have remarkable normalization effect as compared with raw citations, but neither of them can realize the ideal goal of “universality of citation distributions”. The results also suggest that m-score is generally preferable to z-score. The essential cause that m-score has an edge over z-score as a whole has a direct relationship with the characteristics of skewed citation distributions in which case m-score is more applicable than z-score.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. The values of mean-based indicator are also called m-score in this paper. The specific meaning of m-score can be clearly inferred from the context.

  2. Mean with zero cited (or uncited) publications refers to average citation rates of publications in the case of including uncited publications. Median with zero cited publication, mean/median without zero cited publications have similar meanings in this paper.

  3. In a small field, especially when z takes small values, even the nearest observed global top z % percentage will deviate from the theoretical z % significantly. This deviation introduces a significant error for small fields which makes top z % method less robust.

  4. Though the MDs of z-score are much smaller than those of raw citations at the first seven top z % sections, MD of z-score at top 80 % is slightly greater than that of raw citations.

  5. In the commonly used mean-based normalization, the reference standard for the citation count of a publication is the average citation rates of the reference set to which the publication under consideration belongs. The operation of averaging implies that the citation counts of publications within a reference set are comparable and additive.

References

  • Abramo, G., Cicero, T., & D’Angelo, C. A. (2012a). How important is choice of the scaling factor in standardizing citations? Journal of Informetrics, 6(4), 645–654.

    Article  Google Scholar 

  • Abramo, G., Cicero, T., & D’Angelo, C. A. (2012b). Revisiting the scaling of citations for research evaluation. Journal of Informetrics, 6(4), 470–479.

    Article  Google Scholar 

  • Albarrán, P., Crespo, J. A., Ortuño, I., & Ruiz-Castillo, J. (2011). The skewness of science in 219 sub-fields and a number of aggregates. Scientometrics, 88(2), 385–397.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2009). Universality of citation distributions—A validation of Radicchi et al.’s relative indicator c f = c/c 0 at the micro level using data from chemistry. Journal of the American Society for Information Science and Technology, 60(8), 1664–1670.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H.-D. (2008). Citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Enviromental Politics, 8, 93–102.

    Article  Google Scholar 

  • Braun, T., & Glänzel, W. (1990). United Germany: The new scientific superpower? Scientometrics, 19(5–6), 513–521.

    Article  Google Scholar 

  • Castellano, C., & Radicchi, F. (2009). On the fairness of using relative indicators for comparing citation performance in different disciplines. Archivum Immunologiae et Therapiae Experimentalis, 57(2), 85–90.

    Article  Google Scholar 

  • Crespo, J. A., Li, Y., & Ruiz-Castillo, J. (2013). The measurement of the effect on citation inequality of differences in citation practices across scientific fields. PLoS One, 8(3), e58727.

    Article  Google Scholar 

  • De Bruin, R. E., Kint, A., Luwel, M., & Moed, H. F. (1993). A study of research evaluation and planning: The University of Ghent. Research Evaluation, 3(1), 25–41.

    Article  Google Scholar 

  • Garfield, E. (1979a). Citation indexing—Its theory and application in Science, Technology and Humanities. New York: Wiley & Sons.

    Google Scholar 

  • Garfield, E. (1979b). Is citation analysis a legitimate evaluation tool? Scientometrics, 1(4), 359–375.

    Article  Google Scholar 

  • Glänzel, W., Thijs, B., Schubert, A., & Debackere, K. (2009). Subfield-specific normalized relative indicators and a new generation of relational charts: Methodological foundations illustrated on the assessment of institutional research performance. Scientometrics, 78(1), 165–188.

    Article  Google Scholar 

  • Katz, J. S. (1999). Bibliometric indicators and the social sciences. Brighton, UK: SPRU, University of Sussex.

    Google Scholar 

  • Katz, J. S. (2000). Scale-independent indicators and research evaluation. Science and Public Policy, 27(1), 23–36.

    Article  Google Scholar 

  • Larivière, V., & Gingras, Y. (2011). Average of ratios vs. ratio of averages: An empirical analysis of four levels of aggregation. Journal of Informetrics, 5(3), 392–399.

    Article  Google Scholar 

  • Lee, G. J. (2010). Assessing publication performance of research units: Extensions through operational research and economic techniques. Scientometrics, 84(3), 717–734.

    Article  Google Scholar 

  • Li, Y., Radicchi, F., Castellano, C., & Ruiz-Castillo, J. (2013). Quantitative evaluation of alternative field normalization procedures. Journal of Informetrics, 7(3), 746–755.

    Article  Google Scholar 

  • Lillquist, E., & Green, S. (2010). The discipline dependence of citation statistics. Scientometrics, 84(3), 749–762.

    Article  Google Scholar 

  • Lundberg, J. (2007). Lifting the crown—citation z-score. Journal of Informetrics, 1(2), 145–154.

    Article  Google Scholar 

  • McAllister, P. R., Narin, F., & Corrigan, J. G. (1983). Programmatic evaluation and comparison based on standardized citation scores. IEEE Transactions on Engineering Management, 30(4), 205–211.

    Article  Google Scholar 

  • Moed, H. F. (2010). CWTS crown indicator measures citation impact of a research groups’s publication oeuvre. Journal of Informetrics, 4(3), 436–438.

    Article  Google Scholar 

  • Moed, H. F., De Bruin, R. E., & van Leeuwen, T. N. (1995). New bibliometric tools for the assessment of national research performance: Database description, overview of indicators and first applications. Scientometrics, 33(3), 381–422.

    Article  Google Scholar 

  • Narin, F. (1976). Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity. Washington DC: National Science Foundation.

    Google Scholar 

  • Nederhof, A. J. (2006). Bibliometric monitoring of performance in the social sciences and the humanities: A review. Scientometrics, 66(1), 81–100.

    Article  MathSciNet  Google Scholar 

  • Opthof, T., & Leydesdorff, L. (2010). Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance. Journal of Informetrics, 4(3), 423–430.

    Article  Google Scholar 

  • Radicchi, F., & Castellano, C. (2011). Rescaling citations of publications in physics. Pysical Review E, 83(4), 046116.

    Google Scholar 

  • Radicchi, F., & Castellano, C. (2012). A reverse engineering approach to the suppression of citation biases reveals universal properties of citation distributions. PLoS One, 7(3), e33833.

    Article  Google Scholar 

  • Radicchi, F., Fortunato, S., & Castellano, C. (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences of the United States of America, 105(45), 17268–17272.

    Article  Google Scholar 

  • Schubert, A., & Braun, T. (1986). Relative indicators and relational charts for comparative assessment of publication output and citation impact. Scientometrics, 9(5–6), 281–291.

    Article  Google Scholar 

  • Schubert, A., & Braun, T. (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36(3), 311–324.

    Article  Google Scholar 

  • Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628–638.

    Article  Google Scholar 

  • Spaan, J. A. E. (2010). The danger of pseudoscience in informetrics. Journal of Informetrics, 4(3), 439–440.

    Article  Google Scholar 

  • Thompson, B. (1993). GRE percentile ranks cannot be added or averaged: A position paper exploring the scaling characteristics of percentile ranks, and the ethical and legal culpabilities created by adding percentile ranks in making “high-stakes” admission decisions. Paper presented at the Annual Meeting of the Mid-South Educational Research Association, New Orleans, LA.

  • van Raan, A. F. J., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & Waltman, L. (2010a). The new set of bibliometric indicators of CWTS. Paper presented at the Eleventh International Conference on Science and Technology Indicators, Leiden, The Netherlands.

  • van Raan, A. F. J., van Leeuwen, T. N., Visser, M. S., van Eck, N. J., & Waltman, L. (2010b). Rivals for the crown: Reply to Opthof and Leydesdorff. Journal of Informetrics, 4(3), 431–435.

    Article  Google Scholar 

  • Vinkler, P. (2012). The case of scientometricians with the “absolute relative” impact indicator. Journal of Informetrics, 6(2), 254–264.

    Article  Google Scholar 

  • Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. J. (2011). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5(1), 37–47.

    Article  Google Scholar 

  • Waltman, L., van Eck, N. J., & van Raan, A. F. J. (2012). Universality of citation distributions revisited. Journal of the American Society for Information Science and Technology, 63(1), 72–77.

    Article  Google Scholar 

Download references

Acknowledgments

This study is supported by a grant from the National Education Sciences Planning program during the 12th Five-Year period (No. CIA110141).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zhihui Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Cheng, Y. & Liu, N.C. Comparison of the effect of mean-based method and z-score for field normalization of citations at the level of Web of Science subject categories. Scientometrics 101, 1679–1693 (2014). https://doi.org/10.1007/s11192-014-1294-7

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-014-1294-7

Keywords

Navigation