Abstract
Journal impact factor (JIF) quartiles are often used as a convenient means of conducting research evaluation, abstracting the underlying JIF values. We highlight and investigate an intrinsic problem associated with this approach: the differences between quartile boundary JIF values are usually very small and often so small that journals in different quartiles cannot be considered meaningfully different with respect to impact. By systematically investigating JIF values in recent editions of the Journal Citation Reports (JCR) we determine it is typical to see between 10 and 30% poorly differentiated journals in the JCR categories. Social sciences are more affected than science categories. However, this global result conceals important variation and we also provide a detailed account of poor quartile boundary differentiation by constructing in-depth local quartile similarity profiles for each JCR category. Further systematic analyses show that poor quartile boundary differentiation tends to follow poor overall differentiation which naturally varies by field. In addition, in most categories the journals that experience a quartile shift are the same journals that are poorly differentiated. Our work provides sui generis documentation of the continuing phenomenon of impact factor inflation and also explains and reinforces some recent findings on the ranking stability of journals and on the JIF-based comparison of papers. Conceptually there is a fundamental problem in the fact that JIF quartile classes artificially magnify underlying differences that can be insignificant. We in fact argue that the singular use of JIF quartiles is a second order ecological fallacy. We recommend the abandonment of the quartiles reification as an independent method for the research assessment of individual scholars.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Since 2018 China has surpassed the United States of America and is the top ranking country with respect to research articles and reviews indexed in the Science Citation Index Expanded (Liu 2020). The above-mentioned policies have no doubt contributed to this performance. However, Zhu (2020) has recently reported that 2020 is a year of major reform for the evaluation of research in China: recent policy documents jointly issued by the Ministry of Education and the Ministry of Science and Technology explicitly de-emphasize Science Citation Index publications.
It is worth noting in passing that Elsevier launched in late 2016 a direct rival to the JIF in the form of the “CiteScore” (Zijlstra and McCullough 2016) together with an accompanying set of CiteScore metrics. The use of a compromise three-year citation window (recently changed to four years) was the main argument offered in support of the alternative metric’s “robust approach”.
From the current JCR scope notes for category box plot it seems this specific article is the inspiration for the current implementation of the JIF quartile classes as a way to compare journal performance across categories.
The account provided in this paragraph follows the description provided in the InCites Indicators Handbook (Clarivate Analytics 2018, p. 10).
Hypothetical JIF values were generated in the R language and environment for statistical computing (R Core Team 2020) and follow a standard lognormal distribution; the lognormal distribution is known to adequately approximate various scientometric quantities, especially citations (Brito and Rodríguez-Navarro 2018; Thelwall 2016; Vîiu 2018).
For the fictitious category from Table 1 there are (262 – 26) / 2 = 325 JIF differences. Only 6.77% of these are not meaningful with δ = 0.1. Some of these non-meaningful differences are precisely those at the quartile boundaries (Q1, Q2) and (Q3, Q4) which lead to the 23% poorly differentiated journals in the category.
This can also be seen in the LQS profiles of the subject categories in Electronic Supplementary Material 2.
Although not our primary line of inquiry, we also looked at the evolution of mean JIF values in each JCR category between 2015 and 2018: all except 8 of the 234 categories that could be compared experienced an increase in the mean JIF, typically ranging between 12 and 43%.
References
Adigozalova, N. A. (2019). Quartile weighted impact factor. COLLNET Journal of Scientometrics and Information Management, 13(2), 365–386. https://doi.org/10.1080/09737766.2020.1716646.
Aksnes, D. W. (2003). A macro study of self-citation. Scientometrics, 56(2), 235–246. https://doi.org/10.1023/A:1021919228368.
Albarrán, P., Crespo, J. A., Ortuño, I., & Ruiz-Castillo, J. (2011). The skewness of science in 219 sub-fields and a number of aggregates. Scientometrics, 88(2), 385–397. https://doi.org/10.1007/s11192-011-0407-9.
Althouse, B. M., West, J. D., Bergstrom, C. T., & Bergstrom, T. (2009). Differences in impact factor across fields and over time. Journal of the American Society for Information Science and Technology, 60(1), 27–34. https://doi.org/10.1002/asi.20936.
Archambault, É., & Larivière, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635–649. https://doi.org/10.1007/s11192-007-2036-x.
Bornmann, L. (2017). Confidence intervals for journal impact factors. Scientometrics, 111(3), 1869–1871. https://doi.org/10.1007/s11192-017-2365-3.
Bornmann, L., & Marx, W. (2014). How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations. Scientometrics, 98(1), 487–509. https://doi.org/10.1007/s11192-013-1161-y.
Brito, R., & Rodríguez-Navarro, A. (2018). Research assessment by percentile-based double rank analysis. Journal of Informetrics, 12(1), 315–329. https://doi.org/10.1016/j.joi.2018.01.011.
Brito, R., & Rodríguez-Navarro, A. (2019). Evaluating research and researchers by the journal impact factor: Is it better than coin flipping? Journal of Informetrics, 13(1), 314–324. https://doi.org/10.1016/j.joi.2019.01.009.
Campanario, J. M. (2014). The effect of citations on the significance of decimal places in the computation of journal impact factors. Scientometrics, 99(2), 289–298. https://doi.org/10.1007/s11192-013-1206-2.
Chorus, C., & Waltman, L. (2016). A large-scale analysis of impact factor biased journal self-citations. PLoS ONE, 11(8), e0161021. https://doi.org/10.1371/journal.pone.0161021.
Collier, K. (2019). Announcing the 2019 Journal Citation Reports. https://clarivate.com/webofsciencegroup/article/announcing-the-2019-journal-citation-reports/
Clarivate Analytics. (2018). InCites Indicators Handbook. http://help.incites.clarivate.com/inCites2Live/8980-TRS/version/default/part/AttachmentData/data/InCites-Indicators-Handbook - June 2018.pdf
Curry, S. (2018). Let’s move beyond the rhetoric: It’s time to change how we judge research. Nature, 554(7691), 147–147. https://doi.org/10.1038/d41586-018-01642-w.
Fernández-Ríos, L., & Rodríguez-Díaz, J. (2014). The “impact factor style of thinking”: A new theoretical framework. International Journal of Clinical and Health Psychology, 14(2), 154–160. https://doi.org/10.1016/S1697-2600(14)70049-3.
García, J. A., Rodriguez-Sánchez, R., Fdez-Valdivia, J., & Martinez-Baena, J. (2012). On first quartile journals which are not of highest impact. Scientometrics, 90(3), 925–943. https://doi.org/10.1007/s11192-011-0534-3.
Garfield, E. (1972). Citation analysis as a tool in journal evaluation: Journals can be ranked by frequency and impact of citations for science policy studies. Science, 178(4060), 471–479. https://doi.org/10.1126/science.178.4060.471.
Garfield, E. (1990). How ISI selects journals for coverage: Quantitative and qualitative considerations. Current Contents, 13(22), 185–193.
Garfield, E. (2006). The History and meaning of the journal impact factor. Journal of the American Medical Association, 295(1), 90–93. https://doi.org/10.1001/jama.295.1.90.
Greenwood, D. C. (2007). Reliability of journal impact factor rankings. BMC Medical Research Methodology, 7(1), 48. https://doi.org/10.1186/1471-2288-7-48.
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). The leiden manifesto for research metrics. Nature, 520(7548), 9–11. https://doi.org/10.1038/520429a.
Hintze, J. L., & Nelson, R. D. (1998). Violin plots: A box plot-density trace synergism. American Statistician, 52(2), 181–184. https://doi.org/10.1080/00031305.1998.10480559.
Larivière, V., Kiermer, V., MacCallum, C. J., McNutt, M., Patterson, M., Pulverer, B., et al. (2016). A simple proposal for the publication of journal citation distributions. BioRxiv. https://doi.org/10.1101/062109.
Larivière, V., & Sugimoto, C. R. (2019). The journal impact factor: A brief history, critique, and discussion of adverse effects. In W. Glänzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 3–24). Cham: Springer.
Leydesdorff, L., & Bornmann, L. (2016). The operationalization of “fields” as WoS subject categories (WCs) in evaluative bibliometrics: The cases of “library and information science” and “science & technology studies.” Journal of the Association for Information Science and Technology, 67(3), 707–714. https://doi.org/10.1002/asi.23408.
Leydesdorff, L., Bornmann, L., & Adams, J. (2019). The integrated impact indicator revisited (I3*): A non-parametric alternative to the journal impact factor. Scientometrics, 119(3), 1669–1694. https://doi.org/10.1007/s11192-019-03099-8.
Leydesdorff, L., Wouters, P., & Bornmann, L. (2016). Professional and citizen bibliometrics: Complementarities and ambivalences in the development and use. Scientometrics, 109(3), 2129–2150. https://doi.org/10.1007/s11192-016-2150-8.
Liu, W. (2020). China’s SCI-indexed publications: Facts, feelings, and future directions. ECNU Review of Education, 3(3), 562–569. https://doi.org/10.1177/2096531120933902.
Liu, W., Hu, G., & Gu, M. (2016). The probability of publishing in first-quartile journals. Scientometrics, 106(3), 1273–1276. https://doi.org/10.1007/s11192-015-1821-1.
Lozano, G. A., Larivière, V., & Gingras, Y. (2012). The weakening relationship between the impact factor and papers’ citations in the digital age. Journal of the American Society for Information Science and Technology, 63(11), 2140–2145. https://doi.org/10.1002/asi.22731.
Lundberg, J. (2007). Lifting the crown—citation z-score. Journal of Informetrics, 1(2), 145–154. https://doi.org/10.1016/j.joi.2006.09.007.
Madhan, M., Gunasekaran, S., Rani, M. T., Arunachalam, S., & Abinandanan, T. A. (2020). Chemistry research in India in a global perspective- A scientometrics profile, (February), 1–39. arXiv preprint arXiv:2002.03093v2.
Magri, M.-H., & Solari, A. (1996). The SCI journal citation reports: A potential tool for studying journals? Scientometrics, 35(1), 93–117. https://doi.org/10.1007/BF02018235.
McGill, R., Tukey, J. W., & Larsen, W. A. (1978). Variations of box plots. The American Statistician, 32(1), 12–16. https://doi.org/10.1080/00031305.1978.10479236.
McVeigh, M. E., & Mann, S. J. (2009). The Journal impact factor denominator. Journal of the American Medical Association, 302(10), 1107–1109. https://doi.org/10.1001/jama.2009.1301.
Milojević, S. (2020). Practical method to reclassify web of science articles into unique subject categories and broad disciplines. Quantitative Science Studies, 1(1), 183–206. https://doi.org/10.1162/qss_a_00014.
Miranda, R., & Garcia-Carpintero, E. (2019). Comparison of the share of documents and citations from different quartile journals in 25 research areas. Scientometrics, 121(1), 479–501. https://doi.org/10.1007/s11192-019-03210-z.
Müller, R., & de Rijcke, S. (2017). Thinking with indicators. exploring the epistemic impacts of academic performance indicators in the life sciences. Research Evaluation, 26(3), 157–168. https://doi.org/10.1093/reseval/rvx023.
Pajić, D. (2015). On the stability of citation-based journal rankings. Journal of Informetrics, 9(4), 990–1006. https://doi.org/10.1016/j.joi.2015.08.005.
Pudovkin, A. I., & Garfield, E. (2012). Rank normalization of impact factors will resolve Vanclay’s dilemma with TRIF. Scientometrics, 92(2), 409–412. https://doi.org/10.1007/s11192-012-0634-8.
Quan, W., Chen, B., & Shu, F. (2017). Publish or impoverish. Aslib Journal of Information Management, 69(5), 486–502. https://doi.org/10.1108/AJIM-01-2017-0014.
R Core Team. (2020). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. https://www.r-project.org/.
Rafols, I., & Robinson-Garcia, N. (2016). On the dominance of quantitative evaluation in peripherall countries: Auditing research with technologies of distance. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2818335.
Ruiz-Castillo, J., & Costas, R. (2018). Individual and field citation distributions in 29 broad scientific fields. Journal of Informetrics, 12(3), 868–892. https://doi.org/10.1016/j.joi.2018.07.002.
Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ, 314(7079), 497–497. https://doi.org/10.1136/bmj.314.7079.497.
Shu, F., Quan, W., Chen, B., Qiu, J., Sugimoto, C. R., & Larivière, V. (2020). The role of web of science publications in China’s tenure system. Scientometrics, 122(3), 1683–1695. https://doi.org/10.1007/s11192-019-03339-x.
Stern, D. I. (2013). Uncertainty measures for economics journal impact factors. Journal of Economic Literature, 51(1), 173–189. https://doi.org/10.1257/jel.51.1.173.
Thelwall, M. (2016). Are the discretised lognormal and hooked power law distributions plausible for citation data? Journal of Informetrics, 10(2), 454–470. https://doi.org/10.1016/j.joi.2016.03.001.
van Raan, A. (2019). Measuring science: Basic principles and application of advanced bibliometrics. In W. Glänzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 237–280). Cham: Springer.
Vanclay, J. K. (2009). Bias in the journal impact factor. Scientometrics, 78(1), 3–12. https://doi.org/10.1007/s11192-008-1778-4.
Vanclay, J. K. (2012). Impact factor: Outdated artefact or stepping-stone to journal certification? Scientometrics, 92(2), 211–238. https://doi.org/10.1007/s11192-011-0561-0.
Vîiu, G.-A. (2018). The lognormal distribution explains the remarkable pattern documented by characteristic scores and scales in scientometrics. Journal of Informetrics, 12, 401–415. https://doi.org/10.1016/j.joi.2018.02.002.
Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365–391. https://doi.org/10.1016/j.joi.2016.02.007.
Wang, Q., & Waltman, L. (2016). Large-scale analysis of the accuracy of the journal classification systems of web of science and scopus. Journal of Informetrics, 10(2), 347–364. https://doi.org/10.1016/j.joi.2016.02.003.
Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. New York: Springer.
Wilhite, A. W., & Fong, E. A. (2012). Coercive citation in academic publishing. Science, 335(6068), 542–543. https://doi.org/10.1126/science.1212540.
Wouters, P., Sugimoto, C. R., Larivière, V., McVeigh, M. E., Pulverer, B., de Rijcke, S., & Waltman, L. (2019). Rethinking impact factors: Better ways to judge a journal. Nature, 569(7758), 621–623. https://doi.org/10.1038/d41586-019-01643-3.
Zhu, J. (2020). Evaluation of scientific and technological research in China’s colleges: A review of policy reforms, 2000–2020. ECNU Review of Education, 3(3), 556–561. https://doi.org/10.1177/2096531120938383.
Zijlstra, H., & McCullough, R. (2016). CiteScore: a new metric to help you track journal performance and make decisions. https://www.elsevier.com/editors-update/story/journal-metrics/citescore-a-new-metric-to-help-you-choose-the-right-journal
Acknowledgements
The authors express their gratitude to the anonymous reviewers whose comments helped to improve significant aspects of the initial manuscript. This paper was financially supported by the Human Capital Operational Program 2014-2020, co-financed by the European Social Fund, under the project POCU/380/6/13/124708 no. 37141/23.05.2019 with the title “Researcher-Entrepreneur on Labour Market in the Fields of Intelligent Specialization (CERT-ANTREP)”, coordinated by the National University of Political Studies and Public Administration.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Vȋiu, GA., Păunescu, M. The lack of meaningful boundary differences between journal impact factor quartiles undermines their independent use in research evaluation. Scientometrics 126, 1495–1525 (2021). https://doi.org/10.1007/s11192-020-03801-1
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-020-03801-1