Abstract
The use of the journal impact factor for evaluation purposes has been widely challenged because of the simplistic algorithm involved in impact factor calculation. As a result, a journal's impact factor may be affected by many variables, such as highly cited review articles. In the present study, we aimed to investigate whether and to what extent review articles affect the impact factors of journals. We examined the citations of original research articles and review articles in the field of physics, and compared the impact factors of the journals Science, Nature, and Cell with and without citation counts of review articles included. The results showed that review articles consistently received higher numbers of citations than did original research articles. In addition, review articles accounted for 3–10% of the impact factors of these top journals. Based on our findings, we suggest that sophisticated methods, in addition to the impact factor, be employed for research evaluation purposes.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Journal Impact Factor is a copyright of Clarivate Analytics, formerly known as The Institute for Scientific Information (ISI).
References
Abbott, B. P., Abbott, R., Abbott, T. D., Acernese, F., Ackley, K., Adams, C., et al. (2017a). GW170104: Observation of a 50-solar-mass binary black hole coalescence at redshift 0.2. Physical Review Letters,118(22), 221101. https://doi.org/10.1103/PhysRevLett.118.221101.
Abbott, B. P., Abbott, R., Abbott, T. D., Acernese, F., Ackley, K., Adams, C., et al. (2017b). GW170814: A three-detector observation of gravitational waves from a binary black hole coalescence. Physical Review Letters,119(14), 141101. https://doi.org/10.1103/PhysRevLett.119.141101.
Abbott, B. P., Abbott, R., Abbott, T. D., Acernese, F., Ackley, K., Adams, C., et al. (2017c). GW170817: Observation of gravitational waves from a binary neutron star inspiral. Physical Review Letters,119(16), 161101. https://doi.org/10.1103/PhysRevLett.119.161101.
Akerib, D. S., Alsum, S., Araújo, H. M., Bai, X., Bailey, A. J., Balajthy, J., et al. (2017). Results from a search for dark matter in the complete LUX exposure. Physical Review Letters,118(2), 21303. https://doi.org/10.1103/PhysRevLett.118.021303.
Amiri, M., & Michel, M. C. (2018). Do review articles boost journal impact factors? A longitudinal analysis for five pharmacology journals. Naunyn-Schmiedeberg’s Archives of Pharmacology,391(9), 1033–1035. https://doi.org/10.1007/s00210-018-1529-x.
Bornmann, L., & Marx, W. (2018). Critical rationalism and the search for standard (field-normalized) indicators in bibliometrics. Journal of Informetrics,12(3), 598–604. https://doi.org/10.1016/j.joi.2018.05.002.
Clarivate Analytics. (2018). 2017 Journal impact factor. Journal Citation Reports Science Edition. Retrieved from: http://ipscience-help.thomsonreuters.com/incitesLiveJCR/JCRGro up/jcrOverview.html.
Dimitrov, J. D., Kaveri, S. V., & Bayry, J. (2010). Metrics: Journal’s impact factor skewed by a single paper. Nature,466, 179. https://doi.org/10.1038/466179b.
Falagas, M. E., & Alexiou, V. G. (2008). The top-ten in journal impact factor manipulation. Archivum Immunologiae Et Therapiae Experimentalis,56(4), 223–226. https://doi.org/10.1007/s00005-008-0024-5.
Foo, J. Y. A. (2013). Implications of a single highly cited article on a journal and its citation indexes: A tale of two journals. Accountability in Research,20(2), 93–106. https://doi.org/10.1080/08989621.2013.767124.
Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science,178(4060), 471–479. https://doi.org/10.1300/J123v20n02_05.
Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association,295(1), 90–93. https://doi.org/10.1001/jama.295.1.90.
Hanahan, D., & Weinberg, R. A. (2011). Hallmarks of cancer: The next generation. Cell,144(5), 646–674. https://doi.org/10.1016/j.cell.2011.02.013.
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature,520(7548), 429–431. https://doi.org/10.1038/520429a.
InCites Help. (2018). Highly cited papers. Retrieved from http://ipscience-help.thomsonreuters.com/inCites2Live/indicatorsGroup/aboutHandbook/usingCitationIndicatorsWisely/highlyCitedPapers.html.
Krauskopf, E. (2013). Deceiving the research community through manipulation of the impact factor. Journal of the American Society for Information Science and Technology,64(11), 2403. https://doi.org/10.1002/asi.22905.
Lei, L., & Yan, S. (2016). Readability and citations in information science: Evidence from abstracts and articles of four journals (2003–2012). Scientometrics,107(3), 1361–1372. https://doi.org/10.1007/s11192-016-2036-9.
Liu, W., Hu, G., & Gu, M. (2016). The probability of publishing in first-quartile journals. Scientometrics,106(3), 1273–1276. https://doi.org/10.1007/s11192-015-1821-1.
Liu, W., Liu, F., Zuo, C., & Zhu, J. (2018a). The effect of publishing a highly cited paper on a journal’s impact factor: A case study of the review of particle physics. Learned Publishing,31(3), 261–266. https://doi.org/10.1002/leap.1156.
Liu, W., Zhu, J., Zuo, C., & Wang, H. (2018b). Is it suitable for a journal to bid for publishing a review that is likely to be highly cited? Science and Engineering Ethics. https://doi.org/10.1007/s11948-017-0016-y.
Martin, B. R. (2016). Editors’ JIF-boosting stratagems–Which are appropriate and which not? Research Policy,45(1), 1–7. https://doi.org/10.1016/j.respol.2015.09.001.
Noguchi, J. (2006). The science review article: An opportune genre in the construction of science. Bern: Peter Lang.
Paulus, F. M., Cruz, N., & Krach, S. (2018). The impact factor fallacy. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2018.01487.
Sheldrick, G. M. (2008). A short history of SHELX. Acta Crystallographica Section A,64(1), 112–122. https://doi.org/10.1107/S0108767307043930.
Shi, D., Rousseau, R., Yang, L., & Li, J. (2017). A journal’s impact factor is influenced by changes in publication delays of citing journals. Journal of the Association for Information Science and Technology,68(3), 780–789. https://doi.org/10.1002/asi.23706.
Stephan, P., Veugelers, R., & Wang, J. (2017). Reviewers are blinkered by bibliometrics. Nature,544(7651), 411–412. https://doi.org/10.1038/544411a.
Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics,10(2), 365–391. https://doi.org/10.1016/j.joi.2016.02.007.
Yang, D.-H., Li, X., Sun, X., & Wan, J. (2016). Detecting impact factor manipulation with data mining techniques. Scientometrics,109(3), 1989–2005. https://doi.org/10.1007/s11192-016-2144-6.
Yu, G., Yang, D.-H., & He, H.-X. (2011). An automatic recognition method of journal impact factor manipulation. Journal of Information Science,37(3), 235–245. https://doi.org/10.1177/0165551511400954.
Yu, G., Yang, D.-H., & Liang, W. (2010). Reliability-based citation impact factor and the manipulation of impact factor. Scientometrics,83(1), 259–270. https://doi.org/10.1007/s11192-009-0083-1.
Acknowledgements
We very much appreciate the reviewers’ comments and suggestions. The research work reported in this article was supported in part by the National Social Science Fund of China (Grant No. 17BYY097).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lei, L., Sun, Y. Should highly cited items be excluded in impact factor calculation? The effect of review articles on journal impact factor. Scientometrics 122, 1697–1706 (2020). https://doi.org/10.1007/s11192-019-03338-y
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-019-03338-y