Skip to main content
SpringerLink
Log in

Grim tales about the impact factor and the h-index in the Web of Science and the Journal Citation Reports databases: reflections on Vanclay’s criticism

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This paper reflects on the most current and some of the recent contributions of JK Vanclay, focusing on his methods, findings, and criticism about the journal citations reports and the web of science databases, the journal impact factor and the h-index. It is argued and demonstrated that some of the recent papers of the author about scientometric issues, measures and sources show so much demagoguery, ignorance and arrogance, have so much prejudice and bias, so profound errors in using the databases, calculating metrics, and interpreting search results that the papers are very unlikely to be meant as a genuine contribution from an academic who is a graduate of—among others—Oxford University, professor and dean in a respected university, a well-published and well-cited author and a recipient of the Queen’s Award (all the above in forest science). The papers are much more likely to serve as props for a staged, mock-up scenario based on slipshod research in an experiment, to illustrate the deficiencies in the processes and in the assessment of scholarly publishing productivity and impact in order to present the idealized solution of Vanclay: using the h-index, portrayed as the Prince, mounted on the shoulder of the White Horse, Google Scholar.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Abraham, P. (2000). Duplicate and salami publications. Journal of Postgraduate Medicine, 46(2), 67–69.

    Google Scholar 

  • Alonso, S., Cabrerizo, F., Herrera-Viedma, E., & Herrera, F. (2009). H-index: a review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics, 3(4), 273–289.

    Article  Google Scholar 

  • Balaban, A. T. (2012). Positive and negative aspects of citation indices and journal impact factors. Scientometrics . doi: 10.1007/s11192-012-0637-5.

  • Bar-Ilan, J. (2008). Which h-index?—A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257–271. Accessed April 3, 2012, from http://150.214.190.154/hindex/pdf/Bar-Ilan2008.pdf.

  • Bar-Ilan, J. (2010). Ranking of information and library science journals by JIF and by h-type indices. Journal of Informetrics, 4(2), 141–147.

    Article  Google Scholar 

  • Bensman, S. J. (1998). Scientific and technical serials holdings optimization in an inefficient market: A LSU serials redesign project exercise. Library Resources & Technical Services, 42(3), 147–242.

    Google Scholar 

  • Bensman, S. J. (2007). Garfield and the impact factor: The creation, utilization, and validation of a citation measure. Annual Review of Information Science and Technology, 41(1), 93–155.

    Article  Google Scholar 

  • Bensman, S. J. (2011). The impact factor: its place in Garfield’s thought in science evaluation and in library collection. Scientometrics. doi: 10.1007/s11192-011-0601-9.

  • Bollen, J., Rodriquez, M. A., & Van De Sompel, H. (2006). Journal status. Scientometrics, 69(3), 669–687.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H. D. (2008). Citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8(1), 93–102.

    Article  Google Scholar 

  • Bornmann, L., Rüdiger, M., & Daniel, H.-D. (2009). Do we need the h index and its variants in addition to standard bibliometric measures? Journal of the American Society for Information Science and Technology, 60(6), 1286–1289.

    Article  Google Scholar 

  • Braun, T. (2007). The impact factor of scientific and scholarly journals: Its use and misuse in research evaluation. Akadémiai Kiadó, Budapest. ISBN 963-05-8438-7.

  • Braun, T., & Glänzel, W. (1995). On a source of errors in computing journal impact factors. Chemical Intelligencer, 1, 31–32.

    Google Scholar 

  • Braun, T., Glänzel, W., & Schubert, A. (2005). A Hirsch-type index for journals. The Scientist, 19(11), 8.

    Google Scholar 

  • Brumback, R. A. (2009a). Impact factor wars: Episode V.: The empire strikes back. Journal of Child Neurology, 24(3), 260–262.

    Article  MathSciNet  Google Scholar 

  • Brumback, R. A. (2009b). Impact factor: Let’s be unreasonable! Epidemiology, 20(6), 932–933.

    Article  Google Scholar 

  • Buchanan, R. A. (2006). Accuracy of cited references: The role of citation databases. College and Research Libraries, 67, 292–303.

    Google Scholar 

  • Butler, L. (2011). The devil is in the detail: Concerns about Vanclay’s analysis of Australian journal rankings. Journal of Informetrics, 5(4), 693–694.

    Article  Google Scholar 

  • Campanario, J. M. (2011). Empirical study of journal impact factors obtained using the classical two-year citation window versus a five-year citation window. Scientometrics, 87(1), 189–204.

    Article  Google Scholar 

  • Dong, P., Loh, M., & Mondry, A. (2005). The “impact factor” revisited. Biomedical Digital Libraries, 2(7), 1–8.

    Google Scholar 

  • Falagas, M. E., & Alexiou, V. G. (2008). The top-ten in journal impact factor manipulation. Archivum Immunologiae et Therapiae Experimentalis, 56(4), 223–226.

    Article  Google Scholar 

  • Garfield, E. (1994). The Thomson Reuters impact factor. Accessed April 3, 2012, from http://thomsonreuters.com/products_services/science/free/essays/impact_factor/.

  • Garfield, E. (1996). How can impact factors be improved? British Medical Journal, 313, 411–413.

    Article  Google Scholar 

  • Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161(8), 979–980.

    Google Scholar 

  • Garfield, E. (2005). The agony and the ecstasy—The history and meaning of the journal impact factor. Chicago: International Congress on Peer Review and Biomedical Publication.

  • Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295(1), 90–93.

    Article  Google Scholar 

  • Glänzel, W. (2009). The multi-dimensionality of journal impact. Scientometrics, 78(2), 355–374.

    Article  Google Scholar 

  • Gorman, G. E. (2008). They can’t read, but they sure can count—Flawed rules of the journal rankings game. Online Information Review, 32(6), 705–708.

    Article  Google Scholar 

  • Harnad, S. (2008). Validating research performance metrics against peer rankings. Ethics in Science and Environmental Politics, 8(1), 103–107.

    Article  MathSciNet  Google Scholar 

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academies of Science, 102(46), 16569–16572.

    Article  Google Scholar 

  • Hirst, G. (1978). Discipline impact factors: A method for determining core journal lists. Journal of the American Society for Information Science, 29(4), 171–172.

    Article  Google Scholar 

  • Jacso, P. (2000). The number game. Online Information Review, 24(2), 180–183.

    Google Scholar 

  • Jacso, P. (2001). A deficiency in the algorithm for calculating the impact factor of scholarly journals: The journal impact factor. Cortex, 37(4), 590–594.

    Article  Google Scholar 

  • Jacso, P. (2004). WorldCat, Sociological Abstracts on CSA. e-Psyche, 28(2), 54–58.

    Google Scholar 

  • Jacso, P. (2005a). Google Scholar (Redux). Gale Reference Reviews (online), June 2005. Accessed March 31, 2012, from http://www.gale.cengage.com/reference/peter/index.htm.

  • Jacso, P. (2005b). Google Scholar: The pros and the cons. Online Information Review, 29(2), 208–214.

    Article  Google Scholar 

  • Jacso, P. (2006). Deflated, inflated and phantom citation counts. Online Information Review, 30(3), 297–309.

    Article  Google Scholar 

  • Jacso, P. (2007a). How big is a database versus how is a database big. Online Information Review, 31(4), 533–536.

    Article  Google Scholar 

  • Jacso, P. (2007b). Software issues related to cited references. Online Information Review, 31(6), 892–905.

    Article  Google Scholar 

  • Jacso, P. (2007c). The dimensions of cited reference enhanced database subsets. Online Information Review, 31(5), 695–705.

    Article  Google Scholar 

  • Jacso, P. (2007d). Trends in professional and academic online information services. (Keynote Address) at INFORUM 13th conference on professional information resources prague May 22–24, 2007. Accessed April 3, 2012, from http://www.inforum.cz/pdf/2007/jacso-peter.pdf.

  • Jacso, P. (2008a). Google Scholar revisited. Online Information Review, 32(1), 102–114. Accessed March 31, 2012, from http://www.jacso.info/OIR/2008-01-jacso-Google-Scholar-revisited.doc (pre-print version).

  • Jacso, P. (2008b). Testing the calculation of a realistic h-index in Google Scholar, Scopus and Web of Science for F.W. Lancaster. Library Trends, 56(4), 784–815.

    Article  Google Scholar 

  • Jacso, P. (2008c). The plausibility of computing the h-index of scholarly productivity and impact using reference-enhanced databases. Online Information Review, 32(2), 266–283. Accessed April 3, from, http://www.jacso.info/PDFs/jacso-plausibility-of-computing-the-h-index-jav.pdf.

  • Jacso, P. (2008d). The pros and cons of computing the h-index using Google Scholar. Online Information Review, 32(3), 437–452. Accessed March 31, 2012, from http://www.jacso.info/PDFs/jacso-GS-for-h-index-unabridged.pdf (unabridged manuscript).

  • Jacso, P. (2008e). The pros and cons of computing the h-index using Web of Science. Online Information Review, 32(5), 673–688. Accessed March 31, 2012, from http://www.jacso.info/OIR/2008-05-jacso-pros-and-cons-of-computing-the-h-index-using-Web-of-Science.doc (pre-print version).

  • Jacso, P. (2009a). Calculating the h-index and other bibliometric and scientometric indicators from Google Scholar with the Publish or Perish software. Online Information Review, 33(6), 1189–1200. Accessed March 31, 2012, from http://www.jacso.info/OIR/2009-06-jacso-calculating-H-index-and-other-bibliometric-and-scientometric-indicators.doc (pre-print version).

  • Jacso, P. (2009b). Database source coverage: hypes, vital signs and reality checks. Online Information Review, 33(5), 997–1007. Accessed March 31, 2012, fromhttp://www.jacso.info/OIR/2009-05-jacso-database-source-coverage-hypes-and-reality-checks.doc (pre-print version).

  • Jacso, P. (2009c). Errors of omission and their implication for computing scientometric measures in evaluating the publishing productivity and impact of countries. Online Information Review, 33(2), 376–385. Accessed March 31, 2012, from http://www.jacso.info/OIR/2009-02-jacso-errors-of-omission-and-their-implications-for-computing-scientometrics-measures.doc (pre-print version).

  • Jacso, P. (2009d). Five-year impact factor data in the Journal Citation Reports. Online Information Review, 33(3), 603–614. Accessed February 5, 2012, from http://www.jacso.info/OIR/2009-03-five-year-impact-factor-data-in-the-Journal-Citation-Reports.doc (pre-print version).

    Google Scholar 

  • Jacso, P. (2009e). Google Scholar’s ghost authors. Library Journal, 134(18), 26–27. Accessed April 3, 2012, from http://www.libraryjournal.com/article/CA6698580.html?q=jacso.

    Google Scholar 

  • Jacso, P. (2009f). The h-index for countries in Web of Science and Scopus. Online Information Review, 33(4), 831–837. Accessed March 31, 2012, from http://www.jacso.info/OIR/2009-04-jacso-h-index-for-countries-in-WoS-and-Scopus.doc (pre-print version).

  • Jacso, P. (2009g). Web of Science. Gale Reference Reviews (online), July 2009. Accessed March 31, 2012, from http://www.gale.cengage.com/reference/peter/200907/science.html.

  • Jacso, P. (2010a). Eigenfactor and Article Influence Score in the Journal Citation Reports. Online Information Review, 34(2), 339–348. Accessed March 31, 2012, from http://www.jacso.info/jcr-eigenfactor/.

  • Jacso, P. (2010b). Metadata mega mess in Google Scholar. Online Information Review, 34(1), 175–191. Accessed March 31, 2012, from http://www.jacso.info/OIR/2010-01-jacso-metadata-mega-mess-in-Google-Scholar.doc.

  • Jacso, P. (2011). Google Scholar duped and deduped—the aura of ‘robometrics’. Online Information Review, 35(1), 154–160. Accessed March, 31, 2012 fromhttp://www.jacso.info/OIR/2011-01-jacso-Google-Scholar-duped-deduped.doc (pre-print version).

  • Jacso, P. (2012a). Google Scholar author citation tracker: Is it too little, too late? Online Information Review, 36(1), 126–141. Accessed March 31, 2012, from http://www.jacso.info/OIR/2012-01-jacso-Google-Scholar-author-citation-tracker.doc (pre-print version).

    Google Scholar 

  • Jacso, P. (2012b). Using Google Scholar for journal impact factors and the h-index in nationwide publishing assessments in academia—Siren songs and air-raid sirens. Online Information Review, 36(3), (in press).

  • Martin, B. R. (1996). The use of multiple indicators in the assessment of basic research. Scientometrics, 36(3), 343–362.

    Article  Google Scholar 

  • Mc Veigh, M., & Mann, S. J. (2009). The journal impact factor denominator: Defining citable (countable) items. Journal of the American Medical Association, 302(10), 1107–1109.

    Article  Google Scholar 

  • Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105–2125.

    Article  Google Scholar 

  • Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277. Accessed April 3, 2012, from http://arxiv.org/ftp/arxiv/papers/0911/0911.2632.pdf (pre-print version).

    Google Scholar 

  • Moed, H. F., & Plume, A. (2011). The multi-dimensional research assessment matrix. Research Trends, 23, 5–7.

    Google Scholar 

  • Moed, H. F., & Van Leeuwen, T. N. (1995). Improving the accuracy of (the) Institute for Scientific Information’s journal impact factors. Journal of the American Society for Information Science and Technology, 46(6), 461–467.

    Article  Google Scholar 

  • Moed, H. F., Van Leeuwen, T. N., & Reedijk, J. (1996). A critical analysis of the journal impact factors of Angewandte Chemie and the Journal of the American Chemical Society: Inaccuracies in published impact factors based on overall citations only. Scientometrics, 37(1), 105–116.

    Article  Google Scholar 

  • Moed, H. F., Van Leeuwen, T. N., & Reedijk, J. (1999). Towards appropriate indicators of journal impact. Scientometrics, 46(3), 575–589.

    Article  Google Scholar 

  • Nisonger, T. E. (1994). A methodological issue concerning the use of Social Sciences Citation Index Journal Citation Reports impact factor data for journal ranking. Library Collections, Acquisition and Technical Services, 18(4), 447–458.

    Google Scholar 

  • Nisonger, T. E. (2004). The benefits and drawbacks of impact factor for journal collection management in libraries. Serials Librarian, 47(1–2), 57–75.

    Article  Google Scholar 

  • Nunberg, G. (2009). Google’s book search: A disaster for scholars. The Chronicle of Higher Education, August 31. Accessed April 3, 2012, from http://chronicle.com/article/Googles-Book-Search-A/48245/.

  • Oppenheim, C. (1996). Do citations count? Citation indexing and the Research Assessment Exercise (RAE). Serials: The Journal for the Serials Community, 9(2), 155–161.

  • Opthof, T. (1999). Submission acceptance rate rapid review system and impact factor. Cardiovascular Research, 41(1), 1–4.

    Article  Google Scholar 

  • Pendlebury, D. A. (2009). The use and misuse of journal metrics and other citation indicators. Archivum Immunologiae Et Therapiae Experimentalis, 57(1), 1–11.

    Article  Google Scholar 

  • Pendlebury, D. A., & Adams, J. (2012). Comments on a critique of the Thomson Reuters journal impact factor. Scientometrics,. doi:10.1007/s11192-012-0689-6.

    Google Scholar 

  • Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: Theory with application to the literature of physics. Information Processing and Management, 12(5), 297–312.

    Article  Google Scholar 

  • Pudovkin, A. I., & Garfield, E. (2004). Rank-normalized impact factor: A way to compare journal performance across subject categories. Proceedings of the ASIST Annual Meeting, 41, 507–515.

    Google Scholar 

  • Pudovkin, A. I., & Garfield, E. (2012). Rank normalization of impact factors will resolve Vanclay’s dilemma with TRIF. Scientometrics. doi:10.1007/s11192-012-0634-8.

  • Rousseau, R. (2001). Journal evaluation: Technical and practical issues. Library Trends, 50(3), 418–439.

    Google Scholar 

  • Rousseau, R. (2005). Median and percentile impact factors: A set of new indicators. Scientometrics, 63(3), 431–441.

    Article  Google Scholar 

  • Schubert, A., & Braun, T. (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36(3), 311–324.

    Article  Google Scholar 

  • Schubert, A., & Glänzel, W. (1983). Statistical reliability of comparisons based on the citation impact of scientific publications. Scientometrics, 5(1), 59–73.

    Article  Google Scholar 

  • Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 497–502.

    Article  Google Scholar 

  • Sombatsompop, N., Markpin, T., & Premkamolnetr, N. (2004). A modified method for calculating the impact factors of journals in ISI journal citation reports: Polymer science category in 1997–2001. Scientometrics, 60(2), 217–235.

    Article  Google Scholar 

  • Vanclay, J. K. (2008a). Gauging the impact of journals. Forest Ecology and Management, 256(4), 507–509. Accessed April 3, 2012, from http://epubs.scu.edu.au/cgi/viewcontent.cgi?article=1321&context=esm_pubs.

  • Vanclay, J. K. (2008b). Ranking forestry journals using the h-index. Journal of Informetrics, 2(4), 326–334. Accessed March 31, 2012, from http://arxiv.org/ftp/arxiv/papers/0712/0712.1916.pdf.

    Google Scholar 

  • Vanclay, J. K. (2009). Bias in the journal impact factor. Scientometrics, 78(1), 3–12. Accessed March 31, 2012, from http://arxiv.org/ftp/cs/papers/0612/0612091.pdf.

    Google Scholar 

  • Vanclay, J. K. (2011). An evaluation of the Australian Research Council’s journal ranking. Journal of Informetrics, 5(2), 265–274. Accessed April 3, 2012, from http://arxiv.org/ftp/arxiv/papers/1009/1009.3359.pdf.

    Google Scholar 

  • Vanclay, J. K. (2012). Impact factor: Outdated artefact or stepping-stone to journal certification? Scientometrics, (online). Accessed March 31, 2012, from http://arxiv.org/ftp/arxiv/papers/1201/1201.3076.pdf.

  • Williams, G. (2007). Should we ditch impact factors? British Medical Journal, 334(7593), 568. Accessed April 3, 2012, from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1828313/.

  • Wilson, C. S. (1999). Informetrics. Annual Review of Information Science and Technology, 34, 107–247.

    Google Scholar 

  • Yin, C.-Y., Aris, M. J., & Chen, X. (2010). Combination of Eigenfactor™ and h-index to evaluate scientific journals. Scientometrics, 84(3), 639–648.

    Article  Google Scholar 

  • Yu, G., & Wang, L. (2007). The self-cited rate of scientific journals and the manipulation of their impact factors. Scientometrics, 73(3), 321–330.

    Article  Google Scholar 

  • Yu, G., Wang, X.-H., & Yu, D.-R. (2005). The influence of publication delays on impact factors. Scientometrics, 64(2), 235–246.

    Article  Google Scholar 

Download references

Acknowledgments

The author appreciates the help received from John A. Adams, Jeff Clovis, Mary Mc Weigh and Judit Tiszai in calculating, interpreting and/or corroborating some of the journal performance indicators in my paper and is solely responsible for any miscalculations, misinterpretations or factual errors.

Author information

Authors and Affiliations

  1. Department of Information and Computer Sciences, University of Hawaii, Honolulu, HI, 96822, USA

    Peter Jacso

Authors
  1. Peter Jacso
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peter Jacso.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jacso, P. Grim tales about the impact factor and the h-index in the Web of Science and the Journal Citation Reports databases: reflections on Vanclay’s criticism. Scientometrics 92, 325–354 (2012). https://doi.org/10.1007/s11192-012-0769-7

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-012-0769-7

Keywords

Search

Navigation