A combined bibliometric indicator to predict article impact

https://doi.org/10.1016/j.ipm.2010.09.005Get rights and content

Abstract

In both the UK and Australia there has been a recent move to use citation analysis in the evaluation of the research of individuals. In particular, the future UK Research Excellence Framework (REF), proposes using citation data in the research evaluation of articles published as recently as the year prior to the evaluation. In response to this move, this research develops an indicator at the level of individual articles that, when normalized, can supplement peer review. The new hybrid indicator is the weighted sum of two indicators in common usage: the article’s total number of citations in a citation window, and the Impact Factor of the journal in which the article was published. This research compares this new indicator with the article’s total number of citations in a longer citation window (the standard indicator of article impact). For citation windows of 0 or 1 years, the correlation of the simplified weighted sum with long-term citation is substantially higher than the correlation of the standard indicator of article citation with long-term citation. Moreover, for citation windows of as long as 3 years the standard indicator of citation correlates significantly with the month of publication, in that articles published earlier in the year are on average more highly cited than those published later in the year. By contrast, the skewing of the simplified weighted sum towards articles published early in the year is considerably less than that of the standard indicator.

Research highlights

► Compared a new indicator of long-term citation with citations in a citation window. ► For short citation windows, found: new indicator correlated more strongly with long-term. ► For short citation windows, found: new indicator is less skewed towards articles published early in the year.

Introduction

In recent years there has been a move in some countries (notably the UK and Australia) towards using bibliometric data to supplement peer review decisions in research evaluation (http://www.hefce.ac.uk/Research/ref/ and http://www.arc.gov.au/era/default.htm) and in Spain financial rewards are given on the basis of the Impact Factor (IF) of journals (López-Cózar, Ruiz Pérez, & Jiménez Contreras, 2007).

The Higher Education Funding Council for England (HEFCE) stated that bibliometrics are likely to be used to supplement peer review in the UK’s Research Excellence Framework (REF): “The quality of research outputs will be assessed by the expert panels against international standards of excellence. We expect that some of the panels will make use of citation information to inform their review of outputs” (http://www.hefce.ac.uk/Research/ref/). The REF is the latest of a series of assessments of UK university departments that are conducted by HEFCE approximately every 6 years to determine part of the research funding from the UK government.

When evaluating the impact of research using citations for articles published over a range of years, steps must be taken to ensure that articles published earlier do not have the advantage of having a longer period in which to attract citations. In the context of the REF this is a serious issue and hence the REF framework suggests that only articles from the same year should have their citations compared (HEFCE, 2009b). Another way to attempt to redress, and understand, this issue is to predict the future citations of each article using a simple formula that adjusts for the time interval since publication. This could be done by multiplying the citation count of an article by a number dependant on the number of months or years since it was published: more recent papers would have a higher multiplier to compensate for fewer years elapsing since publication. This approach would not work well for very recent articles, however, as several years may be needed to get a realistic indication of the rate at which they are likely to attract citations. An alternative approach is to incorporate the Impact Factor of the journal in which each article is published. This is reasonable on the basis that articles in journals with a higher IF are, on average, likely to be more highly cited than other articles. Moreover, publication in a high impact journal arguably indicates research quality irrespective of whether an article attracts citations.

This research investigates a new indicator, the weighted sum of two readily available indicators that are in common use: the number of citations of the article in a citation window and the Impact Factor (IF) of the journal in which the article is published. The number of citations in a citation window is the standard indicator for gauging long-term citation. The IF was introduced in Garfield (1955), and is used in Thomson-Reuters’ Journal Citation Reports (JCRs); the history of the Impact Factor is described by Garfield (2006). JCRs present data on 2-year and 5-year IFs; this research uses 2-year IFs, as it investigates articles published between 2000 and 2004 and the JCRs do not present 5-year IFs in Economics for years prior to 2007. For brevity, this article refers to the ‘two-year IF’ as ‘IF’. The IF of a journal for 2000 is the sum of the citations to the articles published in the journal in 1998 or 1999, from articles published in 2000, divided by the number of articles in the journal in 1998 or 1999.

In the UK’s 2008 Research Assessment Exercise, peer review panels evaluated articles published as recently as 2007; accordingly, it would be useful to have some indication of long-term citation for very recently published articles. For this reason, this study examines whether for very short citation windows, the proposed indicator correlates more highly with long-term citation counts than does the total citations in a citation window.

Section snippets

Background

The concept of using bibliometrics to supplement peer review is long established within scientometrics. For example, Moed suggested three ways in which bibliometrics and peer review are related: “Bibliometric indicators are applied as supplementary tools in peer review processes. The outcomes of peer reviews are used as a validation instrument of bibliometric indicators. Bibliometric indicators are applied as tools for monitoring and studying peer review processes” (Moed, 2005, p. 229).

Some

Research questions

This study investigates the weighted sum of article citation and journal impact, W(c,t) (called the ‘weighted sum indicator’), defined byW(c,t)=c×At+(1-c)×IFwhere At is the cumulative number of citations of the article t years after publication, IF is the impact factor of the journal in which the article was published (evaluated for the year of publication), and c is a constant weight with 0  c  1. Note that W(1,t) is the standard indicator of article citation (the number of citations of the

Method

The research questions were addressed by examining the correlations of W(c,t) with the article’s long-term citation, the journal’s IF and an article’s month of publication. The first part investigates all UK SSCI Economics articles published in 2000 and the second part all UK SSCI Economics articles recorded in the SSCI as published in the years 2001, 2002, 2003 and 2004. The first part examines the extent to which the correlations of W(c,t) vary with c; the second part compares the

Analysis of articles published in 2000

This section addresses the first research question: For articles published in a single year, how do the correlations of W(c,t) with long-term citation, with journal IFs, and with months of publication depend on the values of c and t?

As background for the first research question and in order to illustrate the skewing of citation towards articles published early in the year, Table 1 presents for every month and for diverse time periods the percentage of articles cited in the month. For 2000, the

Discussion

Section 3 defined W(c,t) byW(c,t)=c×At+(1-c)×IFwhere At is the number of citations of the article t years after publication, IF is the impact factor of the journal in which the article was published, and c is the weighting constant.

The first sub-question asks: For what values of t are there values of c for which W(c,t) correlates substantially more highly with long-term citation than does W(1,t)? Table 6 indicates that the merits of using c < 1 in place of W(1,t) (citation in the citation window

Limitations and conclusions

An obvious limitation is that the findings are on only one subject category and may not apply to other subjects. Further research is needed to establish the extent to which the results obtained for Economics apply to other subjects. Another limitation is that the findings are confined to articles published in the years 2000–2004 and may not apply to articles published in other years. A third limitation is that comparisons between the weighed sum indicator and the indicators from which it is

Acknowledgements

The research is supported by the Economic and Social Research Council [Grant number PTA-026-27-2228].

We wish to thank Professor Charles Oppenheim, Department of Information Science, Loughborough University, for his very useful feedback.

References (25)

  • J. Adams

    Early citation counts correlate with accumulated impact

    Scientometrics

    (2005)
  • D.W. Aksnes et al.

    Peer reviews and bibliometric indicators: a comparative study at Norwegian university

    Research Evaluation

    (2004)
  • K.W. Boyack et al.

    Indicator-assisted evaluation and funding of research: Visualizing the influence of grants on the number and citation counts of research papers

    Journal of the American Society for Information Science and Technology

    (2003)
  • Costas, R., van Leeuwen, T. N., & Bordons, M. (2009). A bibliometric methodology for supporting research assessment at...
  • B. Cronin et al.

    Citation, funding acknowledgement and author nationality relationships in four information science journals

    Journal of Documentation

    (1999)
  • E. Garfield

    The history and meaning of the journal impact factor

    Journal of the American Medical Association (JAMA)

    (2006)
  • E. Garfield

    Citation indexes for science

    Science

    (1955)
  • M. Gaughan et al.

    Using curriculum vitae to compare some impacts of NSF research grants with research Center funding

    Research Evaluation

    (2002)
  • HEFCE (2009a). Interim report of the REF bibliometrics pilot exercise....
  • HEFCE (2009b). Research excellence framework: second consultation on the assessment and funding of research....
  • J.E. Hirsch

    An index to quantify an individual’s scientific research output

    Proceedings of the National Academy of Sciences of the United States of America

    (2005)
  • Ingwersen, P., Schneider, J. W., Scharff, M., & Larsen, B. (2007). A national research profile-based immediacy index...
  • Cited by (80)

    • Measuring risk in science

      2023, Journal of Informetrics
    • A bibliometric analysis and cutting-edge overview on fuzzy techniques in Big Data

      2020, Engineering Applications of Artificial Intelligence
    • Improving the reliability of short-term citation impact indicators by taking into account the correlation between short- And long-term citation impact

      2020, Journal of Informetrics
      Citation Excerpt :

      Bornmann, Leydesdorff, and Wang (2014) explored the possibility of improving the prediction of long-term citation impact based on the early citations, journal impact factor, and other variables such as the number of authors, the number of references, and the number of pages. At present, the predictive variables most commonly used in current existing literature is the early citations of a paper (Abrishami & Aliakbary, 2019) or a combination of the early citations of the paper and the impact factor of the hosting journal (Abramo et al., 2019; Levitt & Thelwall, 2011; Stegehuis, Litvak, & Waltman, 2015; Stern, 2014). Furthermore, in addition to the early citations of the paper and the journal impact factor, some studies added other predictive variables to the prediction model of the long-term citation impact of a paper.

    View all citing articles on Scopus
    1

    Tel.: +44 1902 321470; fax: +44 1902 321478.

    View full text