Positive correlation between quality and quantity in academic journals

https://doi.org/10.1016/j.joi.2016.02.002Get rights and content

Highlights

  • High impact journals publish more articles.

  • Quality and quantity are positively correlated.

  • A new plot of article number versus impact factor is proposed.

Abstract

This study reports a positive correlation between impact factor and article number in scholarly journals. High impact journals publish more articles. Quality and quantity are positively correlated. This is a common trend in different disciplines as revealed by empirical data. The correlation is obscure in a direct plot of article number versus impact factor. A plot of accumulated article number versus normalized rank of impact factor clearly demonstrates the correlation. The reasons behind this correlation are discussed.

Introduction

The trade-off between quality and quantity in scholarly work has long been discussed. Cole and Cole (1967) studied the scientific output of university physicists and found that quality and quantity tend to be positively correlated. They argued that, in the case of an inconsistency, quality is more important than quantity. Smith and Fiedler (1970) studied the scholarly performance of university departments and showed that the relationship between quality and quantity is less clear. Their study indicated that citation count is the least biased measure of scholar work. Later on, Katz (1999) studied the publishing and citing activities as a self-similar social process. A positive correlation between quality and quantity was demonstrated in various science communities. Recently, many studies focused on evaluating the academic performance of individuals or groups. The results were mixed with regards to the correlation between quality and quantity. A positive correlation had been reported in the study of 500 largest universities worldwide (van Raan, 2013), top 100 European universities (van Raan, 2008a) and Italian universities ([Abramo et al., 2014], [Abramo et al., 2010b]). On the contrary, a negative correlation had been reported in the study of Australian academics (Harzing, 2005), UK scientists (Moed, 2008), Iranian institutes (Hayati & Ebrahimy, 2009), and US physicians (Tchetchik, Grinstein, Manes, Shapira, & Durst, 2015). The discrepancy indicates that the correlation can be influenced significantly by research policy and funding guidelines. A biased evaluation might reward quantity rather than quality ([Butler, 2002], [King, 2004]).

Evaluating the quality of a research work has not been a simple task. With an emphasis on peer review, the number of citations is basically a good indicator to measure the impact of a published article. In practice, impact factor of the journal publishing the article is frequently used as a simple indicator to the long-term citation count. The advantage of impact factor is the simplicity. However, some caution has been raised against the use of impact factor to measure the quality (Editorial, 2005; Seglen, 1997). Citations reflect not only the quality of the research work, but also the practice of the field and the popularity of the topic. Citation analysis cannot be justified to compare different disciplines. Besides, there are large fluctuations between the citations to a particular article and the average citations per article in that journal. An article published in a low impact-factor journal might receive more citations than an article published in a high impact-factor journal. In contrast, more recent studies showed that the impact factor of the journal can still be a preferable proxy of an article's quality. The long-term citations of a published article can be predicted more accurately by impact factor of the journal than by early citations of the article ([Abramo et al., 2010a], [Levitt and Thelwall, 2008]). More complicated relations among impact factor, early citation, and long-term citation had been investigated ([Stegehuis et al., 2015], [Stern, 2014]). Alternative indicators of quality had also been proposed ([Frey and Rost, 2010], [Sidiropoulos et al., 2015], [van Raan, 2008b]). Compared to the ambiguity in measuring the quality, counting the number of articles published by a scholar seems to be a simple task. Nowadays, various kinds of collaborations among researchers are essential and encouraged. The issue of co-authorship becomes more and more complicated. A clear-cut definition of quantity is difficult due to the overlap in different scholars’ publications. Measuring the quantity can be as ambiguous as measuring the quality.

It is interesting to note that the ambiguity disappears when the issue is addressed from the perspective of publishers. An article can be co-authored by a number of scholars; each published article belongs only to one journal. The impact factor was originally created to help classify the journals, not the articles. Compared to the citation count of an article, the impact factor of a journal is relatively stable over the years. To our knowledge, there is no systematic study on the correlation between impact factor and article number of the scholarly publishers. It is a reasonable opinion that quality and quantity are negatively correlated. The simple assumption is that rare things are precious. On the other hand, it is also plausible to argue that quality and quantity are not correlated. Since the total citations have already been normalized by the number of articles, the impact factor should be independent of the article number. In this work, empirical data are collected to reveal a positive correlation between quality and quantity in the professional journals, i.e., high impact journals publish more papers. A new plot is proposed to give a clear demonstration of this positive correlation. Data analysis and results are presented in Section 2, followed by the discussions in Section 3.

Section snippets

Analysis and results

Data are collected from the 2011 Journal Citation Reports (JCR) published by Thomson Reuters. The database includes more than ten thousand scholarly journals, which are divided into more than two hundred subject categories. To avoid comparing journals from different research fields, data are compiled from the same subject category. To have sufficient statistics, the subject categories consisting of less than one hundred journals are excluded from consideration. To include a variety of quality,

Discussions

This study reveals an interesting correlation between impact factor and article number in scholarly journals. Quality and quantity are positively correlated. High impact journals publish more articles. Data from 24 subject categories cover more than four thousand scholarly journals. This correlation is a common trend in different subject categories. The correlation is obscure in a direct plot of article number versus impact factor, as shown in Fig. 6. This study proposes to plot the accumulated

References (22)

  • G. Abramo et al.

    Are the authors of highly cited articles also the most productive ones?

    Journal of Informetrics

    (2014)
  • B.S. Frey et al.

    Do rankings reflect research quality?

    Journal of Applied Economics

    (2010)
  • J.S. Katz

    The self-similar science system

    Research Policy

    (1999)
  • C. Stegehuis et al.

    Predicting the long-term citation impact of recent publications

    Journal of Informetrics

    (2015)
  • G. Abramo et al.

    Citations versus journal impact factor as proxy of quality: Could the latter ever be preferable?

    Scientometrics

    (2010)
  • G. Abramo et al.

    Testing the trade-off between productivity and quality in research activities

    Journal of the American Society for Information Science and Technology

    (2010)
  • L. Butler

    A list of published papers is no measure of value

    Nature

    (2002)
  • S. Cole et al.

    Scientific output and recognition: A study in the operation of the reward system in science

    American Sociological Review

    (1967)
  • Editorial

    Not-so-deep impact

    Nature

    (2005)
  • A.W. Harzing

    Australian research output in economics and business: High volume, low impact?

    Australian Journal of Management

    (2005)
  • Z. Hayati et al.

    Correlation between quality and quantity in scientific production: A case study of Iranian organizations from 1997 to 2006

    Scientometrics

    (2009)
  • Cited by (33)

    • SCiMet: Stable, sCalable and reliable Metric-based framework for quality assessment in collaborative content generation systems

      2021, Journal of Informetrics
      Citation Excerpt :

      The paper published in a journal with a high impact factor is known to be a paper with high quality. Some research (e.g., Huang, 2016; Rousseau, 2016) claimed that there exists a positive correlation between impact factor and article number in scholarly journals. In other words, the quality of the venue is in direct relationship with the quantity of its published papers.

    • The Brazilian scientific output published in journals: A study based on a large CV database

      2017, Journal of Informetrics
      Citation Excerpt :

      Across the different disciplines, high impact journals display more articles. As a result, a journal’s expected impact and quantity of papers become positively correlated (Huang, 2016). However, easily quantifiable measures of publication quality—such as number of citations and quantity of published papers in journals of high impact—have been increasingly questioned (Beets, Lewis, & Browe, 2016; Frey & Rost, 2010; Genova, Astudillo, & Fraga, 2016; Seglen, 1997).

    View all citing articles on Scopus
    View full text