Positive correlation between quality and quantity in academic journals
Introduction
The trade-off between quality and quantity in scholarly work has long been discussed. Cole and Cole (1967) studied the scientific output of university physicists and found that quality and quantity tend to be positively correlated. They argued that, in the case of an inconsistency, quality is more important than quantity. Smith and Fiedler (1970) studied the scholarly performance of university departments and showed that the relationship between quality and quantity is less clear. Their study indicated that citation count is the least biased measure of scholar work. Later on, Katz (1999) studied the publishing and citing activities as a self-similar social process. A positive correlation between quality and quantity was demonstrated in various science communities. Recently, many studies focused on evaluating the academic performance of individuals or groups. The results were mixed with regards to the correlation between quality and quantity. A positive correlation had been reported in the study of 500 largest universities worldwide (van Raan, 2013), top 100 European universities (van Raan, 2008a) and Italian universities ([Abramo et al., 2014], [Abramo et al., 2010b]). On the contrary, a negative correlation had been reported in the study of Australian academics (Harzing, 2005), UK scientists (Moed, 2008), Iranian institutes (Hayati & Ebrahimy, 2009), and US physicians (Tchetchik, Grinstein, Manes, Shapira, & Durst, 2015). The discrepancy indicates that the correlation can be influenced significantly by research policy and funding guidelines. A biased evaluation might reward quantity rather than quality ([Butler, 2002], [King, 2004]).
Evaluating the quality of a research work has not been a simple task. With an emphasis on peer review, the number of citations is basically a good indicator to measure the impact of a published article. In practice, impact factor of the journal publishing the article is frequently used as a simple indicator to the long-term citation count. The advantage of impact factor is the simplicity. However, some caution has been raised against the use of impact factor to measure the quality (Editorial, 2005; Seglen, 1997). Citations reflect not only the quality of the research work, but also the practice of the field and the popularity of the topic. Citation analysis cannot be justified to compare different disciplines. Besides, there are large fluctuations between the citations to a particular article and the average citations per article in that journal. An article published in a low impact-factor journal might receive more citations than an article published in a high impact-factor journal. In contrast, more recent studies showed that the impact factor of the journal can still be a preferable proxy of an article's quality. The long-term citations of a published article can be predicted more accurately by impact factor of the journal than by early citations of the article ([Abramo et al., 2010a], [Levitt and Thelwall, 2008]). More complicated relations among impact factor, early citation, and long-term citation had been investigated ([Stegehuis et al., 2015], [Stern, 2014]). Alternative indicators of quality had also been proposed ([Frey and Rost, 2010], [Sidiropoulos et al., 2015], [van Raan, 2008b]). Compared to the ambiguity in measuring the quality, counting the number of articles published by a scholar seems to be a simple task. Nowadays, various kinds of collaborations among researchers are essential and encouraged. The issue of co-authorship becomes more and more complicated. A clear-cut definition of quantity is difficult due to the overlap in different scholars’ publications. Measuring the quantity can be as ambiguous as measuring the quality.
It is interesting to note that the ambiguity disappears when the issue is addressed from the perspective of publishers. An article can be co-authored by a number of scholars; each published article belongs only to one journal. The impact factor was originally created to help classify the journals, not the articles. Compared to the citation count of an article, the impact factor of a journal is relatively stable over the years. To our knowledge, there is no systematic study on the correlation between impact factor and article number of the scholarly publishers. It is a reasonable opinion that quality and quantity are negatively correlated. The simple assumption is that rare things are precious. On the other hand, it is also plausible to argue that quality and quantity are not correlated. Since the total citations have already been normalized by the number of articles, the impact factor should be independent of the article number. In this work, empirical data are collected to reveal a positive correlation between quality and quantity in the professional journals, i.e., high impact journals publish more papers. A new plot is proposed to give a clear demonstration of this positive correlation. Data analysis and results are presented in Section 2, followed by the discussions in Section 3.
Section snippets
Analysis and results
Data are collected from the 2011 Journal Citation Reports (JCR) published by Thomson Reuters. The database includes more than ten thousand scholarly journals, which are divided into more than two hundred subject categories. To avoid comparing journals from different research fields, data are compiled from the same subject category. To have sufficient statistics, the subject categories consisting of less than one hundred journals are excluded from consideration. To include a variety of quality,
Discussions
This study reveals an interesting correlation between impact factor and article number in scholarly journals. Quality and quantity are positively correlated. High impact journals publish more articles. Data from 24 subject categories cover more than four thousand scholarly journals. This correlation is a common trend in different subject categories. The correlation is obscure in a direct plot of article number versus impact factor, as shown in Fig. 6. This study proposes to plot the accumulated
References (22)
- et al.
Are the authors of highly cited articles also the most productive ones?
Journal of Informetrics
(2014) - et al.
Do rankings reflect research quality?
Journal of Applied Economics
(2010) The self-similar science system
Research Policy
(1999)- et al.
Predicting the long-term citation impact of recent publications
Journal of Informetrics
(2015) - et al.
Citations versus journal impact factor as proxy of quality: Could the latter ever be preferable?
Scientometrics
(2010) - et al.
Testing the trade-off between productivity and quality in research activities
Journal of the American Society for Information Science and Technology
(2010) A list of published papers is no measure of value
Nature
(2002)- et al.
Scientific output and recognition: A study in the operation of the reward system in science
American Sociological Review
(1967) Not-so-deep impact
Nature
(2005)Australian research output in economics and business: High volume, low impact?
Australian Journal of Management
(2005)
Correlation between quality and quantity in scientific production: A case study of Iranian organizations from 1997 to 2006
Scientometrics
Cited by (33)
Bibliometric evaluation of Forensic Science International as a scholarly journal within the subject category legal medicine
2023, Forensic Science International: SynergySCiMet: Stable, sCalable and reliable Metric-based framework for quality assessment in collaborative content generation systems
2021, Journal of InformetricsCitation Excerpt :The paper published in a journal with a high impact factor is known to be a paper with high quality. Some research (e.g., Huang, 2016; Rousseau, 2016) claimed that there exists a positive correlation between impact factor and article number in scholarly journals. In other words, the quality of the venue is in direct relationship with the quantity of its published papers.
Quality and quantity are not always positively correlated: A case study of Chinese economics journals
2018, Journal of InformetricsImpact Factors and the Central Limit Theorem: Why citation averages are scale dependent
2018, Journal of InformetricsThe Brazilian scientific output published in journals: A study based on a large CV database
2017, Journal of InformetricsCitation Excerpt :Across the different disciplines, high impact journals display more articles. As a result, a journal’s expected impact and quantity of papers become positively correlated (Huang, 2016). However, easily quantifiable measures of publication quality—such as number of citations and quantity of published papers in journals of high impact—have been increasingly questioned (Beets, Lewis, & Browe, 2016; Frey & Rost, 2010; Genova, Astudillo, & Fraga, 2016; Seglen, 1997).