Skip to main content
Springer Nature Link
Log in

Assessing the publication impact using citation data from both Scopus and WoS databases: an approach validated in 15 research fields

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

In a recent paper (https://doi.org/10.1007/s11192-020-03386-9) we proposed a model to estimate the citations of an article in a database (Scopus/Web of Science) in which it is not indexed using the percentile rank of the database (Web of Science/Scopus) in which it is indexed. In this study we supplement the previous work with three advances: (1) by using 15 different research fields, corresponding to over 1 million papers, since we previously used only four fields; (2) by measuring the agreement between the percentile ranks in both databases using Lin’s concordance correlation coefficient, since this coefficient has not been used previously to measure this agreement, but as a test with a sample of 15,400 papers to compare the actual and estimated number of citations; and (3) by using a robust data cleaning procedure. The results revealed a substantial concordance between percentile ranks of papers indexed in these two databases in all the research fields studied, and that this concordance is even stronger for high percentile values. This level of concordance suggests that we can consider the percentile of a paper in a database in which it is not indexed as being equal to the percentile of this paper in a database in which it is indexed. In other words, we increased the reliability of our previous conclusions that the percentile rank can be used as a citation database-normalization. The results of this study contribute to improve the use of citation counts in bibliometric studies, and to calculate research indicators when we need to use both bibliographic databases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Abramo, G., Aksnes, D. W., & D’Angelo, C. A. (2020). Comparison of research performance of Italian and Norwegian professors and universities. Journal of Informetrics, 14(2), 101023.

    Article  Google Scholar 

  • Abramo, G., & D’Angelo, C. A. (2020). A novel methodology to assess the scientific standing of nations at field level. Journal of Informetrics, 14(1), 100986.

    Article  Google Scholar 

  • Bar-Ilan, J. (2008). Which h-index?—A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257–271.

    Article  Google Scholar 

  • Bornmann, L. (2013). How to analyze percentile citation impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes, and top-cited papers. Journal of the American Society for Information Science and Technology, 64(3), 587–595.

    Article  Google Scholar 

  • Bornmann, L., Leydesdorff, L., & Mutz, R. (2013a). The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits. Journal of Informetrics, 7(1), 158–165.

    Article  Google Scholar 

  • Bornmann, L., Leydesdorff, L., & Wang, J. (2013b). Which percentile-based approach should be preferred for calculating normalized citation impact values? An empirical comparison of five approaches including a newly developed citation-rank approach (P100). Journal of Informetrics, 7, 933–944.

    Article  Google Scholar 

  • Bornmann, L., & Williams, R. (2020). An evaluation of percentile measures of citation impact, and a proposal for making them better. Scientometrics, 124(2), 1457–1478.

    Article  Google Scholar 

  • Bornmann, L., & Wohlrabe, K. (2019). Normalisation of citation impact in economics. Scientometrics, 120(2), 841–884. https://doi.org/10.1007/s11192-019-03140-w.

    Article  Google Scholar 

  • Brito, R., & Rodríguez-Navarro, A. (2018). Research assessment by percentile-based double rank analysis. Journal of Informetrics, 12(1), 315–329.

    Article  Google Scholar 

  • da Silva, J. A. T., & Dobránszki, J. (2018). Multiple versions of the h-index: Cautionary use for formal academic purposes. Scientometrics, 115(2), 1107–1113.

    Article  Google Scholar 

  • Donner, P. (2017). Document type assignment accuracy in the journal citation index data of Web of Science. Scientometrics, 113(1), 219–236.

    Article  Google Scholar 

  • Fairclough, R., & Thelwall, M. (2015). More precise methods for national research citation impact comparisons. Journal of Informetrics, 9(4), 895–906.

    Article  Google Scholar 

  • Feng, Y., Zhu, Q., & Lai, K.-H. (2017). Corporate social responsibility for supply chain management: A literature review and bibliometric analysis. Journal of Cleaner Production, 158, 296–307.

    Article  Google Scholar 

  • Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2016). Empirical analysis and classification of database errors in Scopus and Web of Science. Journal of Informetrics, 10(4), 933–953.

    Article  Google Scholar 

  • Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.

    Article  Google Scholar 

  • Harzing, A.-W. (2019). Two new kids on the block: How do Crossref and Dimensions compare with Google Scholar, Microsoft Academic, Scopus and the Web of Science? Scientometrics, 120(1), 341–349.

    Article  Google Scholar 

  • Harzing, A.-W., & Alakangas, S. (2016). Google Scholar, Scopus and the Web of Science: A longitudinal and cross-disciplinary comparison. Scientometrics, 106(2), 787–804.

    Article  Google Scholar 

  • Haunschild, R., & Bornmann, L. (2016). Normalization of Mendeley reader counts for impact assessment. Journal of Informetrics, 10(1), 62–73.

    Article  Google Scholar 

  • Kosteas, V. D. (2018). Predicting long-run citation counts for articles in top economics journals. Scientometrics, 115, 1395–1412.

    Article  Google Scholar 

  • Lin, L. I.-K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255–268.

    Article  Google Scholar 

  • Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & López-Cózar, E. D. (2018). Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics, 12(4), 1160–1177.

    Article  Google Scholar 

  • McBride, G. B. (2005). A proposal for strength-of-agreement criteria for Lin’s concordance correlation coefficient. NIWA Client Report, HAM2005-062.

  • Milojević, S., Radicchi, F., & Bar-Ilan, J. (2017). Citation success index—An intuitive pair-wise journal comparison metric. Journal of Informetrics, 11(1), 223–231.

    Article  Google Scholar 

  • Moed, H. F. (2016). Comprehensive indicator comparisons intelligible to non-experts: The case of two SNIP versions. Scientometrics, 106(1), 51–65.

    Article  Google Scholar 

  • Moed, H. F., Bar-Ilan, J., & Halevi, G. (2016). A new methodology for comparing Google Scholar and Scopus. Journal of Informetrics, 10(2), 533–551.

    Article  Google Scholar 

  • Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: A comparative analysis. Scientometrics, 106(1), 213–228.

    Article  Google Scholar 

  • Pech, G., & Delgado, C. (2019). Method for comparison of the number of citations from papers in different databases. In 17th international conference on scientometrics and informetrics, ISSI 2019—Proceedings 2 (pp. 2419–2429).

  • Pech, G., & Delgado, C. (2020). Percentile and stochastic-based approach to the comparison of the number of citations of articles indexed in different bibliographic databases. Scientometrics, 123(1), 223–252.

    Article  Google Scholar 

  • Pech, G., Delgado, C., & Vieira, N. (2019). Percentile citation-based method for screening the most highly cited papers in longitudinal bibliometric studies and systematic literature reviews. In 12th annual conference and proceedings of the EuroMed academy of business, EUROMED 2019 (pp. 911–923).

  • Petersen, A. M., Pan, R. K., Pammolli, F., & Fortunato, S. (2019). Methods to account for citation inflation in research evaluation. Research Policy, 48(7), 1855–1865.

    Article  Google Scholar 

  • Radicchi, F., & Castellano, C. (2012). A reverse engineering approach to the suppression of citation biases reveals universal properties of citation distributions. PLoS ONE, 7(3), e33833.

    Article  Google Scholar 

  • Rodríguez-Navarro, A., & Brito, R. (2018). Double rank analysis for research assessment. Journal of Informetrics, 12(1), 31–41.

    Article  Google Scholar 

  • Sangwal, K. (2013). Citation and impact factor distributions of scientific journals published in individual countries. Journal of Informetrics, 7(2), 487–504.

    Article  Google Scholar 

  • Valderrama-Zurián, J.-C., Aguilar-Moya, R., Melero-Fuentes, D., & Aleixandre-Benavent, R. (2015). A systematic analysis of duplicate records in Scopus. Journal of Informetrics, 9(3), 570–576.

    Article  Google Scholar 

  • Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10, 365–391.

    Article  Google Scholar 

  • Waltman, L., & Schreiber, M. (2013). On the calculation of percentile-based bibliometric indicators. Journal of the American Society for Information Science and Technology, 64(2), 372–379.

    Article  Google Scholar 

  • Xu, S., Hao, L., An, X., et al. (2019). Types of DOI errors of cited references in Web of Science with a cleaning method. Scientometrics, 120(3), 1427–1437.

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge the support of ERDF—European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation—COMPETE 2020 Programme and the Portuguese funding agency, FCT—Fundação para a Ciência e a Tecnologia within project POCI-01-0145-FEDER-031821. We would like to thank two anonymous referees for their valuable comments on an earlier version of this paper.

Author information

Authors and Affiliations

  1. Department of Nuclear Physics and High Energies, Rio de Janeiro State University, Rio de Janeiro, 20550-900, Brazil

    Gerson Pech

  2. LIAAD/INESC TEC, University of Porto, 4200-465, Porto, Portugal

    Catarina Delgado

  3. Faculty of Economics, University of Porto, 4200-464, Porto, Portugal

    Catarina Delgado

Authors
  1. Gerson Pech
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Catarina Delgado
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Gerson Pech.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pech, G., Delgado, C. Assessing the publication impact using citation data from both Scopus and WoS databases: an approach validated in 15 research fields. Scientometrics 125, 909–924 (2020). https://doi.org/10.1007/s11192-020-03660-w

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-020-03660-w

Keywords