Abstract
This paper studied whether opening up review reports benefits science in terms of citations by taking Nature Communications as an example. To address this question, we collected the bibliographic records of 7614 papers published by Nature Communications in 2016 and 2017 from the Web of Science database and the disclosed reviewers’ comments and authors’ responses of a subset of 2293 papers. Using a linear regression model, we found no evidence of a citation advantage for the articles which disclosed their peer review documents. We concluded that opening peer review reports did not benefit papers in Nature Communications in terms of citations. We further examined whether the length of the comments and the number of rounds of the review process are associated with the papers’ citations. We found no evidence that the number of rounds is associated with the citations of the articles in Nature Communications. However, longer comments are associated with fewer citations, although the effect is weak.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adie, E. (2014). "Attention! A study of open access versus non-open access articles." Figshare. Available at https://figshare.com/articles/journal_contribution/Attention_A_study_of_open_access_vs_non_open_access_articles/1213690.
Besançon, L. et al. (2020a). Open science saves lives: Lessons from the COVID-19 pandemic. Available at www.biorxiv.org/content/https://doi.org/10.1101/2020.08.13.249847v2.
Besançon, L., et al. (2020). Open up: A survey on open and non-anonymised peer reviewing. Research Integrity and Peer Review, 5, 8.
Bornmann, L., & Daniel, H. D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80.
Bornmann, L., Mutz, R., & Daniel, H. D. (2010a). A reliability-generalisation study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE, 5, e14331.
Bornmann, L., Marx, W., Schier, H., Thor, A., & Daniel, H. D. (2010b). From black box to white box at open access journals: Predictive validity of manuscript reviewing and editorial decisions at atmospheric chemistry and physics. Research Evaluation, 19(2), 105–118.
Brooks, T. A. (1986). Evidence of complex citer motivations. Journal of the Association for Information Science, 37(1), 34–36.
Correa, J. C., Laverde-Rojas, H., Tejada, J., et al. (2021). The Sci-Hub effect on papers’ citations. Scientometrics. https://doi.org/10.1007/s11192-020-03806-w
Elliott, K. C., & Resnik, D. B. (2019). Making open science work for science and society. Environmental Health Perspectives, 127(7), 075002.
Ford, E. (2013). Defining and characterising open peer review: A review of the literature. Journal of Scholarly Publishing, 44(4), 311–326.
Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., & Miyao, Y. (2019). Does my rebuttal matter? Insights from a major NLP conference. http://arxiv.org/abs/1903.11367
Godlee, F. (2002). Making reviewers visible: Openness, accountability, and credit. JAMA, 287(21), 2762–2765.
Ghosal, T., Verma, R., Ekbal, A., & Bhattacharyya, P. (2019, July). DeepSentiPeer: Harnessing sentiment in review texts to recommend peer review decisions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 1120–1130.
Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019). Argument mining for understanding peer reviews. http://arxiv.org/abs/1903.10104
Kang, D., Ammar, W., Dalvi, B., van Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R. (2018). A dataset of peer reviews (peerread): Collection, insights and nlp applications. http://arxiv.org/abs/1804.09635.
Lipworth, W., Kerridge, I., Carter, S., & Little, M. (2011). Should biomedical publishing be “opened up”? toward a values-based peer-review process. Journal of Bioethical Inquiry, 8(3), 267–280.
Ma, C., Li, Y., Guo, F., & Si, K. (2019). The citation trap: Papers published at year-end receive systematically fewer citations. Journal of Economic Behavior & Organization, 166, 667–687.
Maharg, P., & Duncan, N. (2007). Black box, Pandora’s box or virtual toolbox? An experiment in a journal’s transparent peer review on the web. International Review of Law Computers & Technology, 21(2), 109–128.
Nature. (2018). Opening up peer review. Retrieved September 21, 2021 from https://www.nature.com/articles/d41586-018-06045-5
Ni, J., Zhao, Z., Shao, Y., Liu, S., Li, W., & Li, J. (2020). Does opening up peer review benefit science in terms of citations? Proceedings of the Association for Information Science and Technology, 57(1), e258. https://doi.org/10.1002/pra2.258
Pucker, B., Schilbert, H. M., & Schumacher, S. F. (2019). Integrating molecular biology and bioinformatics education. Journal of Integrative Bioinformatics, 16(3), 20190005.
Ross-Hellauer, T. (2017). What is open peer review? A systematic review. F1000Research, 6, 588.
Rooyen, S. V., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers’ recommendations: A randomised trial. The BMJ, 318(7175), 23–27.
Siler, K., Lee, K., & Bero, L. (2015). Measuring the effectiveness of scientific gatekeeping. Proceedings of the National Academy of Sciences, 112(2), 360–365.
Spier, R. (2002). The history of the peer-review process. Trends in Biotechnology, 20(8), 357–358.
Squazzoni, F., Brezis, E., & Marušić, A. (2017). Scientometrics of peer review. Scientometrics, 113, 501–502.
Taylor & Francis. (2015). Peer review in 2015: A global view. Taylor & Francis Group.
UNESCO. (2021). Open science. Retrieved September 21, 2021 from https://en.unesco.org/science-sustainable-future/open-science
Wang, X., Liu, C., Mao, W., & Fang, Z. (2015). The open access advantage considering citation, article usage and social media attention. Scientometrics, 103(2), 555–564.
Walker, R., & Rocha da Silva, P. (2015). Emerging trends in peer review: A survey. Frontiers in Neuroscience, 9, 169.
Wang, K., & Wan, X. (2018, June). Sentiment analysis of peer review texts for scholarly papers. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (pp. 175–184).
Acknowledgements
The manuscript is a new and extended version of our previous work which is accepted by the Proceedings of the 83rd annual meeting of the Association for Information Science and Technology (ASIS&T 2020) (Ni et al., 2020). We declare that we submit this manuscript to Scientometrics with the permission from ASIS&T. We gratefully acknowledge the financial support of the National Natural Science Foundation of China (Grant Number: 71673242). We thank anonymous reviewers for their helpful comments.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The corresponding author (Jiang Li) is member of the Distinguished Reviewers Board of Scientometrics.
Rights and permissions
About this article
Cite this article
Ni, J., Zhao, Z., Shao, Y. et al. The influence of opening up peer review on the citations of journal articles. Scientometrics 126, 9393–9404 (2021). https://doi.org/10.1007/s11192-021-04182-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-021-04182-9