Abstract
Computational support has been applied in different stages for automation of the peer review process, such as reviewer assignment to the article, review of content of the scientific article, detection of plagiarism and bias, all applying Machine Learning (ML) techniques. However, there is a lack of studies which identify the instruments used to evaluate the reviewers’ reports. This systematic literature review aims to find evidence about which techniques have been applied in the assessment of the reviewers’ reports. Therefore, six online databases were evaluated, in which 55 articles were identified, all published since 2000, meeting the inclusion criteria of this review. The result shows 6 relevant studies, which address models of assessment of scientific article reviews. Nevertheless, the use of ML was not identified in any case. Therefore, our findings demonstrate that there are a few instruments used to assess the reviewers’ reports and furthermore, they cannot be reliably used to extensively automate the review process.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chauvin, A., Moher, D., Altman, D., Schriger, D.L., Alam, S., Hopewell, S., Shanahan, D.R., Recchioni, A., Ravaud, P., Boutron, I.: A protocol of a cross-sectional study evaluating an online tool for early career peer reviewers assessing reports of randomised controlled trials. BMJ Open 7, 10 (2017)
Neuhauser, D., Koran, C.J.: Calling Medical Care reviewers first: a randomized trial. Med Care. 27, 664–666 (1989)
DeMaria, A.N.: What constitutes a great review? J. Am. Coll. Cardiol. 42, 1314–1315 (2003)
Ward, P., Graber, K.C., van der Mars, H.: Writing quality peer reviews of research manuscripts. J. Teach. Phys. Educ. 34, 700–715 (2015)
McGaghie, W.C., Bordage, G., Shea, J.A.: Problem statement, conceptual framework, and research question. Acad. Med. 76, 923–924 (2001)
Jefferson, T., Wager, E., Davidoff, F.: Measuring the quality of editorial peer review. JAMA 287, 2786 (2002)
Burley, R., Moylan, E.: What might peer review look like in 2030? (2017)
Price, S., Flach, P.A.: Computational support for academic peer review: a perspective from artificial intelligence. Commun. ACM 60, 70–79 (2017)
Thompson, S.R., Agel, J., Losina, E.: The JBJS peer-review scoring scale: a valid, reliable instrument for measuring the quality of peer review reports. Learn. Publ. 29, 23–25 (2016)
Landkroon, A.P., Euser, A.M., Veeken, H., Hart, W., Overbeke, A.J.P.M.: Quality assessment of reviewers’ reports using a simple instrument. Obstet. Gynecol. 108, 979–985 (2006)
Callaham, M., McCulloch, C.: Longitudinal trends in the performance of scientific peer reviewers. Ann. Emerg. Med. 57, 141–148 (2011)
Fortanet, I.: Evaluative language in peer review referee reports. J. Engl. Acad. Purp. 7, 27–37 (2008)
Henly, S.J., Dougherty, M.C.: Quality of manuscript reviews in nursing research. Nurs. Outlook 57, 18–26 (2009)
Priatna, W.S., Manalu, S.R., Sundjaja, A.M.: Development of review rating and reporting in open journal system. Procedia Comput. Sci. 116, 645–651 (2017)
Van Rooyen, S., Black, N., Godlee, F.: Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J. Clin. Epidemiol. 52, 625–629 (1999)
Li, X., Watanabe, T.: Automatic paper-to-reviewer assignment, based on the matching degree of the reviewers. Procedia Comput. Sci. 22, 633–642 (2013)
Marshall, I.J., Kuiper, J., Wallace, B.C.: RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J. Am. Med. Inf. Assoc. 23, 193–201 (2016)
Tennant, J.P., Dugan, J.M., Graziotin, D., Jacques, D.C., Waldner, F., Mietchen, D., Elkhatib, Y., B. Collister, L., Pikas, C.K., Crick, T., Masuzzo, P., Caravaggi, A., Berg, D.R., Niemeyer, K.E., Ross-Hellauer, T., Mannheimer, S., Rigling, L., Katz, D.S., Greshake Tzovaras, B., Pacheco-Mendoza, J., Fatima, N., Poblet, M., Isaakidis, M., Irawan, D.E., Renaut, S., Madan, C.R., Matthias, L., Nørgaard Kjær, J., O’Donnell, D.P., Neylon, C., Kearns, S., Selvaraju, M., Colomb, J.: A multi-disciplinary perspective on emergent and future innovations in peer review. F1000Research 6, 64 (2017)
Acknowledgement
We appreciate the financial support of AISTI (Iberian Association for Information Systems and Technologies), which permitted the registration in the WorldCIST’18 (6th World Conference on Information Systems and Technologies), held in Naples, Italy, 27–29 March 2018, and consequently this publication.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Sizo, A., Lino, A., Rocha, Á. (2018). Assessing Review Reports of Scientific Articles: A Literature Review. In: Rocha, Á., Adeli, H., Reis, L.P., Costanzo, S. (eds) Trends and Advances in Information Systems and Technologies. WorldCIST'18 2018. Advances in Intelligent Systems and Computing, vol 745. Springer, Cham. https://doi.org/10.1007/978-3-319-77703-0_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-77703-0_14
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-77702-3
Online ISBN: 978-3-319-77703-0
eBook Packages: EngineeringEngineering (R0)