Skip to main content
SpringerLink
Log in

A quantitative study on the effectiveness of peer review for academic journals

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Peer review is a classic method in the field of quality analysis, but the effectiveness of peer reviewing has never been researched using quantity analysis indicators. A new indicator for academic journals, Effectiveness of Peer Review (EPR), is defined for evaluating the effectiveness of peer reviewing. If the assumption is valid, EPR could be a simple indicator of such effectiveness. In a sample experiment, 28 academic journals were tested, and the EPR indicator was able to reflect accurately the academic impact of those journals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Aarssen, L. W. (2008). Bang for your buck: Rejection rates and impact factors in ecological journals. The Open Ecology Journal, 1, 14–19.

    Article  Google Scholar 

  • American Psychological Association. (2009). Summary report of journal operations, 2008. American Psychologist, 64(5), 504–505.

    Article  Google Scholar 

  • American Psychological Association. (2010). Summary report of journal operations, 2009. American Psychologist, 65(5), 524–525.

    Article  Google Scholar 

  • American Psychological Association. (2011). Summary report of journal operations, 2010. American Psychologist, 66(5), 405–406.

    Article  Google Scholar 

  • Berk, R. N., et al. (1995). New category of peer review decision: Rejection with opportunity to revise and resubmit. American Journal of Roentgenology, 164, 235.

    Article  Google Scholar 

  • Carrió, I. (2006). On cloning research, peer review and the possibility of fraud. European Journal of Nuclear Medicine and Molecular Imaging, 33, 235–236.

    Article  Google Scholar 

  • Cicchetti, D. V. (1997). Referees, editors, and publication practices: Improving the reliability and usefulness of the peer review system. Science and Engineering Ethics, 3, 51–62.

    Article  Google Scholar 

  • Eisenhart, M. (2002). The paradox of peer review: Admitting too much or allowing too little. Research in Science Education, 32(2), 241–255.

    Article  Google Scholar 

  • Elsevier (2008). Elsevier editorial system (EES) and peer review. A guide to publishing in scholar journals.

  • Fletcher, R. H. (1997). Evidence for the effectiveness of peer review. Science and Engineering Ethics, 3, 35–50.

    Article  Google Scholar 

  • Glänzel, W., et al. (2003). Better late than never? On the chance to become highly cited only beyond the standard bibliometric time horizon. Scientometrics, 58(3), 571–586.

    Article  Google Scholar 

  • Godlee, F., et al. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: A randomized controlled trial. Journal of American Medical Association, 280, 237–240.

    Article  Google Scholar 

  • Guo, Yu., et al. (2007). Biblio-metrilogical analysis on development trend of computer science in China. Application Research of Computer, 24(12), 28–31.

    Google Scholar 

  • Hojat, M., et al. (2003). Impartial judgment by the “gatekeepers” of Science: Fallibility and accountability in the peer review process. Advances in Health Sciences Education, 8, 75–96.

    Article  Google Scholar 

  • Jefferson, T., et al. (2007). Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database System Review, 18(2), MR000016.

    Google Scholar 

  • Jefferson, T., et al. (2002a). Effects of editorial peer review. JAMA, 287(21), 2784–2786.

    Article  Google Scholar 

  • Jefferson, T., et al. (2002b). Measuring quality of editorial peer review. JAMA, 287(21), 2786–2790.

    Article  Google Scholar 

  • Južnic, P., et al. (2010). Scientometric indicators: Peer-review, bibliometric methods and conflict of interests. Scientometrics, 85, 429–441.

    Article  Google Scholar 

  • Kundzewicz, Z. W., et al. (2005). Editorial—the peer-review system: Prospects and challenges. Hydrological Sciences–Journal–des Sciences Hydrologiques, 50(4), 577–590.

    Google Scholar 

  • Larivière, V., et al. (2009). The decline in the concentration of citations, 1900–2007. Journal of the American Society for Information Science and Technology, 60, 858–862.

    Google Scholar 

  • Li, X., (2008). On the peer review in international journals. http://www.sciencenet.cn/m/user_content.aspx?id=31867. Accessed on 2008-7-11.

  • Liang, L., et al. (2009). Delayed recognition: Sleeping beauties in science. Journal of Dialectics of Nature, 31(1), 39–45.

    Google Scholar 

  • Lock, S. (1994). Does editorial peer review work? Annuals of Internal Medicine, 212(1), 59–61.

    Google Scholar 

  • Ma, Z., et al. (2012). Effectiveness of peer review (EPR): A new indicator of academic journal. Journal of the China Society for Scientific and Technical Information, 30(3), 303–308.

    Google Scholar 

  • Ren, S. (2004). Academic paper’s peer review. The Chinese Journal of Nonferrous Metals, 14(11), F002–F003.

    Google Scholar 

  • Ren, S. (2009). Defects of peer review. Scientific News, 23, 59.

    Google Scholar 

  • Rennie, D. (2003). Editorial peer review: Its development and rationale. In F. Godlee, T. Jefferson (Eds.), Peer Review in Health Sciences (pp. 1–13). London: BMJ Publishing Group.

  • Rotton, J., et al. (1993). Citation impact, rejection rates, and journal value. American Psychologist, 48(8), 911-912

    Google Scholar 

  • Schroeder, W. H. (1988). Peer Review: A foundation of science. Environmental Science and Technology, 22(3), 235.

    Article  Google Scholar 

  • Schultz, D. M. (2010). Rejection rates for journal publishing in the atmospheric science. Bulletin of American Meteorological Society, 91, 231–243.

    Article  Google Scholar 

  • Stefanou, S. E. (2008). Applied production analysis unveiled in open peer review: Introductory remarks. Journal of Productivity Analysis, 20, 1–6.

    Article  Google Scholar 

  • Thatje, S. (2010). The multiple faces of journal peer review. Naturwissenschaften, 97, 237–239.

    Article  Google Scholar 

  • Tobin, M. J. (2002). Rigor of peer review and the standing of a journal. American Journal of Respiratory and Critical Care Medicine, 166, 1013–1018.

    Article  Google Scholar 

  • Van Raan, A. F. J. (2004). Sleeping beauties in science. Scientometrics, 59(3), 467–472.

    Article  Google Scholar 

  • Veitch, E. (2001). Reviewing peer review: The fourth international congress on peer review in biomedical publication. BMC News and Views, 2, 7–8.

    Google Scholar 

  • Warner, J. O. (2007). Increasing rejection rates for a successful journal. Pediatric Allergy Immunology, 18, 179.

    Article  Google Scholar 

  • Yoshida, Y. (2006). Peer review system: Any other choice? International Journal of Hematology, 83, 191–192.

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by the National Natural Science Foundation of China (No. 70973118).

Author information

Authors and Affiliations

  1. Institute of Scientific and Technical Information of China, Beijing, 100038, China

    Zheng Ma, Yuntao Pan, Zhenglu Yu, Jingting Wang & Yishan Wu

  2. ISTIC-Thomson Reuters Joint Lab for Scientometrics Research, Beijing, 100038, China

    Zheng Ma & Yuntao Pan

  3. Peking University, Beijing, 100859, China

    Jia Jia

Authors
  1. Zheng Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuntao Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenglu Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jingting Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jia Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yishan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zheng Ma.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ma, Z., Pan, Y., Yu, Z. et al. A quantitative study on the effectiveness of peer review for academic journals. Scientometrics 95, 1–13 (2013). https://doi.org/10.1007/s11192-012-0879-2

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-012-0879-2

Keywords

Search

Navigation