Skip to main content
SpringerLink
Log in

Assessing precision in the manuscript review process: A little better than a dice roll

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

The credibility of the publication system in science is determined in large part by the precision of the manuscript review process. Studies on the precision of the review process in scientific journals have reported conflicting results. This paper reviews those studies and re-examines the data reported. The findings indicate that highly selective decision-making with imprecise reviewers results in outcomes that are only slightly better than chance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Y. M. M. BISHOP, S. E. FIENBERG, P. W. HOLLAND,Discrete Multivariate Analysis, Cambridge, Mass., The MIT Press, 1975.

    Google Scholar 

  2. C. M. BONJEAN, J. HULLUM, Reasons for journal rejection: An analysis 600 manuscripts,PS, 10 (1978) 480.

    Google Scholar 

  3. D. D. BOWEN, R. RERLOFF, J. JACOBY, Improving manuscript evaluation procedures,American Psychologist, 27 (1972) 22.

    Google Scholar 

  4. R. CRANDALL, Interrater agreement on manuscripts is not so bad!American Psychologist, 33, (1978) 623.

    Google Scholar 

  5. A. COURNAND, M. MEYER, The scientists code,Minerva, 14 (1976) 79.

    Google Scholar 

  6. N. D. GLENN, The journal article review process: Some proposals for change,American Sociologist, 11 (1976) 179.

    Google Scholar 

  7. L. A. GOODMAN, The analysis of cross-classified data: Independence, quasi-independence, and interactions in contigency tables with or without missing entries,Journal of the American Statistical Association 63 (1968) 109.

    Google Scholar 

  8. C. HENDRICK, Editorial comment,Personality and Social Psychology Bulletin, 2 (1976) 207.

    Google Scholar 

  9. D. KLAHR, Insiders, outsiders, and efficiency in a National Science Foundation panel,American Psychologist 40 (1985) 148.

    Google Scholar 

  10. M. KOCHEN, B. PERKEL Improving referee-33 selection and manuscript evaluation. In: J. McCARTNEY (Ed.), Proceedings of the First International Conference of Scientific Editors, April 24–29, 1977, Jerusalem, Dordrecht, Reidel, 1977.

  11. D. LINDSEY,The Scientific Publication System in Social Science, San Francisco, Jossey-Bass, 1978.

  12. J. B. LODAHL, G. GORDON, The structure of scientific fields and the functioning of university graduate departments,American Sociological Review, 35 (1972) 57.

    Google Scholar 

  13. M. J. MAHONEY, Publication prejudices: An experimental study of confirmatory bias in the peer review system,Cognitive Therapy and Research, 1 (1977) 161.

    Google Scholar 

  14. M. J. MAHONEY, Open exchange and epistemic progress,American Psychologist, 40 (1985) 29.

    Google Scholar 

  15. P. McREYNOLDS, Reliability of ratings of research papers, American Psychologist, 26 (1971) 400.

    Google Scholar 

  16. A. C. MILLER, S. L. SERZAN, Criteria for identifying a refereed journal,Journal of Higher Education 35 (1984) 673.

    Google Scholar 

  17. J. PFEFFER, G. R. SALANCIK, H. LEBLEBICI, The effects of uncertainty on the use of social influence in organizational decision-making,Administrative Science Quarterly 21 (1976) 227.

    Google Scholar 

  18. J. PFEFFER, A. LEONG, K. STREHL, Paradigm development and particularism: Journal publication in three scientific disciplines,Social Forces, 55 (1977) 938.

    Google Scholar 

  19. I. PIPKE, The gatekeepers A multivariate study of accepted and rejected adult-education research conference abstracts (1978–80),Adult Education Quarterly 34 (1984) 71.

    Google Scholar 

  20. S. SCARR, B. L. R. WEBER, The reliability of reviews for the American Psychologists,American Psychologist 33 (1978) 935.

    Google Scholar 

  21. W. A. SCOTT, Interreferee agreement of some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology, American Psychologist 29 (1974) 698.

    Google Scholar 

  22. N. J. SPENCER, J. HARTNETT, M. MAHONEY, Problems with reviews in the standard editorial practice,Journal of Social Behavior and Personality 1 (1986) 21.

    Google Scholar 

  23. A. T. STINCHCOMBE, R. OFSHE, On journal editing as a probabilistic process,American Sociologist, 4 (1969) 116.

    Google Scholar 

  24. A. W. WARD, B. W. HALL, C. F. SCHRAM, Evaluation of published educational research-national survey,American Educational Research Journal 12 (1975) 109.

    Google Scholar 

  25. G. J. WHITEHURST, Interrater agreement for journal manuscript reviews,American Psychologist 39 (1984) 22.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Human Development and Performance University of Oregon, (USA)

    D. Lindsey

Authors
  1. D. Lindsey
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lindsey, D. Assessing precision in the manuscript review process: A little better than a dice roll. Scientometrics 14, 75–82 (1988). https://doi.org/10.1007/BF02020243

Download citation

  • Received:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02020243

Keywords

Search

Navigation