Abstract
The credibility of the publication system in science is determined in large part by the precision of the manuscript review process. Studies on the precision of the review process in scientific journals have reported conflicting results. This paper reviews those studies and re-examines the data reported. The findings indicate that highly selective decision-making with imprecise reviewers results in outcomes that are only slightly better than chance.
Similar content being viewed by others
References
Y. M. M. BISHOP, S. E. FIENBERG, P. W. HOLLAND,Discrete Multivariate Analysis, Cambridge, Mass., The MIT Press, 1975.
C. M. BONJEAN, J. HULLUM, Reasons for journal rejection: An analysis 600 manuscripts,PS, 10 (1978) 480.
D. D. BOWEN, R. RERLOFF, J. JACOBY, Improving manuscript evaluation procedures,American Psychologist, 27 (1972) 22.
R. CRANDALL, Interrater agreement on manuscripts is not so bad!American Psychologist, 33, (1978) 623.
A. COURNAND, M. MEYER, The scientists code,Minerva, 14 (1976) 79.
N. D. GLENN, The journal article review process: Some proposals for change,American Sociologist, 11 (1976) 179.
L. A. GOODMAN, The analysis of cross-classified data: Independence, quasi-independence, and interactions in contigency tables with or without missing entries,Journal of the American Statistical Association 63 (1968) 109.
C. HENDRICK, Editorial comment,Personality and Social Psychology Bulletin, 2 (1976) 207.
D. KLAHR, Insiders, outsiders, and efficiency in a National Science Foundation panel,American Psychologist 40 (1985) 148.
M. KOCHEN, B. PERKEL Improving referee-33 selection and manuscript evaluation. In: J. McCARTNEY (Ed.), Proceedings of the First International Conference of Scientific Editors, April 24–29, 1977, Jerusalem, Dordrecht, Reidel, 1977.
D. LINDSEY,The Scientific Publication System in Social Science, San Francisco, Jossey-Bass, 1978.
J. B. LODAHL, G. GORDON, The structure of scientific fields and the functioning of university graduate departments,American Sociological Review, 35 (1972) 57.
M. J. MAHONEY, Publication prejudices: An experimental study of confirmatory bias in the peer review system,Cognitive Therapy and Research, 1 (1977) 161.
M. J. MAHONEY, Open exchange and epistemic progress,American Psychologist, 40 (1985) 29.
P. McREYNOLDS, Reliability of ratings of research papers, American Psychologist, 26 (1971) 400.
A. C. MILLER, S. L. SERZAN, Criteria for identifying a refereed journal,Journal of Higher Education 35 (1984) 673.
J. PFEFFER, G. R. SALANCIK, H. LEBLEBICI, The effects of uncertainty on the use of social influence in organizational decision-making,Administrative Science Quarterly 21 (1976) 227.
J. PFEFFER, A. LEONG, K. STREHL, Paradigm development and particularism: Journal publication in three scientific disciplines,Social Forces, 55 (1977) 938.
I. PIPKE, The gatekeepers A multivariate study of accepted and rejected adult-education research conference abstracts (1978–80),Adult Education Quarterly 34 (1984) 71.
S. SCARR, B. L. R. WEBER, The reliability of reviews for the American Psychologists,American Psychologist 33 (1978) 935.
W. A. SCOTT, Interreferee agreement of some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology, American Psychologist 29 (1974) 698.
N. J. SPENCER, J. HARTNETT, M. MAHONEY, Problems with reviews in the standard editorial practice,Journal of Social Behavior and Personality 1 (1986) 21.
A. T. STINCHCOMBE, R. OFSHE, On journal editing as a probabilistic process,American Sociologist, 4 (1969) 116.
A. W. WARD, B. W. HALL, C. F. SCHRAM, Evaluation of published educational research-national survey,American Educational Research Journal 12 (1975) 109.
G. J. WHITEHURST, Interrater agreement for journal manuscript reviews,American Psychologist 39 (1984) 22.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Lindsey, D. Assessing precision in the manuscript review process: A little better than a dice roll. Scientometrics 14, 75–82 (1988). https://doi.org/10.1007/BF02020243
Received:
Issue Date:
DOI: https://doi.org/10.1007/BF02020243