Skip to main content
Springer Nature Link
Log in

Precision in the manuscript review process: Hargens and Herting revisited

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Lindsey recently examined the precision of the manuscript review process using a stochastic model. The study reported that the low reliability found by previous studies results in journals publishing a large number of papers that should otherwise be rejected and rejecting an equally large number of papers that should be accepted.Hargens andHerting have criticized this view. This paper addresses their criticisms. The paper includes an examination of sociology journals usingimpact scores. The differences between journals is noted. Part of the variation between sociology journals derives from their editorial operations. Central to their editorial operations is the reviewing of manuscripts for publication. Not all journals perform this task equally well. The consequences of poor editorial management are discussed. To improve the quality of published work journals need to reduce the low reliability of the current manuscript review process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes and references

  1. A. Cournand, M. Meyer, The scientist's code,Minerva, 14 (1976) 79.

    Article  Google Scholar 

  2. R.K. Merton,The Sociology of Science: Theoretical and Empirical Investigations. Chicago, University of Chicago Press, 1973.

    Google Scholar 

  3. S.B. Barnes, The beginnings of the learned journalism, 1665–1730. Unpublished doctoral dissertation, Cornell University, 1934.

  4. D. Lindsey,The Scientific Publication System in the Social Sciences: A Study of Leading Professional Journals in Psychology, Sociology, and Social Work, San Francisco, Jossey-Bass, 1978.

    Google Scholar 

  5. D. Lindsey, Assessing precision in the manuscript review process: A little better than chance,Scientometrics, 14 (1988) 75.

    Google Scholar 

  6. L.L. Hargens, J.R. Herting, Neglected considerations in the analysis of agreement among journal referees,Scientometrics, 19 (1990) 91.

    Google Scholar 

  7. I. Mitroff,The Subjective Side of Science, Amsterdam, Elsevier, 1974.

    Google Scholar 

  8. M.J. Mahoney, Publication prejudices: An experimental study of confirmational bias in the peer review system,Cognitive Therapy and Research, 1 (1977) 161.

    Google Scholar 

  9. I. Abramowitz, B. Gomes, Publish or politics: Referee bias in manuscript review,Journal of Applied Social Psychology, 5 (1975) 187.

    Google Scholar 

  10. M.J. Mahoney, Open exchange and epistemic progress,American Psychologist, 40 (1985) 29.

    Article  Google Scholar 

  11. B. Barber, Resistance by scientists to scientific discovery; in:B. Barber, W. Hirsch (Eds),The Sociology of Science, New York, Free Press, 1962.

    Google Scholar 

  12. M.H. Macroberts, B.R. Macroberts, Testing the Ortega Hypothesis: Facts and artifacts,Scientometrics, 12 (1987) 293.

    Article  Google Scholar 

  13. B.F. Reskin, Sex differences and status attainment in science: The case of the postdoctoral fellowship,American Sociological Review, 42 (1977) 491.

    Google Scholar 

  14. C.M. Bowen, R. Perloff, J. Jacoby, Improving manuscript evaluation procedures,American Psychologist, 27 (1972) 22.

    Google Scholar 

  15. Hargens andHerting, op. cit.,, page 92.

    Google Scholar 

  16. Hargens andHerting, op. cit., page 93.

    Google Scholar 

  17. Bowen, et al.,op. cit., page 221.

    Google Scholar 

  18. Y.M.M. Bishop, S.E. Fienberg, P.W. Holland,Discrete Multivariate Analysis: Theory and Practice, Cambridge, Mass., MIT Press, 1975.

    Google Scholar 

  19. S.J. Haberman,The Analysis of Frequency Data, Chicago, University of Chicago Press, 1974.

    Google Scholar 

  20. Hargens andHerting, op. cit.,, page 102.

    Google Scholar 

  21. P. Mcreynolds, Reliability of ratings of research papers,American Psychologist, 25 (1971) 400.

    Google Scholar 

  22. W.A. Scott, Interreferee agreement on some characteristics of manuscripts submitted to theJournal of Personality and Social Psychology, American Psychologist, 29 (1974) 689.

    Article  Google Scholar 

  23. R.O. Lempert, From the editor,Law and Society Review, 19 (1985) 529.

    Google Scholar 

  24. Lindsey, op. cit. reference 5.

    Google Scholar 

  25. Hargens andHerting, op. cit. page 99. Data derived fromV. Bakanic, C. Mcphail, R.J. Simon, The manuscript review and decision-making process,American Sociological Review, 52 (1987) 631.

    Google Scholar 

  26. Hargens andHerting, op. cit., page 99.

    Google Scholar 

  27. Hargens andHerting, op. cit., page 101.

    Google Scholar 

  28. Hargens andHerting, op. cit., page 99.

    Google Scholar 

  29. L.L. Hargens, J.R. Herting, A new approach to referees' assessment of manuscripts,Social Science Research, 19 (1990) 1. This paper is the source for the methodological approach used to examine the argument inLindsey, op. cit. reference 5D. Lindsey, Assessing precision in the manuscript review process: A little better than chance,Scientometrics, 14 (1988) 75.Hargens andHerting compute tests for statistical independence on a number of previously published data sets and report that “one can reject with great confidence the hypothesis that referee recommendations are statistically independent.” However, several of the data sets, even using their approach of collapsing the data along the diagonal, have expected cell frequencies of less than five. The data presented forLaw andSociety Review contains an expected cell frequency of zero.S.E. Fienberg,The Analysis of Cross-Classified Categorical Data, (2nd Ed.) Cambridge, MA, MIT Press, 1981, 142–143, has addressed this issue.Fienberg writes, “Many authors have concerned themselves with the analysis of contingency tables whose entries are missing, a priori zero, or otherwise predetermined. One way to treat such data is to remove these special cells from the model building and analysis process, after which we are left with a contingency table that is said to be structurally incomplete ... One of the most serious problems encountered in the application of incomplete contingency table methodology is that investigators who have data in the form of an incomplete contingency table often fail to recognize that fact. They either fill in the cells containing the fixed zeros using some “appropriate values,” or they collapse the data ... These practices can lead to inappropriate conclusions, general confusion, and bad science.” Also see,S.J. Haberman, Log-linear models and frequency tables with small expected cell counts,Annals of Statistics, 5 (1977) 1148.

    Article  Google Scholar 

  30. Hargens andHerting, op. cit. reference 29 above, page 12. This paper is the source for the methodological approach used to examine the argument inLindsey, op. cit. reference 5D. Lindsey, Assessing precision in the manuscript review process: A little better than chance,Scientometrics, 14 (1988) 75.Hargens andHerting compute tests for statistical independence on a number of previously published data sets and report that “one can reject with great confidence the hypothesis that referee recommendations are statistically independent.” However, several of the data sets, even using their approach of collapsing the data along the diagonal, have expected cell frequencies of less than five. The data presented forLaw and Society Review contains an expected cell frequency of zero.S.E. Fienberg,The Analysis of Cross-Classified Categorical Data, (2nd Ed.) Cambridge, MA, MIT Press, 1981, 142–143, has addressed this issue.Fienberg writes, “Many authors have concerned themselves with the analysis of contingency tables whose entries are missing, a priori zero, or otherwise predetermined. One way to treat such data is to remove these special cells from the model building and analysis process, after which we are left with a contingency table that is said to be structurally incomplete ... One of the most serious problems encountered in the application of incomplete contingency table methodology is that investigators who have data in the form of an incomplete contingency table often fail to recognize that fact. They either fill in the cells containing the fixed zeros using some “appropriate values,” or they collapse the data ... These practices can lead to inappropriate conclussions, general confusion, and bad science.” Also see,S.J. Haberman, Log-linear models and frequency tables with small expected cell counts,Annals of Statistics, 5 (1977) 1148.

    Google Scholar 

  31. Lindsey, op. cit. reference 4 above,, page 33.

    Google Scholar 

  32. E. Garfield, (Ed.),Journal Citation Reports, Social Science Citation Index. Philadelphia, PA, Institute for Scientific Information, 1981–1988.

    Google Scholar 

  33. Lindsey, op. cit. reference 24 above.

    Google Scholar 

  34. Hargens andHerting, op. cit. reference 6 above, page 98.

    Google Scholar 

  35. Hargens andHerting, op. cit. reference 6 above, footnote 31. SeeL.L. Hargens, Scholarly consensus and journal rejection rates,American Sociological Review, 53 (1988) 139. I take issue with use of the term error to define this disagreement. I don't believe eitherHargens orLindsey were in error. In fact, I continue to favor the conventional approach to conceptualizing the reliability of the manuscript review process.

    Google Scholar 

  36. SeeL.A. Goodman, New methods for analyzing the intrinsic character of qualitative variables using cross-classified data,American Journal of Sociology, 93 (1987) 529 andL.A. Goodman,The Analysis of Cross-Classified Data Having Ordered Categories, Cambridge, MA, Harvard University Press, 1984.

    Article  Google Scholar 

  37. Mahoney, op. cit. reference 7 above.

    Article  Google Scholar 

  38. D. Lindsey, Distinction, achievement and editorial board membership,American Psychologist, 32 (1976) 799.

    Google Scholar 

  39. D. Lindsey, Building a child welfare research journal,Children and Youth Services Review, 12 (1990) 185. Also seeD. Lindsey, The role of social work journals in the development of a knowledge base for the profession. Unpublished manuscript, School of Social Welfare, University of California, Berkeley, 1990. I have taken a similar approach to the study of aggregate academic units. SeeD. Lindsey, Building a great public university: A comparison of funding patterns and research in British and American universities,Research in Higher Education, forthcoming.

    Google Scholar 

  40. Hargens andHerting, op. cit. reference 21 above, page 1.

    Google Scholar 

  41. Cournand andMeyer, op. cit. reference 1.

    Article  Google Scholar 

  42. J.R. Ravetz,Scientific Knowledge and Its Social Problems, London, Oxford University Press, 1971.

    Google Scholar 

  43. K. Popper,Conjectures and Reflections, London, Routledge & Kegan Paul, 1963.

    Google Scholar 

  44. Lindsey, op. cit. reference 4 above, pages 77–96 and 133–136.

    Google Scholar 

  45. Lindsey, op. cit. reference 4 above, pages 126–127.

    Google Scholar 

  46. A.L. Stinchcombe, R. Ofshe, On journal editing as a probabilistic process,American Sociologist, 4 (1969) 116.

    Google Scholar 

  47. Hargens andHerting, op. cit. reference 6 above.

    Google Scholar 

  48. Hargens andHerting, op. cit. reference 21 above, page 10.

    Google Scholar 

  49. Lindsey, op. cit. reference 5.

    Google Scholar 

  50. Hargens andHerting, op. cit. reference 6 above, page 94.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of California, Berkeley, (USA)

    D. Lindsey

  2. University of Oregon, (USA)

    D. Lindsey

Authors
  1. D. Lindsey
    View author publications

    You can also search for this author inPubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lindsey, D. Precision in the manuscript review process: Hargens and Herting revisited. Scientometrics 22, 313–325 (1991). https://doi.org/10.1007/BF02020004

Download citation

  • Received:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02020004

Keywords