Skip to main content
SpringerLink
Log in

A web application for aggregating conflicting reviewers’ preferences

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Drawing on social choice theory we derive a rationale in which each reviewer is asked to provide his or her second, third, and fourth choice in addition to his/her first choice recommendation regarding the acceptance/revision/rejection of a given manuscript. All reviewers’ hierarchies of alternatives are collected and combined such that an overall ranking can be computed. Consequently, conflicting recommendations are resolved not by asking a third adjudicating reviewer for his/her recommendation as is usual editorial praxis in many scientific journals, but rather by using more information from the available judges. After a brief introduction into social choice theory and a description and justification of the maximum likelihood rule for ranking alternatives, we describe and demonstrate a public available web application that provides easy-to-use tools to apply these methods for aggregating conflicting reviewers’ recommendations. This application might be accessed by editors to aid their decision process in case they receive conflicting recommendations by their reviewers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, fairness, and innovation. Science and Engineering Ethics 3, 63–84.

    Article  Google Scholar 

  • Arrow, K. (1951) Social Choice and Individual Values. John Wiley, New York

    MATH  Google Scholar 

  • Borda, J. C. (1781). Memoire sur les elections au scrutin. In Histoire de l’Academie Royale des Sciences. Paris:Académie Royale des Sciences

  • Burnham, J. C. (1990). The evolution of editorial peer review. JAMA 263(10), 1323–1329.

    Article  Google Scholar 

  • Campanario, J. M. (1998a). Peer review for journals as it stands today Part 1. Science Communication 19(3), 181–211.

    Article  Google Scholar 

  • Campanario, J. M. (1998b). Peer review for journals as it stands today Part 2. Science Communication 19(4), 277–306.

    Article  Google Scholar 

  • Condorcet, M. (1785). Essai sur l’application de l’analyse a la probabilite des decisions rendue a la pluralite des voix. In: de Bernon O. (Eds) Also reproduced in Condorcet, Sur les elections et autres textes, Fayard 1986. Paris: Imprimerie royale.

  • Conitzer, V., Davenport, A., & Kalagnanam, J. (2006). Improved bounds for computing Kemeny rankings. In Proceedings of the 21st National Conference on Artificial Intelligence (AAAI) (pp. 620–627). AAAI Press.

  • Chubin, D. E., & Hackett, E. J. (1990). Peerless science: Peer review and US science policy. Stony Brook, NY: State University of New York Press.

    Google Scholar 

  • Davenport, A., & Kalagnanam, J. (2004). A computational study of the Kemeny rule for preference aggregation. AAAI’04 Proceedings of the 19th national conference on Artificial intelligence (pp. 697–702).

  • Garcia, J.A., Rodriguez-Sanchez, R., & Fdez-Valdivia, J. (2013). The selection of high quality manuscripts. Scientometrics, doi:10.1007/s11192-013-1034-4

  • Hickok, G., & Poeppel, D. (2011). Should peer reviewers be paid for their work?. Talking Brains: News and views on the neural organization of language. http://www.talkingbrains.org/.

  • Kemeny, J. (1959). Mathematics without numbers. Daedalus 88, 571–591.

    Google Scholar 

  • Kendall, M. (1938). A new measure of rank correlation. Biometrika 30(1–2), 81–89.

    MATH  MathSciNet  Google Scholar 

  • Kravitz, R. L., Franks, P., Feldman, M. D., Gerrity, M., Byrne, C. et al. (2010). Editorial peer reviewers’ recommendations at a general medical journal: Are they reliable and do editors care?. PLoS One 5(4), e10072. doi:10.1371/journal.pone.0010072.

    Article  Google Scholar 

  • Lee, Carole J., Sugimoto, Cassidy R., Zhang, Guo, Cronin, & Blaise (2013). Bias in peer review. Journal of the American Society for Information Science and Technology. 64(1), 2–17.

    Article  Google Scholar 

  • Sen, A. (2008). Social choice. The New Palgrave Dictionary of Economics, 2nd Edition. Abstract & TOC. Palgrave Macmillan (U.S.).

  • Young, P. (1995). Optimal Voting Rules. The Journal of Economic Perspectives 9(1), 51–64.

    Article  Google Scholar 

  • Young, P., & Levenglick, A. (1978). A Consistent Extension of Condorcet’s Election Principle. SIAM Journal on Applied Mathematics 35(2), 285–300.

    Article  MATH  MathSciNet  Google Scholar 

Download references

Acknowledgements

This research was sponsored by the Spanish Board for Science and Technology (MICINN) under grant TIN2010-15157 cofinanced with European FEDER funds. Sincere thanks are due to the reviewers for their insightful comments, constructive suggestions, and help.

Author information

Authors and Affiliations

  1. Departamento de Ciencias de la Computación e I. A, CITIC-UGR, Universidad de Granada, 18071, Granada, Spain

    J. A. García, Rosa Rodriguez-Sánchez & J. Fdez-Valdivia

  2. Consejo Superior de Investigaciones Científicas (CSIC), Instituto de Políticas y Bienes Públicos (IPP), Centro de Ciencias Humanas y Sociales (CCHS), Albasanz 26-28, 28037, Madrid, Spain

    F. de Moya-Anegón

Authors
  1. J. A. García
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rosa Rodriguez-Sánchez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. Fdez-Valdivia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. F. de Moya-Anegón
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to J. A. García.

Rights and permissions

Reprints and permissions

About this article

Cite this article

García, J.A., Rodriguez-Sánchez, R., Fdez-Valdivia, J. et al. A web application for aggregating conflicting reviewers’ preferences. Scientometrics 99, 523–539 (2014). https://doi.org/10.1007/s11192-013-1198-y

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-013-1198-y

Keywords

Search

Navigation