Skip to main content
Springer Nature Link
Log in

On the effectiveness of the scientific peer-review system: a case study of the Journal of High Energy Physics

  • Published:
International Journal on Digital Libraries Aims and scope Submit manuscript

Abstract

The importance and the need for the peer-review system is highly debated in the academic community, and recently there has been a growing consensus to completely get rid of it. This is one of the steps in the publication pipeline that usually requires the publishing house to invest a significant portion of their budget in order to ensure quality editing and reviewing of the submissions received. Therefore, a very pertinent question is if at all such investments are worth making. To answer this question, in this paper, we perform a rigorous measurement study on a massive dataset (29k papers with 70k distinct review reports) to unfold the detailed characteristics of the peer-review process considering the three most important entities of this process—(i) the paper (ii) the authors and (iii) the referees and thereby identify different factors related to these three entities which can be leveraged to predict the long-term impact of a submitted paper. These features when plugged into a regression model achieve a high \(R^2\) of 0.85 and RMSE of 0.39. Analysis of feature importance indicates that reviewer- and author-related features are most indicative of long-term impact of a paper. We believe that our framework could definitely be utilized in assisting editors to decide the fate of a paper.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. https://inspirehep.net.

  2. http://arxiv.org/.

  3. https://ixa2.si.ehu.es/ukb.

  4. http://liwc.wpengine.com/.

  5. http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html.

  6. Refer to [23] for details.

References

  1. Ingelfinger, F.J.: Peer review in biomedical publication. Am. J. Med. 56(5), 686–692 (1974)

    Article  Google Scholar 

  2. Relman, A.S., Angell, M.: How good is peer review? N. Engl. J. Med. 321(12), 827–829 (1989)

    Article  Google Scholar 

  3. Smith, R.: Peer review: a flawed process at the heart of science and journals. J. R. Soc. Med. 99(4), 178–182 (2006)

    Article  Google Scholar 

  4. Cole, S., Simon, G.A., et al.: Chance and consensus in peer review. Science 214(4523), 881–886 (1981)

    Article  Google Scholar 

  5. Braatz, R.D.: Papers receive more citations after rejection [publication activities]. Control Syst. IEEE 34(4), 22–23 (2014)

    Article  MathSciNet  Google Scholar 

  6. Sikdar, S., Marsili, M., Ganguly, N., Mukherjee, A.: Influence of reviewer interaction network on long-term citations: a case study of the scientific peer-review system of the journal of high energy physics. In: 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pp. 1–10 (2017)

  7. Kassirer, J.P., Campion, E.W.: Peer review: crude and understudied, but indispensable. JAMA 272(2), 96–97 (1994)

    Article  Google Scholar 

  8. Lee, C.J., Sugimoto, C.R., Zhang, G., Cronin, B.: Bias in peer review. JASIST 64(1), 2–17 (2013)

    Article  Google Scholar 

  9. McNutt, R.A., Evans, A.T., Fletcher, R.H., Fletcher, S.W.: The effects of blinding on the quality of peer review: a randomized trial. JAMA 263(10), 1371–1376 (1990)

    Article  Google Scholar 

  10. Bjork, B., Roos, A., Lauri, M.: Scientific journal publishing: yearly volume and open access availability. Inf. Res. Int. Electron. J. 14(1), 1–14 (2009)

    Google Scholar 

  11. Craig, L.: Improving peer review. Bull Ecol Soc Am 69(2), 109–111 (1988)

    Google Scholar 

  12. Rodriguez, M.A., Bollen, J.: An algorithm to determine peer-reviewers. In: Proceedings of the 17th ACM Conference on Information and Knowledge Management, pp. 319–328. ACM (2008)

  13. Rodriguez, M.A., Bollen, J., Van de Sompel, H.: Mapping the bid behavior of conference referees. J. Inform. 1(1), 68–82 (2007)

    Article  Google Scholar 

  14. Mimno, D., McCallum, A.: Expertise modeling for matching papers with reviewers. In: Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 500–509. ACM (2007)

  15. Yan, R., Huang, C., Tang, J., Zhang, Y., Li, X.: To better stand on the shoulder of giants. In: Proceedings of the 12th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 51–60. ACM (2012)

  16. Coupé, T.: Peer review versus citations-an analysis of best paper prizes. Res. Policy 42(1), 295–301 (2013)

    Article  Google Scholar 

  17. Kim, S.-M., Pantel, P., Chklovski, T., Pennacchiotti, M.: Automatically assessing review helpfulness. In: EMNLP, pp. 423–430. Association for Computational Linguistics (2006)

  18. Ghose, A., Ipeirotis, P.G.: Estimating the helpfulness and economic impact of product reviews: mining text and reviewer characteristics. TKDD 23(10), 1498–1512 (2011)

    Google Scholar 

  19. Chakraborty, T., Kumar, S., Goyal, P., Ganguly, N., Mukherjee, A.: Towards a stratified learning approach to predict future citation counts. In: Proceedings of the 14th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 351–360. IEEE Press (2014)

  20. Montejo-Ráez, A., Martínez-Cámara, E., Martin-Valdivia, M.T., Urena-Lopez, L.A.: Random walk weighting over sentiwordnet for sentiment polarity detection on twitter. In: WASSA (2012)

  21. Pennebaker, J.W., Chung, C.K., Ireland, M., Gonzales, A., Booth, R.J.: The development and psychometric properties of LIWC2007 (2007)

  22. Sikdar, S., Marsili, M., Ganguly, N., Mukherjee, A.: Anomalies in the peer-review system: A case study of the journal of high energy physics. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 2245–2250. ACM (2016)

  23. Smola, A.J., Schölkopf, B.: A tutorial on support vector regression. Stat. Comput. 14(3), 199–222 (2004)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

We would like to thank the publishing team of JHEP for providing us the data and they were the only people willing to share.

Author information

Authors and Affiliations

  1. Indian Institute of Technology Kharagpur, Kharagpur, India

    Sandipan Sikdar, Paras Tehria, Niloy Ganguly & Animesh Mukherjee

  2. International Centre for Theoretical Physics, Trieste, Italy

    Matteo Marsili

Authors
  1. Sandipan Sikdar
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Paras Tehria
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Matteo Marsili
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Niloy Ganguly
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Animesh Mukherjee
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Sandipan Sikdar.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sikdar, S., Tehria, P., Marsili, M. et al. On the effectiveness of the scientific peer-review system: a case study of the Journal of High Energy Physics. Int J Digit Libr 21, 93–107 (2020). https://doi.org/10.1007/s00799-018-0247-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00799-018-0247-9

Keywords