Abstract
The peer review process is an essential component for the progress of science. The ACM/IEEE International Conference on Human–Robot Interaction is the prime publication channel for the field and this study evaluates its peer review process. The results show that the number of accepted papers are unevenly distributed across countries, organizations and authors. The contributions from the US outweigh all others contributions. A Binary Logistic Regression analysis showed that only for 85.5% of the papers the reviewers’ scores accurately predict its acceptance or rejection. Moreover, there was no significant correlation between the reviewers’ scores and the citations the papers later attract. 73% of the authors only ever submitted one paper and the proportion of newcomers at the conferences ranges from 63–77%.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Profanity removed by author.
The 18th IEEE International Symposium on Robot and Human Interactive Communication.
2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.
References
Bar-Ilan, J. (2008). Which h-index? A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257–271. doi:10.1007/s11192-008-0216-y.
Bartneck, C. (2010). The all-in publication policy. Fourth International Conference on Digital Society (ICDS 2010), IEEE (pp. 37–40). doi:10.1109/ICDS.2010.14.
Bartneck, C. (2011). The end of the beginning—a reflection on the first five years of the HRI conference. Scientometrics, 86(2), 487–504. doi:10.1007/s11192-010-0281-x.
Bartneck, C., & Jun, H. (2010). The fruits of collaboration in a multidisciplinary field. Scientometrics, 85(1), 41–52. doi:10.1007/s11192-010-0242-4.
Bohannon, J. (2013). Who’s afraid of peer review? Science, 342(6154), 60–65. doi:10.1126/science.342.6154.60.
Campanario, J. (2009). Rejecting and resisting nobel class discoveries: Accounts by nobel laureates. Scientometrics, 81(2), 549–565. doi:10.1007/s11192-008-2141-5.
Campanario, J. M. (1995). Commentary: On influential books and journal articles initially rejected because of negative referees’ evaluations. Science Communication 16(3), 304–325. doi:10.1177/1075547095016003004. http://scx.sagepub.com/content/16/3/304.abstract.
Eysenbach, G. (2006). Citation advantage of open access articles. PLoS Biology,. doi:10.1371/journal.pbio.0040157.
Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193. doi:10.1023/A:1014848323806.
Langford, J. (2015). The NIPS experiment. http://cacm.acm.org/blogs/blog-cacm/181996-the-nips-experiment/fulltext.
Lawrence, S. (2001). Free online availability substantially increases a paper’s impact. Nature, 411(6837), 521–521. doi:10.1038/35079151.
Matthews, D. (2016). High rejection rates by journals pointless. https://www.timeshighereducation.com/news/high-rejection-rates-by-journals-pointless.
Meho, L. I., & Rogers, Y. (2008). Citation counting, citation ranking, and h-index of human–computer interaction researchers: A comparison of scopus and web of science. Journal of the American Society for Information Science and Technology, 59(11), 1711–1726. doi:10.1002/asi.20874.
Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105–2125. doi:10.1002/asi.20677.
Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5(2), 187–195.
Price, E. (2014). The NIPS experiment. http://blog.mrtz.org/2014/12/15/the-nips-experiment.html.
Safi, M. (2015). Journal accepts bogus paper requesting removal from mailing list. http://www.theguardian.com/australia-news/2014/nov/25/journal-accepts-paper-requesting-removal-from-mailing-list.
Rocha da Silva, P. (2015). Selecting for impact: new data debunks old beliefs. http://blog.frontiersin.org/2015/12/21/4782/.
Acknowledgements
The author would like to thank Utku Yalcin and Subha Krishna for the data processing.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bartneck, C. Reviewers’ scores do not predict impact: bibliometric analysis of the proceedings of the human–robot interaction conference. Scientometrics 110, 179–194 (2017). https://doi.org/10.1007/s11192-016-2176-y
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-016-2176-y