Abstract
This paper presents a preliminary comparison of citation impact between two types of Best Paper Awards recipients: pre-and post-publication peer-selected best papers. Two hundred and ninety-nine pairs of pre-and post-publication peer-selected best papers from 15 highly prestigious computer science conferences spanning more than two decades were gathered for empirical analysis. Each pair was published in the same proceedings, and their citation information was collected within a 10-year period after their formal publication. A series of pairwise comparisons were performed in order to unbiasedly uncover their differences in citation counts (i.e., a good proxy for citation impact), as well as in citation speed, citation diversity, and citation location distributions (i.e., three characteristic properties of citation impact). The empirical results demonstrate that in computer science, the post-publication peer-selected best papers receive significantly more citations than their pre-publication counterparts, with an estimated probability of up to 0.79 (95% CI: 0.74 ~ 0.83). Meanwhile, they tend to accumulate their citations more quickly in the later (versus earlier) stages—they can scarcely be fast-aging papers, and their accumulated citations tend to cover a broader scope of academic disciplines. Overall, however, the pre-and post-publication peer-selected best papers have very similar citation location distributions. Both of them are most frequently cited in the Background sections, followed by the Methods sections, and finally the Results sections. In addition, the citation impact superiority of the post-publication peer-selected best papers remains consistent across different fields of research and over a long period of time, from 1997 to 2022.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
References
Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533, 452–454. https://doi.org/10.1038/533452a
Brillouin, L. (1956). Science and information theory. Academic Press.
Brooks, T. A. (2000). How good are the best papers of JASIS? Journal of the American Society for Information Science, 51(5), 485–486. https://doi.org/10.1002/(SICI)1097-4571(2000)51:5%3c485::AID-ASI9%3e3.0.CO;2-F
Chen, Y., Wang, S., & Yang, L. (2021, March). Characterizing award-winning papers in Library and Information Science (LIS): A case study of LIS journals published by Emerald Publishing. In Toeppe, K., Yan, H., Chu, S. K. W. (eds.), Diversity, Divergence, Dialogue. iConference 2021. Lecture Notes in Computer Science (vol. 12646, pp. 413–428). Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-030-71305-8_35
Coupé, T. (2013). Peer review versus citations—An analysis of best paper prizes. Research Policy, 42(1), 295–301. https://doi.org/10.1016/j.respol.2012.05.004
Field, A. (2013). Discovering statistics using IBM SPSS statistics (4th ed.). Sage.
Freyne, J., Coyle, L., Smyth, B., & Cunningham, P. (2010). Relative status of journal and conference publications in computer science. Communications of the ACM, 53(11), 124–132. https://doi.org/10.1145/1839676.1839701
Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168. https://doi.org/10.1177/2515245919847202
Ghobadi, S., & Robey, D. (2017). Strategic signalling and awards: Investigation into the first decade of AIS best publications awards. The Journal of Strategic Information Systems, 26(4), 360–384. https://doi.org/10.1016/j.jsis.2017.06.001
Hinkle, D. E., Wiersma, W., & Jurs, S. G. (2003). Applied statistics for the behavioral sciences (5th ed.). Houghton Mifflin.
Huang, J. (2022, April 13). Who wins CS best paper awards?. Computer Science Open Data. Retrieved August 3, 2023, from https://jeffhuang.com/computer-science-open-data/.
Jin, C., Ma, Y., & Uzzi, B. (2021). Scientific prizes and the extraordinary growth of scientific topics. Nature Communications, 12(1), 5619. https://doi.org/10.1038/s41467-021-25712-2
Lee, D. H. (2019). Predictive power of conference-related factors on citation rates of conference papers. Scientometrics, 118(1), 281–304. https://doi.org/10.1007/s11192-016-2162-4
Ma, Y., & Uzzi, B. (2018). Scientific prize network predicts who pushes the boundaries of science. Proceedings of the National Academy of Sciences, 115(50), 12608–12615. https://doi.org/10.1073/pnas.1800485115
Martín-Martín, A., Thelwall, M., Orduna-Malea, E., & Delgado López-Cózar, E. (2021). Google scholar, microsoft academic, scopus, dimensions, web of science, and OpenCitations’ COCI: A multidisciplinary comparison of coverage via citations. Scientometrics, 126(1), 871–906. https://doi.org/10.1007/s11192-020-03690-4
Mixon, F. G., Jr., Torgler, B., & Upadhyaya, K. P. (2022). Committees or markets? An exploratory analysis of best paper awards in economics. Economies, 10(5), 110. https://doi.org/10.3390/economies10050110
Mubin, O., Tejlavwala, D., Arsalan, M., Ahmad, M., & Simoff, S. (2018). An assessment into the characteristics of award winning papers at CHI. Scientometrics, 116, 1181–1201. https://doi.org/10.1007/s11192-018-2778-7
Patil, I. (2021). Visualizations with statistical details: The ‘ggstatsplot’ approach. Journal of Open Source Software, 6(61), 3167. https://doi.org/10.21105/joss.03167
Sen, R., & Patel, P. (2012). Citation rates of award-winning ASCE papers. Journal of Professional Issues in Engineering Education and Practice, 138(2), 107–113. https://doi.org/10.1061/(ASCE)EI.1943-5541.0000092
Vrettas, G., & Sanderson, M. (2015). Conferences versus journals in computer science. Journal of the Association for Information Science and Technology, 66(12), 2674–2684. https://doi.org/10.1002/asi.23349
Wainer, J., Eckmann, M., & Rocha, A. (2015). Peer-selected “best papers”—Are they really that “good”? PloS one, 10(3), e0118446. https://doi.org/10.1371/journal.pone.0118446
Wang, J. (2013). Citation time window choice for research impact evaluation. Scientometrics, 94(3), 851–872. https://doi.org/10.1007/s11192-012-0775-9
Xu, L., Ding, K., Lin, Y., & Zhang, C. (2023). Does citation polarity help evaluate the quality of academic papers? Scientometrics, 128, 4605–4087. https://doi.org/10.1007/s11192-023-04734-1
Zhang, P., Wang, P., & Wu, Q. (2018). How are the best JASIST papers cited? Journal of the Association for Information Science and Technology, 69(6), 857–860. https://doi.org/10.1002/asi.24006
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
Averaged citation accumulation for pre-publication peer-selected best papers (n = 216) and their post-publication counterparts (n = 186)
Pairwise comparisons between pre-and post-publication peer-selected best papers in terms of discipline numbers. Each dashed line between box/violin plots connects a pair of pre-and post-publication peer-selected best papers from the same conference event. Statistical significance is calculated using the Wilcoxon signed-rank test
Multiple comparisons among different citation locations in terms of citation percentages. Statistical significance is calculated using the Friedman test
Multiple comparisons among different research fields in terms of submission numbers and acceptance rates. Each node refers to a specific conference event. Statistical significance is calculated using the Kruskal-Wallis test
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, Y. Comparison of citation impact between pre-and post-publication peer-selected best papers: the case of Best Paper Awards recipients at computer science conferences. Scientometrics 129, 641–662 (2024). https://doi.org/10.1007/s11192-023-04881-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-023-04881-5