Abstract
This paper focuses on ranking computer science departments based on the quality of publications by the faculty in those departments. There are multiple strategies to convert publication lists into ranking scores for the departments. Important open questions include handling multi-author publications, inclusion criteria for publications and publication venues, accounting for the quality of publication venues, and accounting for the sub-areas of computer science. An ablation study is performed to evaluate the importance of different decisions for department ranking. The correlation between the resulting rankings and the peer assessment of computer science departments provided by the U.S. News was measured to evaluate the importance of different decisions. The results show that the selection of publication venues has the highest impact on the ranking. In contrast, decisions related to publication recency, multi-author publications, and clustering publications into subareas have less impact. Overall, Pearson’s correlation coefficient between the publication-based scores and the U.S. News ranking is above 0.90 for a large range of decisions, indicating a strong agreement between the objective measure and the subjective opinion of peers.





Similar content being viewed by others
Notes
http://csrankings.org: Downloaded in July, 2017.
https://dabi.temple.edu/external/vucetic/CSranking/: Downloaded in June 2017.
https://dblp.org/: Downloaded in July, 2017.
http://csmetrics.net/.: Accessed in July, 2017)
http://csmetrics.net/.: Accessed in July, 2017)
References
Bar-Ilan, J. (2010). Web of science with the conference proceedings citation indexes: The case of computer science. Scientometrics, 83(3), 809–824.
Berger E (2017) Csrankings: Computer science rankings. http://csrankings.org, accessed: 2017-05-01
Beyer, J. M., & Snipper, R. (1974). Objective versus subjective indicators of quality in graduate education. Sociology of Education. https://doi.org/10.2307/2112220
Biryukov M, Dong C (2010) Analysis of computer science communities based on dblp. In: International Conference on Theory and Practice of Digital Libraries, Springer, pp 228–235
Blackburn S (2017) Institutional publication metrics for computer science. https://github.com/csmetrics/csmetrics.net/blob/master/docs/Overview.md
Bradford S. C (1934) Sources of information on specific subjects. Engineering 137:85–86. https://cir.nii.ac.jp/crid/1570854175272181632
Butler D (2011) Computing giants launch free science metrics
Cartter AM (1966) An assessment of quality in graduate education. ERIC
Chen X, Zhou M, Carin L (2012) The contextual focused topic model. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, pp 96–104
Colledge, L., de Moya-Anegón, F., Guerrero-Bote, V. P., et al. (2010). Sjr and snip: two new journal metrics in elsevier’s scopus. Insights, 23(3), 215.
Da Silva, J. A. T., & Memon, A. R. (2017). Citescore: A cite for sore eyes, or a valuable, transparent metric? Scientometrics, 111(1), 553–556.
Deng H, Han J, Zhao B, et al (2011) Probabilistic topic models with biased propagation on heterogeneous information networks. In: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, pp 1271–1279
Drew, D. E., & Karpf, R. (1981). Ranking academic departments: Empirical findings and a theoretical perspective. Research in Higher Education, 14(4), 305–320.
Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69, 131–152.
Elton, C. F., & Rodgers, S. A. (1971). Physics department ratings: Another evaluation. Science, 174(4009), 565–568.
Hagstrom, W. O. (1971). Inputs, outputs, and the prestige of university science departments. Sociology of Education. https://doi.org/10.2307/2112029
Ji M, Han J, Danilevsky M (2011) Ranking-based classification of heterogeneous information networks. In: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, pp 1298–1306
Jones, T., Huggett, S., & Kamalski, J. (2011). Finding a way through the scientific literature: Indexes and measures. World Neurosurgery, 76(1), 36–38. https://doi.org/10.1016/j.wneu.2011.01.015
Keselman L (2019) Venue analytics: A simple alternative to citation-based metrics. arXiv preprint arXiv:1904.12573
Kuno, M., Prorok, M., Zhang, S., et al. (2022). Deciphering the us news and world report ranking of us chemistry graduate programs. Scientometrics, 127(5), 2131–2150.
Kurtz, M. J., & Henneken, E. A. (2017). Measuring metrics-a 40-year longitudinal cross-validation of citations, downloads, and peer review in astrophysics. Journal of the Association for Information Science and Technology, 68(3), 695–708.
Lazaridis, T. (2010). Ranking university departments using the mean h-index. Scientometrics, 82(2), 211–216.
Ley M (2002) The dblp computer science bibliography: Evolution, research issues, perspectives. In: International symposium on string processing and information retrieval, Springer, pp 1–10
Ley M (2005) Dblp computer science bibliography. University of Trier
Ostriker, J. P., Kuh, C. V., & Voytuk, J. A. (2011). A data-based assessment of research-doctorate programs in the United States. National Academies Press.
Prisyanti, A., Nurhayati, O., & Widodo, A. (2020). Evaluation university ranking system using quacquarelli symonds and integrated performance measurement system approach. Journal of Physics, 1524(1), 012098.
Ranking CLCL (2020) University rankings
Ross, D. (2017). Times higher education world university rankings. Research Analytics (pp. 137–146). Boca Raton: Auerbach Publications.
Scholar G (2011) Google scholar citations open to all. https://scholar.googleblog.com/2011/11/google-scholar-citations-open-to-all.html, accessed: 2023-03-20
Shanghai Jiaotong University C (2021) 2021 academic ranking of world universities (arwu). https://www.shanghairanking.com/rankings/arwu/2021, online; accessed 28-Jan-2021
Sun Y, Han J, Zhao P, et al (2009a) Rankclus: integrating clustering with ranking for heterogeneous information network analysis. In: Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, ACM, pp 565–576
Sun Y, Yu Y, Han J (2009b) Ranking-based clustering of heterogeneous information networks with star network schema. In: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, pp 797–806
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the royal statistical society series b-methodological, 58, 267–288.
Vucetic, S., Chanda, A. K., Zhang, S., et al. (2018). Peer assessment of cs doctoral programs shows strong correlation with faculty citations. Communications of the ACM, 61(9), 70–76.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Aniruddha Maiti and Sai Shi have contributed equally in this work.
Ranking comparison list
Ranking comparison list
See Table 10.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Maiti, A., Shi, S. & Vucetic, S. An ablation study on the use of publication venue quality to rank computer science departments. Scientometrics 128, 4197–4218 (2023). https://doi.org/10.1007/s11192-023-04733-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-023-04733-2