Skip to main content
Log in

An ablation study on the use of publication venue quality to rank computer science departments

Publication quality is strongly correlated with the subjective perception of research strength

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This paper focuses on ranking computer science departments based on the quality of publications by the faculty in those departments. There are multiple strategies to convert publication lists into ranking scores for the departments. Important open questions include handling multi-author publications, inclusion criteria for publications and publication venues, accounting for the quality of publication venues, and accounting for the sub-areas of computer science. An ablation study is performed to evaluate the importance of different decisions for department ranking. The correlation between the resulting rankings and the peer assessment of computer science departments provided by the U.S. News was measured to evaluate the importance of different decisions. The results show that the selection of publication venues has the highest impact on the ranking. In contrast, decisions related to publication recency, multi-author publications, and clustering publications into subareas have less impact. Overall, Pearson’s correlation coefficient between the publication-based scores and the U.S. News ranking is above 0.90 for a large range of decisions, indicating a strong agreement between the objective measure and the subjective opinion of peers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://www.usnews.com/best-graduate-schools/top-science-schools/computer-science-rankings.

  2. https://app.neilpatel.com/en/traffic_analyzer/overview?domain=https%3A%2F%2Fwww.usnews.com%2Fbest-graduate-schools%2Ftop-science-schools%2Fcomputer-science-rankings.

  3. https://scholar.google.com/.

  4. https://dabi.temple.edu/external/vucetic/CSranking/.

  5. http://csrankings.org.

  6. https://dblp.org/.

  7. https://app.neilpatel.com/en/traffic_analyzer/overview?lang=en &locId=2840 &domain=csrankings.org.

  8. https://app.neilpatel.com/en/traffic_analyzer/overview?domain=https%3A%2F%2Fcsmetrics.net%2F &lang=en &locId=2840 &mode=url.

  9. http://csrankings.org: Downloaded in July, 2017.

  10. https://dabi.temple.edu/external/vucetic/CSranking/: Downloaded in June 2017.

  11. https://dblp.org/: Downloaded in July, 2017.

  12. http://csmetrics.net/.: Accessed in July, 2017)

  13. http://csmetrics.net/.: Accessed in July, 2017)

References

  • Bar-Ilan, J. (2010). Web of science with the conference proceedings citation indexes: The case of computer science. Scientometrics, 83(3), 809–824.

    Article  Google Scholar 

  • Berger E (2017) Csrankings: Computer science rankings. http://csrankings.org, accessed: 2017-05-01

  • Beyer, J. M., & Snipper, R. (1974). Objective versus subjective indicators of quality in graduate education. Sociology of Education. https://doi.org/10.2307/2112220

    Article  Google Scholar 

  • Biryukov M, Dong C (2010) Analysis of computer science communities based on dblp. In: International Conference on Theory and Practice of Digital Libraries, Springer, pp 228–235

  • Blackburn S (2017) Institutional publication metrics for computer science. https://github.com/csmetrics/csmetrics.net/blob/master/docs/Overview.md

  • Bradford S. C (1934) Sources of information on specific subjects. Engineering 137:85–86. https://cir.nii.ac.jp/crid/1570854175272181632

  • Butler D (2011) Computing giants launch free science metrics

  • Cartter AM (1966) An assessment of quality in graduate education. ERIC

  • Chen X, Zhou M, Carin L (2012) The contextual focused topic model. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, pp 96–104

  • Colledge, L., de Moya-Anegón, F., Guerrero-Bote, V. P., et al. (2010). Sjr and snip: two new journal metrics in elsevier’s scopus. Insights, 23(3), 215.

    Article  Google Scholar 

  • Da Silva, J. A. T., & Memon, A. R. (2017). Citescore: A cite for sore eyes, or a valuable, transparent metric? Scientometrics, 111(1), 553–556.

    Article  Google Scholar 

  • Deng H, Han J, Zhao B, et al (2011) Probabilistic topic models with biased propagation on heterogeneous information networks. In: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, pp 1271–1279

  • Drew, D. E., & Karpf, R. (1981). Ranking academic departments: Empirical findings and a theoretical perspective. Research in Higher Education, 14(4), 305–320.

    Article  Google Scholar 

  • Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69, 131–152.

    Article  Google Scholar 

  • Elton, C. F., & Rodgers, S. A. (1971). Physics department ratings: Another evaluation. Science, 174(4009), 565–568.

    Article  Google Scholar 

  • Hagstrom, W. O. (1971). Inputs, outputs, and the prestige of university science departments. Sociology of Education. https://doi.org/10.2307/2112029

    Article  Google Scholar 

  • Ji M, Han J, Danilevsky M (2011) Ranking-based classification of heterogeneous information networks. In: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, pp 1298–1306

  • Jones, T., Huggett, S., & Kamalski, J. (2011). Finding a way through the scientific literature: Indexes and measures. World Neurosurgery, 76(1), 36–38. https://doi.org/10.1016/j.wneu.2011.01.015

    Article  Google Scholar 

  • Keselman L (2019) Venue analytics: A simple alternative to citation-based metrics. arXiv preprint arXiv:1904.12573

  • Kuno, M., Prorok, M., Zhang, S., et al. (2022). Deciphering the us news and world report ranking of us chemistry graduate programs. Scientometrics, 127(5), 2131–2150.

    Article  Google Scholar 

  • Kurtz, M. J., & Henneken, E. A. (2017). Measuring metrics-a 40-year longitudinal cross-validation of citations, downloads, and peer review in astrophysics. Journal of the Association for Information Science and Technology, 68(3), 695–708.

    Article  Google Scholar 

  • Lazaridis, T. (2010). Ranking university departments using the mean h-index. Scientometrics, 82(2), 211–216.

    Article  Google Scholar 

  • Ley M (2002) The dblp computer science bibliography: Evolution, research issues, perspectives. In: International symposium on string processing and information retrieval, Springer, pp 1–10

  • Ley M (2005) Dblp computer science bibliography. University of Trier

  • Ostriker, J. P., Kuh, C. V., & Voytuk, J. A. (2011). A data-based assessment of research-doctorate programs in the United States. National Academies Press.

    Google Scholar 

  • Prisyanti, A., Nurhayati, O., & Widodo, A. (2020). Evaluation university ranking system using quacquarelli symonds and integrated performance measurement system approach. Journal of Physics, 1524(1), 012098.

    Google Scholar 

  • Ranking CLCL (2020) University rankings

  • Ross, D. (2017). Times higher education world university rankings. Research Analytics (pp. 137–146). Boca Raton: Auerbach Publications.

    Chapter  Google Scholar 

  • Scholar G (2011) Google scholar citations open to all. https://scholar.googleblog.com/2011/11/google-scholar-citations-open-to-all.html, accessed: 2023-03-20

  • Shanghai Jiaotong University C (2021) 2021 academic ranking of world universities (arwu). https://www.shanghairanking.com/rankings/arwu/2021, online; accessed 28-Jan-2021

  • Sun Y, Han J, Zhao P, et al (2009a) Rankclus: integrating clustering with ranking for heterogeneous information network analysis. In: Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, ACM, pp 565–576

  • Sun Y, Yu Y, Han J (2009b) Ranking-based clustering of heterogeneous information networks with star network schema. In: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, pp 797–806

  • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the royal statistical society series b-methodological, 58, 267–288.

    MathSciNet  MATH  Google Scholar 

  • Vucetic, S., Chanda, A. K., Zhang, S., et al. (2018). Peer assessment of cs doctoral programs shows strong correlation with faculty citations. Communications of the ACM, 61(9), 70–76.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Aniruddha Maiti.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Aniruddha Maiti and Sai Shi have contributed equally in this work.

Ranking comparison list

Ranking comparison list

See Table 10.

Table 10 Ranking of 100 U.S. CS graduate programs in 2018 (Size: number of faculty in a department; Ab-1: Ranking obtained through grouping 67 venues by the k-means clustering; Ab-2: Ranking obtained through grouping 290 venues by the k-means clustering)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Maiti, A., Shi, S. & Vucetic, S. An ablation study on the use of publication venue quality to rank computer science departments. Scientometrics 128, 4197–4218 (2023). https://doi.org/10.1007/s11192-023-04733-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-023-04733-2

Keywords

Navigation