Skip to main content
SpringerLink
Log in

Research-driven classification and ranking in higher education: an empirical appraisal of a Romanian policy experience

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

In this paper we investigate the problem of university classification and its relation to ranking practices in the policy context of an official evaluation of Romanian higher education institutions and their study programs. We first discuss the importance of research in the government-endorsed assessment process and analyze the evaluation methodology and the results it produced. Based on official documents and data we show that the Romanian classification of universities was implicitly hierarchical in its conception and therefore also produced hierarchical results due to its close association with the ranking of study programs and its heavy reliance on research outputs. Then, using a distinct dataset on the research performance of 1385 faculty members working in the fields of political science, sociology and marketing we further explore the differences between university categories. We find that our alternative assessment of research productivity—measured with the aid of Hirsch’s (Proc Natl Acad Sci 102(46):16569–16572, 2005) h-index and with Egghe’s (Scientometrics 69(1):131–152, 2006) g-index—only provides empirical support for a dichotomous classification of Romanian institutions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. This provision was later annulled through a Government Emergency Ordinance adopted in December 2013.

  2. For all ranking domains belonging to natural sciences, mathematics, biomedical sciences and engineering—in total 33 out of the 60 domains—the research criterion had a weight of 0.60 whereas for most domains belonging to social sciences (for example political science or administrative science) it had a weight of 0.50 and for those within humanities (for example philosophy or history) it had a weight of 0.40.

  3. According to the methodology the h-index is to be calculated based on the following databases for academics belonging to natural sciences, mathematics, biomedical sciences and engineering: Thompson Reuters’ Web of Science, Elsevier’s Scopus, Google Scholar. Different weights are attached to the h-index from each database: 0.5 for Web of Science, 0.3 for Scopus and 0.2 for Google Scholar. For academics in social sciences and humanities the h-index is to be calculated only from Google Scholar.

  4. There is of course one logical exception: in the case of index values normalized with reference to academic titles the intra-university tests of difference which compare index values between academic titles within each of the three university types are rendered obsolete. In the case of this specific analysis conducted under this particular normalization technique the differences in index values are naturally very small and statistically insignificant.

References

  • Albarrán, P., Crespo, J. A., Ortuño, I., & Ruiz-Castillo, J. (2011). The skewness of science in 219 sub-fields and a number of aggregates. Scientometrics, 88(2), 385–397.

    Article  Google Scholar 

  • Andreescu, L., Gheorghiu, R., Irimia, A., & Curaj, A. (2015). Mergers and classifications in Romania: Opportunities and obstacles. In A. Curaj, L. Georghiou, J. C. Harper, & E. Egron-Polak (Eds.), Mergers and alliances in higher education. International practice and emerging opportunities (pp. 33–55). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Andreescu, L., Gheorghiu, R., Proteasa, V., & Curaj, A. (2012). Institutional diversification and homogeneity in Romanian higher education: The larger picture. In A. Curaj, P. Scott, L. Vlăsceanu, & L. Wilson (Eds.), European higher education at the crossroads: Between the Bologna process and national reforms (pp. 863–885). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Billaut, J.-C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? An MCDM view. Scientometrics, 84(1), 237–263.

    Article  Google Scholar 

  • Bornmann, L., & Leydesdorff, L. (2014). On the meaningful and non-meaningful use of reference sets in bibliometrics. Journal of Informetrics, 8(1), 273–275.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H.-D. (2011). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5(3), 346–359.

    Article  Google Scholar 

  • Buela-Casal, G., Gutiérrez-Martínez, O., Bermúdez-Sánchez, M. P., & Vadillo-Muñoz, O. (2007). Comparative study of international academic rankings of universities. Scientometrics, 71(3), 349–365.

    Article  Google Scholar 

  • Cheng, Y., & Liu, N. C. (2006). A first approach to the classification of the top 500 world universities by their disciplinary characteristics using scientometrics. Scientometrics, 68(1), 135–150.

    Article  Google Scholar 

  • Costas, R., & Bordons, M. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1(3), 193–203.

    Article  Google Scholar 

  • Dill, D., & Soo, M. (2005). Academic quality, league tables, and public policy: A crossnational analysis of university ranking systems. Higher Education, 49(4), 495–533.

    Article  Google Scholar 

  • Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69(1), 131–152.

    Article  MathSciNet  Google Scholar 

  • García, J. A., Rodríguez-Sánchez, R., Fdez-Valdivia, J., Robinson-García, N., & Torres-Salinas, D. (2012). Mapping academic institutions according to their journal publication profile: Spanish universities as a case study. Journal of the American Society for Information Science and Technology, 63(11), 2328–2340.

    Article  Google Scholar 

  • Geuna, A. (2001). The changing rationale for European university research funding: Are there negative unintended consequences? Journal of Economic Issues, 35(3), 607–632.

    Article  Google Scholar 

  • Glänzel, W., & Henk, F. M. (2013). Opinion paper: Thoughts and facts on bibliometric indicators. Scientometrics, 96(1), 381–394.

    Article  Google Scholar 

  • Harzing, A. W. (2007). Publish or Perish. http://www.harzing.com/pop.htm.

  • Harzing, A.-W., Alakangas, S., & Adams, D. (2014). hIa: An individual annual h-index to accommodate disciplinary and career length differences. Scientometrics, 99(3), 811–821.

    Article  Google Scholar 

  • Hazelkorn, E. (2013). How rankings are reshaping higher education. In V. Climent, F. Michavila, & M. Ripolles (Eds.), Los rankings univeritarios: Mitos y realidades. Ed. Tecnos.

  • Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251–261.

    Article  MathSciNet  Google Scholar 

  • Hirsch, J. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572.

    Article  Google Scholar 

  • Huisman, J., Meek, L., & Wood, F. (2007). Institutional diversity in higher education: A cross-national and longitudinal analysis. Higher Education Quarterly, 61(4), 563–577.

    Article  Google Scholar 

  • Huisman, J., & van Vught, F. (2009). Diversity in European higher education: Historical trends and current policies. In F. van Vught (Ed.), Mapping the higher education landscape. Towards a European classification of higher education (pp. 17–38). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Kaiser, F., Faber, M., & Jongbloed, B. (2012). U-Map, university activity profiles in practice. In A. Curaj, P. Scott, L. Vlăsceanu, & L. Wilson (Eds.), European higher education at the crossroads: Between the Bologna process and national reforms (pp. 887–903). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Kaur, J., Radicchi, F., & Menczer, F. (2013). Universality of scholarly impact metrics. Journal of Informetrics, 7(4), 924–932.

    Article  Google Scholar 

  • Longden, B. (2011). Ranking indicators and weights. In J. C. Shin, R. K. Toutkoushian, & U. Teichler (Eds.), University rankings. Theoretical basis, methodology and impacts on global higher education (pp. 73–104). New York: Springer.

    Google Scholar 

  • McCormick, A. (2008). The Complex interplay between classification and ranking of colleges and universities: Should the Berlin principles apply equally to classification? Higher Education in Europe, 33(2–3), 209–218.

    Article  Google Scholar 

  • McCormick, A., & Zhao, C. (2005). Rethinking and reframing the Carnegie Classification. Change, 37(5), 51–57.

    Article  Google Scholar 

  • Miroiu, A., & Andreescu, L. (2010). Goals and instruments of diversification in higher education. Quality Assurance Review for Higher Education, 2(2), 89–101.

    Google Scholar 

  • Miroiu, A., Păunescu, M., & Vîiu, G.-A. (2015). Ranking Romanian academic departments in three fields of study using the g-index. Quality in Higher Education, 21(2), 189–212.

    Article  Google Scholar 

  • Ortega, J. S., López-Romero, E., & Fernández, I. (2011). Multivariate approach to classify research institutes according to their outputs: The case of the CSIC’s institutes. Journal of Informetrics, 5(3), 323–332.

    Google Scholar 

  • Păunescu, M., Florian, B., & Hâncean, M.-G. (2012). Internalizing quality assurance in higher education: Challenges of transition in enhancing the institutional responsibility for quality. In A. Curaj, P. Scott, L. Vlăsceanu, & L. Wilson (Eds.), European higher education at the crossroads: Between the Bologna process and national reforms (pp. 317–338). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Rauhvargers, A. (2011). Global university rankings and their impact. Brussels: European University Association.

    Google Scholar 

  • Ruiz-Castillo, J., & Costas, R. (2014). The skewness of scientific productivity. Journal of Informetrics, 8(4), 917–934.

    Article  Google Scholar 

  • Sadlak, J., & Liu, N. C. (2007). The world-class university and ranking: Aiming beyond status. Bucharest: UNESCO-CEPES.

    Google Scholar 

  • Salmi, J. (2009). The challenge of establishing world-class universities. Washington, DC: World Bank.

    Book  Google Scholar 

  • Salmi, J., & Saroyan, A. (2007). League tables as policy instruments: Uses and misuses. Higher Education Management and Policy, 19(2), 31–68.

    Article  Google Scholar 

  • Schreiber, M. (2008). An empirical investigation of the g-index for 26 physicists in comparison with the h-index, the a-index, and the r-index. Journal of the American Society for Information Science and Technology, 59(9), 1513–1522.

    Article  Google Scholar 

  • Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628–638.

    Article  Google Scholar 

  • Shin, J. C. (2009). Classifying higher education institutions in Korea: A performance-based approach. Higher Education, 57(2), 247–266.

    Article  Google Scholar 

  • Shin, J. C., & Kehm, B. (Eds.). (2013). Institutionalization of world-class university in global competition. Dordrecht: Springer.

    Google Scholar 

  • Shulman, L. S. (2005). Classification’s complexities. The Chronicle of Higher Education B, 52, 20.

    MathSciNet  Google Scholar 

  • Stensaker, B., & Gornitzka, A. (2009). The ingredients of trust in European higher education’. In B. M. Kehm, J. Huisman, & B. Stensaker (Eds.), The European higher education area: Perspectives on a moving target (pp. 125–139). Rotterdam: Sense Publishers.

    Google Scholar 

  • Usher, A., & Medow, J. (2009). A global survey of university rankings and league tables. In B. M. Kehm & B. Stensaker (Eds.), University rankings, diversity, and the new landscape of higher education (pp. 3–18). Rotterdam: Sense Publishers.

    Google Scholar 

  • van der Wende, M. (2008). Rankings and classifications in higher education: A European perspective. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 23, pp. 49–72). Berlin: Springer.

    Chapter  Google Scholar 

  • van Raan, A. F. J. (2005a). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143.

    Article  Google Scholar 

  • van Raan, A. F. J. (2005b). Measurement of central aspects of scientific research: Performance, interdisciplinarity, structure. Measurement, 3(1), 1–19.

    Google Scholar 

  • van Vught, F. (2009). Diversity and differentiation in higher education. In F. van Vught (Ed.), Mapping the higher education landscape. Towards a European classification of higher education (pp. 1–16). Dordrecht: Springer.

    Chapter  Google Scholar 

  • van Vught, F., & Ziegele, F. (Eds.). (2011). Design and testing the feasibility of a multidimensional global university ranking. Final report. Consortium for Higher Education and Research Performance Assessment, CHERPA-Network.

  • Vîiu, G.-A., Vlăsceanu, M., & Miroiu, A. (2012). Ranking political science departments: The case of Romania. Quality Assurance Review for Higher Education, 4(2), 79–97.

    Google Scholar 

  • Vîiu, G.-A., Păunescu, M., & Miroiu, A. (2015). Research-driven classification and ranking in higher education. An empirical appraisal of a Romanian policy experience. In A. A. Salah, Y. Tonta, A. A. A. Salah, A. Sugimoto, & U. Al (Eds.), Proceedings of ISSI 2015 Istanbul (pp. 622–633). Istanbul: Boğaziçi University Printhouse.

    Google Scholar 

  • Vlăsceanu, L., Miroiu, A., Păunescu, M., & Hâncean, M.-G. (Eds.). (2011). Barometrul calităţii 2010. Starea calităţii în învăţământul superior din România. Braşov: Editura Universităţii Transilvania din Braşov.

    Google Scholar 

Download references

Acknowledgments

Financial support from the National Research Council (Grant Number PN-II-ID-PCE-2011-3-0746) is gratefully acknowledged by Gabriel Vîiu and Adrian Miroiu. The authors would also like to express their gratitude for the helpful comments and suggestions provided to them during the anonymous peer review process, both for the original paper submitted for the 2015 ISSI conference as well as for the subsequent manuscript submitted to Scientometrics.

Author information

Authors and Affiliations

  1. Department of Political Science, National School of Political and Administrative Studies, Povernei Street 6, 010643, Bucharest, Romania

    Gabriel-Alexandru Vîiu, Mihai Păunescu & Adrian Miroiu

Authors
  1. Gabriel-Alexandru Vîiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mihai Păunescu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adrian Miroiu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gabriel-Alexandru Vîiu.

Additional information

The present paper is a revised and expanded version of the work (Vîiu et al. 2015) previously published in the Proceedings of ISSI 2015 Istanbul.

Appendices

Appendix 1

See Table 5.

Table 5 Results of nonparametric tests for comparison of h and g-indices between the three university types
Full size table

Appendix 2

See Table 6.

Table 6 Results of nonparametric tests for comparison of g-indices between academic staff within the three classes of universities
Full size table

Appendix 3

See Table 7.

Table 7 Intra-university comparison of g-indices across staff types
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vîiu, GA., Păunescu, M. & Miroiu, A. Research-driven classification and ranking in higher education: an empirical appraisal of a Romanian policy experience. Scientometrics 107, 785–805 (2016). https://doi.org/10.1007/s11192-016-1860-2

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-016-1860-2

Keywords

Search

Navigation