Skip to main content
SpringerLink
Log in

A performance indicator for academic communities based on external publication profiles

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Studying research productivity is a challenging task that is important for understanding how science evolves and crucial for agencies (and governments). In this context, we propose an approach for quantifying the scientific performance of a community (group of researchers) based on the similarity between its publication profile and a reference community’s publication profile. Unlike most approaches that consider citation analysis, which requires access to the content of a publication, we only need the researchers’ publication records. We investigate the similarity between communities and adopt a new metric named Volume Intensity. Our goal is to use Volume Intensity for measuring the internationality degree of a community. Our experimental results , using Computer Science graduate programs and including both real and random scenarios, show we can use publication profile as a performance indicator.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. Google Scholar: http://scholar.google.com.

  2. Microsoft Academic Search: http://academic.research.microsoft.com.

  3. AMiner: http://aminer.org.

  4. DBLP: http://dblp.uni-trier.de/.

  5. CAPES: http://www.capes.gov.br.

  6. The choice of these programs follows Laender et al. (2008), which did not include Eastern institutions because their cultural profile is completely different from Brazil’s.

  7. Agreement has been used in academic ranking evaluation by Silva et al. (2014).

  8. DCG has been used in academic ranking evaluation by other studies such as (Lima et al. 2013) and (Ribas et al. 2015).

  9. Volume intensity is called as intensity in the results.

References

  • Abramo, G., D’Angelo, C., & Di Costa, F. (2011). National research assessment exercises: A comparison of peer review and bibliometrics rankings. Scientometrics, 89(3), 929–941.

    Article  Google Scholar 

  • Bollen, J., Van de Sompel, H., Hagberg, A., & Chute, R. (2009). A principal component analysis of 39 scientific impact measures. PLoS One, 4(6), 6022.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H. D. (2008). What do citation counts measure? a review of studies on citing behavior. Journal of Documentation, 64(1), 45–80.

    Article  Google Scholar 

  • Brandão, M. A., Moro, M. M., & Almeida, J. M. (2014). Experimental evaluation of academic collaboration recommendation using factorial design. Journal of Information and Data Management, 5(1), 52–63.

    Google Scholar 

  • Digiampietri, L. A., Mena-Chalco, J. P., Vaz de Melo, P. O. S., Malheiro, A. P. R., Meira, D. N. O., Franco, L. F., et al. (2014). BraX-ray: An X-ray of the Brazilian computer science graduate programs. PLoS One, 9(4), e94541.

    Article  Google Scholar 

  • Freire, V. P., & Figueiredo, D. R. (2011). Ranking in collaboration networks using a group based metric. Journal of the Brazilian Computer Society, 17(4), 255–266.

    Article  MathSciNet  Google Scholar 

  • Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161(8), 979–980.

    Google Scholar 

  • Gonçalves, G. D., Figueiredo, F., Almeida, J. M., & Gonçalves, M. A. (2014). Characterizing scholar popularity. A case study in the computer science research community. In: JCDL, London, pp. 57–66.

  • Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). The leiden manifesto for research metrics. Nature, 520, 429–431.

    Article  Google Scholar 

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. PNAS, 102(46), 16,569–16,572.

    Article  Google Scholar 

  • Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4), 422–446.

    Article  Google Scholar 

  • Laender, A. H. F., de Lucena, C. J. P., Maldonado, J. C., de Souza e Silva, E., & Ziviani, N. (2008). Assessing the research and education quality of the top brazilian computer science graduate programs. SIGCSE Bulletin, 40(2), 135–145.

    Article  Google Scholar 

  • Lee, D., Kang, J., Mitra, P., Giles, C. L., & On, B. W. (2007). Are your citations clean? Communications of the ACM, 50(12), 33–38.

    Article  Google Scholar 

  • Lima, H., Silva, T. H. P., Moro, M. M., Santos, R. L. T, Jr., & Meira, W., Laender AHF,. (2013). Aggregating productivity indices for ranking researchers across multiple areas. JCDL (pp. 97–106). USA: Indianapolis.

  • Lima, H., Silva, T. H. P., Moro, M. M., Santos, R. L. T., Wagner, Meira J., & Laender, A. H. F. (2015). Assessing the profile of top brazilian computer science researchers. Scientometrics, 103(3), 879–896.

    Article  Google Scholar 

  • Lopes, G. R., Moro, M. M., da Silva, R., Barbosa, E. M., & de Oliveira, J. P. M. (2011). Ranking strategy for graduate programs evaluation. In: ICITA, Sydney, Australia, pp 59–64.

  • Mena-Chalco, J. P., Digiampietri, L. A., Lopes, F. M., & Cesar, R. M. (2014). Brazilian bibliometric coauthorship networks. Journal of the Association for Information Science and Technology, 65(7), 1424–1445.

    Article  Google Scholar 

  • Menezes, G. V., Ziviani, N., & Laender, A. H., Almeida, V. (2009). A geographical analysis of knowledge production in computer science. In: WWW, Madrid, Spain, pp 1041–1050.

  • Molinari, J. F., & Molinari, A. (2008). A new methodology for ranking scientific institutions. Scientometrics, 75(1), 163–174.

    Article  Google Scholar 

  • Ortega, J. L., López-Romero, E., & Fernández, I. (2011). Multivariate approach to classify research institutes according to their outputs: The case of the csic’s institutes. Journal of Informetrics, 5(3), 323–332.

    Google Scholar 

  • Podlubny, I. (2005). Comparison of scientific impact expressed by the number of citations in different fields of science. Scientometrics, 64(1), 95–99.

    Article  Google Scholar 

  • Ribas, S., Ribeiro-Neto, B., de Souza e Silva, E., Ueda, A. H., & Ziviani, N. (2015). Using reference groups to assess academic productivity in computer science. In: WWW Companion, pp 603–608.

  • Sheskin, D. J. (2007). Handbook of parametric and nonparametric statistical procedures (4th ed.). Boca Raton: Chapman & Hall/CRC.

    MATH  Google Scholar 

  • Silva T. H. P., Moro, M. M., Silva, A. P. C., Meira, W. Jr., & Laender, A. H. F. (2014) Community-based endogamy as an influence indicator. In: JCDL, London, UK, pp 67–76.

  • Silva, T. H. P,, Moro, M. M., & Silva, A. P. C. (2015a). Authorship contribution dynamics on publication venues in computer science: An aggregated quality analysis. In: SAC, Salamanca, Spain, pp 1142–1147.

  • Silva, T. H. P., Moro, M. M., & Silva, A. P. C. (2015b) Tc-index: A new research productivity index based on evolving communities. In: TPDL, Poznań, Poland, pp 209–221.

  • Vieira, E., & Gomes, J. (2010). A research impact indicator for institutions. Journal of Informetrics, 4(4), 581–590.

    Article  Google Scholar 

  • Wainer, J., Eckmann, M., Goldenstein, S., & Rocha, A. (2013). How productivity and impact differ across computer science subareas. Communicatoins of the ACM, 56(8), 67–73.

    Article  Google Scholar 

  • Waltman, L., & van Eck, N. J. (2012). The inconsistency of the h-index. Journal of the American Society for Information Science and Technology, 63(2), 406–415.

    Article  Google Scholar 

  • Waltman, L., van Eck, N. J., & Wouters, P. (2013). Counting publications and citations: Is more always better? Journal of Informetrics, 7(3), 635–641.

    Article  Google Scholar 

Download references

Acknowledgments

This work was funded by the authors’ individual grants from CNPq and FAPEMIG.

Author information

Authors and Affiliations

  1. Computer Science Department, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil

    Thiago H. P. Silva, Gustavo Penha, Ana Paula Couto da Silva & Mirella M. Moro

Authors
  1. Thiago H. P. Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gustavo Penha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ana Paula Couto da Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mirella M. Moro
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mirella M. Moro.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Silva, T.H.P., Penha, G., da Silva, A.P.C. et al. A performance indicator for academic communities based on external publication profiles. Scientometrics 107, 1389–1403 (2016). https://doi.org/10.1007/s11192-016-1901-x

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-016-1901-x

Keywords

Search

Navigation