Abstract
Quantifying the scientific performance of investigators has become an integral part of decision-making in research policy. The aim of the present study was to evaluate if there is a correlation between journal impact factor (IF) and researchers’ influence among a selected group of Brazilian investigators in the fields of clinical nephrology and neurosciences. This study was based on 94 senior investigators (36 in clinical nephrology and 58 in clinical neurosciences) receiving productivity scholarships from the Brazilian Council for Scientific and Technological Development (CNPq) according to a list provided by the agency in February 2009. Scientific performance indicators included in the analysis were: number of papers indexed by the Web of Science and Scopus databases, number of citations, h- and m-index. IFs were analyzed as (1) cumulative IF (∑IF), (2) IF adjusted by time (IF/t), and (3) average IF. There was a moderate positive correlation only between ∑IF and two indicators: total number of citations (P < 0.001) and h-index (P < 0.001). There was also a positive correlation between IF/t and m-index (P < 0.001). There was an agreement in these correlations between both groups (clinical nephrology and neurosciences). No significant correlation between the average IF and any of the scientific indicators was detected. A cut-off of 10.53 for IF/t showed the best performance in predicting researchers with m-index equal to or greater than 1. According to our findings, other qualitative and quantitative instruments rather than IF are clearly needed for identifying researchers with outstanding scientific output.


Similar content being viewed by others
References
Abramo, G., D’Angelo, C. A., & Costa, F. (2011). National research assessment exercises: The effects of changing the rules of the game during the game. Scientometrics, 88, 229–238.
Abt, H. A. (2011). A publication index that is independent of age. Scientometrics,. doi:10.1007/s11192-011-0525-4.
Balaban, A. T. (2012). Positive and negative aspects of citation indices and journal impact factors. Scientometrics, 92, 241–247.
Benitez-Bribiesca, L. (1999). The impact factor of medical journals: Its use and misuse. Archives of Medical Research, 30(3), 161–162.
Boell, S. K., & Wilson, C. S. (2010). Journal Impact Factors for evaluating scientific performance: Use of h-like indicators. Scientometrics, 82, 613–626.
Bordons, M., Fernandez, M. T., & Gomes, I. (2002). Advantages and limitations in the use of impact factor measures for the assessment of research performance in a peripheral country. Scientometrics, 53(2), 195–206.
Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80.
Bornmann, L., Mutz, R., & Daniel, H.-D. (2008a). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. Journal of the American Society for Information Science and Technology, 59(5), 830–837.
Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H.-D. (2008b). Citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8, 93–102.
Bornmann, L., Wallon, G., & Ledin, A. (2008c). Is the h index related to (standard) bibliometric measures and to the assessments by peers? An investigation of the h index by using molecular life sciences data. Research Evaluation, 17(2), 149–156.
Braun, T. (2012). Editorial. Scientometrics, 92, 207–208.
Brink, A. J. (2004). Impact factor: Use and abuse. Cardiovascular Journal of South Africa, 15(1), 5–7.
Brown, H. (2007). How impact factors changed medical publishing—and science. BMJ, 334(7593), 561–564. doi:10.1136/bmj.39142.454086.AD.
Cardoso, S. C., & Gattas, G. J. (2009). The scientific production of full professors of the Faculdade de Medicina da Universidade de Sao Paulo: A view of the period of 2001–2006. Clinics (Sao Paulo), 64(9), 903–909. doi:34910.1590/S1807-59322009000900012.
CNPq, Ministério de Ciência e Tecnologia (2012). Critérios de Julgamento—CA-MD. Retrieved September 4, 2012 from http://www.cnpq.br/cas/ca-md.htm#criterios.
Correa, A. A., Moreira-Almeida, A., Meneze, P. R., Vallada, H., & Scazufca, M. (2011). Investigating the role played by social support in the association between religiosity and mental health in low income older adults: Results from the Sao Paulo Ageing and Health Study (SPAH). Revista Brasileira de Psiquiatria, 33(2), 157–164.
Dong, P., Loh, M., & Mondry, A. (2005). The “impact factor” revisited. Biomedical Digital Libraries, 2, 7. doi:10.1186/1742-5581-2-7.
Editorial. (2012). Count on me. Nature, 489, 177.
Franceschini, F., & Maisano, D. (2011). Proposals for evaluating the regularity of a scientist’s research output. Scientometrics, 88, 279–295.
Grzybowski, A. (2009). The journal impact factor: How to interpret its true value and importance. Medical Science Monitor, 15(2), SR1–SR4.
Grzybowski, A. (2010). Impact factor—strengths and weaknesses. Clinics in Dermatology, 28(4), 455–457. doi:10.1016/j.clindermatol.2010.01.002.
Haeffner-Cavaillon, N., & Graillot-Gak, C. (2009). The use of bibliometric indicators to help peer-review assessment. Archivum Immunolgiae et Therapiae Experimentalis, 57(1), 33–38. doi:10.1007/s00005-009-0004-2.
Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of United States of America, 102(46), 16569–16572. doi:10.1073/pnas.0507655102.
Jeang, K. T. (2008). H-index, mentoring-index, highly-cited and highly-accessed: How to evaluate scientists? Retrovirology, 5, 106. doi:10.1186/1742-4690-5-106.
Kelly, C. D., & Jennions, M. D. (2006). The h index and career assessment by numbers. Trends in Ecology and Evolution, 21(4), 167–170. doi:10.1016/j.tree.2006.01.005.
Kumar, V., Upadhyay, S., & Medhi, B. (2009). Impact of the impact factor in biomedical research: Its use and misuse. Singapore Medical Journal, 50(8), 752–755.
Lehmann, S., Jackson, A. D., & Lautrup, B. E. (2006). Measures for measures. Nature, 444(7122), 1003–1004. doi:10.1038/4441003a.
Martelli-Júnior, H., Martelli, D. R., Quirino, I. G., Oliveira, M. C. L., Lima, L. S., & Oliveira, E. A. (2010). CNPq researchers in medicine: A comparative study of research areas. Revista da Associação Médica Brasile, 56(4), 478–483.
Metze, K. (2010). Bureaucrats, researchers, editors, and the impact factor: A vicious circle that is detrimental to science. Clinics (Sao Paulo), 65(10), 937–940.
Oliveira, E. A., Colosimo, E. A., Martelli, D. R., Quirino, I. G., Oliveira, M. C. L., Silva, L. S., et al. (2011a). Comparison of Brazilian researchers in clinical medicine: Are criteria for ranking well-adjusted? Scientometrics, 90, 429–443. doi:10.1007/s11192-011-0492-9.
Oliveira, E. A., Peicots-Filho, R., Quirino, I. G., Oliveira, M. C. L., Martelli, D. R., Lima, L. S., et al. (2011b). Perfil e produção científica dos pesquisadores do CNPq nas áreas de Nefrologia e Urologia. Jornal Brasileiro de Nefrologia, 33, 31–37.
Oliveira, E. A., Ribeiro, A. L. P., Quirino, I. G., Oliveira, M. C. L., Martelli, D. R., Lima, L. S., et al. (2011c). Perfil e produção científica dos pesquisadores do Conselho Nacional de Desenvolvimento Científico e Tecnológico na área de Cardiologia. Arquivos Brasileiro de Cardiologia, 97(3), 186–193.
Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research, 33(1), 1–7.
Panaretos, J., & Malesios, C. (2009). Assessing scientific research performance and impact with single indices. Scientometrics, 81(3), 635–670.
Rothenberg, R. (2008). The impact factor follies. Epidemiology, 19(3), 372. doi:10.1097/EDE.0b013e31816b6a8c.
Santos, N. C. F., Candido, L. F. O., & Kuppens, C. L. (2010). Produtividade em pesquisa do CNPq: análise do perfil dos pesquisadores da química. Quimica Nova, 33(2), 489–495.
Seglen, P. O. (1994). Causal relationship between article citedness and journal impact. Journal of the American Society for Information Science, 45(1), 1–11.
Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ, 314(7079), 498–502.
Shibayama, S. (2011). Distribution of academic research funds: A case of Japanese national research grant. Scientometrics, 88, 43–60.
Simons, K. (2008). The misused impact factor. Science, 322(5899), 165. doi:10.1126/science.1165316.
Taylor, M., Perakakis, P., & Trachana, V. (2008). The siege of science. Ethics in Science and Environmental Politics, 8, 17–40.
Tess, B. H., Furuie, S. S., Castro, R. C., Barreto Mdo, C., & Nobre, M. R. (2009). Assessing the scientific research productivity of a Brazilian healthcare institution: A case study at the Heart Institute of Sao Paulo, Brazil. Clinics (Sao Paulo), 64(6), 571–576.
van Leeuwen, T. (2012). Discussing some basic critique on Journal Impact Factors: Revision of earlier comments. Scientometrics, 92, 443–455.
Vinkler, P. (1986). Evaluation of some methods for the relative assessment of scientific publications. Scientometrics, 10, 157–177.
Zitt, M. (2012). The journal impact factor: Angel, devil, or scapegoat? A comment on J. K. Vanclay’s article 2011. Scientometrics, 92, 485–503.
Acknowledgments
This study was partially supported by CNPq (National Council for Scientific and Technological Development) and FAPEMIG (Research Support Foundation of Minas Gerais). Eduardo A. Oliveira, Ana Cristina Simões e Silva, and Roberto Peicots-Filho were the recipients of CNPq scholarships in the area of medicine and their main area of investigation is clinical nephrology. Hercílio Martelli-Júnior and Enrico A. Colosimo were the recipients of CNPq scholarships in the areas of Dentistry and Mathematics, respectively.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Oliveira, E.A., Peicots-Filho, R., Martelli, D.R. et al. Is there a correlation between journal impact factor and researchers’ performance? A study comprising the fields of clinical nephrology and neurosciences. Scientometrics 97, 149–160 (2013). https://doi.org/10.1007/s11192-013-0992-x
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-013-0992-x