Skip to main content
SpringerLink
Log in

A multivariate model for evaluating emergency medicine journals

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This paper proposes a multivariate model for the evaluation of international emergency medicine journals using the most widely used, efficient and representative productivity and citation indicators. This is a descriptive observational evaluation study based on a sample of 24 journals included in the emergency medicine category of the Journal Citation Reports 2015. The sample is evaluated based on seven evaluation indicators: Journal Citation Reports impact factor, three H-indices (Web of Science, Scimago and Google Scholar), the Scimago Journal Rank and two altmetric scores (3-month and any time). The bivariate correlations between the diverse distributions of evaluation indicators and a multivariate metric reliability index are calculated. The factorial structure of the indicators is explored and clusters of journals are defined. A factor score is assigned to each journal. The correlations between the seven evaluation indicators are high and statistically significant. The metric reliability of the multivariate analysis calculated using Cronbach’s alpha is .97. A general factor explains the 84.64% variance of the factor space of the seven evaluation indicators, representing the ‘quality of EM journals’ construct. The most well-represented indicator, the one with the highest communality, is H_Index_WoS (h = .922), which is also the indicator with the greatest weight in the general factor (a = .96). The journals can be classified in four clusters according to their quality. A factor score is generated for each journal that could be used as a multivariate meta-index to evaluate journal quality and to define a ranking of journals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Adam, D. (2002). The counting house. Nature, 415, 726–729.

    Article  Google Scholar 

  • Altmetrics. (2016). Altmetrics explorer in these journals. Available at: https://www.altmetric.com. Accessed 26 Apr 2016.

  • Barbic, D., Tubman, M., Lam, H., & Barbic, S. (2016). An analysis of altmetrics in emergency medicine. Academic Emergency Medicine, 23, 251–268.

    Article  Google Scholar 

  • Barnaby, D. P., & Gallagher, E. J. (1998). Alternative to the science citation index impact factor as an assessment of emergency medicine’s scientific contributions. Annals of Emergency Medicine, 31, 78–82.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., & Daniel, H. D. (2008). Are there better indices for evaluation purposes than the H-index? A comparison of nine different variants of the H-index using data from biomedicine. Journal of the American Society for Information Science, 59, 830–837.

    Article  Google Scholar 

  • Bounes, V., Dehours, E., Houze-Cerfon, V., Valle, B., Lipton, R., & Ducasse, J. L. (2013). Quality of publications in emergency medicine. American Journal of Emergency Medicine, 31, 297–301.

    Article  Google Scholar 

  • Callaham, M., Wears, R. L., & Weber, E. (2002). Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA Journal of the American Medical Association, 287, 2847–2850.

    Article  Google Scholar 

  • Callaham, M., Weber, E., & Wears, R. L. (2001). Citation characteristics of research published in emergency medicine versus other scientific journals. Annals of Emergency Medicine, 38, 513–517.

    Article  Google Scholar 

  • Carpenter, C. R., Cone, D. C., & Sarli, C. C. (2014). Using publication metrics to highlight academic productivity and research impact. Academic Emergency Medicine, 21, 1160–1172.

    Article  Google Scholar 

  • Cone, D. C., & Gerson, L. W. (2012). Measuring the measurable: A commentary on impact factor. Academic Emergency Medicine, 19, 1297–1299.

    Article  Google Scholar 

  • De Marchi, M., & Lorenzetti, E. (2016). Measuring the impact of scholarly journals in the humanities field. Scientometrics, 106, 253–261.

    Article  Google Scholar 

  • Delgado López-Cózar, E. (1999). ISO standards for the presentation of scientific periodicals: Little known and little used by Spanish biomedical journals. Journal of Documentation, 55, 288–309.

    Article  Google Scholar 

  • DeLuca, L. A., St John, A., Stolz, U., Matheson, L., Simpson, A., & Denninghoff, K. R. (2013). The distribution of the H-index among academic emergency physicians in the United States. Academic Emergency Medicine, 20, 997–1003.

    Article  Google Scholar 

  • Ennas, G., Biggio, B., & Di Guardo, M. C. (2015). Data-driven journal meta-ranking in business and management. Scientometrics, 105, 1911–1929.

    Article  Google Scholar 

  • Fernández-Cano, A., & Bueno, A. (2002). Multivariate evaluation of Spanish educational research journals. Scientometrics, 55, 87–102.

    Article  Google Scholar 

  • Fernández-Guerrero, I. M., Burbano, P., Martín-Sánchez, F. J., Hidalgo-Rodríguez, A., Leal-Lobato, M. M., Rivilla-Doce, C., et al. (2016). Producción científica de los urgenciólogos españoles durante el quinquenio 2010–2014 y comparación con el quinquenio 2005–2009. Emergencias, 28, 153–166.

    Google Scholar 

  • Gallagher, E. J., & Barnaby, D. P. (1998). Evidence of methodologic bias in the derivation of the science citation index impact factor. Annals of Emergency Medicine, 31, 83–86.

    Article  Google Scholar 

  • Garfield, E. (1983). Citation indexing—Its theory and application in science, technology, and humanities. Philadelphia: ISI Press.

    Google Scholar 

  • Google. (2016). Google Académico. Publicaciones principales—Ciencias Médicas y de la Salud (Spanish Access). Available at: https://scholar.google.com. Accessed 27 Apr 2016.

  • Haynes, B. (2007). Reliance on bibliometric databases can let you down. Nature, 446, 725.

    Article  Google Scholar 

  • Hirsch, J. E. (2010). An index to quantify an individual’s scientific research output that takes into account the effect of multiple coauthorship. Scientometrics, 85, 741–754.

    Article  Google Scholar 

  • InCites. (2016). Journal Citation Reports. Available at: https://jcr.incites.thomsonreuters.com/. Accessed 25 Apr 2016.

  • Jolliffe, I. T. (2002). Principal component analysis (2nd ed.). New York: Springer.

    MATH  Google Scholar 

  • Lee, C. H. (2015). Top cited articles and journal impact factors. American Journal of Emergency Medicine, 33, 1837–1838.

    Article  Google Scholar 

  • Lee, C.-H., Shih, C.-P., Chang, Y.-C., & Chaou, C.-H. (2011). The evolution of academic performance in emergency medicine journals: Viewpoint from 2000 to 2009 Journal Citation Reports. Academic Emergency Medicine, 18, 898–904.

    Article  Google Scholar 

  • Lee, J., Kang, H. G., Lim, T. H., Oh, J., Cho, Y., & Cho, J. H. (2014). The development of emergency medicine in South Korea: Academic productivity in highly cited journals. Hong Kong Journal of Emergency Medicine, 21, 205–212.

    Google Scholar 

  • Li, Q., Jiang, Y., & Zhang, M. (2012). National representation in the emergency medicine literature: A bibliometric analysis of highly cited journals. American Journal of Emergency Medicine, 30, 1530–1534.

    Article  Google Scholar 

  • Liaw, Y.-C., Chan, T.-Y., Fan, C.-Y., & Chiang, C.-H. (2014). Can the technological impact of academic journals be evaluated? The practice of non-patent reference (NPR) analysis. Scientometrics, 101, 17–37.

    Article  Google Scholar 

  • Lu, K., Ajiferuke, I., & Wolfram, D. (2014). Extending citer analysis to journal impact evaluation. Scientometrics, 100, 245–260.

    Article  Google Scholar 

  • Merton, R. K. (2010). The Matthew effect in science. Berliner Journal fur Soziologie, 20, 285–308.

    Article  Google Scholar 

  • Miró, O., Brown, A. F. T., Graham, C. A., Ducharme, J., Martín-Sanchez, F. J., & Cone, D. C. (2015a). Relationship between category size and journals’ impact factor: Implications for emergency medicine journals and researchers. European Journal of Emergency Medicine, 22, 355–362.

    Article  Google Scholar 

  • Miró, O., Fernández-Guerrero, I. M., Burillo-Putze, G., & Martín-Sanchez, F. J. (2015b). Análisis de los artículos de alta repercusión publicados en la revista EMERGENCIAS. Emergencias, 27, 379–385.

    Google Scholar 

  • Miró, O., Martín-Sánchez, F. J., Burillo-Putze, G., Julián-Jiménez, A., Tomás, S., & Pacheco, A. (2010). Evolución de diferentes marcadores bibliométricos y de calidad de la revista EMERGENCIAS entre 2005 y 2009 y comparación con las revistas de su especialidad incluidas en Journal Citation Reports. Emergencias, 22, 165–174.

    Google Scholar 

  • Mooi, E., & Sarstedt, M. (2001). Cluster analysis. A concise guide to market research (pp. 237–284). Berlin: Springer.

    Google Scholar 

  • Reynolds, J. C., Menegazzi, J. J., & Yealy, D. M. (2012). Emergency medicine journal impact factor and change compared to other medical and surgical specialties. Academic Emergency Medicine, 19, 1248–1254.

    Article  Google Scholar 

  • Schriger, D. L., Kadera, S. P., & von Elm, E. (2015). Are reviewers’ scores influenced by citations to their own work? An analysis of submitted manuscripts and peer reviewer reports. Annals of Emergency Medine, 67(401–406), e6.

    Google Scholar 

  • Schubert, A. (2014). Multidimensional journal evaluation. Analyzing scientific periodicals beyond the impact factor. Scientometrics, 98, 763–764.

    Article  Google Scholar 

  • Scimago Journal and Country Rank. (2016). Journals ranks. Available at: http://www.scimagojr.com. Accessed 26 Apr 2016.

  • Shideler, G. S., & Araujo, R. J. (2016). Measures of scholarly journal quality are not universally applicable to determining value of advertised annual subscription price. Scientometrics, 107, 963–973.

    Article  Google Scholar 

  • Shuaib, W., Acevedo, J. N., Khan, M. S., Santiago, L. J., & Gaeta, T. J. (2015). The top 100 cited articles published in emergency medicine journals. American Journal of Emergency Medicine, 33, 1066–1071.

    Article  Google Scholar 

  • Thoma, B., Chan, T. M., Paterson, Q. S., Milne, K., Sanders, J., & Lin, M. (2015a). Emergency medicine and critical care blogs and podcasts: Establishing an international consensus on quality. Annals of Emergency Medicine, 66, 396–402.

    Article  Google Scholar 

  • Thoma, B., Sanders, J. L., Lin, M., Paterson, Q., Steeg, J., & Chan, T. (2015b). The social media index: Measuring the impact of emergency medicine and critical care websites. Western Journal of Emergency Medicine, 16, 242–249.

    Article  Google Scholar 

  • Thorp, A. W., & Schriger, D. L. (2011). Citations to web pages in scientific articles: The permanence of archived references. Annals of Emergency Medicine, 57, 165–168.

    Article  Google Scholar 

  • Trueger, N. S., Thoma, B., Hsu, C. H., Sullivan, D., Peters, L., & Lin, M. (2015). The altmetric score: A new measure for article-level dissemination and impact. Annals of Emergency Medicine, 66, 549–553.

    Article  Google Scholar 

  • Vana, L., Hochreiter, R., & Hornik, K. (2016). Computing a journal meta-ranking using paired comparisons and adaptive lasso estimators. Scientometrics, 106, 229–251.

    Article  Google Scholar 

  • Web of Science. (2016). Science citation index-expanded. Available at: http://apps.webofknowledge.com. Accessed 25 Apr 2016.

  • Weingart, S. D., & Faust, J. S. (2014). Future evolution of traditional journals and social media medical education future evolution of traditional journals and social media medical education. Emergency Medicine Australasia, 26, 62–66.

    Article  Google Scholar 

  • West, J. D., Bergstrom, T. C., & Bergstrom, C. T. (2010). The eigenfactor metrics (TM): A network approach to assessing scholarly journals. College and Research Libraries, 71, 236–244.

    Article  Google Scholar 

  • Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics, 101, 1491–1513.

    Article  Google Scholar 

Download references

Acknowledgements

To Thomson-Reuters, Elsevier-Scimago, and Altmetrics databases.

Author information

Authors and Affiliations

  1. Department of Research Methods and Diagnostics in Education, University of Granada, Granada, Spain

    Antonio Fernandez-Cano

  2. Emergency Medicine Service, General University Hospital “Virgen de las Nieves”, Granada, Spain

    Inés M. Fernández-Guerrero

Authors
  1. Antonio Fernandez-Cano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Inés M. Fernández-Guerrero
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Antonio Fernandez-Cano.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fernandez-Cano, A., Fernández-Guerrero, I.M. A multivariate model for evaluating emergency medicine journals. Scientometrics 110, 991–1003 (2017). https://doi.org/10.1007/s11192-016-2197-6

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-016-2197-6

Keywords

Search

Navigation