Skip to main content
Springer Nature Link
Log in

Journal ratings: a paper affiliation methodology

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Journal ratings are a key factor when individuals or institutions assess research and select journals. Despite their significant development in recent years, ratings are still not sufficiently precise or updated enough, depending on the subject and method used. In this paper, we present a new methodology, called paper affiliation index, with which to create subject journal ratings using expert judgment and research impact, both of which are based on secondary, objective measures, thus making it possible to produce lists every year without human manipulation at virtually no cost. We compare the results of our methodology with the selection of top journals by the ShanghaiRanking’s Global Ranking of Academic Subjects in the Social Sciences, the Excellence in Research for Australia, Australian Business Dean Council, the Academic Journal Guide release by Chartered Association of Business Schools lists, and the Journal Impact Factor calculated by Clarivate Analytics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. Some research has put forward mixed methods. For example, Chen and Chen (2011) proposed the Evolutionary PageRank algorithm, which first uses the PageRank algorithm to assess the journal’s prestige and then uses the Multi-Objective Particle Swarm Optimization method to balance the citation analysis and the experts’ opinions. More recently, Yuan et al. (2020) have proposed a mixed method where the weights of the objective indicators are obtained from experts’ ratings previously published in ranking lists which were based on opinions. Mixed methods can also be used for pooling different lists to obtain meta-rankings (Herrmann et al., 2011).

  2. In Docampo and Safón (in review) we have developed a less sophisticated first version of the PAI methodology for finance journals.

  3. The full results are available upon request.

  4. Retrieved December 25th, 2020 from http://www.shanghairanking.com/subject-survey/survey-methodology-2020.html.

  5. 2010 Final Journal List. Retrieved October 27th, 2020 from https://www.righttoknow.org.au/request/journal_list_relating_to_the_201

  6. These percentages are similar to the proportion of top journals in ABDC (A* = 7.4%, ABDC 2019), and AJG (4 and 4* = 7.6%, AJG 2018) lists.

  7. We would like to thank a referee for considering this factor.

References

  • ABDC (2019). Australian Business Deans Council 2019 Journal Quality List Review. Final Report 6 December 2019. Retrieved August 25th, 2020 from www.abdc.edu.au.

  • AJG (2018). Academic Journal Guide 2018. Chartered Association of Business Schools (ABS). Retrieved November 26th, 2020 from www.charteredabs.org

  • Alexander, J. C., & Mabry, R. H. (1994). Relative significance of journal authors, and articles cited in financial research. Journal of Finance, 49, 697–712.

    Article  Google Scholar 

  • Alexander, J. K., Scherer, R. F., & Lecoutre, M. (2007). A global comparison of business journal ranking systems. Journal of Education for Business, 82(6), 321–328.

    Article  Google Scholar 

  • Arnold, T., Butler, A. W., Crack, T. F., & Altintig, A. (2003). Impact: What influences finance research? Journal of Business, 76(2), 343–361.

    Article  Google Scholar 

  • Baccini, A., & De Nicolao, G. (2016). Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise. Scientometrics, 108(3), 1651–1671.

    Article  Google Scholar 

  • Bajo, E., Barbi, M., & Hillier, D. (2020). Where should I publish to get promoted? A finance journal ranking based on business school promotions. Journal of Banking & Finance, 114, 105780.

    Article  Google Scholar 

  • Barman, S., Tersine, R. J., & Buckley, M. R. (1991). An empirical assessment of the perceived relevance and quality of POM-related journals by academicians. Journal of Operations Management, 10(2), 194–212.

    Article  Google Scholar 

  • Barnes, S. J. (2005). Assessing the value of IS journals. Communications of the ACM, 48(1), 110–112.

    Article  Google Scholar 

  • Baumgartner, H., & Pieters, R. (2003). The structural influence of marketing journals: A citation analysis of the discipline and its subareas over time. Journal of Marketing, 67(2), 123–139.

    Article  Google Scholar 

  • Beattie, V., & Goodacre, A. (2006). A new method for ranking academic journals in accounting and finance. Accounting and Business Research, 36(2), 65–91.

    Article  Google Scholar 

  • Benati, S., & Stefani, S. (2011). The academic journal ranking problem: A fuzzy-clustering approach. Journal of Classification, 28(1), 7–20.

    Article  MathSciNet  Google Scholar 

  • Borokhovich, K. A., Lee, A. A., & Simkins, B. J. (2011). A framework for journal assessment: The case of the Journal of Banking and Finance. Journal of Banking & Finance, 35(1), 1–6.

    Article  Google Scholar 

  • Brown, L. D., & Huefner, R. J. (1994). The familiarity with and perceived quality of accounting journals: Views of senior accounting faculty in leading US MBA programs. Contemporary Accounting Research, 11(1), 223–250.

    Article  Google Scholar 

  • Bryce, C., Dowling, M., & Lucey, B. (2020). The journal quality perception gap. Research Policy, 49(5), 103957.

    Article  Google Scholar 

  • Chan, K. C., Chang, C. H., & Chang, Y. (2013). Ranking of finance journals: Some google scholar citation perspectives. Journal of Empirical Finance, 21, 241–250.

    Article  Google Scholar 

  • Chan, K. C., Chang, C. H., & Chang, Y. C. (2013). Ranking of finance journals: Some Google Scholar citation perspectives. Journal of Empirical Finance, 21, 241–250.

    Article  Google Scholar 

  • Chan, K. C., Chen, C. R., & Steiner, T. L. (2002). Production in the finance literature, institutional reputation, and labor mobility in academia: A global perspective. Financial Management, 31(4), 131–156.

    Article  Google Scholar 

  • Chan, K. C., Fung, A., Fung, H. G., & Yau, J. (2016). Ranking of institutions and academic journals. Managerial Finance, 42(4), 292–302.

    Article  Google Scholar 

  • Chen, Y. L., & Chen, X. H. (2011). An evolutionary PageRank approach for journal ranking with expert judgements. Journal of Information Science, 37(3), 254–272.

    Article  Google Scholar 

  • Cudd, M., & Morris, J. (1988). Bias in journal ratings. Financial Review, 23(1), 117–125.

    Article  Google Scholar 

  • Currie, R. R., & Pandher, G. S. (2011). Finance journal rankings and tiers: An active scholar assessment methodology. Journal of Banking and Finance, 35(1), 7–20.

    Article  Google Scholar 

  • Currie, R. R., & Pandher, G. S. (2020). Finance journal rankings: Active scholar assessment revisited. Journal of Banking and Finance, 111, 105717.

    Article  Google Scholar 

  • Docampo, D., and Safón, V. (in review): Finance journal rankings: A paper affiliation methodology. Journal of International Financial Management & Accounting.

  • Eisend, M. (2011). Is VHB-JOURQUAL2 a good measure of scientific quality? Assessing the validity of the major business journal ranking in German-speaking countries. Business Research, 4(2), 241–274.

    Article  Google Scholar 

  • Eleftheriou, K., & Polemis, M. (2020). One list to fit them all: What do we learn from journal ranking? Finance Research Letters, 35, 101278.

    Article  Google Scholar 

  • Engemann, K. M., & Wall, H. J. (2009). A journal ranking for the ambitious economist. Federal Reserve Bank of St. Louis Review, 91(3), 127–139.

    Google Scholar 

  • Ennas, G., Biggio, B., & Di Guardo, M. C. (2015). Data-driven journal meta-ranking in business and management. Scientometrics, 105(3), 1911–1929.

    Article  Google Scholar 

  • ERA (2010). Excellence in Research for Australia. Retrieved October 27th, 2020 from https://www.arc.gov.au/excellence-research-australia/era-reports

  • Extejt, M. M., & Smith, J. E. (1990). The behavioral sciences and management: An evaluation of relevant journals. Journal of Management, 16(3), 539–551.

    Article  Google Scholar 

  • Fei, Q., & Bell, R. (2013). Marketing journal ranking, celebrity authors, and the diminishing quality gap. Marketing journal ranking, celebrity authors, and the diminishing quality gap. Mustang Journal of Management and Marketing, 3, 16–30.

    Google Scholar 

  • Gordon, M. D. (1982). Citation ranking versus subjective evaluation in the determination of journal hierarchies in the social sciences. Journal of the American Society for Information Science, 33(1), 55–57.

    Article  Google Scholar 

  • Gorman, M. F., & Kanet, J. J. (2005). Evaluating operations management-related journals via the author affiliation index. Manufacturing and Service Operations Management, 7(1), 3–19.

    Article  Google Scholar 

  • GRAS (2020). Academic Ranking of World Universities: Methodology for ShanghaiRanking's Global Ranking of Academic Subjects 2020. Retrieved November 26th, 2020, from http://www.shanghairanking.com/Shanghairanking-Subject-Rankings/Methodology-for-ShanghaiRanking-Global-Ranking-of-Academic-Subjects-2020.html.

  • Guidry, J. A., Guidry Hollier, B. N., Johnson, L., Tanner, J. R., & Veltsos, C. (2004). Surveying the cites: A ranking of marketing journals using citation analysis. Marketing Education Review, 14(1), 45–59.

    Article  Google Scholar 

  • Guo, H., Wang, B., Qiao, X., & Liu, R. (2016). A review of studies on citations and journal ranking in finance. Managerial Finance, 42(4), 303–311.

    Article  Google Scholar 

  • Haddawy, P., Hassan, S. U., Asghar, A., & Amin, S. (2016). A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality. Journal of Informetrics, 10(1), 162–173.

    Article  Google Scholar 

  • Harzing, A. (2020). Journal quality list. Retrieved August 25th, 2020 from www.harzing.com.

  • Heckman, J. J., & Moktan, S. (2020). Publishing and promotion in economics: The tyranny of the top five. Journal of Economic Literature, 58(2), 419–470.

    Article  Google Scholar 

  • Herrmann, R., Berg, E., Dabbert, S., Pöchtrager, S., & Salhofer, K. (2011). Going beyond impact factors: A survey-based journal ranking by agricultural economists. Journal of Agricultural Economics, 62(3), 710–732.

    Article  Google Scholar 

  • Hirschberg, J. G., & Lye, J. N. (2020). Grading journals in economics: The ABCs of the ABDC. Journal of Economic Surveys, 34(4), 876–921.

    Article  Google Scholar 

  • Horowitz, I. (2003). Preference-neutral attribute weights in the journal-ranking problem. Journal of the Operational Research Society, 54(5), 452–457.

    Article  MATH  Google Scholar 

  • Jaafar, R., Pereira, V., Saab, S. S., & El-Kassar, A. N. (2020). Which journal ranking list? A case study in business and economics. EuroMed Journal of Business.

  • Jarvis, R. M., & Coleman, P. (2007). Ranking law reviews by author prominence-Ten years later. Law Library Journal, 99, 573–588.

    Google Scholar 

  • Jarvis, R. M., & Coleman, P. G. (1997). Ranking law reviews: An empirical analysis based on author prominence. Arizona Law Review, 39, 15–23.

    Google Scholar 

  • Kao, E. H., Hsu, C. H., Lu, Y., & Fung, H. G. (2016). Ranking of finance journals: A stochastic dominance analysis. Managerial Finance, 42(4), 312–323.

    Article  Google Scholar 

  • Katerattanakul, P., Han, B., & Hong, S. (2003). Objective quality ranking of computing journals. Communications of the ACM, 46(10), 111–114.

    Article  Google Scholar 

  • Krippendorff, K. (2004). Content analysis: An introduction to its methodology. Sage.

    Google Scholar 

  • Krippendorff, K. (2011). Computing Krippendorff's Alpha-Reliability. Retrieved August 25th, 2020 from http://repository.upenn.edu/asc_papers/43

  • Koojaroenprasit, N., Weinstein, A., Johnson, W. C., & Remington, D. O. (1998). Marketing journal rankings revisited: Research findings and academic implications. Marketing Education Review, 8(1), 95–102.

    Article  Google Scholar 

  • Koseoglu, M. A. (2018). A new approach to journal ranking: Social structure in hospitality and tourism journals. International Journal of Contemporary Hospitality Management, 32(2), 389–424.

    Article  Google Scholar 

  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

    Article  MATH  Google Scholar 

  • Larivière, V., & Gingras, Y. (2010). The impact factor’s Matthew effect: A natural experiment in bibliometrics. Journal of the American Society for Information Science and Technology, 61(2), 424–427.

    Google Scholar 

  • Lowry, P. B., Gaskin, J., Humpherys, S. L., Moody, G. D., Galletta, D. F., Barlow, J. B., et al. (2013). Evaluating journal quality and the association for information systems senior scholars' journal basket via bibliometric measures: Do expert journal assessments add value? MIS quarterly, 37(4), 993–1012.

    Article  Google Scholar 

  • Mabry, R. H., & Sharplin, A. D. (1985). The relative importance of journals used in financial research. Journal of Financial Research, 8, 287–296.

    Article  Google Scholar 

  • Mingers, J., & Harzing, A. W. (2007). Ranking journals in business and management: A statistical analysis of the Harzing data set. European Journal of Information Systems, 16(4), 303–316.

    Article  Google Scholar 

  • Mingers, J., & Yang, L. (2017). Evaluating journal quality: A review of journal citation indicators and ranking in business and management. European Journal of Operational Research, 257(1), 323–337.

    Article  MathSciNet  MATH  Google Scholar 

  • Moosa, I. (2011). The demise of the ARC journal ranking scheme: An ex post analysis of the accounting and finance journals. Accounting & Finance, 51(3), 809–836.

    Article  Google Scholar 

  • Moosa, I. A. (2016). A critique of the bucket classification of journals: The ABDC list as an example. Economic Record, 92(298), 448–463.

    Article  Google Scholar 

  • Moussa, S., & Touzani, M. (2010). Ranking marketing journals using the google scholar-based hg-index. Journal of Informetrics, 4(1), 107–117.

    Article  Google Scholar 

  • Peffers, K., & Ya, T. (2003). Identifying and evaluating the universe of outlets for information systems research: Ranking the journals. Journal of Information Technology Theory and Application, 5(1), 63–84.

    Google Scholar 

  • Peters, K., Daniels, K., Hodgkinson, G. P., & Haslam, S. A. (2014). Experts’ judgments of management journal quality: An identity concerns model. Journal of Management, 40(7), 1785–1812.

    Article  Google Scholar 

  • Petersen, C. G., Aase, G. R., & Heiser, D. R. (2011). Journal ranking analyses of operations management research. International Journal of Operations and Production Management., 31(4), 405–422.

    Article  Google Scholar 

  • Podsakoff, P. M., MacKenzie, S. B., Bachrach, D. G., & Podsakoff, N. P. (2005). The influence of management journals in the 1980s and 1990s. Strategic Management Journal, 26(5), 473–488.

    Article  Google Scholar 

  • Rahal, A., & Zainuba, M. (2019). The rating dilemma of academic management journals: Attuning the perceptions of peer rating. Advances in Business Research, 9(1), 26–45.

    Google Scholar 

  • Ren, L. Y. L. (2016). A note about the finance journal rankings and citation counts. Asian Academy of Management Journal of Accounting and Finance, 12(Suppl. 1), 183–194.

    Google Scholar 

  • Saladin, B. (1985). Operations management research: Where should we publish. Operations Management Review, 3(4), 3–9.

    Google Scholar 

  • Serenko, A., & Bontis, N. (2013). First in, best dressed: The presence of order-effect bias in journal ranking surveys. Journal of Informetrics, 7(1), 138–144.

    Article  Google Scholar 

  • Serenko, A., & Dohan, M. (2011). Comparing the expert survey and citation impact journal ranking methods: Example from the field of artificial intelligence. Journal of Informetrics, 5(4), 629–648.

    Article  Google Scholar 

  • Sheskin, D. J. (2003). Handbook of parametric and nonparametric statistical procedures. Chapman and Hall.

    Book  MATH  Google Scholar 

  • Soteriou, A. C., Hadjinicola, G. C., & Patsia, K. (1999). Assessing production and operations management related journals: the European perspective. Journal of Operations Management, 17(2), 225–238.

    Article  Google Scholar 

  • Steward, M. D., & Lewis, B. R. (2010). A comprehensive analysis of marketing journal rankings. Journal of Marketing Education, 32(1), 75–92.

    Article  Google Scholar 

  • Swidler, S., & Goldreyer, E. (1998). The value of a finance journal publication. Journal of Finance, 53, 351–363.

    Article  Google Scholar 

  • Tadajewski, M. (2016). Academic labour, journal ranking lists and the politics of knowledge production in marketing. Journal of Marketing Management, 32(1–2), 1–18.

    Article  Google Scholar 

  • Tahai, A., & Meyer, M. J. (1999). A revealed preference study of management journals’ direct influences. Strategic Management Journal, 20(3), 279–296.

    Article  Google Scholar 

  • Vanclay, J. K. (2011). An evaluation of the Australian Research Council’s journal ranking. Journal of Informetrics, 5(2), 265–274.

    Article  Google Scholar 

  • Vogel, R., Hattke, F., & Petersen, J. (2017). Journal rankings in management and business studies: What rules do we play by? Research Policy, 46(10), 1707–1722.

    Article  Google Scholar 

  • Wilhite, A. W., & Fong, E. A. (2012). Coercive citation in academic publishing. Science, 335(6068), 542–543.

    Article  Google Scholar 

  • Wilsdon, J., et al. (2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management. https://doi.org/10.13140/RG.2.1.4929.1363

  • Wouters, P. et al. (2015). The Metric Tide: Literature Review (Supplementary Report I to the Independent Review of the Role of Metrics in Research Assessment and Management). HEFCE. DOI:https://doi.org/10.13140/RG.2.1.5066.3520

  • Xu, N., Chan, K. C., & Chang, C. H. (2016). A quality-based global assessment of financial research. Review of Quantitative Finance and Accounting, 46(3), 605–631.

    Article  Google Scholar 

  • Xu, F., Liu, W., & Rousseau, R. (2015). Introducing sub-impact factor (SIF-) sequences and an aggregated SIF-indicator for journal ranking. Scientometrics, 102(2), 1577–1593.

    Article  Google Scholar 

  • Yuan, L., Li, J., Li, R., Lu, X., & Wu, D. (2020). Mapping the evaluation results between quantitative metrics and meta-synthesis from experts’ judgements: Evidence from the supply chain management and logistics journals ranking. Soft Computing, 24(9), 6227–6243.

    Article  Google Scholar 

  • Yuyuenyongwatana, R. P., & Carraher, S. M. (2008). Academic journal ranking: Important to strategic management and general management researchers? Journal of Business Strategies, 25(2), 1–9.

    Google Scholar 

  • Zainuba, M., & Rahal, A. (2015). Assessing the validity of business and management journals ranking list: An alternative approach for determining journal quality. Annals of Management Science, 4(2), 1–28.

    Article  Google Scholar 

Download references

Author information

Author notes

  1. Vicente Safón

    Present address: , Ivie, Spain

Authors and Affiliations

  1. Atlantic Research Center for Information and Communication Technologies, University of Vigo, Vigo, Spain

    Domingo Docampo

  2. Department of Management, University of Valencia, Valencia, Spain

    Vicente Safón

Authors
  1. Domingo Docampo
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Vicente Safón
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Vicente Safón.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Docampo, D., Safón, V. Journal ratings: a paper affiliation methodology. Scientometrics 126, 8063–8090 (2021). https://doi.org/10.1007/s11192-021-04045-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-021-04045-3

Keywords