Skip to main content
SpringerLink
Log in

Analysis of journal evaluation indicators: an experimental study based on unsupervised Laplacian score

  • Published:
Scientometrics Aims and scope Submit manuscript

A Correction to this article was published on 03 June 2020

This article has been updated

Abstract

Academic journal rankings have always been a hot topic because the journals ranking will be significantly affected by using different ranking systems or bibliometrics indicators. Many studies have shown that feature selection methods are beneficial to categorize and rank journals according to the different evaluation indicators. However, such methods are limited to analyzing the journals published in a single discipline. In this article, we conduct an experiment based on unsupervised Laplacian score to analyze journal evaluation indicators between three different disciplines (Biology, Artificial Intelligence, and Mathematics). The journal’s features considered in the study include: JIF, 5-Year JIF, CiteScore, SJR, SNIP, and H-index. Based on the results of the feature selection, we first use the spectral clustering technique to verify the impact of different feature combinations on journals categorization. Furthermore, k-Nearest Neighbor (k-NN) and Back Propagation Neural Network classifiers are trained to measure the classification performance by using each single feature or feature subsets. The experimental results demonstrate that the two classifiers achieve the superior classification performance when the input of the classifier is the set of JIF, CiteScore, and H-index. Finally, we design a subjective questionnaire to collect the opinion of the volunteers with regard to the six evaluation indiactors in terms of familiarity, complexity, and usability. According to the vote results of a seven-point Likert scale, JIF and H-index receive the relatively higher ratings in comparison to other bibliometric indicators.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Change history

  • 03 June 2020

    The original publication of the article contained an error in Table 6, in header column. The correct table has been given below.

Notes

  1. https://journalmetrics.scopus.com/.

  2. https://clarivate.com/.

  3. www.las.ac.cn/.

  4. https://publons.com/.

  5. www.medsciediting.com/.

  6. www.letpub.com.cn/.

  7. https://ieeexplore.ieee.org/.

  8. https://archive.ics.uci.edu/ml/datasets/Iris/.

  9. www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html.

References

  • Ahlgren, P., & Waltman, L. (2014). The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. norwegian quality assessments. Journal of Informetrics, 8(4), 985–996.

    Google Scholar 

  • Belkin, M., & Niyogi, P. (2002). Laplacian eigenmaps and spectral techniques for embedding and clustering. In Advances in neural information processing systems (pp. 585–591).

  • Bergstrom, C. (2007). Eigenfactor: Measuring the value and prestige of scholarly journals. College & Research Libraries News, 68(5), 314–316.

    Google Scholar 

  • Bezdek, J. C., & Pal, N. R. (1998). Some new indexes of cluster validity. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 28(3), 301–315.

    Google Scholar 

  • Bordons, M., Fernández, M., & Gómez, I. (2002). Advantages and limitations in the use of impact factor measures for the assessment of research performance. Scientometrics, 53(2), 195–206.

    Google Scholar 

  • Bornmann, L., & Marx, W. (2016). The journal impact factor and alternative metrics. EMBO Reports, 17(8), 1094–1097.

    Google Scholar 

  • Bornmann, L., Schier, H., Marx, W., & Daniel, H. D. (2012). What factors determine citation counts of publications in chemistry besides their quality? Journal of Informetrics, 6(1), 11–18.

    Google Scholar 

  • Braun, T., Glänzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169–173.

    Google Scholar 

  • Cai, L., Tian, J., Liu, J., Bai, X., Lee, I., Kong, X., et al. (2019). Scholarly impact assessment: A survey of citation weighting solutions. Scientometrics, 118(2), 453–478.

    Google Scholar 

  • Campanario, J. M. (2011). Empirical study of journal impact factors obtained using the classical two-year citation window versus a five-year citation window. Scientometrics, 87(1), 189–204.

    Google Scholar 

  • Chen, X., Yuan, G., Wang, W., Nie, F., Chang, X., & Huang, J. Z. (2018). Local adaptive projection framework for feature selection of labeled and unlabeled data. IEEE Transactions on Neural Networks and Learning Systems, 29(12), 6362–6373.

    MathSciNet  Google Scholar 

  • Da Silva, J. A. T., & Memon, A. R. (2017). Citescore: A cite for sore eyes, or a valuable, transparent metric? Scientometrics, 111(1), 553–556.

    Google Scholar 

  • Fernandez-Llimos, F. (2018). Differences and similarities between journal impact factor and citescore. Pharmacy Practice, 16(2), 1282.

    Google Scholar 

  • Forman, G. (2003). An extensive empirical study of feature selection metrics for text classification. Journal of Machine Learning Research, 3(Mar), 1289–1305.

    MATH  Google Scholar 

  • Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.

    Google Scholar 

  • Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA, 295(1), 90–93.

    Google Scholar 

  • González-Pereira, B., Guerrero-Bote, V. P., & Moya-Anegón, F. (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of Informetrics, 4(3), 379–391.

    Google Scholar 

  • Goodman, S. N. (2018). A quality-control test for predatory journals. Nature, 553(7687), 155.

    Google Scholar 

  • Habibzadeh, F., & Yadollahie, M. (2008). Journal weighted impact factor: A proposal. Journal of Informetrics, 2(2), 164–172.

    Google Scholar 

  • Halim, Z., & Khan, S. (2019). A data science-based framework to categorize academic journals. Scientometrics, 119(1), 393–423.

    Google Scholar 

  • Halim, Z., & Muhammad, T. (2017). Quantifying and optimizing visualization: An evolutionary computing-based approach. Information Sciences, 385, 284–313.

    Google Scholar 

  • He, X., Cai, D., & Niyogi, P. (2006). Laplacian score for feature selection. In Advances in neural information processing systems (pp. 507–514).

  • He, X., & Niyogi, P. (2004). Locality preserving projections. In Advances in neural information processing systems (pp. 153–160).

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572.

    MATH  Google Scholar 

  • Hoffman, M., Steinley, D., & Brusco, M. J. (2015). A note on using the adjusted rand index for link prediction in networks. Social Networks, 42, 72–79.

    Google Scholar 

  • Huang, S. H. (2015). Supervised feature selection: A tutorial. Artificial Intelliegence Research, 4(2), 22–37.

    Google Scholar 

  • Jenssen, R., Hild, K., Erdogmus, D., Principe, J. C., & Eltoft, T. (2003). Clustering using Renyi’s entropy. In Proceedings of the international joint conference on neural networks, 2003, vol. 1 (pp. 523–528).

  • Jiang, X., Sun, X., Yang, Z., Zhuge, H., & Yao, J. (2016). Exploiting heterogeneous scientific literature networks to combat ranking bias: Evidence from the computational linguistics area. Journal of the Association for Information Science and Technology, 67(7), 1679–1702.

    Google Scholar 

  • Jin, B., Liang, L., Rousseau, R., & Egghe, L. (2007). The R-and AR-indices: Complementing the h-index. Chinese Science Bulletin, 52(6), 855–863.

    Google Scholar 

  • Krishnapuram, B., Harternink, A., Carin, L., & Figueiredo, M. A. (2004). A Bayesian approach to joint feature selection and classifier design. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(9), 1105–1111.

    Google Scholar 

  • Kuncheva, L. I., & Whitaker, C. J. (2003). Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning, 51(2), 181–207.

    MATH  Google Scholar 

  • Leydesdorff, L. (2009). How are new citation-based journal indicators adding to the bibliometric toolbox? Journal of the American Society for Information Science and Technology, 60(7), 1327–1336.

    Google Scholar 

  • Li, A., Cai, Z., & Cao, W. (2017). Correlation analysis between lunar surface roughness and other land–surface parameters using bpnn. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10(12), 5647–5656.

    Google Scholar 

  • Li, Z., Liu, J., Yang, Y., Zhou, X., & Lu, H. (2013). Clustering-guided sparse structural learning for unsupervised feature selection. IEEE Transactions on Knowledge and Data Engineering, 26(9), 2138–2150.

    Google Scholar 

  • Liu, H., & Yu, L. (2005). Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on Knowledge and Data Engineering, 17(4), 491–502.

    Google Scholar 

  • Liu, S., Feng, L., & Qiao, H. (2014). Scatter balance: An angle-based supervised dimensionality reduction. IEEE Transactions on Neural Networks and Learning Systems, 26(2), 277–289.

    MathSciNet  Google Scholar 

  • Liu, Y., & Rousseau, R. (2013). Interestingness and the essence of citation. Journal of Documentation, 69(4), 580–589.

    Google Scholar 

  • Mathur, V. P., Sharma, A., et al. (2009). Impact factor and other standardized measures of journal citation: A perspective. Indian Journal of Dental Research, 20(1), 81.

    Google Scholar 

  • Moed, H. F. (2016). Comprehensive indicator comparisons intelligible to non-experts: The case of two SNIP versions. Scientometrics, 106(1), 51–65.

    Google Scholar 

  • Moustafa, K. (2015). The disaster of the impact factor. Science and Engineering Ethics, 21(1), 139–142.

    Google Scholar 

  • Ng, A. Y., Jordan, M. I., & Weiss, Y. (2002). On spectral clustering: Analysis and an algorithm. In Advances in neural information processing systems (pp. 849–856).

  • Pajić, D. (2015). On the stability of citation-based journal rankings. Journal of Informetrics, 9(4), 990–1006.

    Google Scholar 

  • Rousseau, R., et al. (2009). What does the web of science five-year synchronous impact factor have to offer? Chinese Journal of Library and Information Science, 2(3), 1–7.

    Google Scholar 

  • Saad, G. (2006). Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics, 69(1), 117–120.

    Google Scholar 

  • Santos, J. M., & Embrechts, M. (2009). On the use of the adjusted rand index as a metric for evaluating supervised classification. In International conference on artificial neural networks (pp. 175–184). Springer.

  • Serenko, A., & Bontis, N. (2009). Global ranking of knowledge management and intellectual capital academic journals. Journal of Knowledge Management, 13(1), 4–15.

    Google Scholar 

  • Tan, Z. Y., Cai, N., Zhou, J., & Zhang, S. G. (2019). On performance of peer review for academic journals: Analysis based on distributed parallel system. IEEE Access, 7, 19024–19032.

    Google Scholar 

  • Thelwall, M. (2019). Are classic references cited first? An analysis of citation order within article sections. Scientometrics, 120(2), 723–731.

    Google Scholar 

  • Valderrama, P., Escabias, M., Jiménez-Contreras, E., Rodríguez-Archilla, A., & Valderrama, M. J. (2018). Proposal of a stochastic model to determine the bibliometric variables influencing the quality of a journal: Application to the field of dentistry. Scientometrics, 115(2), 1087–1095.

    Google Scholar 

  • Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365–391.

    Google Scholar 

  • Wang, D., Nie, F., & Huang, H. (2015). Feature selection via global redundancy minimization. IEEE Transactions on Knowledge and Data Engineering, 27(10), 2743–2755.

    Google Scholar 

  • Wang, M., Wang, Z., & Chen, G. (2019). Which can better predict the future success of articles? Bibliometric indices or alternative metrics. Scientometrics, 119(3), 1575–1595.

    Google Scholar 

  • Wolf, L., & Shashua, A. (2005). Feature selection for unsupervised and supervised inference: The emergence of sparsity in a weight-based approach. Journal of Machine Learning Research, 6(Nov), 1855–1887.

    MathSciNet  MATH  Google Scholar 

  • Yang, D. H., Li, X., Sun, X., & Wan, J. (2016). Detecting impact factor manipulation with data mining techniques. Scientometrics, 109(3), 1989–2005.

    Google Scholar 

  • Yu, D., Wang, W., Zhang, S., Zhang, W., & Liu, R. (2017). A multiple-link, mutually reinforced journal-ranking model to measure the prestige of journals. Scientometrics, 111(1), 521–542.

    Google Scholar 

  • Zhang, J., Luo, Z., Li, C., Zhou, C., & Li, S. (2019). Manifold regularized discriminative feature selection for multi-label learning. Pattern Recognition, 95, 136–150.

    Google Scholar 

  • Zhou, J., Cai, N., Tan, Z. Y., & Khan, M. J. (2019). Analysis of effects to journal impact factors based on citation networks generated via social computing. IEEE Access, 7, 19775–19781.

    Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation (NNSF) of China (Grants 61672130 and 61602082), by Fundamental Research Funds for the Central Universities (Grants 2019RC29 and 31920180115), and by LiaoNing Revitalization Talents Program (Grant XLYC1806006).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Dalian University of Technology, Dalian, 116024, China

    Lin Feng, Jian Zhou & Sheng-Lan Liu

  2. School of Automation, Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Ning Cai

  3. Research Institute of Information Technology, Tsinghua University, Beijing, 100084, China

    Jie Yang

  4. Key Laboratory of China’s Ethnic Languages and Information Technology for Ministry of Education, Northwest Minzu University, Lanzhou, 730030, China

    Ning Cai

Authors
  1. Lin Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jian Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sheng-Lan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ning Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jie Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jian Zhou or Ning Cai.

Ethics declarations

Conflict of interest

The authors declare that they have no competing interests regarding the publication of this paper.

Additional information

The original version of this article was revised due to an error in Table 6.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 347 KB)

Supplementary material 2 (pdf 124 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feng, L., Zhou, J., Liu, SL. et al. Analysis of journal evaluation indicators: an experimental study based on unsupervised Laplacian score. Scientometrics 124, 233–254 (2020). https://doi.org/10.1007/s11192-020-03422-8

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-020-03422-8

Keywords

Search

Navigation