Abstract
With the intention to evaluate a current h(fa)-index which considered the relative contribution of each author, Sebo P and colleagues selected senior hospital physicians from six college hospitals in Switzerland between 1 and 14 March 2020. Using Web of Science, the authors assessed the h(fa)-index of all physicians with acquired non-zero h-index. They applied Cohen’s kappa statistic to calculate the inter-rater agreement between the h-index and the h(fa)-index. We argue that the agreement between the results of the two metrics is disputed by Cohen’s kappa statistic. Therefore, we recommend that an additional appropriate statistical approach, the weighted Cohen’s kappa statistic, should be used in this study.
This is a preview of subscription content, log in via an institution to check access.
References
Hernández-Hernández, V., Ferraz-Amaro, I., & Díaz-González, F. (2014). Influence of disease activity on the physical activity of rheumatoid arthritis patients. Rheumatology (oxford), 53, 722–731. https://doi.org/10.1093/rheumatology/ket422
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. https://doi.org/10.2307/2529310
McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica (zagreb), 22, 276–282. https://doi.org/10.11613/BM.2012.031
Sebo, P., & de Lucia, S. (2021). Evaluation of the productivity of hospital-based researchers: Comparative study between the h-index and the h(fa)-index. Scientometrics, 126, 7087–7096. https://doi.org/10.1007/s11192-021-04040-8
Sebring, D., Kvist, T., Buhlin, K., Jonasson, P. E., & Lund, H. (2021). Calibration improves observer reliability in detecting periapical pathology on panoramic radiographs. Acta Odontologica Scandinavica, 79, 554–561. https://doi.org/10.1080/00016357.2021.1910728
Wang, J., Chen, S., Liang, H., Zhao, Y., Xu, Z., Xiao, W., et al. (2022). Fully automatic classification of brain atrophy on NCCT images in cerebral small vessel disease: A pilot study using deep learning models. Frontiers in Neurology, 13, 846348. https://doi.org/10.3389/fneur.2022.846348
Acknowledgements
This work was supported by the Heilongjiang Province Higher Education Teaching Reform Project (Grant No. SJGY20200799) and Qiqihar University Degree and Postgraduate Education and Teaching Reform Research Project (Grant No. JGXM_QUG_Z2019003).
Author information
Authors and Affiliations
Contributions
ML drafted the manuscript. QG analyzed and interpreted the data. TY developed the study concept and design. The study was supervised by ML and TY. All authors read and approved the final manuscript.
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Rights and permissions
About this article
Cite this article
Li, M., Gao, Q. & Yu, T. Methodological issues on statistical rigor of agreement analysis. Scientometrics 128, 2025–2027 (2023). https://doi.org/10.1007/s11192-022-04591-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-022-04591-4