Abstract
Annotation quality measurement is crucial when building a supervised dataset for either general purpose research or domain applications. Inter-rater agreement measure is one of the most vital aspects in terms of establishing annotation quality. The traditional inter-rater agreement measures cannot address the issue in multi-label scenario. To adapt to multi-label annotations, the recent research has developed a bootstrapping method to measure the level of agreement between two raters. In this paper we propose a fine-grained multi-label agreement measure MLA, which attends to discover slight differences in inter-rater agreement across different annotations when multiple raters are involved. We demonstrate its compatibility with traditional measures through mathematics and experiments. The experimental results show it can interpret the agreement more accurately and consistently with intuitive understanding. In addition, a toolset is provided to enable users to generate the multi-label annotations that mimic different annotators, and calculate various agreement coefficients for several scenarios.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
PaddlePaddle AI studio. https://aistudio.baidu.com/aistudio/datasetdetail/181754
Liu, W.W., Wang, H.B., Shen, X.B., Tsang, I.W.: The emerging trends of multi-label learning, arXiv preprint arXiv: 2011.11197 (2021)
Xu, D., Shi, Y., Tsang, I.W., Ong, Y.S., Gong, C., Shen, X.B.: A survey on multi-output learning, arXiv preprint arXiv: 1901.00248 (2019)
Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Measur. 20(1), 37–46 (1960)
Scott, W.: Reliability of content analysis: the case of nominal scale coding. Public Opin. Q. 19(3), 321–325 (1955)
Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychol. Bull. 76(5), 378–382 (1971)
Cohen, J.: Weighed kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol. Bull. 70(4), 213–220 (1968)
Krippendorff, K.: Computing krippendorff’s alpha-reliability. https://repository.upenn.edu/asc_papers/43. Accessed 25 Jan 2011
Marchal, M., Scholman, M., Yung, F., Demberg, V.: Establishing annotation quality in multi-label annotations. In: Proceedings of the 29th International Conference on Computational Linguistics, pp. 3659–3668, International Committee on Computational Linguistics, Gyeongju, Republic of Korea (2022)
Ji, A.Y., et al.: Abstract visual reasoning with tangram shapes. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 582–601, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (2022)
Beck, C., Booth, H., El-Assady, M., Butt, M.: Representation problems in linguistic annotations: ambiguity, variation, uncertainty, error and bias. In: 14th Linguistic Annotation Workshop, pp. 60–73, Association for Computational Linguistics, Barcelona, Spain (2020)
Zapf, A., Castell, S., Morawietz, L., Karch, A.: Measuring inter-rater reliability for nominal data – which coefficients and confidence intervals are appropriate? BMC Med. Res. Methodol. 16, 93 (2016)
Zettersten, M., Lupyan, G.: Finding categories through words: more nameable features improve category learning. Cognition 196, 104135 (2020)
Passonneau, R.: Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. Communicative Events, Columbia University New York, New York, USA (2006)
Mohammadreza, H., Doyle, T.E., Samavi, R.: MLCM: multi-label confusion matrix. IEEE Access 10, 19083–19095 (2022)
Kim, Y., Kim, J.M., Akata, Z., Lee, J.: Large loss matters in weakly supervised multi-label classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14136–14146 (2022)
Acknowledgements
Thank all the anonymous reviewers and chairs for their meaningful suggestions.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, S. et al. (2023). Annotation Quality Measurement in Multi-Label Annotations. In: Liu, F., Duan, N., Xu, Q., Hong, Y. (eds) Natural Language Processing and Chinese Computing. NLPCC 2023. Lecture Notes in Computer Science(), vol 14303. Springer, Cham. https://doi.org/10.1007/978-3-031-44696-2_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-44696-2_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44695-5
Online ISBN: 978-3-031-44696-2
eBook Packages: Computer ScienceComputer Science (R0)
