Abstract
Validation of information retrieval(IR) systems represents an inherently difficult task. We present a study that uses the Kappa measure for inter-judge agreement for establishing a reference quality benchmark for responses provided by a custom developed IR system in a comparative analysis with already existing search mechanism. Experiments show that it is difficult to assess the relevance of responses as human judges do not always easily agree on what is relevant and what is not. The results prove that when judges agree the responses from our system are mostly better than those returned by existing mechanism. This bench-marking mechanism opens the way for further detailed investigation of responses that were not relevant and possible improvement of the IR system design.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
State-of-the-art multilingual lemmatization. https://towardsdatascience.com/state-of-the-art-multilingual-lemmatization-f303e8ff1a8, Accessed 29 June 2021
Blackman, N.J.M., Koval, J.J.: Interval estimation for Cohen’s kappa as a measure of agreement. Stat. Med. 19(5), 723–741 (2000)
Aker, A., Petrak, J., Sabbah, F.: An extensible multilingual open source lemmatizer. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pp. 40–45. ACL (2017)
Bafna, P., Pramod, D., Vaidya, A.: Document clustering: TF-IDF approach. In: 2016 International Conference on Electrical, Electronics, and Optimization Techniques (ICEEOT), pp. 61–66. IEEE (2016)
Basu, S., Yu, Y., Singh, V.K., Zimmermann, R.: Videopedia: lecture video recommendation for educational blogs using topic modeling. In: Tian, Q., Sebe, N., Qi, G.-J., Huet, B., Hong, R., Liu, X. (eds.) MMM 2016. LNCS, vol. 9516, pp. 238–250. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-27671-7_20
Bleoancă, D.I., Heras, S., Palanca, J., Julian, V., Mihăescu, M.C.: LSI based mechanism for educational videos retrieval by transcripts processing. In: Analide, C., Novais, P., Camacho, D., Yin, H. (eds.) IDEAL 2020. LNCS, vol. 12489, pp. 88–100. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-62362-3_9
Deerwester, S., Dumais, S.T., Landauer, T.K., Furnas, G., Beck, F.D.L., Leighton-Beck, L.: Improvinginformation-retrieval with latent semantic indexing (1988)
Galanopoulos, D., Mezaris, V.: Temporal lecture video fragmentation using word embeddings. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, W.-H., Vrochidis, S. (eds.) MMM 2019. LNCS, vol. 11296, pp. 254–265. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05716-9_21
Gutiérrez, L., Keith, B.: A systematic literature review on word embeddings. In: Mejia, J., Muñoz, M., Rocha, Á., Peña, A., Pérez-Cisneros, M. (eds.) CIMPS 2018. AISC, vol. 865, pp. 132–141. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-01171-0_12
Kastrati, Z., Kurti, A., Imran, A.S.: Wet: word embedding-topic distribution vectors for mooc video lectures dataset. Data Brief 28, 105090 (2020)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)
Ramos, J., et al.: Using TF-IDF to determine word relevance in document queries. In: Proceedings of the First Instructional Conference on Machine Learning, vol. 242, pp. 29–48. Citeseer (2003)
Umesh, U.N., Peterson, R.A., Sauber, M.H.: Interjudge agreement and the maximum value of kappa. Educ. Psychol. Meas. 49(4), 835–850 (1989)
Zhu, H., Dong, L., Wei, F., Qin, B., Liu, T.: Transforming wikipedia into augmented data for query-focused summarization (2019). arXiv:1911.03324
Acknowledgements
This work was partially supported by the grant 135C/ 2021 “Development of software applications that integrate machine learning algorithms", financed by the University of Craiova.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Bleoancă, D., Heras, S., Palanca, J., Julian, V., Mihăescu, M.C. (2021). Validation of Video Retrieval by Kappa Measure for Inter-Judge Agreement. In: Yin, H., et al. Intelligent Data Engineering and Automated Learning – IDEAL 2021. IDEAL 2021. Lecture Notes in Computer Science(), vol 13113. Springer, Cham. https://doi.org/10.1007/978-3-030-91608-4_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-91608-4_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-91607-7
Online ISBN: 978-3-030-91608-4
eBook Packages: Computer ScienceComputer Science (R0)