Abstract
This paper introduces a framework for evaluating the performance of information retrieval systems. Current evaluation metrics provide an average score that does not consider performance variability across the query set. In this manner, conclusions lack of any statistical significance, yielding poor inference to cases outside the query set and possibly unfair comparisons. We propose to apply statistical methods in order to obtain a more informative measure for problems in which different query classes can be identified. In this context, we assess the performance variability on two levels: overall variability across the whole query set and specific query class-related variability. To this end, we estimate confidence bands for precision-recall curves, and we apply ANOVA in order to assess the significance of the performance across different query classes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Everingham, M., Ali Eslami, S.M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: Assessing the significance of performance differences on the pascal voc challenges via bootstrapping. Tech. rep. (2013)
Bertail, P., Clemencon, S., Vayatis, N.: On bootstrapping the roc curve. In: NIPS, pp. 137–144. Curran Associates Inc. (2008)
Clémençon, S., Vayatis, N.: Nonparametric estimation of the precision-recall curve. In: ICML, pp. 185–192 (2009)
Macskassy, S.A., Provost, F.J.: Confidence bands for roc curves: Methods and an empirical study. In: ROCAI, pp. 61–70 (2004)
Brughi, F., Gil, D., Ramos Terrades, O.: Artistic heritage motive retrieval: an explorative study. Tech. rep. (2013)
Crowley, E.J., Zisserman, A.: Of gods and goats: Weakly supervised learning of figurative art. In: BMVC (2013)
Bamber, D.: The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J. Math. Psy. 12, 387–415 (1975)
DeLong, E.R., DeLong, D.M., Clarke-Pearson, D.L.: Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics 44, 837–845 (1988)
Wieand, S., Gail, M.H., James, B.R., James, K.L.: A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika 76(3), 585–592 (1989)
Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press (2008)
Davis, J., Goadrich, M.: The relationship between precision-recall and roc curves. In: ICML, pp. 233–240 (2006)
Casella, G., Berger, R.: Statistical inference. Duxbury Press (1990)
Hochberg, Y., Tamhane, A.C.: Multiple Comparison Procedures. John Wiley & Sons Inc. (1987)
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004)
Sivic, J., Zisserman, A.: Video google: A text retrieval approach to object matching in videos. In: ICCV, pp. 1470–1477 (2003)
Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: CVPR (2007)
Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Lost in quantization: Improving particular object retrieval in large scale image databases. In: CVPR, pp. 1–8 (2008)
Lebeda, K., Matas, J., Chum, O.: Fixing the locally optimized ransac. In: BMVC, pp. 1–11 (2012)
Badiella, L., Puig, P., Leton, E.: Evaluacion diagnostica mediante curvas roc. Tech. rep. (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Brughi, F., Gil, D., Badiella, L., Casabella, E.J., Terrades, O.R. (2014). Exploring the Impact of Inter-query Variability on the Performance of Retrieval Systems. In: Campilho, A., Kamel, M. (eds) Image Analysis and Recognition. ICIAR 2014. Lecture Notes in Computer Science(), vol 8814. Springer, Cham. https://doi.org/10.1007/978-3-319-11758-4_45
Download citation
DOI: https://doi.org/10.1007/978-3-319-11758-4_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-11757-7
Online ISBN: 978-3-319-11758-4
eBook Packages: Computer ScienceComputer Science (R0)