Abstract
This paper highlights a number of problems which exist in the evaluation of existing image annotation and tag recommendation methods. Crucially, the collections used by these state-of-the-art methods contain a number of biases which may be exploited or detrimental to their evaluation, resulting in misleading results. In total we highlight seven issues for three popular annotation evaluation collections, i.e. Corel5k, ESP Game and IAPR, as well as three issues with collections used in two state-of-the-art photo tag recommendation methods. The result of this paper is two freely available Flickr image collections designed for the fair evaluation of image annotation and tag recommendation methods called Flickr-AIA and Flickr-PTR respectively. We show through experimentation and demonstration that these collection are ultimately fairer benchmarks than existing collections.
This research was supported by the the European Community’s FP7 Programme under grant agreements nr 288024 (LiMoSINe).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Athanasakos, K., Stathopoulos, V., Jose, J.M.: A Framework for Evaluating Automatic Image Annotation Algorithms. In: Gurrin, C., He, Y., Kazai, G., Kruschwitz, U., Little, S., Roelleke, T., Rüger, S., van Rijsbergen, K. (eds.) ECIR 2010. LNCS, vol. 5993, pp. 217–228. Springer, Heidelberg (2010)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: A Large-Scale Hierarchical Image Database. In: IEEE CVPR 2009 (2009)
Duygulu, P., Barnard, K., de Freitas, J.F.G., Forsyth, D.: Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part IV. LNCS, vol. 2353, pp. 97–112. Springer, Heidelberg (2002)
Eickhoff, C., Vries, A.P.: Increasing Cheat Robustness of Crowdsourcing Tasks. Inf. Retr. 2013 (2013)
Garg, N., Weber, I.: Personalized, Tnteractive Tag Recommendation for Flickr. In: ACM RecSys 2008 (2008)
Grubinger, M., Clough, P., Mller, H., Deselaers, T.: The IAPR TC-12 Benchmark - A New Evaluation Resource for Visual Iinformation Systems (2006)
Hirth, M., Hoßfeld, T., Tran-Gia, P.: Cheat-detection Mechanisms for Crowdsourcing. Technical Report, University of Würzburg, 8 (2010)
Huiskes, M.J., Lew, M.S.: The MIR Flickr Retrieval Evaluation. In: MIR 2008 (2008)
Vuurens, A.P.D.V.J., Eickhoff, C.: How much Spam can you take? An Analysis of Crowdsourcing Results to Increase Accuracy. In: ACM SIGIR 2011 (2011)
Jeon, J., Lavrenko, V., Manmatha, R.: Automatic Image Annotation and Retrieval using Cross-media Relevance Models. In: ACM SIGIR 2003 (2003)
Liu, D., Hua, X.-S., Yang, L., Wang, M., Zhang, H.-J.: Tag ranking. In: WWW 2009 (2009)
Makadia, A., Pavlovic, V., Kumar, S.: Baselines for image annotation. In: IJCV 2010 (2010)
Mark, B.T., Huiskes, J., Lew, M.S.: New trends and Ideas in Visual Concept Detection: The MIR Flickr Retrieval Evaluation Initiative. In: MIR 2010 (2010)
McParlane, P.J., Moshfeghi, Y., Jose, J.M.: On Contextual Photo Tag Recommendation. In: SIGIR 2013 (2013)
Miller, G.A.: Wordnet: A lexical database for english. Communications of the ACM (1995)
Müller, H., Marchand-Maillet, S., Pun, T.: The Truth about Corel - Evaluation in Image Retrieval. In: Lew, M., Sebe, N., Eakins, J.P. (eds.) CIVR 2002. LNCS, vol. 2383, pp. 38–49. Springer, Heidelberg (2002)
Nowak, S., Rüger, S.: How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation. In: MIR 2010 (2010)
Sigurbjörnsson, B., van Zwol, R.: Flickr Tag Recommendation based on Collective Knowledge. In: WWW 2008 (2008)
von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: CHI 2004 (2004)
Westerveld, T., de Vries, A.P.: Experimental Evaluation of a generative probabilistic image retrieval model on ’easy’ data. In: ACM SIGIR 2003 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
McParlane, P.J., Moshfeghi, Y., Jose, J.M. (2014). Collections for Automatic Image Annotation and Photo Tag Recommendation. In: Gurrin, C., Hopfgartner, F., Hurst, W., Johansen, H., Lee, H., O’Connor, N. (eds) MultiMedia Modeling. MMM 2014. Lecture Notes in Computer Science, vol 8325. Springer, Cham. https://doi.org/10.1007/978-3-319-04114-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-04114-8_12
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-04113-1
Online ISBN: 978-3-319-04114-8
eBook Packages: Computer ScienceComputer Science (R0)