Skip to main content
SpringerLink
Log in
Book cover

Workshop of the Cross-Language Evaluation Forum for European Languages

CLEF 2009: Multilingual Information Access Evaluation II. Multimedia Experiments pp 94–109Cite as

Overview of the CLEF 2009 Large-Scale Visual Concept Detection and Annotation Task

  • Conference paper

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 6242))

Abstract

The Large-Scale Visual Concept Detection and Annotation Task (LS-VCDT) in ImageCLEF 2009 aims at the detection of 53 concepts in consumer photos. These concepts are structured in an ontology which can be utilized during training and classification of the photos. The dataset consists of 18,000 Flickr photos which were manually annotated with 53 concepts. 5,000 photos were used for training and 13,000 for testing. Two evaluation paradigms have been applied, the evaluation per concept and the evaluation per photo. The evaluation per concept was performed by calculating the Equal Error Rate (EER) and the Area Under Curve (AUC). For the evaluation per photo a recently proposed ontology-based measure was utilized that takes the hierarchy and the relations of the ontology into account and calculates a score per photo. Altogether 19 research groups participated and submitted 73 runs. For the concepts, an average AUC of 84% could be achieved, including concepts with an AUC of 95%. The classification performance for each photo ranged between 68.7% and 100% with an average score of 89.6%.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Huiskes, M.J., Lew, M.S.: The MIR Flickr Retrieval Evaluation. In: MIR 2008: Proceedings of the 2008 ACM International Conference on Multimedia Information Retrieval. ACM, New York (2008)

    Google Scholar 

  2. Nowak, S., Dunker, P.: A Consumer Photo Tagging Ontology: Concepts and Annotations. In: THESEUS/ImageCLEF Pre-Workshop 2009, Co-located with the Cross Language Evaluation Forum (CLEF) Workshop and 13th European Conference on Digital Libraries ECDL, Corfu, Greece (2009)

    Google Scholar 

  3. Nowak, S., Lukashevich, H.: Multilabel Classification Evaluation using Ontology Information. In: The 1st Workshop on Inductive Reasoning and Machine Learning on the Semantic Web -IRMLeS 2009, co-located with the 6th Annual European Semantic Web Conference (ESWC), Heraklion, Greece (2009)

    Google Scholar 

  4. Nowak, S., Lukashevich, H., Dunker, P., Rüger, S.: Performance Measures for Multilabel Classification - A Case Study in the Area of Image Classification. In: ACM SIGMM International Conference on Multimedia Information Retrieval (ACM MIR), Philadelphia, Pennsylvania (2010)

    Google Scholar 

  5. Glotin, H., Fakeri-Tabrizi, A., Mulhem, P., Ferecatu, M., Zhao, Z., Tollari, S., Quenot, G., Sahbi, H., Dumont, E., Gallinari, P.: Comparison of Various AVEIR Visual Concept Detectors with an Index of Carefulness. In: CLEF Working Notes 2009, Corfu, Greece (2009)

    Google Scholar 

  6. Ngiam, J., Goh, H.: Learning Global and Regional Features for Photo Annotation. In: Peters, C., Tsikrika, T., Müller, H., Kalpathy-Cramer, J., Jones, J., Gonzalo, J., Caputo, B. (eds.) CLEF 2009 Workshop, Part II. LNCS, vol. 6242, pp. 287–290. Springer, Heidelberg (2010)

    Google Scholar 

  7. Binder, A., Kawanabe, M.: Enhancing Recognition of Visual Concepts with Primitive Color Histograms via Non-sparse Multiple Kernel Learning. In: Peters, C., Tsikrika, T., Müller, H., Kalpathy-Cramer, J., Jones, J., Gonzalo, J., Caputo, B. (eds.) CLEF 2009 Workshop, Part II. LNCS, vol. 6242, pp. 269–276. Springer, Heidelberg (2010)

    Google Scholar 

  8. Hare, J., Lewis, P.: IAM@ImageCLEFPhotoAnnotation 2009: Naive application of a linear-algebraic semantic space. In: CLEF Working Notes 2009, Corfu, Greece (2009)

    Google Scholar 

  9. Douze, M., Guillaumin, M., Mensink, T., Schmid, C., Verbeek, J.: INRIA-LEARs participation to ImageCLEF 2009. In: CLEF Working Notes 2009, Corfu, Greece (2009)

    Google Scholar 

  10. van de Sande, K., Gevers, T., Smeulders, A.: The University of Amsterdam’s Concept Detection System at ImageCLEF 2009. In: Peters, C., Tsikrika, T., Müller, H., Kalpathy-Cramer, J., Jones, J., Gonzalo, J., Caputo, B. (eds.) CLEF 2009 Workshop, Part II. LNCS, vol. 6242, pp. 261–268. Springer, Heidelberg (2010)

    Google Scholar 

  11. Sarin, S., Kameyama, W.: Joint Contribution of Global and Local Features for Image Annotation. In: CLEF Working Notes 2009, Corfu, Greece (2009)

    Google Scholar 

  12. Dumont, E., Zhao, Z.Q., Glotin, H., Paris, S.: A new TFIDF Bag of Visual Words for Concept Detection. In: Peters, C., Tsikrika, T., Müller, H., Kalpathy-Cramer, J., Jones, J., Gonzalo, J., Caputo, B. (eds.) CLEF 2009 Workshop, Part II. LNCS, vol. 6242. Springer, Heidelberg (2010)

    Google Scholar 

  13. Llorente, A., Motta, E., Rüger, S.: Exploring the Semantics Behind a Collection to Improve Automated Image Annotation. In: Peters, C., Tsikrika, T., Müller, H., Kalpathy-Cramer, J., Jones, J., Gonzalo, J., Caputo, B. (eds.) CLEF 2009 Workshop, Part II. LNCS, vol. 6242, pp. 307–314. Springer, Heidelberg (2010)

    Google Scholar 

  14. Pham, T., Maisonnasse, L., Mulhem, P., Chevallet, J.P., Quénot, G., Al Batal, R.: MRIM-LIG at ImageCLEF 2009: Robot Vision, Image annotation and retrieval tasks. In: Peters, C., Tsikrika, T., Müller, H., Kalpathy-Cramer, J., Jones, J., Gonzalo, J., Caputo, B. (eds.) CLEF 2009 Workshop, Part II. LNCS, vol. 6242, pp. 324–331. Springer, Heidelberg (2010)

    Google Scholar 

  15. Daróczy, B., Petrás, I., Benczúr, A., Fekete, Z., Nemeskey, D., Siklósi, D., Weiner, Z.: Interest Point and Segmentation-Based Photo Annotation. In: Peters, C., Tsikrika, T., Müller, H., Kalpathy-Cramer, J., Jones, J., Gonzalo, J., Caputo, B. (eds.) CLEF 2009 Workshop, Part II. LNCS, vol. 6242, pp. 340–347. Springer, Heidelberg (2010)

    Google Scholar 

  16. Ferecatu, M., Sahbi, H.: TELECOM ParisTech at ImageClef 2009: Large Scale Visual Concept Detection and Annotation Task. In: CLEF working notes 2009, Corfu, Greece (2009)

    Google Scholar 

  17. Escalante, H., Gonzalez, J., Hernandez, C., Lopez, A., Montex, M., Morales, E., Ruiz, E., Sucar, L., Villasenor, L.: TIA-INAOE’s Participation at ImageCLEF 2009. In: CLEF working notes 2009, Corfu, Greece (2009)

    Google Scholar 

  18. Iftene, A., Vamanu, L., Croitoru, C.: UAIC at ImageCLEF 2009 Photo Annotation Task. In: Peters, C., Tsikrika, T., Müller, H., Kalpathy-Cramer, J., Jones, J., Gonzalo, J., Caputo, B. (eds.) CLEF 2009 Workshop, Part II. LNCS, vol. 6242, pp. 283–286. Springer, Heidelberg (2010)

    Google Scholar 

  19. Fakeri-Tabrizi, A., Tollari, S., Usunier, N., Gallinari, P.: Improving Image Annotation in Imbalanced Classification Problems with Ranking SVM. In: Peters, C., Tsikrika, T., Müller, H., Kalpathy-Cramer, J., Jones, J., Gonzalo, J., Caputo, B. (eds.) CLEF 2009 Workshop, Part II. LNCS, vol. 6242, pp. 291–294. Springer, Heidelberg (2010)

    Google Scholar 

  20. Ah-Pine, J., Clinchant, S., Csurka, G., Liu, Y.: XRCE’s Participation in ImageCLEF 2009. In: CLEF Working Notes 2009, Corfu, Greece (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Audio-Visual Systems, Fraunhofer IDMT, Ilmenau, Germany

    Stefanie Nowak

  2. Media Technology Lab, Gracenote Inc, Emeryville, CA, USA

    Peter Dunker

Authors
  1. Stefanie Nowak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Dunker
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. ISTI-CNR, Area Ricerca CNR, Via Moruzzi, 1, 56124, Pisa, Italy

    Carol Peters

  2. Idiap Research Institute, Rue Marconi 19, 1920, Martigny, Switzerland

    Barbara Caputo

  3. LSI-UNED, Juan del Rosal, 16, 28040, Madrid, Spain

    Julio Gonzalo

  4. Centre for Digital Video Processing, School of Computing, Dublin City University, Dublin 9, Ireland

    Gareth J. F. Jones

  5. Oregon Health and Science University, 3181 SW Sam Jackson Park Road, 97239-3098, Portland, OR, USA

    Jayashree Kalpathy-Cramer

  6. University of Applied Sciences Western Switzerland, TechnoArk 3, 3960, Sierre, Switzerland

    Henning Müller

  7. Centrum Wiskunde and Infoormatica, Science Park 123, 1098, Amsterdam, XG, The Netherlands

    Theodora Tsikrika

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Nowak, S., Dunker, P. (2010). Overview of the CLEF 2009 Large-Scale Visual Concept Detection and Annotation Task. In: Peters, C., et al. Multilingual Information Access Evaluation II. Multimedia Experiments. CLEF 2009. Lecture Notes in Computer Science, vol 6242. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15751-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-15751-6_10

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-15750-9

  • Online ISBN: 978-3-642-15751-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation