Skip to main content
SpringerLink
Log in

Creating Realistic Topics for Image Retrieval Evaluation

  • Chapter
ImageCLEF

Part of the book series: The Information Retrieval Series ((INRE,volume 32))

  • 995 Accesses

Abstract

This chapter describes the various ways for creating realistic query topics in the context of image retrieval evaluation campaigns such as ImageCLEF. A short overview describes general ways of creating topics, from complete laboratory style evaluations based on the technical capabilities of systems to real–world applications with real end users. The chapter offers help to those planning to evaluate systems on how to develop challenging and realistic topics based on knowledge of the users and of the capabilities of systems. Information sources for created topics are detailed. The main analysis will be the ImageCLEF tasks, and especially the medical retrieval tasks, where many different ways for creating topics have been analyzed over the years.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Caputo B, Pronobis A, Jensfelt P (2010) Overview of the CLEF 2009 robot vision task. In: Multilingual Information Access Evaluation Vol. II Multimedia Experiments: Proceedings of the 10th Workshop of the Cross–Language Evaluation Forum (CLEF 2009), Revised Selected Papers. Lecture Notes in Computer Science (LNCS). Springer

    Google Scholar 

  • Cleverdon CW (1962) Report on the testing and analysis of an investigation into the comparative efficiency of indexing systems. Tech. rep., Aslib Cranfield Research Project, Cranfield, USA

    Google Scholar 

  • Clough PD, Müller H, Sanderson M (2005) The CLEF cross–language image retrieval track (ImageCLEF) 2004. In: Peters C, Clough PD, Gonzalo J, Jones GJF, Kluck M, Magnini B (eds) Multilingual Information Access for Text, Speech and Images: Result of the fifth CLEF evaluation campaign. Lecture Notes in Computer Science (LNCS), vol 3491. Springer, Bath, UK, pp 597–613

    Google Scholar 

  • Clough PD, Müller H, Deselaers T, Grubinger M, Lehmann TM, Jensen J, Hersh W (2006) The CLEF 2005 cross–language image retrieval track. In: Cross–Language Evaluation Forum (CLEF 2005). Lecture Notes in Computer Science (LNCS). Springer, pp 535–557

    Google Scholar 

  • Deselaers T, Müller H, Clough PD, Ney H, Lehmann TM (2007) The CLEF 2005 automatic medical image annotation task. International Journal in Computer Vision 74(1):51–58

    Article  Google Scholar 

  • Enser PGB (1995) Pictorial information retrieval. Journal of Documentation 51(2):126–170

    Article  Google Scholar 

  • Everingham M, Zisserman A, Williams CKI, van Gool L, Allan M, Bishop CM, Chapelle O, Dalal N, Deselaers T, Dorko G, Duffner S, Eichhorn J, Farquhar JDR, Fritz M, Garcia C, Griffiths T, Jurie F, Keysers D, Koskela M, Laaksonen J, Larlus D, Leibe B, Meng H, Ney H, Schiele B, Schmid C, Seemann E, Shawe-Taylor J, Storkey A, Szedmak S, Triggs B, Ulusoy I, Viitaniemi V, Zhang J (2006) The 2005 PASCAL visual object classes challenge. In: Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment (PASCAL Workshop 05). Lecture Notes in Artificial Intelligence (LNAI), vol. 3944. Southampton, UK, pp 117–176

    Google Scholar 

  • Goodrum A (2000) Image information retrieval: An overview of current research. Informing Science 3(2):63–66

    Google Scholar 

  • Grubinger M, Clough PD (2007) On the creation of query topics for ImageCLEFphoto. In: MUSCLE/ImageCLEF workshop 2007, Budapest, Hungary

    Google Scholar 

  • Grubinger M, Clough P, Hanbury A, Müller H (2008) Overview of the ImageCLEF 2007 photographic retrieval task. In: CLEF 2007 Proceedings. Lecture Notes in Computer Science (LNCS), vol 5152. Springer, Budapest, Hungary, pp 433–444

    Google Scholar 

  • Güld MO, Kohnen M, Keysers D, Schubert H, Wein BB, Bredno J, Lehmann TM (2002) Quality of DICOM header information for image categorization. In: International Symposium on Medical Imaging. SPIE Procweedings, vol 4685, San Diego, CA, USA, pp 280–287

    Google Scholar 

  • Gunther NJ, Beretta G (2001) A benchmark for image retrieval using distributed systems over the Internet: BIRDS–I. Tech. rep., HP Labs, Palo Alto, Tech. Rep. HPL–2000–162, San Jose

    Google Scholar 

  • Harman D (1992) Overview of the first Text REtrieval Conference (TREC–1). In: Proceedings of the first Text REtrieval Conference (TREC–1), Washington DC, USA, pp 1–20

    Google Scholar 

  • Hersh W, Jensen J, Müller H, Gorman P, Ruch P (2005) A qualitative task analysis for developing an image retrieval test collection. In: ImageCLEF/MUSCLE workshop on image retrieval evaluation, Vienna, Austria, pp 11–16

    Google Scholar 

  • Jörgensen C (1999) Retrieving the unretrievable in electronic imaging systems: emotions, themes and stories. In: Rogowitz B, Pappas TN (eds) Human Vision and Electronic Imaging IV. SPIEProc, vol 3644, San Jose, California, USA. (SPIE Photonics West Conference)

    Google Scholar 

  • Law-To J, Joly LCA, Laptev I, Buisson O, Nozha Boujemaa VGB, Stentifordl F (2007) Video copy detection: a comparative study. In: Proceedings of the 6th ACM international conference on Image and video retrieval. ACM press, pp 371–378

    Google Scholar 

  • Lestari Paramita M, Sanderson M, Clough P (2010) Diversity in Photo Retrieval: Overview of the ImageCLEFphoto Task 2009. In: Peters C, Tsikrika T, Müller H, Kalpathy-Cramer J, Jones JFG, Gonzalo J, Caputo B (eds) Multilingual Information Access Evaluation Vol. II Multimedia Experiments: Proceedings of the 10th Workshop of the Cross–Language Evaluation Forum (CLEF 2009), Revised Selected Papers. Lecture Notes in Computer Science (LNCS), Corfu, Greece

    Google Scholar 

  • Markkula M, Sormunen E (1998) Searching for photos — journalists’ practices in pictorial IR. In: Eakins JP, Harper DJ, Jose J (eds) The Challenge of Image Retrieval, A Workshop and Symposium on Image Retrieval. Electronic Workshops in Computing. The British Computer Society, Newcastle upon Tyne

    Google Scholar 

  • Mizzaro S (1997) Relevance: The whole (hi)story. Journal of the American Society for Information Science 48(9):810–832

    Article  Google Scholar 

  • Müller H, Müller W, Squire DM, Marchand-Maillet S, Pun T (2001) Performance evaluation in content–based image retrieval: Overview and proposals. PRL 22(5):593–601. Special Issue on Image and Video Indexing

    MATH  Google Scholar 

  • Müller H, Marchand-Maillet S, Pun T (2002) The truth about Corel – Evaluation in image retrieval. In: Lew MS, Sebe N, Eakins JP (eds) Proceedings of the International Conference on the Challenge of Image and Video Retrieval (CIVR 2002). Lecture Notes in Computer Science (LNCS), vol 2383. Springer, London, England, pp 38–49. 3-540-43899-8

    Google Scholar 

  • Müller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content–based image retrieval systems in medicine—clinical benefits and future directions. International Journal of Medical Informatics 73(1):1–23

    Article  Google Scholar 

  • Müller H, Despont-Gros C, Hersh W, Jensen J, Lovis C, Geissbuhler A (2006) Health care professionals’ image use and search behaviour. In: Proceedings of the Medical Informatics Europe Conference (MIE 2006). IOS Press, Studies in Health Technology and Informatics, Maastricht, The Netherlands, pp 24–32

    Google Scholar 

  • Müller H, Boyer C, Gaudinat A, Hersh W, Geissbuhler A (2007) Analyzing web log files of the Health On the Net HONmedia search engine to define typical image search tasks for image retrieval evaluation. In: MedInfo 2007. IOS press, Studies in Health Technology and Informatics, vol 12, Brisbane, Australia, pp 1319–1323

    Google Scholar 

  • Müller H, Deselaers T, Kim E, Kalpathy-Cramer J, Deserno TM, Clough PD, Hersh W (2008) Overview of the ImageCLEFmed 2007 medical retrieval and annotation tasks. In: CLEF 2007 Proceedings. Lecture Notes in Computer Science (LNCS), vol 5152. Springer, Budapest, Hungary, pp 473–491

    Google Scholar 

  • Müller H, Kalpathy-Cramer J, Hersh W, Geissbuhler A (2008) Using Medline queries to generate image retrieval tasks for benchmarking. In: Medical Informatics Europe (MIE2008). IOS press, Gothenburg, Sweden, pp 523–528

    Google Scholar 

  • Müller H, Kalpathy-Cramer J, Eggel I, Bedrick S, Said R, Bakke B, Kahn Jr. CE, Hersh W (2009) Overview of the CLEF 2009 medical image retrieval track. In: Working Notes of CLEF 2009, Corfu, Greece

    Google Scholar 

  • Nowak S, Dunker P (2009) Overview of the CLEF 2009 Large–Scale Visual Concept Detection and Annotation Task. In: Peters C, Tsikrika T, Müller H, Kalpathy-Cramer J, Jones J, Gonzalo J, Caputo B (eds) Multilingual Information Access Evaluation Vol. II Multimedia Experiments: Proceedings of the 10th Workshop of the Cross–Language Evaluation Forum (CLEF 2009), Revised Selected Papers. LNCS, Corfu, Greece

    Google Scholar 

  • Radhouani S, Kalpathy-Cramer J, Bedrick S, Hersh W (2009) Medical image retrieval, a user study. Tech. rep., Medical Inforamtics and Outcome Research, OHSU, Portland, OR, USA

    Google Scholar 

  • Savoy J (2002) Report on CLEF–2001 experiments. In: Report on the CLEF Conference 2001 (Cross Language Evaluation Forum). Lecture Notes in Computer Science (LNCS), vol 2406. Springer, Darmstadt, Germany, pp 27–43

    Google Scholar 

  • Smeaton AF, Kraaij W, Over P (2003) TRECVID 2003: An overview. In: Proceedings of the TRECVID 2003 conference

    Google Scholar 

  • Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content–based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12):1349–1380

    Article  Google Scholar 

  • Sparck Jones K, van Rijsbergen C (1975) Report on the need for and provision of an ideal information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge

    Google Scholar 

  • Tsikrika T, Kludas J (2009) Overview of the wikipediaMM task at ImageCLEF 2008. In: Peters C, Giampiccolo D, Ferro N, Petras V, Gonzalo J, Peñas A, Deselaers T, Mandl T, Jones G, Kurimo M (eds) Evaluating Systems for Multilingual and Multimodal Information Access — 9th Workshop of the Cross–Language Evaluation Forum. Lecture Notes in Computer Science (LNCS), Aarhus, Denmark

    Google Scholar 

  • Voorhees EM, Harmann D (2000) Overview of the ninth Text REtrieval Conference (TREC–9). In: The Ninth Text Retrieval Conference, Gaithersburg, MD, USA, pp 1–13

    Google Scholar 

  • Westerveld T, van Zwol R (2007) The INEX 2006 Multimedia track. In: Fuhr N, Lalmas M, Trotman A (eds) Comparative Evaluation of XML Information Retrieval Systems, Proceedings of the 5th International Workshop of the Initiative for the Evaluation of XML Retrieval (INEX 2006), Revised Selected Papers. Lecture Notes in Computer Science (LNCS), vol 4518. Springer, pp 331–344

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. TechnoArk 3, Business Information Systems, University of Applied Sciences Western Switzerland (HES–SO), 3960, Sierre, Switzerland

    Henning Müller

Authors
  1. Henning Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Henning Müller .

Editor information

Editors and Affiliations

  1. HES-SO Business Information Systems, TechnoArk 3, Sierre, 3960, Switzerland

    Henning Müller

  2. Dept. Information Studies, University of Sheffield, Portobello Street 211, Sheffield, S1 4DP, United Kingdom

    Paul Clough

  3. , Computer Vision Lab/ETF-C 113.2, ETH Zürich, Zürich, 8092, Switzerland

    Thomas Deselaers

  4. Idiap Research Institute, rue Marconi 19, Martigny, 1920, Switzerland

    Barbara Caputo

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Müller, H. (2010). Creating Realistic Topics for Image Retrieval Evaluation. In: Müller, H., Clough, P., Deselaers, T., Caputo, B. (eds) ImageCLEF. The Information Retrieval Series, vol 32. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15181-1_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-15181-1_3

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-15180-4

  • Online ISBN: 978-3-642-15181-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation